Companies are learning how to unlock the potential windfall of insights that are available in their big data stores (e.g. Apache Hadoop) and more recently in their enterprise big data streams. In a recent article on CNBC.com (Big data’s big stumbling block), Nyshka Chandran points out that there simply are not enough skilled engineers to fill data scientist roles for all of the projects companies want to initiate in 2015… and this problem will continue until at least 2018. The looming data scientist shortage of 2015 stems from the complexity of the current generation of big data management and analysis tools.
“McKinsey showed only 18 percent of companies believe they have the skills to gather and use data effectively while 19 percent are confident that their data-gathering processes contribute to sales effectiveness.”
International Data Corporation (IDC) believes the shortage of skilled staff will persist next year.
“In the U.S. alone, there will be 181,000 deep analytics roles in 2018 and five times that many positions requiring related skills in data management and interpretation,” the firm said in a report on its ‘Big Data and Analytics Predictions for 2015’ conference earlier this month. – CNBC: Big data’s big stumbling block
Big Data Analytics, A Victim of It’s Own Success?
The demand for skilled engineers is compounding at an alarming rate as more companies explore and exploit the potential of big data analytics in areas such as: competitive advantage, operational intelligence, risk management, customer experience, and real-time contextual marketing. Most current technology tools used to discover big data insights are built by engineers for engineers. Said more simply: the tools get the job done, but are not easy to use by those without a CS degree.
This brings to mind one of my favorite “for the engineers, by the engineers” products of all time, the BMW Z3 Coupe. A performance car of the highest calibre on every metric, but not user-friendly or beautiful. Similar to the Z3 Coupe, these Open Source tools are adept, but are realistically designed to be used by a very small segment of the population working with big data. What’s worse is that once you get over the hurdle of development and get into the production rhythm, your highly-valuable Java engineers will now be working on mundane repeat tasks as your specialized data scientists refine and develop new hypotheses.
As Technology Moves Toward Maturity
While this process seems inefficient (and it is), it is the normal evolution as any technology moves toward maturity, big data and streaming analytics are no exception. The first iteration of access brings the custom-built tools. Exceptionally talented engineers build custom software tools from the ground up to address their specific needs. The next level of maturity has these very talented engineers opening up their specialized tools to be used for additional use cases, becoming more accessible and easier to use as they are combined with other software tools. This is where many streaming big data analytics solutions are positioned right now, taking an Open Source base and refining the interfaces and usability by adding components. An end-to-end analytic solution built in this manner becomes many point solutions strung together and maintained by your staff, rather than finding the best-fit engineering solution, which frees your talented staff to focus on high value tasks. The right tool for the right job. Striim takes the next step with an end-to-end real-time platform delivering continuous insights across your enterprise.
A Streaming Big Data Management Platform… No Java Skills Required
The real magic happens in this next step when software makes data more accessible, allowing a larger community of business analysts and managers to directly access and be enabled by streaming big data analytics. Platforms are built to handle and hide the underlying complexity of streaming analytics and make feel easy to use, allowing people to focus on the creative endeavor of exploring correlated data from every source (transactional, structured, semi-structured, machine data, IoT sensors, etc.). At this point insights flourish and new ways of securing competitive advantage are discovered. This is what Striim enables.
Striim: A Solution to the Data Scientist Shortage
Striim offers this next generation software solution for streaming big data analytics. Anyone familiar with SQL now has access to streaming big data analytics through a unique combination continuous data acquisition, processing, and analysis capabilities along with an agile drag-and-drop development environment. Need a more complex Java algorithm? No problem, just import it into our Java platform and make it available to the end-users. The Striim platform handles all of the end-to-end complexity of streaming (and static) big data management, allowing your talented engineers and analysts to focus on exploring and solving your business problems, not performing repeatable technical processes.
Learn More About Striim
For more information on the Striim platform, download the recent Bloor Group white paper: The Genesis of the Real-time Enterprise: How WebAction Enables Truly Responsive IT Applications
