The big data phenomenon is being driven by mobile computing, social computing and cloud computing. In the next three years, businesses globally are expected to invest more than $120-billion in data analytics to gain insight into big data.
“There is little doubt that big data has entered the mainstream of information management,” says Sean Paine, COO of information solutions specialist, EnterpriseWorx. “This brings new business opportunities. Vast amounts of semi-structured and unstructured data are being generated at network speed, and new technologies enable businesses to access and analyse this data. This provides a valuable source of information on customers’ buying patterns.”
However, big data is not without its challenges.
“For the first time since 1965, when Gordon Moore hypothesised that the number of transistors on an integrated circuit doubles every two years, data volume is increasing faster than processing power,” says Paine. “Moore’s ‘law’, the bedrock of the IT establishment, governing processing speed and memory capacity, is under threat.
“We have reached a tipping point. In the old world, there was no means of storing massive XML databases. The cost of modelling big data and understanding what business could do with it may have outweighed the benefits. But the old ways of storing data in a relational database are not keeping up.
“We’ve entered a new paradigm. New technologies have been developed for big data that enable information to be stored relatively inexpensively on a per terabyte basis. However, that doesn’t mean we have to analyse it all at the outset. Our mindset needs to shift to one of ‘get it all in, and then we’ll have it if we need it’.”
The exponential increase in data means an endless supply of information, says Paine. “However, it’s not just about the volume of data, it’s about the complexity and variety – structured and unstructured such as text, voice, image, video, multi-media. “The question is: How do you even begin to get value from this data?
“Two technologies have come together to achieve this. First, multi-core, big-memory servers make it possible to crunch huge volumes of data. Leading data warehouse appliances like Kognitio can handle huge volumes of data in seconds. Second, there’s advanced analytics, a collection of different tools that can analyse data in a flash.”
According to Paine, it is possible to extract relevant nuggets of information using a mature data warehouse appliance like Kognitio running on top of Hadoop’s open source software framework. Both support data-intensive applications and petabytes of data. The data can then be analysed using online analytic processing tools.
“Big data technology has come to the rescue,” he says. “In the past, the concept of big data washed over us and was ignored. Now we have a game plan and a strategy for how to manage it.
“Organisations, however, continue to spend a great deal of money on storing data in structured SQL databases. But the volume of data in most organisations has quadrupled in the past two or three years. The question needs to be asked: is it not time to bring a no-SQL platform such as Hadoop into the organisation? A cost-benefit exercise may demonstrate that, from a business perspective, it is prudent to adopt this new technology. The question is: how much is the business prepared to invest in gleaning profound insights from its data.
“Big data should be regarded as an opportunity, not a problem, and we should focus on identifying ways of leveraging the data and extracting valuable insights through advanced analytics. It’s time to look at the whole picture – not only to access what we already know, but also to find relationships that cast light on what we don’t know. It’s possible, for example, to do sophisticated analysis in regard to customers’ needs, product defects and supply chain lapses. The ability to store and analyse this data is now available. We have reached the stage of analysis at the speed of thought.”