The concept of big data is steadily gaining acceptance around the world, even though its origins can be traced back to the 1960s and ’70s with the development of the relational database and the first data centres to handle large data sets. The early and mid-2000s saw the emergence of social media and with it, the realisation of just how much data could be generated. This led to the development of open-source frameworks to handle the growth of big data, essentially making it easier for companies and individuals to make sense of it.
In the years since, a lot of research and development has gone into big data storage, analysis and use cases. The growth of cloud computing has also played an important role in this, offering developers the ability to scale applications and frameworks that rely on big data. The Internet of Things (IoT) and machine learning have also helped, mostly by using collected data on consumer preferences to create solutions that meet the needs of modern customers and businesses.
Making sense of big data requires an understanding of the three main attributes that define it: Variety, Volume and Velocity. Together, these three Vs encompass the phenomenon that is larger, complex data sets that are being used to address business problems in ways never thought of before. Big data is both structured and unstructured, received in high volumes and at a faster rate. For some organisations, this can be terabytes of data in a single day and an assortment of multiple data types.
For individuals such as Thibaut de Roux, a former senior banking executive with more than 30 years of global capital markets management and technical experience, the growth of big data has brought more opportunities to learn and invest. He is currently involved in a start-up business that is linked to technology, artificial intelligence (AI) and big data.
The Changes Over Time
Developers have played an important role in the rise of big data, especially in overcoming the boundaries of traditional closed-source software to rely on open-source frameworks. While information technology (IT) departments have primarily been the consumers of software, developers plug into the open-source world and integrate the solutions into their projects, which then become part of the development cycles of organisations. The result is working software that has a place in the technology ecosystem and is appreciated by other developers in different sectors, many of whom use it for varying use cases.
As the solutions evolve, so too has the technology and the people who use them. Many organisations continue to incorporate operational analytics into their production systems, primarily to evaluate important aspects of their business in real-time. Data lakes have emerged for storing different types of data before analysis. Streaming data sources have made it desirable to analyse data as it is received, while the hardware environment has changed to accommodate analytics. The outcome is faster hardware that significantly improves the time it takes to perform and report on analytical calculations.
While big data promises a bright and revolutionary future for technology and business, it has its challenges. For starters, data volumes continue to grow at exponential levels, with sizes doubling every two years. Storage is an issue for many companies, especially those keen on making effective use of the data.
Even with good storage capabilities, the data must be valuable to the users. As big data encompasses both structured and unstructured data types, being able to clean it all up and organise it in a meaningful way requires a lot of effort. Beyond analysing the data for trends and patterns, the modern data scientist requires good data curation skills to prepare data.
With technology, what’s good for the industry today may be overtaken by another solution tomorrow. Solutions continue to be upgraded and improved. Keeping up with these changes and new solutions is in itself work that businesses must be ready for.