Big Data has been a trend for a while and many of the user cases for early adopters are widely available. Big Data usually refers to large or time sensitive data that cannot be handled by relational databases. The main characteristics of Big Data include the mountainous volume of data with multiple types, often generated at a high velocity and requiring advanced analytical tools to gain value from the data. Some of the newer types of data include web applications, social media, video, internet, & wearables, but also machine data.
Many are overwhelmed by the amount of data which is in multiple places and are looking for a solution that organizes the data and makes it easy to find documents and surface real-time information. Being able to see relationships between the data and their clients is increasingly becoming more important. It is estimated that the volume of data doubles every 3 years, therefore, business leaders should develop their strategies to harness existing data, in addition to looking for ways to be disruptive with the new data.
Over the last 10 years, some of the barriers to adoption have disappeared –
- The cost and availability data storage is now cheap
- Compute power has increased dramatically
- Plethora of easy-to-use, smart Analytical Tools, to aggregate and visualize the data
Machine Learning will tame Big Data
One of the other barriers to entry is the lack of talent to implement the new change, but the need for this is quickly eroding with the growth of Artificial Intelligence and Machine Learning tools which can automate many of the user cases. Some of the low hanging fruit in this area include:
- Automatically detect and find anomalies in the data, especially security
- Bots that find patterns in data and suggest best and personalised approach (used in customer service and sales)
- Providing sentiment analysis by combing through social media (e.g. Is there a problem with a product)
- Predicting the probability of customer churn
How do you get ready for Big Data?
The steps to take will be different for each business, but the general steps are:
- Define an approach to collect and ingest most important data (e.g. from website, CRM, ERP, logs, unstructured docs, sensors, sentiment and geo data). Create a Meta Data Dictionary that shows importance of different data and apply at point of ingestion.
- Use Analytical tools to gather insights
- Define security and privacy structure to protect data
One of the companies that have figured out how to make Big Data easier is Elastic. With over 200 million downloads of its’ Elastic Stack, they are rapidly making it easier for businesses ingest, aggregate, store, search, visualize and gain insights out of their data. Check out Elastic!
1. See EY Website for Big Data overview.
2. See article on Big Data on Techopedia
3. The Age of Analytics: Competing in a Data Driven World. McKenzie Global institute, December 2016
4. For an updated overview of Big Data Trends, see “Enterprise data sovereignty: If you you’re your data, set it free, Deloitte Insights, Tech Trends 2018.
5. IBM’s Graphic on Big Data:
6. See article on Elastic on Datanami website.