Big Data & Digital Marketing

After giving us the definition of Big Data, Gartner is now giving us 3 predictions on what will happen with Big Data in Business Intelligence:

1. By 2020,

information will be used to reinvent, digitalize or eliminate 80% of business processes and products from a decade earlier.

2. By 2017,

more than 30% of enterprise access to broadly based big data will be via intermediary data broker services, serving context to business decisions.

3. By 2017,

more than 20% of customer-facing analytic deployments will provide product tracking information leveraging the IoT.

There is a lot of content in this (long) article published by the Harvard Magazine.

Here are my main 3 takeaways:

1. "The Big Data revolution lies in improved statistical and computational methods, not in the exponential growth of storage or even computational capacity" by Gary King

2. Big Data isn't everything: "We had petabytes of data and yet we were building models that were fundamentally flawed, because we didn't have insights about what was happening" by Nathan Eagle

3. "No matter how much data exists, researchers still need to ask the right questions to create a hypothesis, design a test, and use the data to determine whether the hypothesis is true." by Nathan Eagle

Today's buyers are increasingly coming from the business side of the house and not from corporate IT and self-service analytics is growing while traditional dashboard BI is in remission.
Self-service analytic tools allow power users to quickly explore, blend and visualize data to produce new business insights and to validate business data requirements to support application development and data management. 
Is the pendulum swinging in the direction of analytics empowerment and reduced time-to-answer and away from cost control and data quality management?
Gartner thought leader, Frank Buytendijk, suggested that we look to the business model that cracked the code on optimizing the centralization versus decentralization trade-off; namely, franchising.
This implies standardization of tools and enterprise licensing to drive down costs, tool-specific skilling to create larger pools of skilled workers to be shared across projects and centralized provisioning of compute infrastructure to save time and money.

Another way to look at this is:

Since the market offers fewer data scientists than needed, new tools are being developed so that less experienced professionals can analyse data in a productive way.

Will this "Data Discovery" be an evolution of self-service BI that we see emerging today?

I think that Microsoft products such as Power BI, or Excel Power Query and Power Pivot, are worth mentioning in this context.

I was not aware of the existence of economic theories that disagree with the positive effect of the digital revolutions. Their main points are:

1. The impact of the current ICT revolution is not as radical as the previous one (steam engine, railways, telephone ...)

2. Too many traditional businesses are disappearing, and the average required level of skills to enter the market is going up

3. The peak of the revolution has passed, and the progress is slowing down.

Mark Cliffe, ING's chief economist, replies like this:

1. Part of the current growth slowdown is due to the financial crisis

2. One needs more time to evaluate the impact of new technologies (even electricity took decades to have its full effect)

3. A key aspect of the "Anytime Anywhere" technology is its network effects, i.e. benefits spread faster with adoption

4. Hundreds of millions of people in the emerging world are being lifted out of poverty

5. The digital revolution is actually increasing, becoming the Internet Of Things, or "Anytime, Anywhere, Anything" economy

In such a scenario machine learning and predictive anlytics will be essential to not drown in Big Data. But keep this in mind: "Machines are for answers, Humans are for questions".

Up to now the Internet of Things has mainly focused on the data generation part (sensors and devices).

It is now time for the Analytics side to take over. Here is where Hadoop can make the difference.

However, my expectations are that IoT will boom with Real-Time analytics, and Hadoop can be of little use in this scenario.