Big Data & Digital Marketing

In my opinion, Spark should NOT be compared with Hadoop but with MapReduce. However, people usually compare Hadoop and Spark (probably because they are buzzwords).

5 things to keep in mind:

1. They do different things -

Hadoop is a distributed data infrastructure (HDFS),

Spark is a data-processing tool.

2. Hadoop is more complete -

Hadoop also includes a data-processing tool (MapReduce),

Spark does not have its own filesystem and needs to be integrated with some.

3. Spark is (much) faster -

MapReduce operates in step;

Spark operates in one shot (because it is in-memory).

4. Speed is not always what you need -

For batch processing you do not need Spark's high velocity;

Common applications for Spark are those requiring real-time analysis.

5. Failure recovery -

both Hadoop and Spark are resilient to failures.

Up to now the Internet of Things has mainly focused on the data generation part (sensors and devices).

It is now time for the Analytics side to take over. Here is where Hadoop can make the difference.

However, my expectations are that IoT will boom with Real-Time analytics, and Hadoop can be of little use in this scenario.

120 pages for professional programmers on how Microsoft is using Hadoop to dive into Big Data:

1. Intro to Big Data

2. Intro to HDInsight

3. Programmgin with HDInsight

4. HDInsight and Data

5. Customisation

At the moment, companies are nowhere near reaching their individual data frontiers. They only use 12% of the data they already have, because data are siloed and they have a portfolio of hundreds of applications!

Data science is very different from traditional analytics.

Traditional analytics are based on managers' theories and is a human-driven approach. On the data science side, it's very different. We don't need a big meeting. We don't need your hypotheses. We don't need your ideas. What we need is all the data you've got.

SAP HANA and Hadoop are very different, that's why they could be a good combination in terms of complementing each other.

For example, SAP HANA is in-memory and uses predefined schema, while Hadoop is on disk and has no schema.

A scalable column-oriented database management for real-time analytics (SAP HANA) meets a technology platform that supports any kind of data for analyzing massive amount of data (Hadoop).

I cannot think of any other place for Big Data but the cloud!

Amazon and Microsoft are clear leaders at the moment, but there is a lot of movements and Oracle might well catch up.

My Hadoop clusters have always been running on the cloud and will always be (I am currently using HDInsight on Azure).

This is an extremely basic description of Hadoop, and yet it introduces relevant concepts: parallelism and MapReduce.

By 2015, 65 percent of applications with advanced analytics will come embedded with Hadoop. There's never been a better time to unlock the power of your Big Data. 

The Hadoop Innovation Summit returns to San Diego at the Marriott Marquis & Marina, on February 19 & 20, 2014.