Big Data & Digital Marketing

In my opinion, Spark should NOT be compared with Hadoop but with MapReduce. However, people usually compare Hadoop and Spark (probably because they are buzzwords).

5 things to keep in mind:

1. They do different things -

Hadoop is a distributed data infrastructure (HDFS),

Spark is a data-processing tool.

2. Hadoop is more complete -

Hadoop also includes a data-processing tool (MapReduce),

Spark does not have its own filesystem and needs to be integrated with some.

3. Spark is (much) faster -

MapReduce operates in step;

Spark operates in one shot (because it is in-memory).

4. Speed is not always what you need -

For batch processing you do not need Spark's high velocity;

Common applications for Spark are those requiring real-time analysis.

5. Failure recovery -

both Hadoop and Spark are resilient to failures.