Big Data & Digital Marketing

There are a lot of secrets around Palantir, and probably you would not expect anything different from a company that has received funding from CIA (actually Q-Tel, CIA's venture capital arm).

What is known is that its data-processing software is being used to fight terror and catch financial criminals.

Palantir is now raising $500 million at a valuation of $20 billion, surpassed only by Uber ($50 billion) and Airbnb ($24 billion).

I say it's time to merge sharing economy and data science!

After giving us the definition of Big Data, Gartner is now giving us 3 predictions on what will happen with Big Data in Business Intelligence:

1. By 2020,

information will be used to reinvent, digitalize or eliminate 80% of business processes and products from a decade earlier.

2. By 2017,

more than 30% of enterprise access to broadly based big data will be via intermediary data broker services, serving context to business decisions.

3. By 2017,

more than 20% of customer-facing analytic deployments will provide product tracking information leveraging the IoT.

Data is certainly growing at an exponential rate, and the Internet Of Things is just making the process stronger. However, the buzz has created some false myths that need to be busted.

Five biggest myths about big Data:

1. The Entire Data is Accessible

2. Entire Data is Needed

3. Offers Certainty

4. Big Data isn't Here to Stay

5. Granular Data is Better

Top Performers consistently apply analytics in almost every activity across their organization. They prefer Analytics over Intuition 5 times more than Low Performers.

This Survey by MIT Sloan, in collaboration with IBM, draws a clear picture on how organizations can approach big data, what the major challenges are and how a successful analytics culture can be established.

Organizations are usually found in one of these 3 stages:

1. Aspirational: just started with analytics. The main target is to improve cost efficiency. Are not very rigorous.

2. Experienced: use analytics to guide actions, target at growing revenues, some use of rigorous approaches, applications are limited for future strategies.

3. Transformed: use analytics to prescribe actions, use analytics at all levels also in day-by-day activities, use rigorous approaches.

Sometimes organizations transition from state 1 to 2 to 3.

The Survey suggests a 5-step methodology for successfully implementing analytics-driven management:

1. Focus on the biggest and highest value opportunities

2. Within each opportunity, start with questions, not data

3. Embed insights to drive actions and deliver value

4. Keep existing capabilities while adding new ones

5. Use an information agenda to plan the future

Nowadays, companies in virtually any field have the capability to gather, store and take advantage of massive amounts of information.But not all of them have the capability to manage the data in the proper way.
Here are 5 signs that it could be time to outsource your data management:1. Overpriced in-house management costs.2. Lack of in-house Big Data management experts.3. Constant need to re-deploy employees to do other tasks.4. Inability to comply with regulatory requirements.5. Failure to respond quickly to technological changes.
Avoid creating strong dependency on the third party expert and keep in mind these three things that can massively help you:1. Have a clear plan on how you want to use the data2. Embrace cloud technologies3. Develop in-house skills

As organizations become more and more data-driven, it becomes progressively more important to set up a healthy and productive way to manage data.

Here are 4 major steps to follow to help you improve on this:

1. Data Management

2. Data Security

3. Data Quality

4. The Team

-----

1.

Data management is the “administrative process by which the required data is acquired, validated, stored, protected, and processed, and by which its accessibility, reliability, and timeliness is ensured to satisfy the needs of the data users.”Basic pillars are: provisioning, protection, replication and recovery.Evaluate data before engaging in big data analytics.Have a maintenance plan.
2.

Data security must be prioritized by any organization to enable it to function properly and for operations to flow efficiently. It also provides stockholders and executive teams peace of mind of knowing that the information they have stored in their servers will not be easily exploited by hackers or cyber-criminals.
3.

A study conducted by Experian Data Quality shows that outstanding data quality has a direct correlation to an increase in company profits. 4 steps to reduce incidence of human error (cited by 65 percent of organizations to be the main cause of data problems): Identify data entry points, train staff, Automated verification, clean data over time.
4.

Hire a competent team of professionals who know their roles very well: data management supervisor, data entry staff, data analyst, quality and training staff

Up to now the Internet of Things has mainly focused on the data generation part (sensors and devices).

It is now time for the Analytics side to take over. Here is where Hadoop can make the difference.

However, my expectations are that IoT will boom with Real-Time analytics, and Hadoop can be of little use in this scenario.

Lots of information about Big Data from New Jersey Institute of Technology.

The upper half of the infographics introduces Big Data with the usual buzz words. The remaining has a lot of interesting statistics.

My 6 takeaways from the infographics:

1. $300 billion, what US could save in Healthcare

2. +60% in operating margin (retail)

3. 73% of companies have already increased revenues thanks to big data

4. 56% of IT decision makers believe that finding the right staff is the biggest challenge

5. "Query and reporting" is the Top 1 capability currently available

6. "Transactions" and "Log Data" are the most common source of data currently collected and analysed (88% and 73% respectively).

When a successful Data Science project is well presented, it usually looks very simple. It reminds me of some of the proofs in Calculus or Physics that I studied when an undergrad.

In fact, they just *look* simple, and they do so because someone has done an incredibly hard and difficult job before hand.

In Data Science projects, the hidden job is usually related to data: looking for data, cleaning the data, joining the data, realising you are missing some data and iterate.

Here is a list of 7 "new" companies that might make the difference in the realm of Big Data.

1. Palantir Technologies

Founded in 2004 and evaluated at $20B it is probably not a startup anymore

2. Crayon Data 

Big data for decision making

3. Neo Technology

Best known for its graph db Neo4j

4. Couchbase

The company behind the NoSQL db Couchbase server.

5. PromptCloud

Specialised in Data-as-a-Service

6. Snowflake

Big Data technologies for SMBs.

7. Saama Technologies

Is this a startup? Founded 18 years ago, it has clients such as Apple and Cisco

At the Wimbledon tennis tournament, IBM will use a solution based on InfoSphere + Watson. This will make it possible not only to collect information just about everything that happens to the players and to the balls, but also to relate this large set of data with historical information and to query it using Natural Language.

There's more.

With so much data about each shot and player, it will be possible to analyse players' playing style. This will in turn allow the players to study their opponents' weakness and strengths to better prepare next matches.

A very interesting note.

IBM is training tennis professionals to use the data entry system, rather than training data scientists on tennis.

Yet another proof of the importance of having field experts (about the specific industry under consideration).

Five mistakes to avoid to fail a big data project:

1. Going at Big Data alone

2. Using outdated data management practices

3. Ignoring Big Data best practices

4. Failing to understand the importance of Big Data governance

5. Understanding the power of Big Data, aka concentrating at the finger and missing the moon.

Most organizations have realized both the potentials and the difficulties of Big Data.

Here is a TDWI checklist report that can help to get organized before beginning a new project:

1. Identify key business drivers

2. Create an effective team

3. Emphasize communication skills

4. Embrace visualization and storytelling

5. Access all the data

6. Operationalize Analytics

7. Improve governance

The 3 elements that I particularly consider crucial are:

A. Set clear goals

B. Invest on the team, not on individuals

C. Communicate and operationalize your project findings 

Based on Big Data analysis, one has the ability to capture and uncover meaningful insights that can be translated into improved ROI on marketing activity.

Here are a few lessons learned in the performance marketing world that small businesses can apply to their own efforts to exploit the potential of Big Data:

1. Use Active Tracking Links Where Possible

2. Get Granular

3. Analyse Holistically, aka avoid silos!

4. Analyse in Real-time

5. Visualise

I was not aware of the existence of economic theories that disagree with the positive effect of the digital revolutions. Their main points are:

1. The impact of the current ICT revolution is not as radical as the previous one (steam engine, railways, telephone ...)

2. Too many traditional businesses are disappearing, and the average required level of skills to enter the market is going up

3. The peak of the revolution has passed, and the progress is slowing down.

Mark Cliffe, ING's chief economist, replies like this:

1. Part of the current growth slowdown is due to the financial crisis

2. One needs more time to evaluate the impact of new technologies (even electricity took decades to have its full effect)

3. A key aspect of the "Anytime Anywhere" technology is its network effects, i.e. benefits spread faster with adoption

4. Hundreds of millions of people in the emerging world are being lifted out of poverty

5. The digital revolution is actually increasing, becoming the Internet Of Things, or "Anytime, Anywhere, Anything" economy

In such a scenario machine learning and predictive anlytics will be essential to not drown in Big Data. But keep this in mind: "Machines are for answers, Humans are for questions".

Big Data is a big opportunity in banking, but there are a lot of things to sort out along the way.
In the UK, banks are teaming up with retailers to offer personalised discounts to customers through their mobile applications based on an individual’s spending patterns.Banks are starting to think of their small business customers and using their data to see in real time what is happening to their businesses and where banks can step in to help.
More than a third of banks still buy data on clients from third parties, because it is too hard to extract the same data from their own systems.Most banks are held back by concerns over privacy and the technical challenges stemming from their ageing and complex systems.

There is a lot of content in this (long) article published by the Harvard Magazine.

Here are my main 3 takeaways:

1. "The Big Data revolution lies in improved statistical and computational methods, not in the exponential growth of storage or even computational capacity" by Gary King

2. Big Data isn't everything: "We had petabytes of data and yet we were building models that were fundamentally flawed, because we didn't have insights about what was happening" by Nathan Eagle

3. "No matter how much data exists, researchers still need to ask the right questions to create a hypothesis, design a test, and use the data to determine whether the hypothesis is true." by Nathan Eagle

A common challenge when dealing with the Internet Of Things is the lack of standardization. This, in turns, makes it difficult for all of the sensors to communicate, for all of the data to be gathered, analysed and eventually leveraged.

IBM and ARM have teamed up to lower the barrier for developers in IoT: the mBed IoT Starter Kit costs just around 100eur, and can be directly connected to the IBM IoT Foundation (to channel data into IBM Bluemix serivces).

Today's buyers are increasingly coming from the business side of the house and not from corporate IT and self-service analytics is growing while traditional dashboard BI is in remission.
Self-service analytic tools allow power users to quickly explore, blend and visualize data to produce new business insights and to validate business data requirements to support application development and data management. 
Is the pendulum swinging in the direction of analytics empowerment and reduced time-to-answer and away from cost control and data quality management?
Gartner thought leader, Frank Buytendijk, suggested that we look to the business model that cracked the code on optimizing the centralization versus decentralization trade-off; namely, franchising.
This implies standardization of tools and enterprise licensing to drive down costs, tool-specific skilling to create larger pools of skilled workers to be shared across projects and centralized provisioning of compute infrastructure to save time and money.

Another way to look at this is:

Since the market offers fewer data scientists than needed, new tools are being developed so that less experienced professionals can analyse data in a productive way.

Will this "Data Discovery" be an evolution of self-service BI that we see emerging today?

I think that Microsoft products such as Power BI, or Excel Power Query and Power Pivot, are worth mentioning in this context.