Big Data is here to stay !!



The amount of data in our world has been exploding, and analyzing large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey’s Business Technology Office. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future.


Deep analytical talent: Where are they now?
Deep analytical talent: Where are they now?

Research by MGI and McKinsey’s Business Technology Office examines the state of digital data and documents the significant value that can potentially be unlocked.

Pop out
MGI studied big data in five domains—healthcare in the United States, the public sector in Europe, retail in the United States, and manufacturing and personal-location data globally. Big data can generate value in each. For example, a retailer using big data to the full could increase its operating margin by more than 60 percent. Harnessing big data in the public sector has enormous potential, too. If US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year. Two-thirds of that would be in the form of reducing US healthcare expenditure by about 8 percent. In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data, not including using big data to reduce fraud and errors and boost the collection of tax revenues. And users of services enabled by personal-location data could capture $600 billion in consumer surplus. The research offers seven key insights.

1. Data have swept into every industry and business function and are now an important factor of production, alongside labor and capital. We estimate that, by 2009, nearly all sectors in the US economy had at least an average of 200 terabytes of stored data (twice the size of US retailer Wal-Mart’s data warehouse in 1999) per company with more than 1,000 employees.

2. There are five broad ways in which using big data can create value. First, big data can unlock significant value by making information transparent and usable at much higher frequency. Second, as organizations create and store more transactional data in digital form, they can collect more accurate and detailed performance information on everything from product inventories to sick days, and therefore expose variability and boost performance. Leading companies are using data collection and analysis to conduct controlled experiments to make better management decisions; others are using data for basic low-frequency forecasting to high-frequency nowcasting to adjust their business levers just in time. Third, big data allows ever-narrower segmentation of customers and therefore much more precisely tailored products or services. Fourth, sophisticated analytics can substantially improve decision-making. Finally, big data can be used to improve the development of the next generation of products and services. For instance, manufacturers are using data obtained from sensors embedded in products to create innovative after-sales service offerings such as proactive maintenance (preventive measures that take place before a failure occurs or is even noticed).


3. The use of big data will become a key basis of competition and growth for individual firms. From the standpoint of competitiveness and the potential capture of value, all companies need to take big data seriously. In most industries, established competitors and new entrants alike will leverage data-driven strategies to innovate, compete, and capture value from deep and up-to-real-time information. Indeed, we found early examples of such use of data in every sector we examined.

4. The use of big data will underpin new waves of productivity growth and consumer surplus. For example, we estimate that a retailer using big data to the full has the potential to increase its operating margin by more than 60 percent. Big data offers considerable benefits to consumers as well as to companies and organizations. For instance, services enabled by personal-location data can allow consumers to capture $600 billion in economic surplus.

5. While the use of big data will matter across sectors, some sectors are set for greater gains. We compared the historical productivity of sectors in the United States with the potential of these sectors to capture value from big data (using an index that combines several quantitative metrics), and found that the opportunities and challenges vary from sector to sector. The computer and electronic products and information sectors, as well as finance and insurance, and government are poised to gain substantially from the use of big data.

6. There will be a shortage of talent necessary for organizations to take advantage of big data. By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.

7. Several issues will have to be addressed to capture the full potential of big data. Policies related to privacy, security, intellectual property, and even liability will need to be addressed in a big data world. Organizations need not only to put the right talent and technology in place but also structure workflows and incentives to optimize the use of big data. Access to data is critical—companies will increasingly need to integrate information from multiple data sources, often from third parties, and the incentives have to be in place to enable this.


Opportunities and Threats in a Brave New Market Research World


Much is written about how much Research has changed over the past few years – mobile is the latest big thing, social media is hovering in the background to be trawled for insights, ethnography is making a comeback, online qual. is proving extremely popular thanks to the speed and ease with which selected consumers can give their opinions on a whole range of subjects. Behavioural Economics has pointed out with much fanfare what qualitative researchers have always known – that context, social influence and emotions play a huge role in influencing what we say and do.

It’s exciting times, and from a Client perspective almost bewildering – the array of options from which to choose from is expanding rapidly, and the new normal seems to be that any one insights challenge requires a mixed methodology approach, using online, offline, qual and quant. We encircle our subject with an ever better (we hope) sense of what’s really going on – making Research an even more powerful tool.

So why the ongoing sense of Angst – that Research is threatened? Shouldn’t we be relishing change as an opportunity to become more influential with an upgraded toolkit?

My sense is it’s actually change per se, that is making many of us feel uncomfortable – margin-destroying, pervasive, ongoing. Low-cost technology software is putting MR tools into the hands of the potentially inexperienced. Our professional status is challenged; our sense of relative immunity to the ups and downs of economic cycles shattered; some of our assumptions on how best to model human behaviour are being shown to be wrong. It’s how we react to this change that will determine whether we will emerge strengthened or elbowed aside in a wave of MR disruption.

Here’s my take on the opportunities and threats.


Data Experts = Insights Experts
Companies are exposed to ever more information, but we still live in a world short on real insights. This is a huge opportunity for Market Researchers: to widen the scope of our mandate – take on sales data, market data, financial data, feedback from Customer Service, sales force reports, and mine them appropriately for a given business question.

Nearer the Action
The traditional structure of a Research programme was invariably quant. survey plus groups/depths – solid, but hardly spicy and often regarded as costly and slow by Marketing people inspired by the speed with which their Internet Marketing analytics were available. All that has changed with a MR powered by technology which can now deliver data (not necessarily insights) in days not weeks, and where the visualisation of evidence produced by Smartphones gets us really up-close and personal.

This ability to be on-the pace pushes market research nearer to decision making – and helps ensure we are an ongoing and valued member of the marketing team.

New MR = Creative and Strategic
Market Research increasingly plays a strategic role in new product development: we are tasked with unearthing unmet needs, leading ideation projects; we often take the lead in multi-functional task forces made up of R&D, marketing and sales personnel.

This is a radically new position: we’re forced to develop hypotheses, not just evaluate them, to be pro-active, engage in lateral thinking, and step out of our analytical comfort zone. Get this right and you automatically upgrade the value of the MR effort.


Market Researchers used to be data-guardians, people respected for their in-depth knowledge of categories and brands, often gained over decades. The power associated with this knowledge primacy has effectively been exploded – data often bypasses the MR department; Marketing people with good business degrees often have a good grasp of how to use Excel, perform simple regression analyses, certainly track data, and establish benchmarks. The black box, if you like, has been replaced by Pandora’s box.

This data-freedom means Researchers need to work harder to be recognised and valued as the true go-to people when it comes to insights.


“D. I. B.” (Do it Badly) Research
Low-cost, easy-to-use survey software effectively allows anyone with a database to do their own research – social media scraping (netnography at its best) is equally a field open to those with the time and inclination. The DIY trend is unquantified, often under-discussed, but a strong one in MR, driven by cost and speed – unstoppable forces, but with a downside: the lack of understanding of what makes good research, and what dangers and biases are involved in the whole Research process. I recently heard the phrase “Do-It -Together” at an Esomar Conference – which nicely encapsulates one way of addressing the danger of botched DIY Market Research, by collaborating and offering training, expertise.

Volume not Value Growth
The biggest single pressure I see on MR of the future is on budgets – either flat, decreasing, or simply not capturing a larger slice of the Marketing pot. More needs to be done with less, effectively – and once Marketing people have discovered the benefits of using a proprietary panel – radically reducing the per-survey cost – the floodgates can and do open.

This can lead to MR departments being bombarded with Survey requests, with less and less time available to evaluate the results. Larger Client side MR departments can split roles into more senior “strategic evaluators” roles and more junior “data-providers”, but for many smaller companies this isn’t an option.


In summary, Research has a broader and arguably superior toolkit than say 5 years ago – we can get closer and closer to an authentic sense of what is driving choice, be it habit, social influence, visceral states or impulse. We have reason to be optimistic, but the hope that methodologies will on their own actually make a massive difference may be naive.

The most pressing challenge for Market Research in future is in my view actually using all the methodological innovations for superior business impact.

The most amazing tools aren’t much use in the hands of mediocre craftsmen, and vice versa: brilliant skills can create much out of nothing. It’s here ,at the coal-face of MR – the area of ROI and impact – that we actually need to see the needle move. I hope that we can look back in 5 years time and say that all our improved insights actually made a bigger difference, and that we captured the recognition for the additional value we bring to the party