The Limits of Big Data, the Strength of Small Data

We are in the data economy. Data is the new oil. We have to make decisions based on data. And data must drive everything we do. From one industry to another, we read how data analysis is improving decision making. There is no problem with using data to make decisions: I support it. The use of data in business is a tool of the time, deployed in different ways. But when entrepreneurs or business leaders begin to lose their intuitive capabilities because of the absence or paucity of data, reliance on data becomes a weak point in running a business.

Let me give two cases:

• Gary Loveman was a professor in Harvard Business School who believed in data. He built business models on how better insights, driven by data, could unlock more value in the casino business and specifically Caesars Entertainment Corporation (then called Harrah’s Entertainment). He was brought to Caesars to apply the models; he later became the overall boss. But a moment came when opportunities were evolving in Macau, a Chinese administrative territory, for casinos and gambling companies. Many American casinos prepared for the moments and invested in the “new Las Vegas”. Gary looked at the data and decided that the data was not solid enough for him to invest. He passed the opportunity, one of the best in the sector, in the global casino industry. Largely, because of that mistake, Caesars lagged competitors and practically went into decline, even as some competitors that went into Macau flourished.
• While Mr. Loveman was looking for data, Sheldon Adelson, an American business magnate, did not bother. He had the instincts that Macau, despite the lack of data, was going to be a refuge for rich Chinese to play games. He invested through his Las Vegas Sands Corporation and flourished. Later, Mr. Adelson mocked Caesars, explaining that those with more mathematics missed the opportunities, and that sometimes, numbers cannot show everything in business.

The Limits of Numbers

Yes, numbers cannot show everything we need in business. Sometimes you just have to go, and build the products and services even when there are no numbers, to support the thesis. If Steve Jobs had waited for numbers from Verizon, we would not have the iPhone today.

Richard Branson, the founder of Virgin, has the same business philosophy.  In an interview, he explained the futility of wasting time, and asking consultants to analyze business plans. He noted that sending a plan to two accounting firms would likely result to two different perspectives. Relying on them will be a mistake. For him, the key is not being totally driven by data, but making a decision as an entrepreneur with elements of risks to move in a territory that is simply not easily understood. The first-mover advantage becomes a huge opportunity.

While I am not saying that entrepreneurs should be careless to enter into new businesses without making efforts to understand the market opportunities through market analysis, my point is that there are many things market analysis cannot tell you. Sometimes, the analysis gives you what you want to hear. The key is to understand that success can happen if your entrepreneurial intuition is allowed to drive business vision, accepting risk, by overlooking some signals from data. Steve Jobs flourished on that by ignoring focus groups and surveys, trusting his instincts to shape a new world.

You need small data (the stories from customers, the one-by-one interface with clients, etc) but you must not always need big data (the massive human-less datasets) to drive everything you do in your startup. The capacity to use that small data is what will decide how far you could go. Small data wins in many ways over big data:

• Small data makes it easier to understand individual customers instead of the averages of multitudes which big data does best. There is an insight you can get by talking to a customer that reports generated from massive datasets will never match.
• By talking to some customers, over reliance on massive datasets driven by averages, you can get better understanding on why the data you are looking at looks the way it looks. Yes, small data answers the Why behind the data.

Simply, you need to go out and meet your customers because despite all the data in this world, that interaction is what will give you a clear understanding of your market. A product is whatever a customer says it is. It is when you interact with your customers that you will have that better understanding.

Our lives and what customers do are not really averages. Analytic solutions make them so. Two people are in a room: one man ate five burgers, the other none. In the analytics software, you will likely get that each ate 2.5 burgers by running the mean. Why that is an insight, it has missed the key element: the marginal outlier which can be gathered by speaking with these two people. Yes, by speaking with these two people, the startup could have seen that one is hungry while the other is fed. And based on that, the company could engineer better solutions for the outliers instead of the averages.

It turns out that products built on averages are mundane and typical. The opportunity for innovation comes by looking at areas no one has looked (or looked but ignored). That means knowing that one man has eaten while the other has not, in the example above.  With averages, the blue ocean strategy opportunity is gone because the analytics has normalized all the data to averages. You build transformational businesses by having insights which are not typical. They do not happen because you have averaged customers and markets. That will not create any new insight to bring change.

All Together

There is no human system that can be effectively represented with all-statistics. Statistics is nothing but averages and most times, it is not very perfect. The best insights in consumer business come from small data which is largely stories customers have told. You need to find ways to collect them, and make use of them, as a startup.

Delivering great experiences to customers will not happen unless you can answer the Why in the data you see. Most times getting that Why insights comes by meeting and speaking with the customers. The stories of the customers are the best insights you can use to shape your products.

The customer story is what drives the best disruptive products and solutions in the market. It is not the averages from the data analytics software. You need to invest to get those stories which are unfiltered and un-cleaned, but simply raw to help engineer the next big product. When you pass datasets from the field through a cleaning process, before feeding them into your analytics software, you are essentially removing the most important elements that will drive the blue oceans or activate the trajectories to disruptions. And because the small data cannot be handled that way, we then tend to diminish its importance. That has to change because for all the great things big data offers, it has a limit: customers are not necessarily averages packaged in nice graphics, sequestered from the Why.

Register for Tekedia Mini-MBA (Feb 10 - May 3, 2025), and join Prof Ndubuisi Ekekwe and our global faculty; click here.