When Small Data Beats Big Data: Small is Beautiful says the Statisticians Around the Globe

Big data isn't always better than small data. A high-quality small sample can yield conclusions that are more reliable than a low-quality large sample. Costs associated with data collecting, computing, and privacy must be weighed against benefits. On limited data, statistical inference is effective; on vast data, not so much.

Big data is legitimately a key area of research. Small data is still around, though. The same sociological and technological forces that have produced data points have also produced a lot of tinier datasets. More data would initially appear to be unquestionably preferable to fewer data. Psychological AI can be implemented. In reality, acquiring more data will increase costs in many ways and make the analysis more challenging. Small data can sometimes outperform big data in terms of speed, accuracy, and cost to draw the appropriate conclusions. In this article, we outline several circumstances in which small data is preferred.

Big Data

Big Data can help businesses operate better by boosting revenue, improving operational effectiveness, refining marketing campaigns and customer service programs, responding more swiftly to new market trends, and obtaining an advantage over rivals.

Will One Data Be Able to Beat Big Data?

The discovery by Google engineers of a way to anticipate the early spread of the flu was widely reported in the media in 2008. The concept seems sound. Google's search engine is probably used by people who have the flu to identify their symptoms and find treatments. These inquiries could reveal where the flu is spreading immediately. Engineers looked through 50 million search phrases to determine which ones were related to the flu to identify the most appropriate searches. Then, after testing 450 million alternative algorithms, they developed a covert method that made use of 45 search phrases that best fit the data. After that, each region's doctor visits related to the flu were predicted using the algorithm.

Everything went smoothly at first. A few months later, in the spring of 2009, an unexpected event occurred. The swine flu epidemic began. With the initial instances appearing in March and a peak in October, it arrived unexpectedly. Because it had learned from previous years that flu cases peaked in the winter and declined in the summer, Google Flu Trend missed the outbreak. The predictions failed.

The engineers started working on the algorithm's improvement after this setback. There are two alternative strategies for doing this. Combating complexity with complexity is one strategy. The underlying principle is that complicated issues require complex solutions, and if a complex method fails, more complexity is required. The second strategy adheres to the stable-world principle, which states that complicated algorithms perform at their peak in clearly defined, stable environments with abundant data. Whether there are vast data or small data, human intellect has evolved to deal with uncertainty. The theory behind it is that a complicated algorithm using historical big data may not accurately forecast the future in uncertain situations and should be simplified. The engineers at Google aimed for additional complexity. They increased them to roughly 160 instead of reducing the 45 search phrases and kept betting on big data.

For a short while, the improved algorithm performed well in predicting new situations. In 100 out of 108 weeks between August 2011 and September 2013, it overstated the percentage of anticipated doctor visits attributable to the flu. The flu's inherent volatility was a significant factor. Since influenza viruses are ever-evolving chameleons, it is incredibly challenging to forecast their spread. In comparison to other flu strains, the swine flu's symptoms, such as diarrhea, were different from those of previous years, and younger people were more likely to contract the illness. The unpredictability of human conduct was a second factor. Many others searched for information about the flu not because they were ill but simply out of curiosity. However, the system was unable to discern between different search intentions. The engineers kept fiddling with the improved algorithm in an attempt to figure out if their model was too simple. Google Flu Trends was quietly discontinued in 2015.

Google Flu Trends serves as an example of how, in an uncertain world, lowering data volume and complexity can result in more precise forecasts. In some circumstances, it may be best to disregard everything that has occurred in the past and only depend on the most recent data point. It also demonstrates how psychological AI, in this case, the recency heuristic, may forecast just as well as sophisticated machine learning systems. My main argument is that psychological AI may be implemented using "fast-and-frugal" heuristics that require little data.

However, many of us find it challenging to avoid the thought of purposefully omitting information when we're attempting to make an informed decision. The flu example, however, is neither an aberration nor an exception.