[♪ INTRO ] Data. The word is everywhere these days. Every company is dying to tell you about its
big data, data analytics, data privacy, data warehouse, data lake, data data data data. At the center of the data mania is data mining—the
practice of sifting through all those piles of information for insights. Data mining recently made big news with the
Cambridge Analytica scandal. The political consultancy reportedly sucked
up data about millions of Facebook users without their knowledge, then used it to profile and
sway voters in the US, UK, and elsewhere. And similar techniques let companies like
Amazon, Facebook, and Google work out what we want to see or buy—sometimes with shocking
accuracy. It’s a little creepy. It’s not just ads and politics, either. Data mining allows airlines to predict who’s
going to miss a flight; it tells big-box stores who’s pregnant; it helps doctors spot fatal
infections; and it’s even enabled cell phone companies to predict massacres in the Congo. The power of data mining and the hype surrounding
it can make it sound like a magic wand—one that will either save your business or sink
democracy. Of course, data mining doesn’t really involve
any unicorn hair or phoenix tail feathers. It’s just applied statistics, searching
lots of data points for patterns that humans might not spot. Those patterns are based not on human intuition,
but on whatever the data suggests, so sometimes they can seem incredibly subtle or even alien. But there’s no more magic in data mining
than there is in a weather forecast. In fact, data mining is a lot like meteorology. Meteorologists aim for two things: first,
they want to describe patterns in the weather—to boil down its massive complexity into a few
numbers and equations. And second, they want to predict Tuesday’s
weather. That’s the whole point. Similarly, Spotify’s data scientists might
be interested in describing medieval rock fans, recognizing them as a group distinct
from nerdcore or freak folk fans — yes, that's a real subgenre. Ultimately, though, what’s most important
to companies like Spotify is predicting what each person wants to listen to. The key with data mining is that it achieves
description and prediction not through careful study by experts, but by analyzing large amounts
of data. In Spotify’s case, that might mean scanning
for patterns in genre labels, acoustic attributes, Internet reviews, and anything else about
each track, plus the age, location, friend group, and other scraps of information about
each user. Data mining is more about spotting patterns
than explaining them. Of course, the words “pattern” and “data”
can mean just about anything. There are no clear definitions for data mining,
data science, or big data, and they’re sometimes used interchangeably with each other or with
machine learning. That’s why it’s so easy to slap these
buzzwords onto any project for instant venture capital karma. That being said, a few types of techniques
consistently earn the “data mining” label. The most broadly applicable one is classification,
where you try to categorize things. For example, Target famously realized as early
as 2002 that they could guess who was pregnant and send them baby-related coupons. That’s a textbook classification problem:
Target needed to assign each customer to one of two categories: either “probably pregnant”
or “probably not pregnant.” Classification typically works in several
stages. First, each example, or instance, has to be
broken down into a collection of numerical attributes, or features. For a store like Target, an instance might
be your mom 7 months before you were born. The features would be things like “How many
bottles of unscented lotion did she buy in the last three months? How about in the quarter before that?” And likewise for zinc supplements, Asian pears,
and every other product in the inventory. The store would also need labels for some
chunk of the data—the ground truth about whether those customers were pregnant. Target got those labels from baby registries
and due dates customers had shared. Once the data’s all lined up, it’s time
for training. That’s where the system tries to tease out
patterns from all the labeled examples. Learning to classify is such a basic, common
need that dozens of algorithms, the mathematical procedures computer programs follow, have
been devised for it. Which algorithm works best depends on all
kinds of factors, like how many categories there are and how different features are connected
to each other. But many classification algorithms are similar
in that they treat each feature as a drop of evidence for one category or the other. The features get weights indicating how strongly
they boost or weaken someone’s chances of falling into the “yes” category — that
they are pregnant, for example. Those weights are what the system learns during
training. Basically, it’s figuring out how informative
each attribute is. Finally, to classify instances the system
hasn’t seen before, it puts together all the weighted contributions, and maybe stuffs
the resulting number through a bit of mathematical machinery to slide it up or down. If the result is negative, that instance goes
in the “no” bucket. If it’s positive—load up the crib coupons! Each individual feature doesn’t tell you
much. In fact, many turn out to be irrelevant. But together they can be really powerful. Target’s approach worked so well that when
one customer complained that his teenage daughter was getting coupons for baby clothes, he ended
up apologizing to Target. Turned out the company knew about his daughter’s
pregnancy before he did! Classification is useful any time you want
to tell one group of things from another. Insurance companies use it to guess which
elderly patients will die soon so that they can start end-of-life counseling. Doctors use it to check whether premature
babies are developing dangerous infections, since the classifier can put together subtle
disease indicators before humans would notice any signs. I could spend all day listing uses for classification,
but it’s far from the only type of data mining. One close cousin is known as regression. And no, that doesn’t mean deciding you like
Limp Bizkit again. In regression, instead of predicting a category,
the goal is to predict a number. Take Target again. They wanted to know not just whether each
customer was pregnant, but when to send each coupon So they managed to estimate due dates, too. That’s a regression question—how many
weeks until the customer gives birth. Regression often depends on dozens or even
thousands of variables—the features that describe each example. It finds an equation or curve to fit the data
points, telling you how high you’d expect the curve to be given any arbitrary input. Or in this case, how far away you’d expect
the customer’s due date to be. Like in classification, many regression techniques
give each feature a weight, then combine the positive and negative contributions from the
weighted features to get an estimate. And, also like classification, regression
is used everywhere. One of the better-known examples is Google
Flu Trends. In 2008, it began publishing real-time estimates
of how many people had the flu based on searches for words like “fever” and “cough.” Regression is also part of predictive policing
software — programs that look at historical data to guess how likely a crime is to occur
in each area. The third major data mining technique is clustering. As the name suggests, the goal here is to
group data points in a way that helps with the analysis In the marketing world, clustering emerged
in the 1980s—well before data mining—with the work of a market researcher named Howard
Moskowitz. He struck gold when he realized there wasn’t
one best pasta sauce. Consumers showed three distinct types of preferences—and
the previously unrecognized group that craved extra-chunky turned out to be worth millions. Clustering is often used to analyze market
segmentation like this, but to understand how the techniques work, let’s take a different
example: eBay. On ebay, you can get millions of products,
from antiques to zip ties. Even within a single category, like electronics,
the selection is overwhelming. So eBay organizes things into subcategories. But it’s a pain for humans to trawl through
all the electronics, identify subcategories, and assign every product to a subcategory. Instead, the company can use clustering to
automatically group the products. Again, each product first has to be broken
down into numerical features, like how many times “printer” appears in the description,
or who manufactured it. The simplest clustering method is to guess
how many distinct subcategories there should be. Then you randomly lump items together into
that many clusters, and keep shifting items between groups to make each cluster tighter. In the end, similar products end up settling
into clusters together. But we don’t have to stop there! The blue and silver versions of the same camera
don’t really deserve separate listings; they’re variants of the same product. So in addition to subcategories, it would
be nice to find listings to merge. Sites like eBay can do both simultaneously
with a technique called hierarchical clustering. Rather than a single set of categories, hierarchical
clustering produces a sort of taxonomic tree. For example, it might find that cameras are
much more like each other than like TVs. But within cameras, the DSLRs and point-and-shoots
each get their own subgroup, albeit slightly less distinct ones. And within those are many different models,
each with a few variants. (on image) Companies like Cambridge Analytica use these
techniques to look for groups of voters who will respond to the same kinds of advertising,
and Spotify can use them to guess who will like similar music. The fourth staple of data mining is anomaly
detection. It’s basically a special case of classification—identifying
instances that are unusual or worrisome. The IRS uses anomaly detection to spot likely
tax evaders, and credit card companies use it to flag transactions that don’t fit your
usual buying habits. It also helps industries with heavy-duty equipment. For instance, power companies and airlines
can see when a generator or jet engine is starting to vibrate differently than usual. Some anomalies can be detected just by looking
for deviations from averages. Fancier techniques include looking for instances
that don’t match any cluster, or comparing instances with the closest other examples
to see if their feature values are far off. Finally, association learning reveals which
birds are of a feather. The idea is to look through, say, millions
of grocery store purchases to see what gets bought together and when. A classic example is the Osco drug store chain,
which once found that many customers bought beer and diapers together on Friday evenings. Contrary to popular legend, the store never
acted on this profound insight, but stores regularly use observations like this to optimize
their floor layouts and inventory. For instance, Walmart discovered that shoppers
buy lots of Pop Tarts immediately before hurricanes, so it started to stock up. Association learning has broader applications,
too. CellTel, an African cell phone company, realized
it could spot impending massacres in the Congo when everyone nearby started buying prepaid
phone cards. The five strategies we’ve covered—classification,
regression, clustering, anomaly detection, and association learning—form the backbone
of data mining. What makes them so powerful is that they offer
standard mathematical tools you can use for everything from curating Facebook feeds to
optimizing store layouts. But that ease of use can also lead people
astray. Data mining is just one step in the process
of extracting knowledge from data—and it’s all too easy to whip out an algorithm without
carefully selecting the data, massaging it into the right form, and considering how to
interpret the results. Remember Google Flu Trends? It shut down after a few years, but not because
the algorithm was broken. Search auto completion had totally thrown
off the data, and engineers had given it too much leeway to interpret seasonal words like
“snow” as evidence of the flu. Then there are the queasy social implications
of sharing data in the first place, and of letting companies form such an intimate understanding
of our behavior. In other words … the creep factor. So as powerful as it is, the math of data
mining is just the beginning. Sometimes the hardest part is all the messy
human stuff. Thanks for watching this episode of SciShow! If you’re interested in the ways companies
can use psychology to learn even more about you from your data, you can check out our
video about that over on the SciShow Psych channel. [ ♪OUTRO ]
