Data Mining Ready for a Comeback
September 11, 2006 (Computerworld) -- Notwithstanding all the emphasis I've put on text data in my past two columns, enterprises also run on numbers. Yet companies are typically staffed by humans, and most humans are somewhat ill at ease with more advanced forms of mathematics. As a result, most of what passes for quantitative analysis in
organizations is painfully, often misleadingly, simple.
However, there are quite a few exceptions to that rule, including the following:
Data mining/knowledge discovery, which actually includes an increasing amount of text mining.
Predictive analytics, which overlaps heavily with data mining.
Forecasting, which is generally regarded as separate from data mining, even though they both rely on related statistical techniques.
Optimization, which generally refers to the use of operations research techniques that contain a specific concept of "maximization
How sophisticated you need to be about these techniques depends in large part on your industry segment, unit sales (if applicable) and the size of your organization. Many organizations license large suites of products from SAS. Others try to squeak by using just Microsoft Project and Excel. But no matter what tools you use, the basic story remains the same -- enterprises have a lot of quantitative and/or objective data, and if you squeeze that data hard enough, something valuable will probably pop out.
Perhaps the most controversial of these disciplines is data mining. It has already gone through a classic boom/hype phase, complete with breathless business press coverage and widely repeated myths (no, Virginia, there never was a retailer that boosted sales by placing beer next to diapers). As part of that phase, mediocre products were half-heartedly sold by various business intelligence generalists, with predictably disappointing results. And now the doldrums would seem to have set in. Even so, large companies data-mine very profitably, in a broad range of industries, for a broad range of purposes. And considerable innovation is still moving data mining technology forward. For example, both SAS and Oracle are pushing ease-of-use strategies, but with rather different emphases. (Care to guess which of the two is more database-centric?) When vendors sell analytic applications that contain a good deal of statistical analysis, it's reasonable to say that data mining is involved. Text mining continues to boom. And product sectors such as Web search and antispam rely on data mining for large fractions of their overall research and development.
Many of today's data mining applications can be united under the rubric of customer-offer targeting.