This is part two of a three-part blog series. Part one examined how big data has changed our approach to crafting business questions and the expectations of what information we’ll have access to when we make decisions.
Today analytics is moving rapidly into business operating processes and practices. The term being coined to describe this new field is machine-to-machine process intelligence, which at its core is predictive analytics.
What fascinates us as data scientists is the analytics beyond the simple. Our focus on analytics is on the “predictive analytics edge,” where the analysis is based on real-time results and continues to learn from a baseline predictive model that is also flexible.
Financial services shifts
Consider insurance fraud analysis that was traditionally run, say, every two months. At that point the damage was done – the fraudulent insurance claim had already been paid. This approach was slow and passive. Now insurance companies run in-database fraud analysis twice a day, catching fraudulent claims within hours, and increasingly within minutes. Whereas traditional systems were fine-tuned for transactions and batch processing, today we need to sense and respond to changing conditions immediately. Even correlating criminals living in certain neighbourhoods can add to risk insights for insurance providers.
In banking, every banking transaction you place, becomes a permanent record of your customer buying profile, ranging from your credit purchasing trends, to investment profile, to banking wealth. The banking industry has been working diligently to consolidate its internal walls between retail and wholesale banking to get a more consolidated view of its customer profile attributes, and to develop triggers on customer service approaches to provide alternative services offers, whether in the transactional experience moments, or in upselling/cross selling potential solutions based on buyer profile and potential propensity for new solutions.
For example, predictive analytics on credit card purchases notice a stream of purchases over three months on baby products, but there is no bank mortgage on file with their customer record, a predictive trigger could recommend that a call from a mortgage advisor to make an offer and pre-empt the inevitable decision to seek a mortgage elsewhere. More advanced solutions will crawl obituary information and correlate doing risk with customers with certain attributes.
In the past, we focused on structured relational databases, predefined relationships, and structured transactional data. Although those don’t go away, now we have volumes of unstructured data, untold relationships between the data, and new data coming in all the time. Hence we have more dynamic data, with multiple contexts, from many sources, both people and machines.
This shift in how we handle data reflects the fact that our world is complex and connected in ways we cannot imagine. The Internet has brought all of this complexity to us in a flood of data that is massive, diverse, unstructured for the most part, and rich in valuable information.
However the data flood or tsunami is not just Internet based. It is everywhere in corporations, in governments, data has no rules and as we have learned and very limited constraints. Every person can recreate their context or persona within a few minutes, tell a new story, and morph into new bits and bytes.
Further, what happened on the Internet is happening in corporations and governments. There are few rules and virtually no constraints. Every author of content can invent his or her own structure, create his or her own context, and tell his or her own story. Some of the data can be defined as well structured and some is semi-structured. The table below provides a summary of the shift we are experiencing.
Shifting worlds: data evolution
| PAST REALITY | CURRENT REALITY |
| Data is a fact of life | Data is a factor of our supply chain. |
| Collecting data | Connecting data |
| Analyzing and interpreting data | Predicting data and Perpetual Insights |
| Data bits to source Information | Smarter Insights and Pattern Recognition |
| One version of reality – Black and White World | Multiple sources and perspectives – The Endless Grey World |
| Structured | Unstructured |
| Relational | Non-Relational |
| Centralized processing | Distributed parallel processing |
| Terabytes | Petabytes, exabytes, … |
| Analytics for Select Few, Niche Focus | Analytics Everywhere for Everyone |
| Limited Experimentation | Endless Innovation and Agile Experimentation |
| Information Scientists | Big Data Scientists |
In summary, what we have learned is that in this vast data reservoir of diverse data sources, the most important driving force is making the right connections.
This means that predictive analytics approaches are critical to be applied. Methods from artificial intelligence, machine learning, advanced statistics and continual machine learning approaches are needed to enable these smarter connections.
One of the most powerful data connectors is the use of links or hash tags as this codification method can be used to mine and detect early signals or pattern inferences. We also have learning that linking allows us to connect diverse element data sources and that these combinations, together tell a story; further, the same story can be told in many different ways. In addition others can keep adding to the story patterns, so the predictive models continue to shift like different waves.
In conclusion, there is no longer room in our world for rigid data, rather complexity in data and agile or free pattern detection will increasingly become the new norm. With this shifting world context, data pattern detection with scientists that know how to mine it effectively will increasingly enrich our sense making.
In the third and final blog post in this series, we’ll examine the rise of chief data scientists inside of businesses and other ways the data revolution has transformed company structure.