Bertelsmann AG, the German media conglomerate, pays very close attention to what it calls issues management. The company wants to know about trends and developments in the surrounding environment that could adversely affect its fortunes – regulatory changes, negative public opinion, new competition – so it can deal with them proactively.
One way Bertelsmann monitors its environment is by subscribing to electronic news and article aggregators. They currently deliver a mind-boggling 60,000 pieces of text a day. Who has time to read it all? Nobody, of course, which is why Bertelsmann uses a sophisticated enterprise search tool, MindServer from San Francisco-based Recommind Inc., to help filter and categorize documents.
Simple Google-style keyword searching wasn’t going to cut it. “We needed a means of going through the content that would help us find and identify new topics that we didn’t know anything about yet,” explains Matthias Wulff, director, corporate communications at Bertelsmann. “That’s why we chose MindServer.” Wulff, a user, helped develop the company’s issues management system.
The Recommind product, like other high-end search tools, doesn’t just search for keywords. MindServer uses a patented algorithm – probabilistic latent semantic analysis (PLSA) – to search for concepts by analyzing the distribution in a document of concept-related words and using statistical methods to predict an item’s relevance. Result: it finds more of the good stuff faster – more articles relevant to Bertelsmann’s concerns, for example – and clutters it with less irrelevant material.
“Using MindServer has helped us considerably widen our horizons, allowed us to look at a greater variety of sources and topics, and made our media monitoring way more systematic,” Wulff says.
Recommind, headed by Canadian software entrepreneur Robert Tennant, is a relatively small but growing player in a market that IDC and others say is about to explode. More familiar names are Verity Inc., Autonomy Corporation plc and Fast Search & Transfer ASA. They all, like Recommind, claim to have unique search technology that gives them a competitive advantage. “I wouldn’t say [Recommind’s PLSA] is anything stagger-ingly different,” says Nick Patience, an analyst with New York-based research firm The 451 Group. Tennant, however, claims that academic testing of algorithms used in competing products supports Recommind’s claim that searches using MindServer miss fewer relevant documents and place fewer irrelevant documents among the first hits listed.
All of these post-Google search engines can pay big dividends when used in function-specific applications such as Bertelsmann’s issues management system. And the applications are endless. Companies concerned about regulatory compliance, for example, are using search tools to monitor e-mail or instant messaging in real time to make sure employees aren’t communicating in ways that breach regulations. Banks use them to analyze transaction records looking for tell-tale patterns of fraud.
The Australian federal government is using the Recommind product to filter tens of thousands of government documents and forms that relate to programs and issues of interest to small businesses. It posts the content to its own Web site and syndicates it to other sites. The object was to reduce red tape and provide a single entry point where small businesses can quickly find exactly what they need. “MindServer routinely goes out and sweeps vast arrays of information from three levels of government and classifies it into the themes we’ve established,” explains Paul Griffin, general manager of the online e-services branch at the federal Department of Industry, Tourism and Resources.
The search engine also works in the background at the Web site to filter keyword searches so users get information relevant to their local or regional jurisdiction, and to reduce the number of hits to only absolutely relevant items. It’s often just two or three documents, says Griffin. Which means users don’t have to manually sift through dozens or hundreds of extraneous documents, as they do with less effective search tools.
Using sophisticated search engines in this kind of application is a hot trend, says Tamara Alairys, the global practice lead in Accenture Information Management Services (AIMS). “It’s what we’re going to see more of in the future,” Alairys says. “There’s a large opportunity for search based applications – things you build on top of the core search capability that enable different business functions. It’s almost like BI [business intelligence] for the masses.”
Susan Feldman, research vice-president in content technologies at IDC, agrees. IDC says the global enterprise search market is taking off like a rocket. Revenue from software and maintenance hit $950 million in 2005, up 25.9% from $760 million in 2004. And IDC predicts it will grow to $2.6 billion (all figures U.S.) by 2010, with much of that growth coming this year and next.
“These technologies will become pervasive and therefore more difficult to attribute revenue to in the later years of the forecast,” Feldman says. “In other words, as is already happening, they will become embedded in CRM applications, BI tools, and other office applications.”
Search-enabled applications can deliver quantifiable return on investment. Cell phone companies have used sophisticated search tools in the background at their Web sites to help customers zero in more quickly to find exactly the ring tone they want to buy. It may seem a trivial application, but as Alairys explains, if it takes users too many mouse or button clicks to find what they want, they all too often give up.
“By getting searches down to two or three touch points, the sales closure rate goes way up to 60 percent,” she says.
But as powerful as search-enabled vertical applications can be, advanced search engines have much broader application in the enterprise – helping knowledge workers find stuff faster. Not just unstructured information in word processing documents and spreadsheets, but also structured data in ERP and CRM systems. According to IDC, the average knowledge worker spends 9.5 hours each week searching for information. A third of that time is completely wasted because they never find what they wanted.
Lawyers probably spend more time searching than most – for precedents and research. Electronic document stores in big law firms can run into the millions of items. And the cost of search is more direct for lawyers because time is what they sell. This is one reason Recommind focused until recently almost exclusively on the legal vertical.
“If lawyers are doing half-hour network searches that could be reduced to five minutes, that ties directly to time not spent on billable hours,” says Tennant. “The return on investment is very clear.”
It’s not just time savings, which in any case can be a misleading, since lawyers won’t necessarily reinvest all the time they save in billable work. Using less effective search tools, a lawyer may never find the best precedent, which can result in his using an inferior one. Or if finding precedents is too hard or time consuming, he’ll draft the document from scratch, resulting in more wasted time. The same dynamics apply with other knowledge workers.
“Even if you figure they spend 10 percent of their time searching,” Alairys says, “there’s a dollar cost to that. And if you extrapolate it across the organization, it can be huge.” So why aren’t more companies providing these tools to their employees?
“There’s enormous productivity gain to be had by getting information to do a job more quickly,” Tennant argues. “Far more gain than from other types of systems that have been widely deployed. The problem is, you can’t tie that gain to any item on anybody’s budget, or to the P&L sheet. It’s a diffuse benefit but with a very focused cost.”
Alairys is inclined to agree. There are other impediments to getting the technology into many companies as well, she says. Business users when they see what enterprise search can do have no doubt about its benefits, but because it’s an enterprise tool, vendors are typically selling to IT managers. And IT managers may not as readily see the need for an advanced search product, especially if the content management or intranet software they bought already has a rudimentary search function.
Alairys argues that virtually any company can benefit from using enterprise search tools, though clearly some more than others – financial services firms, for example, and oil companies with their terabytes of exploration data. If the IDC forecast is accurate, many more will be using the technology soon.
Ironically, it is Google, the search tool against which vendors like Recommind, Autonomy and Verity always favourably compare their own products, that is driving growth. As Alairys says, “Acceptance of Google has shown people the power of search technology.” Now new desktop search tools for sifting data on a user’s local hard drives, such as the free products from Google and Yahoo and X1 Technologies Inc.’s enterprise software may ratchet up demand even further.