It happens to everyone. As life goes on, you accumulate stuff — clothes, shoes, furniture. First the closets get full, then the basement, then the attic.
Eventually, there’s more stuff than room.
Likewise, corporations are continually accumulating data — transaction records, customer profiles,
competitive intelligence. But the growing enterprise doesn’t accumulate data at a linear rate. Research firm Gartner Inc. projects a 70 per cent increase in the number of terabytes of data installed for 2002 over 2001, on top of a 60 per cent increase the year before. “”It’s a madly, widely escalating curve of installed capacity,”” says Gartner’s Bob Passmore. As the economy recovers — and as the per-megabyte cost of storage continues to decline by 40 per cent per year — Passmore expects the compunding to continue.
The space-challenged homeowner can always thin out her collection. A company can’t throw its
old Dockers into a box and drop it off at the nearest Goodwill. That data is the institutional history of the corporation, the sum of the knowledge of the market collected over the years. Moreover, from a business process perspective, there are many good reasons the volumes of data collected have to be preserved.
For example, a bank can’t just throw out transaction data. While privacy regulations restrict what data can be held for specific reasons, a bank still has to be able to prove a transaction took place on an account. Paper transaction records have to be archived electronically, says Art D’Silva, Royal Bank of Canada’s manager of enterprise information functions and strategies. Customer demand for quick response and the cost associated with searching paper records make them an archaic notion.
The necessity to reflect the minute-by-minute status and history of millions of accounts, the redundancy required for fault tolerance and business continuity, the need to understand customer behaviour historically … all these contribute to the banks’ hunger for data, D’Silva says.
There are other legal issues driving the need for companies to store more data. Recent high-profile corporate scandals have not only shattered investor confidence, they’ve demonstrated the need to archive messaging systems to prove or defend against allegations of playing loose and fast with the rules.
And proposed legislation would require Canadian ISPs to track customer use data and maintain it for six months. That could mean keeping a three- to five-gigabyte file on every customer an ISP has, Bob Carrick, president of ISP help site Carrick Solutions Ltd., told ITBusiness.ca in November.
Even without such legislation, the Internet has possibly had the single most profound impact on the amount of data an enterprise collects. The dot-com boom established new customer touchpoints. When the bust came, those touchpoints didn’t disappear.
A busy Web site generates gigabytes of data daily that aren’t even related to transactions. Graham Rose,
