 |
|
Intelligent Enterprise
January 2000
It's About Data Integration
Richard Winter
Recently I have worked with several clients assisting with data architecture efforts in major enterprises. In each case, the top IT executive responsible for architecture or technology said essentially the same thing to me: "Our data architecture is a major concern." In each case, the executive had a mandate from the top of the organization to figure out what to do and make it happen. In each case, we took a look at what is going on in the industry and in the company.
And, in each case, we came to the same conclusion: The industry doesn't quite offer an answer. Oh, each of the clients figured out what to do. Each client has a plan to do something pragmatic about the situation.
But here is what I think my clients really want: a unified way of managing enterprise data across the entire organization. They want something like the data warehouse, but one that works both for operational systems and decision support. It also has to work for legacy systems as well as new systems, application development in addition to off-the-shelf application acquisition. It has to remain useful when there is yet another corporate acquisition or merger, and must perform on all hardware platforms, operating systems, database systems, and all the other engines of data (such as online analytical processing).
And, oh yes, a major impetus to all this is - what else? - e-commerce. The demands of e-commerce are really shining a spotlight on the inadequacies of existing approaches to data architecture and integration. So, of course, part of the requirement is a way to manage data across new e-commerce applications along with all the existing systems and components with which they must interface.
What do my clients mean by "manage data across the enterprise"? Fundamentally, they want better ways of implementing solutions and better ways, long term, of managing the data resource. It should be quick and cost-effective to identify and source the data needed to support a new packaged application. It should be possible to define a given data element once in the enterprise. It should be possible to know the derivation of a given data element from its root sources. It should be possible to know everything important about a data element. It should be possible to make business rules about data and have them apply across the enterprise. It should be possible to invest in some architecture, direction, and set of standards and - over time - clean up the mess. One study estimated that 30 to 40 percent of the time spent developing applications in a major enterprise was spent on "data issues": identifying the sources of needed data, evaluating them, extracting and transforming data, dealing with data quality problems, and correcting software errors due to data-related problems. All these activities would be completed more rapidly and at lower cost with a good data infrastructure.
Why is this important? Because we need a better approach to data integration in order to implement applications quickly and efficiently. When you install a major purchased application, typically it brings with it hundreds or thousands of data element definitions. It often must interface to a hundred or more existing systems, principally to exchange data. Making this happen in most organizations is a large, costly nightmare. And why do we buy applications, rather than build them? So that we can reduce time to implement, cost and risk.
Purchased applications are just one example: Similar problems exist in building applications, enhancing existing systems, and consolidating systems or databases.
But you know what? I haven't heard of anyone who really knows how to do this across the industry! And, from what I have seen out there, no vendor has convincing plans or proposals on how to accomplish it. You can devise a solution to key elements of this problem for a given enterprise and make progress on behalf of that enterprise. But you really can't lay out the big-picture solution. In fact, as an industry, I don't think we quite have the concepts and architectural principles in hand to really spell out the solution.
Meanwhile, we keep on building databases and files. We keep on installing new, attractively packaged applications. Now we build e-commerce sites. We build data marts by the dozens. In some lucky organizations we build enterprise data warehouses, and that helps - providing someone can get the builders of the data marts to link them to the data warehouse. And, of course, sometimes we build "operational data stores" - these are fascinating things, sometimes clearly useful and valuable. But what they are, exactly, is something about which hardly anyone can agree. And how would they fit into an enterprise data architecture? Most of us don't really have the foggiest idea.
What does all this have to do with scalable systems? Simply this: The business executive's idea of scalability is "a solution I won't outgrow." In our major enterprises, we have outgrown the data in- frastructure. Data warehousing is one important and valuable form of data integration. But as implemented, for the most part, it is focused on historical, static data for query and analysis. E-commerce and other changes have brought us into a new era, the era in which we need data integration across the enterprise.
So here is my challenge for the new millennium: Figure out this data architecture/data integration business, and start building products across the industry according to some data integration framework that more or less works.
Any vendor that can provide a scalable solution to the enterprise data-integration problem is sure to become part of the database dozen in years ahead.
Richard Winter (Richard.Winter@wintercorp.com, fax: 617-338-4499) is a specialist in large database technology and implementation and president of Waltham, Mass.-based Winter Corp (www.wintercorp.com).