“To be or not to be, that is the question” the most famous words from Shakespeare’s play Hamlet aptly summarizes the conundrum faced by BI decision makers faced with the problem of data consolidation within their organization. Essentially, an organization has to consolidate its data repositories for various reasons, viz. Mergers and Acquisitions, Single View of Customer, Regulatory requirements, Resource optimization etc.
The act of creating consolidated physical repositories is a cumbersome task, especially in large enterprises, that such investments need solid financial models, elaborate planning and even then there is no guarantee of success. For many years, the only viable option for enterprises is to perform “physical consolidation”, i.e. create the consolidated logical and physical data models, create hard-wired ETL programs to load data into the consolidated database and use this infrastructure for reporting and analytics. Not anymore! – It is my view that Enterprise Information Integration (EII) provides a viable option for enterprise wide BI consolidation and this post it to initiate practitioners to look at EII as a business solution for problems in the BI domain.
EII falls in the realm of data virtualization and refers to technology behind real-time aggregation of corporate data across multiple, widely disparate data sources. EII delivers comprehensive, reusable “views” that is exposed via SQL and/or web services to whole lot of consuming applications (Reports, Dashboards, etc.). Though the concept of data virtualization is very apt for BI consolidation, there are many other class of problems in which EII technology can play a major role.
At a high level, EII tools work in the following way:
• Designer uses the data connectivity feature of the tool and can potentially connect to a whole lot of data repositories, viz. databases, XML, excel sheets, etc.
• Data Modelers then can combine the required data sets and model them in the EII software
• EII software then creates a virtual data layer and exposes the metadata in the form of views
• The views can be optimized by using the features available within the software
• Views are exposed to consuming applications and at run-time, data is fetched from the underlying data repositories
A Simple yet Powerful Value Proposition!
It is important to understand that EII does not replace EAI or ETL. All 3 technologies can co-exist and each on its own provides solutions for different kinds of problems.
EAI stands for Enterprise Application Integration and is used in cases where different applications (Sales & Inventory application for example) want to talk to each other. Target for EAI is an application.
ETL stands for Extract, Transform and Load and is used in cases where data aggregation is required to provide an integrated, consistent definition of data for decision support process. Target for ETL is a database.
EII stands for Enterprise Information Integration and is used as a framework for real-time integration of data from multiple sources both inside & outside an enterprise, empowering business users to “pull” any kind of data from anywhere in the enterprise. Target for EII is the business end user.
A classic reference architecture in which all 3 tools can play a part is when, transactional applications are integrated thro’ EAI, data from these applications flow into an Enterprise Data Warehouse (EDW) by leveraging ETL capability and then EII tools help to combine data from OLTP applications, EDW, external data repositories and local excel sheets for business decision making.
In terms of technology, there are specialized EII tools like Composite Software and also many of the standard reporting tools have an EII component embedded in them (Data Federator in Business Objects XI, Virtual View Manager in Cognos 8.x etc.). In my view, EII does have a strong business case and BI practitioners would do well to evaluate this option when faced with data consolidation challenges.
Thanks for reading. Please do share your thoughts.