Data Integrator version 6.5

Transcription

1 Data Integrator version 6.5 from Business Objects an extract from the Bloor Research report, Data Integration, Volume 1

2

3 BusinessObjects Data Integrator 6.5 BusinessObjects Data Integrator 6.5 Fast facts BusinessObjects Data Integrator is a general purpose data integration product although it is usually marketed as a conventional ETL (extract, transform and load) tool in conjunction with a data warehousing environment. Despite the apparent success of other vendors in this market, Business Objects is of the opinion (and we agree) that it remains the case that the vast majority of user organisations hand code data movement tasks rather than using tools such as BusinessObjects Data Integrator. A hand-coded approach is inefficient and, in the longer term, expensive. As a result, Business Objects believes that there is a substantial market for its product. Key findings In the opinion of Bloor Research the following represent the key facts of which prospective users should be aware: BusinessObjects Data Integrator has substantial knowledge of the source ERP and CRM systems, such as those from SAP, Oracle, Siebel and People- Soft, that it supports. Since the product closely integrates with other parts of the Business Objects product suite, this has significant advantages: for example, you can generate BusinessObjects Universes directly from the ETL product, which can take into account this knowledge of source systems. This integration means that features such as impact analysis, where used capabilities (which have been introduced in the most recent release, 6.5), dependency diagrams, and data lineage, all of which are a part of Business- Objects Data Integrator, span the entire product set. This is particularly important in the case of data lineage, because of its importance for corporate governance. BusinessObjects Data Integrator is a code generating tool. However, apart from specific capabilities like generating ABAP (for SAP environments) and optimised, native SQL it generates pseudo code, so it is not as portable as products that generate source code. On the other hand, it does not require a compiler. Yet again, the fact that it generates code means that you can inspect, debug, and test it (there are extensive facilities provided) which many will prefer to a black-box approach. The product is not limited to batch operations and also supports real-time capabilities, both directly and by allowing jobs to be published as web services, so that they can be used by external applications. In real-time environments, code is not generated but the same deployment engine is used. Bloor Research 2004 Page 1

4 Data Integration, Volume 1 Business Objects has spent considerable effort, since it acquired Acta Technologies in 2002, in improving Data Integrator s performance and scalability. This has continued with the latest release, with enhancements for distributed load balancing and failover, as well as extended use of caching. Unlike some companies, Business Objects has built parallelism directly into the product so that it comes as standard rather as an optional extra. One performance feature we particularly like is the ability to delegate processing to source or target platforms, where that is appropriate. This can have a significant impact on performance under the right circumstances, and it is a feature offered by very few vendors. In the latest release, Business Objects has announced a partnership with FirstLogic so that pre-built FirstLogic transforms are delivered with the product. However, these are limited to data cleansing, matching and linking functions only. We would have like to see this extended to FirstLogic s data profiling capabilities as well, since Business Objects has only limited capability in this area. The repository in BusinessObjects Data Integrator is CWM (common warehouse metamodel) compliant. This means that it is easy to exchange metadata with other such tools and data warehouses, of which there are a growing number. Very few, if any, other ETL vendors have adopted the CWM standard directly, though some have bridging technology to it. The bottom line The company has done a good job with BusinessObjects Data Integrator. When it was owned by Acta it was interesting but commanded little attention within the market. That has all changed, not only because Business Objects has a much larger user community at which to target the product, but because it has invested the necessary time and effort to make the product perform and scale to the extent that is required by large enterprises. Equally important, BusinessObjects Data Integrator is now closely integrated with the rest of the Business Objects product suite. Given that it is its user population at which the product is primarily aimed: should potential customers opt for BusinessObjects Data Integrator or should they consider an independent third party product? In our view, this is rapidly becoming a no contest. You would have to have some particularly powerful reasons for not using BusinessObjects Data Integrator if you were otherwise going to use Business Objects products such reasons will be increasingly hard to find. Page 2 Bloor Research 2004

5 BusinessObjects Data Integrator 6.5 Vendor Information Background information Business Objects was founded in France, in 1990, as a private company. Although it was not until 1994 that the company went public, from the outset the founders of the company had a global vision. Even at the very earliest stages, the company was talking about itself as a trans-national company, where Business Objects distinguishes between trans-nationalism and internationalism. In its approach, it devolves corporate functions to wherever is most appropriate rather than relying on historical happenstance. Thus the company was floated on the NASDAQ stock exchange and has its main administrative functions centred in the USA. On the other hand, the European support centre is in the UK while product development is done in France and Vancouver. Also, it is noteworthy that, because the business world is English oriented, the products are developed in English and then translated into French and other languages. The company s growth has mostly been organic, primarily led by its direct sales force, although it does have a number of reseller agreements with other software vendors, as well as partnerships with system integrators and the like. In 2002 it acquired Acta Technologies, whose ETL (extract, transform and load) tool formed the basis for what is now BusinessObjects Data Integrator. In addition, the company was also the purveyor of Acta ecaches, which are now known as BusinessObjects Rapid Marts. The company also licenses third-party technology where appropriate: for example, the KXEN data mining engine is embedded within BusinessObjects Application Foundation. BusinessObjects web address: Product availability Business Objects offers a wide range of capabilities, from conventional OLAP to reporting (through the recent acquisition of Crystal Decisions), and from analytic applications to enterprise performance management, scorecards and dashboards, as well as various other capabilities, as is illustrated in the diagram shown left. The wide range of capabilities from Business Objects Business Objects One thing that is important to appreciate in this diagram is the extent to which BusinessObjects Data Integrator is, in fact, integrated into this environment. For example, one of the strengths of Data Integrator, as we shall see, is its understanding of source environments such as those of SAP or PeopleSoft. This can be used to feed context, as well as data, into analytic applications, for ex- Bloor Research 2004 Page 3

6 Data Integration, Volume 1 ample. Similarly, you can generate BusinessObjects Universes directly from the transformations you define using Data Integrator. Perhaps most impressive is that when you do impact analysis in Data Integrator, this isn t just limited to analysing the impact of changes within the ETL environment, as it would be in third party stand-alone products, but it can reach out to the whole BusinessObjects environment, so that you can see the impact on Universes, queries, reports and so on. This is a much more comprehensive capability than is otherwise available. There is also a reverse side to this integration. It also means that you can use Business Objects other tools to monitor BusinessObjects Data Integrator. For example, you could define key performance indicators against data movement tasks and then use the company s metrics management capabilities to illustrate the results in a dashboard-style front-end. The current version of BusinessObjects Data Integrator is 6.5, which was released in March There are, in fact two versions of the product: the Enterprise Edition and the Departmental Edition. The difference is that the former supports SNMP for integration with systems management tools (though the product includes its own management console if you do not have such an environment), grid computing, and provides an additional server environment as well as unlimited relational database and technology interfaces. Both versions of the product are Unicode compliant. Data Integrator runs on Windows platforms and under HP-UX, Sun Solaris and AIX, while the repository, which supports the Common Warehouse Metamodel, may be based on Oracle, DB2 or SQL. The product also provides native connectivity, not only to these databases, but also to Informix, Sybase and Teradata. Otherwise it uses ODBC, though it also supports flat file data sources, XML (both DTDs and XML schemas), JMS (Java message service) and MQSeries, Tibco, and Web Services. Supported enterprise applications include SAP, JD Edwards, Oracle, PeopleSoft and Siebel. The integration with the last of these, in particular, has been enhanced in the most recent release. While Data Integrator includes some limited data quality functions of its own, Business Objects has historically partnered with both Trillium and FirstLogic to provide sophisticated data quality capabilities. However, in the 6.5 release, Business Objects has extended this integration by providing pre-built FirstLogic icons within the transformation logic of Data Integrator, thus extending the company s partnership with FirstLogic. Reviews of both the Trillium and FirstLogic products are available separately from Bloor Research. Financial results As was stated above, Business Objects was originally floated on the NASDAQ. However, its shares are also traded on the Paris Bourse. As Bloor Research does not claim to be financial analysts we will not comment on the company s recent trading figures, since they are obscured by the recent acquisition of Crystal Decisions, which was completed during December This necessarily skews the figures. Suffice it to say that total revenues were $560.8 for the year 2003, which Page 4 Bloor Research 2004

7 BusinessObjects Data Integrator 6.5 represents a substantial increase compared to Profits seem to be similarly on target. Historically, the company has focused on direct sales and it has more than 3,800 employees at offices, in addition to those mentioned, in Australia, Brazil, Canada, China, Hong Kong, India, Japan, South Korea, Mexico and Singapore, and across Europe. In addition, there are distributors around the global globe and, in total, Business Objects is represented in more than 80 countries world-wide, either directly or through its 300+ partners. Bloor Research 2004 Page 5

8 Data Integration, Volume 1 Product description Introduction BusinessObjects Data Integrator is a general purpose data integration platform and Business Objects does sell it as such. However, for the most part, it is seen by the company as a key enabler of its other technologies and it is primarily marketed in conjunction with data warehousing and other business intelligence products, though the company is fully aware that it has a large existing user base with a wide range of data integration issues. Indeed, according to Business Objects, the primary reason for the acquisition of Acta in the first place, was its understanding of the various ERP environments that it supported, and its ability to easily present information from those sources in business rather than technical terms. It is easy to see how such an understanding of the metadata could be useful to the BusinessObjects environment as a whole. Architecture The architecture of BusinessObjects Data Integrator is illustrated in the diagram left, with data sources shown on the left and targets shown on the right. However, as is the way with such diagrams there is quite a lot that is left unsaid. Nevertheless, before discussing these points, there is one thing to note about the existing diagram, which is that Rapid Marts are what Acta used to call ecaches (and before that: Rapid Marts). These are for what their name suggests: rapidly building subject-specific data marts using BusinessObjects Data Integrator as a generator. While not the main subject of this review, we will consider Rapid Marts in the details that follow. Architecture of Data Integrator Business Objects The most obvious architectural omissions from this diagram are: BusinessObjects Data Integrator uses a code generating approach when operated in batch mode. So it generates ABAP code, for example, to support SAP environments, or PeopleSoft or other relevant code when appropriate, and it understands Oracle Flexfields. It also generates native SQL (including stored procedures) for the leading database products. More generally, it generates pseudo code (which is executed by the product s engine [the Data Server] at runtime), which has the advantage that it is not compiler-dependent but the disadvantage that it is not as portable as tools that generate COBOL or C, say. More generally, there is a debate about whether it is preferable to generate code or to offer a black-box approach, which some vendors offer. The advantage of the former, apart from portability, is that you can inspect the Page 6 Bloor Research 2004

9 BusinessObjects Data Integrator 6.5 code, debug it (Business Objects provides a debugger with watch points, the ability to step through the code and so on), and optimise it. Business Objects also provides testing capabilities. BusinessObjects Data Integrator also supports real-time data movement (by listening to MQSeries, JMS, SAP i-docs and so forth) and real-time updates to dashboards and the like. In this case it does not generate code. However, precisely the same engine is used, regardless of whether movement is to be in batch or real-time modes. In version 6.5, a notable new feature is support for Oracle Change Data Capture, which provides real-time information about changes to Oracle databases. While on the subject of the engine, it is also worth noting that the first release of BusinessObjects Data Integrator after it was acquired (version 6.1) concentrated particularly on parallel everything, with parallel sorts, parallel writes, reads and so on (even on a single CPU), in order to provide appropriate levels of performance and scalability. These facilities have been significantly enhanced again in the latest release, with distributed load balancing, improved fault tolerance and increased caching all making their appearance. Note that in the case of Business Objects, parallelism is not an option, as it is with some products, but is built into the standard version of the product. Not all processing within a data movement task has to be done by the Data Server. BusinessObjects Data Integrator lets you push down processing to target or source platforms so that you can execute a join, for example, on a source prior to extracting the results. This can have significant performance advantages and is a relatively unusual facility to offer. There is native support for XML, using both document type definitions (DTDs) and XML schemas, so that you can process XML data directly, with the ability to either flatten hierarchical data or process it as is (not all competitors can do the latter). Hierarchical data can be combined with other data, as required. BusinessObjects Data Integrator has what the company calls an all-in-one design. That is, it has a single engine, a single repository, a single interface for all design and development work (including testing, debugging and monitoring), and so on. We will concentrate upon the Designer and Repository in the sections that follow. Designer The Designer uses a familiar icon-based drag-and-drop style interface, as illustrated on the next page, in which you create workflows and units of work, which you can combine with conditional logic. In this example, the tool is being used to calculate metrics that will be subsequently used to populate a dashboard. Note that although these are weekly metrics, similar workflows can be implemented in real-time if you need up-to-the-minute details within your dashboard. Bloor Research 2004 Page 7

10 Data Integration, Volume 1 The four most notable additional features of Designer are the large number of pre-built transforms are supplied out of the box; the interactive debugging facilities that we have already mentioned, but which have been enhanced in this release with facilities like breakpoints and filters; the ability to publish a job as a web service so that it can be used by an external application; and the product s data profiling capabilities. Screenshot showing Data Integrator Designer UI Business Objects Actually, data profiling is relatively limited and is confined to comparing data sets on a side-by-side basis, and by visual inspection. While this not without value it is not as comprehensive as the sorts of facilities that specialist vendors can provide. As it happens, FirstLogic, whose data cleansing capabilities are integrated with Business- Objects Data Integrator in this release, doers offer such functionality. However, Business Objects has limited itself to integration with the data cleansing, matching and linking aspects of the product in this release. It would be sensible to expand this support to include FirstLogic s data profiling capabilities. Repository The BusinessObjects Data Integrator repository is compliant with the OMG s (Object Management Group) Common Warehouse Metamodel (CWM) standard. This is a significant advantage because the trend is towards adoption of this standard across the data warehousing community, and most ETL vendors can only support it via transforms, if at all. In practice, what support for CWM means is that you get in-built support for the XML metadata interchange (XMI) standard and this in turn means that you automatically get the ability to exchange information with any other tool that uses this standard. Currently, there are upwards of 65 such tools and products, not to mention the data warehouses that support CWM. While on the subject of integration, one that is particularly important to note is support for native interchange with Computer Associates AllFusion ERWin, which you can use to reverse engineer existing data sources and targets so that you can load the relevant metadata into BusinessObjects Data Integrator. In fact, the product has some capabilities of its own in this regard (it can read a database catalog), but they are not as extensive as those provided by ERWin. The repository stores a great deal more information than just metadata about sources and targets. In particular, it also stores object metadata about data flows and workflows (where data flows describe physical movement and workflows are more about logical details) so that these can be reused; planning metadata about jobs, schedules and timings; and performance metadata about actual operations, with detail about runtimes, errors, failures, and so on. It also stores the necessary information so that BusinessObjects Data Integrator can be used to generate BusinessObjects Universes. Page 8 Bloor Research 2004

11 BusinessObjects Data Integrator 6.5 Finally, while we have already mentioned the cross-product impact analysis, we should also say that this is not limited to just analysis: the product will also help you to determine how important a particular dependency, and there is a dependency diagram so that you can visually inspect parent-child and other relationships. In addition, BusinessObjects Data Integrator also supports (in the latest release) the corollary to impact analysis: where-used capabilities. It is also important to note the product s ability to provide data lineage detail. While you would expect any ETL tool to provide an audit capability, the fact that you can not just track data from source to target but also through to the business intelligence environment that exploits that target must be a substantial advantage in these times of increased concerns over corporate governance and regulatory requirements. Rapid Marts Rapid Marts are automatically generated, packaged data marts using a pre-defined target data model with pre-built source to target mappings, and change data capture capabilities to keep the data mart up-to-date on a real-time basis. Currently, Rapid Marts are available for a wide range of SAP R/3 modules, Oracle sales analysis and general ledger, Siebel pipeline/opportunities, PeopleSoft Human Resources, and JD Edwards general ledger and sales analysis. Some but not all of these include pre-built BusinessObjects reports and universes. The company plans to expand the number of Rapid Marts it supports, and their functionality. Summary As we mentioned above, if you want an ETL tool for use within a Business Objects environment, then very few users are going to look further than Business- Objects Data Integrator. Nevertheless, there is the broader question of deploying this product as a general-purpose data integration platform for purposes that have nothing to do with data warehousing or business intelligence, or may involve third party business intelligence suppliers. It is certainly capable of fulfilling this role, and this is supported by the fact that you can use Business Objects dashboards and metrics to monitor the data integration environment. It would be fair to summarise Business Objects position as saying that its primary focus for Data Integrator is on supporting its own business intelligence needs, but that it is alive to the additional data integration requirements of its user base. Bloor Research 2004 Page 9

12 Suite 6, Challenge House, Sherwood Drive, Bletchley, Milton Keynes, MK3 6DP, United Kingdom Tel: +44 (0) Fax: +44 (0) Web: