InDetail. expressor software

Transcription

1 InDetail expressor software A InDetail Paper by Bloor Research Author : Philip Howard Publish date : May 2009

2 Anyone needing to process high volumes of data and/or needing to support complex processes should immediately put expressor on their short list of data integration products to consider. Philip Howard

3 Executive summary Talk to any vendor in the data integration market and they will tell you that their biggest competitor is hand coding. Why? The simple answer is that many users do not believe that the capital costs involved in investing in data integration can be recouped through the reduced development requirements and performance benefits that can be derived from using a tools-based approach. expressor has set out to challenge this assumption through a semantically aware product that makes development (and data governance) much simpler, combined with a product that has been designed from the outset to offer the best possible performance at a readily affordable price. Fast facts expressor is provided by the company of the same name. It is a data integration tool that also provides significant capabilities to support data stewardship. However, it differs significantly from other products within the data integration market. The basic premise behind expressor is that it understands the meaning of things. It understands, for example, that cust_no is the same as cust#, which is the same as cust_id and so on. It is difficult to over-emphasise the importance of this. What it means is that you don t have to discover these identities across your various data sources and you don t have to tell the product about them. This saves considerable manual effort. Moreover, because expressor understands these identities it can create relevant mappings automatically, thereby simplifying the whole development process. In addition, because all equivalent terms are normalised to common business terms, this makes the product very much more amenable to use by business analysts and data stewards who can more easily collaborate with developers because of this common terminology. The other major emphasis within expressor is on performance. The product has been designed from the ground up to make optimal use of parallelisation and it is notable that expressor is directly targeting existing Ab Initio customers. Given that Ab Initio has traditionally been regarded as the best performing data integration product in the market, this is a bold move. Nevertheless, we understand that expressor is out-performing Ab Initio in customer benchmarks. In general, the product is focused on environments where the main issues are performance (typically meaning large volumes of data) and complexity. While the latter is simplified through semantics the former is addressed not only through the product s architecture but also via the company s innovative channelbased pricing model. In essence a channel is a unit of parallelism: thus the idea is that you license just as many channels as you need to get the performance you want: you pay for exactly the performance you need. Key findings In the opinion of Bloor Research, the following represent the key facts of which prospective users should be aware: Optimal use of parallelism has been designed as a fundamental feature of expressor. This differs from most of the rest of the market for which (less efficient) parallelism has been added as an afterthought. As a result, expressor is extremely fast and only requires a small footprint. Semantics is at the heart of expressor. This has significant advantages in terms of both data integration and data governance. In particular, it enables both faster development and improved collaboration. There are facilities provided within expressor for business analysts, data stewards and developers, thus further supporting the notion of collaborative working. These facilities are furthered by the use of a role-based approach that is adopted throughout the product. For development and testing purposes, logic is separated from data. In other words, you don t have to know about data structures in order to develop your processes. A traditional graphical approach to development is provided and you can also use the datascript scripting language that is provided by expressor for complex operations. expressor datascript is an extended subset of the open source Lua language. For companies not having Lua skills this will require some education though Lua has similar semantics (though not syntax) to JavaScript. Lua does have the advantage that it can also be used at the operating system level. A Bloor InDetail Paper Bloor Research

4 Executive summary expressor is not yet complete. This is hardly surprising in a product that is still relatively new. Two major additional facilities that the company plans to introduce in due course are scheduling and data quality modules. expressor supports both batch and real-time processing of data. However, change data capture is not yet fully supported: it currently supports flat-file/table change data capture and is expected to add support for relational data capture later in The pricing model adopted by expressor is very attractive: it means that you pay for exactly the power that you need. The bottom line We are very impressed with expressor. Both in its use of parallelism and semantics it has all the hallmarks of a next generation product We are very impressed with expressor. Both in its use of parallelism and semantics it has all the hallmarks of a next generation product and all the advantages that derive from making the best use of up-to-date technology. As we have stated, the product lacks some features as yet, but this is hardly surprising and, in any case, you can get these facilities through third parties if you really need them now and cannot wait for expressor to introduce them. For most potential users this will not be an issue the cost savings, performance benefits and productivity advantages offered by expressor are significant anyone needing to process high volumes of data and/or needing to support complex processes should immediately put expressor on their short list of data integration products to consider Bloor Research A Bloor InDetail Paper

5 The product The latest release of expressor is 2.0, available from June It runs under Windows, both 32-bit and 64-bit Linux (Red Hat and SUSE), AIX, and Solaris. Because of its physical architecture (see later) and channel-based licensing model, expressor is amenable to deployment via a cloud-based approach. With the exception of the administrator, all enduser modules are Windows based. In terms of connectivity expressor supports ftp; native database access to Oracle, DB2, SQL Server, Sybase, Teradata and Netezza; ODBC; XML and flat files; message queues (JMS, MQ Series, Microsoft MQ and WebLogic JMS); SWIFT; EDI; COBOL copybooks, time series data and SAP IDocs. There is also support for SOA. It plans to add support for rdbms change data capture later in The product consists of three major elements, as follows: 1. integrator, which is a suite of end-use tools consisting of administrator, initiator, constructor, illustrator and communicator. Two further modules: coordinator (which will provide scheduling and related capabilities) and investigator (which will provide data profiling and other data quality functions) are planned for a future release. 2. repositor (sic) is the product s repository, which is a J2EE-based, Geronimo application with an embedded Derby database and a fully integrated Subversion version control system. 3. processor is the expressor processing engine. In the following sections we will discuss expressor in the order indicated, with the exception that we will leave communicator to last. A Bloor InDetail Paper Bloor Research

6 The product integrator Because expressor represents a significant departure from the way that things are normally done when using a data integration tool, it will be easiest to follow a logical progression through the various steps required to develop processes within expressor. However, bear in mind that the following is simplified and that various steps may actually be carried out partly or completely in parallel. The first thing that you will do is to create a project using the web-based administrator. A project might be an ETL job, a data synchronisation task, data migration; whatever. Next you will assign people to that project on a role (data architect, business analyst, developer and so on) basis. Again, this is done using administrator which supports LDAP (though not Active Directory or Kerberos). Figure 1: Editing an image in administrator Following the assignment of roles you will define your data sources, which is, again, done through administrator. The structure of these sources is captured in XML-based image files (see Figure 1). Similar network files are also defined using administrator where these contain details about the location of data and processing and (perhaps added later) the degree of parallelism to be deployed. After this the emphasis shifts to the semantic element of the product. expressor ships with a semantic dictionary that contains more than ten thousand terms and their equivalents. Thus the software already knows that part_#, part_no and pnum are all equivalent to part_ number, which is the business term that this user has settled on. Of course this isn t the end of the story. There may, for example, be industry-specific terminology that the out-of-the-box dictionary does not cover. To enable the inclusion of these the initiator module is used, which allows the bulk import of relevant dictionaries from external sources, and provides syntax checking and data validation. This is illustrated in Figure 2. Nevertheless, pre-shipped and imported dictionaries are not enough on their own. You also need to define the terminology that is specific to your company and organisation. You need to decide that part_number, for example, is your standard business term and that all equivalents should match to that. This is accom- Figure 2: Semantics in expressor Figure 3: expressor initiator plished via the constructor module, typically by a data steward. expressor will attempt to match all terms automatically but it will make appropriate suggestions where it cannot be sure of a match. constructor has two interfaces, one spreadsheet style (illustrated in Figure 3) and one wizard-based. However, it is not limited to Bloor Research A Bloor InDetail Paper

7 The product Figure 4: A drawing created using illustrator use by data stewards. It is also used by business analysts to define target-centric business rules along with relevant test cases. These rules are defined in expressor datascript where datascript is an extended subset of the Lua open source scripting language (which has semantic similarities to JavaScript though rather different syntax; and which can operate at the operating system level as well as within expressor). Note that the creation of image and network files as separate entities isolates the business analyst from needing to know about data structures when developing rules. At the developer level, expressor uses a conventional visual paradigm, based on Microsoft Visio, to capture data processing logic. The developer starts by checking out a project and loading the relevant image and network files that define the data sources, record structures and parallel processing methods to be used. He or she then creates visual representations of the processes to be deployed in one or more drawings, which are created by dragging and dropping relevant shapes from the product s palette onto the canvas, as illustrated in Figure 4. A full set of operators to perform functions such as joining, transforming, sorting, collating (which allows the automated generation of relevant statistics such as standard deviations) and partitioning are provided within illustrator, though you can also drop into datascript (via the editor supplier) for more complex functions if necessary. The resulting scripts can either be embedded within a drawing shape or written to an external file that is called from an appropriate shape. illustrator includes a debugging capability and, in particular, it supports synthetic debugging, which allows you to test processing logic by using text values and without having to connect to the actual data. However, while you don t have to connect to the data you may wish to. To do thus you can leverage network files that point to either production or historical text data and you can choose to use these during development or debugging. The actual data is protected so that, unless you have appropriate privileges, you will not be able to see actual values. Two further points to note are, firstly, during development the software provides you with feedback through colour coding and a message window and, secondly, you can test drawings on a remote system for development purposes. repositor repositor is expressor s repository, which is used to store, manage and support the reuse of all the project artefacts that we have already discussed, including the product s semantic dictionary, business rules, run-time details and so forth. In addition to these being stored by project they are also associated with subject areas (such as banking, account, country and so forth) of which more than one hundred are pre-defined within the product. You can then manage artefacts by subject. As a repository, repositor includes the normal sort of version control, branching, snapshots, check-in/out, and multi-developer support one might expect. A Bloor InDetail Paper Bloor Research

8 The product processor expressor was designed from its inception to support parallel processing and processor is the engine that does this. As such it includes support for both implicit parallelism (whereby the language automatically takes advantage of available parallelism) and explicit parallelism (where it is under the developer s control). Pipeline parallelism, whereby different (but often related) tasks are carried out in parallel, is also supported both for moving the data and processing (depth parallelism) it. Data partitioning is also used. However, perhaps most significant is the product s support for channel-based parallel partitioning. A channel is, effectively, a unit of parallelism. Thus, if you have four parallel processing streams between a source and target you are using four channels. In many traditional environments parallelism is based on multiple processing nodes whereby you have one channel per node. However, in the case of expressor parallelism is an intra-node capability rather than an inter-node one (cross-node support is in development). Thus the key issue is the number of cores not the number of nodes. The big advantages of this approach are that it is less expensive and requires a smaller footprint in hardware terms; it enables the channelbased pricing introduced by expressor whereby you only license the performance that you actually need; and it provides great performance: in its own tests, and using the compression techniques it can employ, expressor has demonstrated load speeds in excess of 10Tb per hour, which is outstanding. communicator communicator represents the product s reporting environment, whereby all relevant data is stored in a relational database that mirrors repositor. This has the advantage that running reports and queries has no impact on the run-time performance of processor. Also, because the communicator database is read-only (in other words the data cannot be changed) this means that it can be used for compliance purposes to abide by regulations such as Sarbanes-Oxley to prove data lineage, for example. You can use any standard reporting tool that is SQL-based or has an ODBC or JDBC connector (including Microsoft Excel) to generate whatever reports you require. A number of sample reports are provided that can be used either directly or as templates for further development. In addition, some twenty pre-written queries are provided, with functions such as who created one of these items, when was one of these items changed and list all the data sources (images) used in this project, that you can drop into reports, therefore helping to simply the process of report creation. In practice there are two types of report you might require, one that works against the repository and the other to provide run-time information. In the former, an example of which is illustrated in Figure 5, this means things like data lineage, impact analysis, dependency tracking, dictionary and artefact usage and so on; while in the latter case it means that you can report on operational statistics. Apart form its performance, processor provides the sorts of capabilities one might expect, such as the ability to run in either intermittent (batch) or perpetual (real-time) modes along with support for functions like exception handling. Figure 5: A semantic report created by communicator Bloor Research A Bloor InDetail Paper

9 The vendor expressor was founded in 2003 and came to market with the first beta version of its eponymous product in March 2008 followed by a general release in June of that year. The company is backed by venture capital and raised a B round of funding in August While the company has a direct sales force in the United States this is relatively small and is currently focused on obtaining reference accounts. More generally, the company s go-to-market strategy is to leverage partnerships, systems integrators and resellers. In the latter category it has partners in the UK and Switzerland (covering Austria and Germany as well) as well as in the United States. In terms of systems integrators there are a half dozen further partners, of which the most notable is BitWise, which has developed a migration tool from Ab Initio and Informatica environments that converts their diagrams into ones that can leveraged by expressor. The company also has a partner specialising in the US federal government market, which is an area that the company is targeting. In terms of technical partnerships the company has partnerships with HP, Intel, Netezza, Dataupia, Kognitio, BIReady, Panopticon and nextanalytics. A number of these partnerships, for example those with Netezza and BIReady, go beyond mere marketing and extend to technical integration. expressor web address: A Bloor InDetail Paper Bloor Research

10 Summary There is a parallel between expressor s releasing a product that challenges the performance leader in its space with Netezza s entry into the data warehousing market a few years ago. The latter has completely changed the landscape of the data warehousing environment, which was previously relatively moribund. The same applies to data integration. While it is too early to say definitively that expressor s new approach, and its latest generation technology, will have the same effect, the signs are encouraging. expressor appears to have marketleading technology backed up a strong marketing program. With those two elements combined we expect it to become a company that everybody will be talking about Of course, success in introducing potentially disruptive technologies is not solely a question of introducing superior technology at a reduced price point; it is also a matter of marketing. Here too, expressor is taking a sensible approach: it is focusing on Ab Initio accounts initially because if it can prove its case there then it should have little difficulty in competing effectively with the likes of IBM and Informatica. Moreover, it is encouraging that it is already building up a partner network that is taking it to Europe as well as further afield, and the fact that its UK partner Emunio is also a leading Ab Initio specialist is telling in its own right. Further, its partnership with Netezza is also driving interest and the company is actively engaging with other leading data warehouse vendors. To conclude, expressor appears to have market-leading technology backed up a strong marketing program. With those two elements combined we expect it to become a company that everybody will be talking about. It represents a significant challenge to the leading incumbents within the data integration space. Further Information Further information about this subject is available from Bloor Research A Bloor InDetail Paper

11 Bloor Research overview Bloor Research has spent the last decade developing what is recognised as Europe s leading independent IT research organisation. With its core research activities underpinning a range of services, from research and consulting to events and publishing, Bloor Research is committed to turning knowledge into client value across all of its products and engagements. Our objectives are: Save clients time by providing comparison and analysis that is clear and succinct. Update clients expertise, enabling them to have a clear understanding of IT issues and facts and validate existing technology strategies. Bring an independent perspective, minimising the inherent risks of product selection and decisionmaking. Communicate our visionary perspective of the future of IT. Founded in 1989, Bloor Research is one of the world s leading IT research, analysis and consultancy organisations distributing research and analysis to IT user and vendor organisations throughout the world via online subscriptions, tailored research services and consultancy projects. About the author Philip Howard Research Director - Data Philip started in the computer industry way back in 1973 and has variously worked as a systems analyst, programmer and salesperson, as well as in marketing and product management, for a variety of companies including GEC Marconi, GPT, Philips Data Systems, Raytheon and NCR. After a quarter of a century of not being his own boss Philip set up what is now P3ST (Wordsmiths) Ltd in 1992 and his first client was Bloor Research (then ButlerBloor), with Philip working for the company as an associate analyst. His relationship with Bloor Research has continued since that time and he is now Research Director. His practice area encompasses anything to do with data and content and he has five further analysts working with him in this area. While maintaining an overview of the whole space Philip himself specialises in databases, data management, data integration, data quality, data federation, master data management, data governance and data warehousing. He also has an interest in event stream/complex event processing. In addition to the numerous reports Philip has written on behalf of Bloor Research, Philip also contributes regularly to and Analysis.com and was previously the editor of both Application Development News and Operating System News on behalf of Cambridge Market Intelligence (CMI). He has also contributed to various magazines and published a number of reports published by companies such as CMI and The Financial Times. Away from work, Philip s primary leisure activities are canal boats, skiing, playing Bridge (at which he is a Life Master) and walking the dog.

12 Copyright & disclaimer This document is copyright 2009 Bloor Research. No part of this publication may be reproduced by any method whatsoever without the prior consent of Bloor Research. Due to the nature of this material, numerous hardware and software products have been mentioned by name. In the majority, if not all, of the cases, these product names are claimed as trademarks by the companies that manufacture the products. It is not Bloor Research s intent to claim these names or trademarks as our own. Likewise, company logos, graphics or screen shots have been reproduced with the consent of the owner and are subject to that owner s copyright. Whilst every care has been taken in the preparation of this document to ensure that the information is correct, the publishers cannot accept responsibility for any errors or omissions.

13 2nd Floor, St John Street LONDON, EC1V 4PY, United Kingdom Tel: +44 (0) Fax: +44 (0) Web: