Evaluation Checklist Data Warehouse Automation

Transcription

1 Evaluation Checklist Data Warehouse Automation October 2017

2 General Principles Requirement Question Ajilius Response Primary Deliverable Is the primary deliverable of the project a data warehouse, or is end user visualisation and analytics more important? At Ajilius we believe that a project with a data warehouse as its primary deliverable is a failure waiting to happen. Business value comes from analysis, not from data waiting for a use case. Time to deliver Can the project stakeholders wait for a data warehouse to be completed before getting results, or do they need reports and data now? Ajilius conforms to the dimensional modelling methodology, which is renowned for providing data for analysis faster than any other approach. You can test this criterion by asking the vendor of each product to show how you query their warehouse in the visualisation tool of your choice. Frequency of change Do the stakeholders have time to wait while new data requirements are analysed, modelled and work their way through the EDW backlog, or do they need results now? Dimensional modelling enables new star schemas and cubes to be built quickly to solve analytics problems at the speed of Business, not IT. A good test of this criterion is to ask the vendor to demonstrate, live and without notice, the integration of a new data source into their target warehouse. Page 2 / 19

3 Dimensional Modelling Requirement Questions Ajilius Responses Load Tables Is a special class of staging table defined for the loading of source data into the DBMS for further processing? Yes. We refer to this class as a Load Table. Staging Tables Are staging tables supported for the transformation of data, ready to be loaded into dimension and fact tables? Yes. Persistent Staging Tables Types of Staging Table Can the data in staging tables be persisted across load schedules? Is there any built-in intelligence associated with specific types of staging table? Yes, at the choice of the data warehouse developer. Ajilius supports transactional, latest and historised persistence. Yes. Ajilius supports six types of staging table. Transactional tables are sets of data with one row for each source row, and may contain joins across tables. Union tables represent the union of two or more predecessor tables. Aggregate tables contain fewer rows than the predecessor table, and may contain typical aggregate values such as sum, count and average. Deduplicate tables remove duplicate rows by selected column values. Page 3 / 19

4 Flatten tables flatten ragged parent-child hierarchies. Unpivot tables convert data from multiple columns to multiple rows. Dimensions What types of dimension table have built-in intelligence supported by the data warehouse automation product? Ajilius generates data structures and code to support Type 0 (date and time) dimensions, Type 1 (overwrite), Type 2 (add row) and Type 3 (add column) dimensions; as well as combinations of these. Fact Tables What types of fact table have built-in intelligence supported by the data warehouse automation product? Ajilius generates data structures and code to support transactional fact tables, periodic snapshots and accumulating snapshots. Data Screens Does the data warehouse automation product have built-in intelligence to support data screens? Yes. Refer to the section on Data Quality. Page 4 / 19

5 Integration Requirement Questions Ajilius Responses Data Warehouse Platform List the target data warehouse platforms supported by the product. Ajilius supports the following target data warehouse platforms in Version 3.x: Microsoft SQL Server Microsoft Azure SQL Microsoft Azure SQL Data Warehouse AWS Redshift Snowflake Elastic Data Warehouse PostgreSQL EnterpriseDB Exasol MariaDB ColumnStore Data Sources List the sources from which data may be loaded by the product. Ajilius supports a large number of data sources in V3.x, including: PostgreSQL EnterpriseDB Microsoft SQL Server Oracle Sybase DB2 MySQL MariaDB AWS Aurora Hadoop MongoDB SQLite Microsoft Excel Microsoft Access Microsoft SharePoint Microsoft Dynamics Page 5 / 19

6 Salesforce Google Sheets Delimited text files XML JSON Note that some data sources require separate licensing of third-party JDBC drivers. Additional data sources may be supported through a JDBC driver if available for the platform. We will validate new data sources on request. Additional Data Sources Does the product support an interface such as ODBC or JDBC for integration with data sources for which native drivers are not provided? Yes. Ajilius supports JDBC interfaces for high quality, vendor supported integration to most data sources. Credentials How are database credentials secured in the metadata database? Ajilius uses a combination of SHA256 hashing and AES-256 encryption to ensure database credentials for data sources and the data warehouse are secure from prying eyes. Job Scheduler Does the data warehouse automation product integrate with enterprise job schedulers, such as <insert name here>? Yes. Ajilius generates scripts and batch files appropriate for the target platform. These can be invoked by any known Job Scheduler. We will validate job scheduler integration on request. Ajilius includes a dependencybased command line Page 6 / 19

7 scheduler which makes integration a breeze. ETL Tool Can the data warehouse automation product integrate with ETL tools such as <insert name here>? Yes. The scripts generated by Ajilius can typically be integrated with job execution components which are common to most ETL tools. We will validate ETL tool integration on request. Existing Data Warehouse Can the data warehouse automation product be integrated into an existing data warehouse? Yes, assuming that the data warehouse platform is one of those supported by Ajilius. Our Virtual Load capability makes it easy to re-use data from an existing warehouse, without having to re-load and re-process it before use. Stored Procedure Dependency Does the data warehouse automation product have a dependency on stored procedures in the target platform? No. Ajilius is designed to be a cross-platform data warehouse product, and some of our supported platforms do not contain a stored procedure language. Beware of products that only generate stored procedures, as you may be locked into a target platform. Data Warehouse Migration Can the data warehouse automation product migrate a data warehouse from one platform to another? Yes, this is a key feature of Ajilius. A data warehouse may be migrated from one platform to Page 7 / 19

8 another say, SQL Server to PostgreSQL with a simple three-click migration process. If a vendor claims to be able to migrate between data warehouse platforms, make sure that you have this capability demonstrated during the evaluation process. It may be a very complex, manual process in some products. BI Product Integration Custom Code Integration Does the data warehouse automation product integrate with any BI tools, and to what extent? Can the data warehouse automation tool integrate custom code? Ajilius generates integration metadata for Yellowfin, Tableau and Qlik. This saves our customers significant time entering BI tool metadata such as table and column definitions, formats, descriptions, rules, etc. Additionally, Ajilius automatically generates a set of named views that use business terminology instead of database table and column names. These are ideally suited to integration with products like PowerBI, Excel, Microstrategy, etc., for which a rich metadata API is not yet available. Ajilius enables custom code before and after all load and transform jobs. For load jobs, custom code is written in the Groovy scripting language. For transform jobs, custom code is written in SQL. Page 8 / 19

9 Data Quality Requirement Questions Ajilius Responses In-built Data Quality Does the product have in-built support for data quality screens? Yes. Validation Capability Is data validated by type? Yes, validation is provided for all Ajilius data types. Is data validated by range? Yes, both discreet and continuous ranges may be specified for all data types. Is regex validation supported? Yes, regex validation is fully supported through the Java regular expression library. No 3 rd. party dependencies Does data quality depend on the integration of any third party software? No. All data quality operations are fully integrated into Ajilius. No additional cost Does data quality incur any additional licensing fees? No. Data quality has been implemented as a standard feature of the Ajilius data warehouse automation platform. Page 9 / 19

10 Usability Requirement Questions Ajilius Responses User Languages Which languages are supported for developers of the data warehouse? Ajilius currently supports English, Malay, Indonesian, and Vietnamese. Adding new languages takes 2-3 days of professional translation. We will undertake this work on request, feel free to request it before a demonstration. User Interface What technology delivers the user interface of the application? Like most modern computer systems, Ajilius is delivered through a web browser. We fully support current versions of Internet Explorer, Firefox, Chrome and Safari. We will test other browsers and/or versions on request. Desktop products lock you into legacy architectures that are costly to deliver and support. Device Support What devices can be used to operate all features of the application? Thanks to our responsive application design within the browser, every end-user feature of Ajilius is supported on desktop, tablet and smartphone devices. If a vendor claims that their application can be delivered on a tablet (or smaller) device, make sure that you test all dialogs for fit on the screen, and that the application UI can be scaled to make it usable at Page 10 / 19

11 high resolutions on small devices. Touch Friendly Does the application be driven entirely by touch on tablet and smartphone devices? Ajilius is fully touch compliant. All operations can be conducted on a tablet or smartphone, without needing an external keyboard connection. If a vendor claims touch compliance, make sure that their application is not just a full screen display squashed into a small tablet screen, and that it has plenty of room around controls for finger operation. Screen Scaling Can the user interface scale to support a variety of screen sizes and resolutions? Can the user interface scale to be more readable by vision impaired users? Ajilius supports browser 'zoom' controls, which typically enable the user to make fonts larger and more readable. Key elements of the user interface, such as the number of rows displayed per page in tables, and the amount of space dedicated to free text editors, may be customised through user settings. If a vendor claims to be able to zoom their display, make sure that all user interface elements scale well, including icons and images, and all text fields. Screen Customisation Can the colour, font and size of user interface elements be customised by the user? Ajilius uses the Bootstrap model for browser interfaces, and all user interface elements can be customised Page 11 / 19

12 through the Bootstrap CSS elements. To evaluate this criterion, ask the vendor to customise specific colour or control elements while you watch. Page 12 / 19

13 Infrastructure Requirement Questions Ajilius Responses Application Dependencies What additional software products must be licensed in order to run the data warehouse automation software? Ajilius is dependent on the Java Runtime Engine Standard Edition JRE8. This is a free download from Oracle. No other downloads and/or installations are required to support Ajilius. Make sure that each vendor accounts for the versions of operating systems that may need to be purchased, application framework dependencies (ie,.net 3.5), database management systems that may be required to host their metadata, and additional drivers that may be necessary to connect to your data sources. Server Operating Systems On which server operating systems is the data warehouse automation product supported? Ajilius is fully supported on all versions of Windows, Linux and OSX that are in mainstream support from their respective providers. Beware of products that do not offer a server-based installation, these are typically priced on an expensive peruser basis. Client Operating Systems From which client operating systems may the data We fully support Ajilius on all versions of Windows, Linux and OSX that are in Page 13 / 19

14 warehouse automation product be accessed? mainstream support from their respective providers. Other operating systems will be validated on request. There is no dependency between the client operating system and the server operating system. An OSX client, for example, may access a Windows server. Web Browsers Which web browsers may be used to access the data warehouse automation product? Ajilius is fully supported on current versions of Chrome, Firefox, Internet Explorer and Safari on desktop devices. Ajilius is fully supported on Chrome, Dolphin and Safari browsers on mobile devices. Other browsers and platforms will be validated on request. Metadata Repository Which DBMS is used to store the metadata managed by the data warehouse automation product? Ajilius uses the SQLite3 DBMS to hold all application control data and metadata. This DBMS requires no licence fee. Beware of the potential cost of products that require their metadata to be stored in commercial DBMS platforms. Metadata Location Is the metadata stored independently from the data warehouse? Ajilius stores its metadata in a separate repository for each data warehouse, and that repository is held separately from the warehouse. Products which store their metadata in the warehouse Page 14 / 19

15 make it difficult to migrate across data warehouse platforms, enforcing a lock-in to your current architecture. Metadata Upgrades How is metadata upgraded to reflect changes in Ajilius? Ajilius metadata upgrades are automatically applied when a new version of Ajilius is installed. Upgrades are cumulative, and will automatically apply earlier metadata changes if required. Page 15 / 19

16 Constraints Requirement Questions Ajilius Responses Maxumum number of users What is the maximum number of users (developers) that may concurrently access the product? Ajilius imposes no limits on the number of developers that may concurrently access the application. Optimistic locking is applied to all metadata updates, to protect against typical concurrency issues. We note that our use of SQLite as a metadata store may slightly lengthen response times where the number of concurrent developers actively updating the metadata is greater than 10. Multiple instances of Ajilius can resolve this problem if it occurs. Maximum number of tables What is the maximum number of tables that may be supported in a data warehouse? Ajilius supports the maximum number of tables that are supported for the target DBMS. Maximum number of columns What is the maximum number of columns that may be supported in a table? Ajilius supports the maximum number of columns that are supported for the target DBMS. Maximum identifier length What is the maximum length of identifiers such as table names and column names? Ajilius sets a maximum identifier length of 63, to support identical migration across supported data warehouse platforms. Page 16 / 19

17 Reserved word detection Does the data warehouse automation product support the detection of reserved words in identifier names? How are reserved words resolved? Ajilius holds an internal dictionary of all reserved words for all supported data warehouse platforms. When defining a new object, names are checked against this list. During manual maintenance, such as creating a new table name, the reserved word will trigger an error and must be corrected by the user. During automatic updates, such as importing metadata from a data source, the reserved word will be flagged with the prefix 'reserved_', and can later be modified by the user. Duplicate identifier detection Does the data warehouse automation product support the detection of duplicate table and column names? How are duplicate names resolved? Ajilius prevents the creation of objects with duplicate names. Attempting to manually create a table or columns with the same name as one already defined in metadata will trigger an error, and must be corrected by the user. During automatic updates, such as merging the metadata of two or more tables during a join, duplicate column names will have an incrementing suffix appended to keep them unique, which may later be updated by the user. Page 17 / 19

18 Costs Requirement Questions Ajilius Responses Public Price List Does the vendor publish their price list? Ajilius' pricing is directly linked from the front page of our web site. We ve got nothing to hide. Cost Per User What is the licence cost per user? What are the break-points at which licence costs per user are reduced? Ajilius is not licensed per user, it is only licensed per site. Cost Per Site What is the licence cost for a site licence? What is the definition of a site? A site licence costs USD 15,000 per site per year. The definition of a site is an organisation sharing a common business identity. This is usually represented by an domain, but in cases like whole-of-government services organisations, the site is the Department, Ministry, or other beneficial entity. Cost Per Warehouse What is the licence cost per data warehouse implementation? Ajilius users can create as many data warehouses as they like, with no additional licence obligation. Cost Per DBMS What is the licence cost per source DBMS? Ajilius imposes no licence cost per data source, regardless of whether the data source is Page 18 / 19

19 What is the licence cost per target DBMS? running on a free or commercial platform. Ajilius imposes no licence cost per target DBMS, other than the applicable site licence fee. Further reducing the DBMS cost of data warehousing, Ajilius supports a number of low-cost and open-source target data warehouse platforms. Additional Unit Costs Are there any additional cost constraints that may require additional licensing? Ajilius has no additional licensing costs. The use of 3 rd. party JDBC drivers, from companies like CData, Simba, Progress, etc., may require payment of additional licence fees to the relevant vendor. Annual Maintenance Is there an annual maintenance fee? Is the annual maintenance fee mandatory? Ajilius does not have any annual maintenance charge, but see below for our annual renewal policy. Annual Renewal Must annual licences be renewed? No. If you choose not to renew your licence, you must discontinue the use of Ajilius, but may continue to use the scripts which have previously been generated. Page 19 / 19