Whitepaper: Reporting on Google Big Query with SAP BusinessObjects BI4.x

Transcription

1 Whitepaper: Reporting on Google Big Query with SAP BusinessObjects BI4.x Applies to: SAP BusinessObjects BI.4x Summary This whitepaper summarizes how to enable connectivity for Google BigQuery for SAP BusinessObjects Enterprise BI4.0 SP4 and higher. This connection is a customization on top of the default connectivity of the SAP BusinessObjects BI4.x Suite. Further investigation is ongoing to formally support Google BigQuery within the BI 4.x Suite however; nothing is confirmed at this point of time. Author: Merlijn Ekkel Director BI Solutions SAP BI GTM Company: SAP Created on: 20 October 2014 Author Bio Merlijn Ekkel is a Director BI Solutions within the SAP BI Go To Market SAP. In this role he focus on driving the adoption of SAP BI Solutions from a global perspective. Merlijn has over 15 years of experience in (SAP) BusinessObjects solutions, implementations, delivery and advisory. As Director BI Solutions, he has become a bridge between the SAP Prodcut Development Teams and out customers, taking in his years of expereince to improve the SAP BI solutions today and in the future. 1

2 Table of Contents Pre-requisites... 8 Enable Google BigQuery API... 8 Validating Google BigQuery Installation of Simba BigQuery ODBC Driver bits ODBC Driver install bits ODBC Driver install Configuring Simba BigQuery ODBC Driver Main ODBC.PRM File Extension Legacy ODBC.prm Building a Universe (UNX) on Google BigQuery using IDT Creating Connection using IDT Building a Universe using IDT Building a Universe (UNV) on Google BigQuery using UDT Creating Connection using UDT Building a Universe using UDT Web Intelligence Rich Client Crystal Reports Lumira 1.x Explorer Dashboards Crystal Report for Enterprise

3 Document update history Version Author Date Comment 0.1 Merlijn Ekkel May 22nd, 2013 Initial Draft Version 0.2 Merlijn Ekkel June 3rd, 2013 Minor Corrections Added Contributors Updated BI4.x Clients Support 0.3 Merlijn Ekkel September 2nd, 2013 Minor Corrections 0.4 Merlijn Ekkel December 19th, 2013 Minor Corrections + BI4.1 Validation 1.0 Merlijn Ekkel October 2014 Final Release 3

4 Background Several customers using SAP BusinessObjects BI4.0 have been asking if there is a way to query against Google BigQuery. This has led to a prototype where SAP BI Solution Management, SAP Development and Google has been working together to validate what can and needs to be done to support Google BigQuery within the SAP BusinessObjects BI4.x product suite. The collaboration between Google and SAP has resulted to a working prototype, by adopting the SIMBA ODBC driver for BigQuery and the modification of the data connectivity configuration files of the SAP BusinessObjects BI4.x suite. This document provides that steps required to enable Google BigQuery as a source for the Universes available within SAP BusinessObjects BI4.0 4

5 Document Contributors Company Name Role SAP Pierpaolo Vezzosi Product Manager Marc Daniau Product Owner David Mobbs Product Owner Google Carlos Cuesta Vishwas Agashe Ryan Boyd Ju-Kay Kwek Michael Manoochehri Simba Technologies Amynmohamed Rajan Kyle Porter 5

6 Disclaimer This whitepaper describes a connectivity option that is UNSUPPORTED by SAP BusinessObjects BI4.x OR any other version / product of the SAP BusinessObjects portfolio. By that the connectivity, as described in this document, is considered as a customization of the product. Any issues that might be arising by using the connectivity as described in this document or following due to implementing the connectivity are on your own risk and not supported by SAP and cannot be addressed by the SAP Support teams. Any customization made within the SAP BusinessObjects BI4.x product line is by default support policy unsupported. In case issues are experienced after implementing the connectivity against Google Big Query, it is required to undo all changes made and validate if the issue still persists. Only when the issue is clearly not directly or indirectly related to the customization, you can open up a support ticket for the issue. 6

7 Pre-Requisites The following Pre-Requisites are in place to enable connectivity of SAP BusinessObjects BI4.x to Google Big Query. - SAP BusinessObjects BI4.x Client and (optional) SAP BusinessObjects BI4.x server deployment - Simba BigQuery ODBC Driver - Google Account - API Access to Google Big Query - Google Chrome Browser 7

8 Enabling Google BigQuery Pre-requisites To enable Google BigQuery for a prototype the following pre-requisites are applied: - An Google Account is available - Credit Card details can be used and entered for billing of BigQuery Usage (linked to the Google Account) Enable Google BigQuery API To enable connection into Google BigQuery, the Google BigQuery API has to be enabled within the Google Account that will be used for the connectivity. Enabling the Google BigQuery API can be achieved via the following steps: 1. Connect with the browser to 2. Login with the Google Account that will be used for the connectivity 3. The Google API Project main page will be shown 8

9 4. Navigate to Services 5. The full list of API services from Google will appear 6. Find the BigQuery API 7. if required: Review the Pricing for BigQuery ( 8. Switch the status of the BigQuery API to ON 9

10 9. Within the list of active API s the BigQuery API has appeared 10. BigQuery Access has now been activated. NOTE: It can take up to 1 hour before the activation is completed! Billing of Google BigQuery The use of Google BigQuery is subject to charge, for details around the pricing please read If the Google Account has not yet been configured for billing, the following steps have to be made additional to the Enablement of the Google BigQuery API. 1. Within the Google API page, select the Billing menu 2. Enter the required details to enable the billing for using BigQuery 10

11 3. If after entering the billing details a warning appears, switch to the profile page to enable your contact details 11

12 4. An confirmation has been send to the entered billing contact 12

13 5. Check with the billing contact entered in the previous steps for receiving of the following , this may take a few minutes before arriving) 13

14 6. After verification has been completed, the Billing details will appear as validated 14

15 Validating Google BigQuery To ensure the Google BigQuery API is successfully activated a simple test can be executed via the Google BigQuery Web Interface. Google provides a set of samples that can be used for the validation of the BigQuery connectivity. To validate the activation of the BigQuery API for the account defined in the previous chapter, navigate to Note: only Google Chrome is supported for the graphical web interface of BigQuery Sample steps to validate the activation of the Google BigQuery API: 1. Using Chrome Browser, navigate to 2. Login with the Google Account registered for the BigQuery API 3. Unfold : Publicdata:samples and select a sample database 4. Review the table structure and run a query against the table by selecting the Query Table Button 15

16 5. The base query will appear in the New Query panel of the screen 6. Modify the query along your wishes to query data from the table. I.E: 7. Click the Run Query button to have the query executed against the table and evaluate the results 8. Additional Sample Queries: a. SELECT * FROM [publicdata:samples.natality] WHERE YEAR = 1980 LIMIT 1000; b. SELECT * FROM [publicdata:samples.natality] WHERE YEAR = 1980 LIMIT 1000; c. SELECT SUM(born_alive_alive) AS ALIVE, SUM(born_alive_dead) AS DIED, SUM(born_dead) AS DEAD FROM [publicdata:samples.natality] WHERE YEAR = 1980 LIMIT 1000; d. SELECT month, SUM(born_alive_alive) AS ALIVE, SUM(born_alive_dead) AS DIED, SUM(born_dead) AS DEAD FROM [publicdata:samples.natality] WHERE YEAR = 1980 GROUP BY month LIMIT 1000; Note: Running these queries will result in a charge against your billing information provided in previous steps. Usefull references: Title BigQuery Browser BigQuery SQL Reference BigQuery Cookbook URL

17 Simba BigQuery ODBC Driver To connect the SAP BusinessObjects BI Suite into Google BigQuery for querying, the SIMBA Google BigQuery ODBC driver is leveraged ( For usage of the Simba BigQuery ODBC Driver, please contact Simba Technologies via Simba BigQuery ODBC drivers is available in 32 bits and 64 Bits. Depending on the SAP BusinessObjects BI40 Suite Client tools used, either the 32 bits AND / OR the 64 Bits driver has to be installed. The table below represents what version needs to be installed for enabling BigQuery ODBC support. Product Client Installation Server Installation Universe Design Tool (UDT) 32 bits n/a Information Design Tool (IDT) 32 bits n/a Explorer 4.x n/a 64 bits Web Intelligence 32 bits 64 bits Crystal Reports for Enterprise* 32 bits 64 bits Crystal Reports bits 32 bits Dashboards (Xcelsius) 4.x 32 bits 64 bits Lumira 1.x** 32 and 64 bits n/a *Not yet validated/working against Google BigQuery ** Via Universe Connection, Data retrieval executed by BI server Installation of Simba BigQuery ODBC Driver 32 bits ODBC Driver install 1. Download / copy 32bits ODBC installer to the system 2. Double Click the installer to launch setup 17

18 3. The Simba BigQuery ODBC Driver Setup is started, continue with next 4. Read & Accept the End-User License Agreement before continuing with next 18

19 5. Accept the default destination folder, or select the required installation folder and continue with next 6. The setup engine is ready for installation, continue with install 7. The installer runs and takes approx. 1 minute 19

20 8. The installer is completed 20

21 64 bits ODBC Driver install 1. Download / copy 64bits ODBC installer to the system 2. Double Click the installer to launch setup 3. The Simba BigQuery ODBC Driver Setup is started, continue with next 4. Read & Accept the End-User License Agreement before continuing with next 21

22 5. Accept the default destination folder, or select the required installation folder and continue with next 6. The setup engine is ready for installation, continue with install 7. The installer runs and takes approx. 1 minute 22

23 8. The installer is completed 23

24 Configuring Simba BigQuery ODBC Driver After the installation of the Simba BigQuery ODBC driver (either 32 or 64 Bits), the ODBC configuration has to be completed before the connection can be used. The configuration withholds the Confirmation Code required to communicate from the system to the BigQuery API. 1. Open the ODBC Administration either via the control panel or via the icon provided in the Simba BigQuery ODBC Driver 1.0 menu from the Start button 2. Within the ODBC Data Source Administrator, navigate to the System DSN tab and locate the Sample Simba BigQuery DSN 24

25 3. Configure the Sample Simba BigQuery DSN 4. Select the Sign In. button to retrieve the Confirmation Code for the BigQuery Access 5. Logon using the account used for the API registration 25

26 6. Access is requested for external access into BigQuery, click Allow Access 7. Once access is accepted, the API code is presented 26

27 8. Copy and paste this code into the confirmation code box of the Simba BigQuery ODBC Connector Setup The Refresh Token will automatically be generated based on the Confirmation Code 9. If required, apply advanced options 27

28 10. Select your catalog i.e. publicdata and test the connection If required, repeat these steps for the x64 Bits Driver when installing them on a BI4.x Server. The x64 Bits Driver configuration is mandatory for using reports via BI Launchpad and or 3-tier mode. 28

29 SAP BusinessObjects BI40 customizations To enable Google BigQuery support by SAP BusinessObjects BI4.0, several data access configuration files have to be modified. These changes have to be made on every deployment of SAP BusinessObjects BI4.0 that is going to connect to Google BigQuery. This would include the SAP BusinessObjects BI4.0 Servers and Clients using the SAP BusinessObjects Semantic Layer. Main ODBC.PRM File Google BigQuery supports complex expressions in the Group By statement. To enable the ODBC driver for supporting this, one of the parameters within the main odbc parameter file needs to be changed. For enabling the support of Complex Group By functions as supported by Big Query, the following changes have to be made in the main ODBC.prm file of SAP BusinessObjects BI Navigate to <BI40 Installation Folder>\SAP BusinessObjects Enterprise XI 4.0\dataAccess\connectionServer\odbc 2. Backup odbc.prm 3. Open the odbc.prm file 4. Find the line : <Parameter Name="GROUP_BY_SUPPORTS_COMPLEX">NO</Parameter> 5. Update NO into YES 6. Save and close the file 29

30 7. Repeat these steps for every other installation that needs to connect to Google Big Query Extension Legacy ODBC.prm Within BigQuery there are no reservation defined in the naming of fields like year or month To overcome issues with reserved words like year or month, the main odbc.prm file has to be modified. For support of BigQuery in this matter the following changes have to be made in the legacy odbc.prm: 1. Navigate to <BI40 Installation Folder>\SAP BusinessObjects Enterprise XI 4.0\dataAccess\connectionServer\odbc\extensions\legacy 2. Backup odbc.prm 3. Open the odbc.prm file 4. Add the following lines before the </Configuration> line (this should be the second last line in the file): <Parameter Name="IDENTIFIER_DELIMITER">"</Parameter> <Parameter Name="DELIMIT_LOWERCASE">Y</Parameter> <Parameter Name="DELIMIT_IDENTIFIERS">Y</Parameter> 5. Save and close the file 6. Repeat these steps for every other installation that needs to connect to Google Big Query 30

31 Building a Universe against Google BigQuery Now we are setup to use the Simba BigQuery ODBC connector and will be able to create a Universe against the Google BigQuery data using the Information Design Tool (IDT) or Universe Design Tool (UDT). Building a Universe (UNX) on Google BigQuery using IDT To create a UNX against Google BigQuery, via the SIMBA BigQuery ODBC the normal steps for building a universe can be obtained. This chapter will describe sample steps for creating a universe against the public data sets available at Google BigQuery. Some steps may be different when using own data sets within your Google BigQuery deployment. As mentioned before this is solely providing a sample creation of the universe and required connection within Information Design Tool. In case items or steps are unclear, it is strongly recommended to read the User Manual of Information Design Tool available on Creating Connection using IDT In case the connection is already created/available, you can skip these steps and continue directly to the next section : 0 Building a Universe 1. Start Information Design Tool 2. Within the Repository Recourse Tab, connect to your SAP BusinessObjects BI Platform 3. Unfold the Connection to you BI Platform and navigate to a folder of choice for creating a new connection 31

32 4. Right mouse click on the folder of choice and select Insert Relational Connection 5. The New Relational Connection Wizard will appear, enter a name for the connection and continue with Next 32

33 6. In the screen Database Middelware Driver Selection choose for -> Generic -> Generic ODBC datasource -> ODBC Drivers and continue with Next 7. In the screen Parameters for Generic ODBC datasource connection (1/3) select the ODBC connection as defined in section 0 Configuring Simba BigQuery ODBC Driver 33

34 8. Test the connection. At a successful test, close the test result and continue with Next In case the connection fails, it is most probably due to in incorrect configuration of the Authentication settings required within the SIMBA BigQuery ODBC driver as described in section 0 Configuring Simba BigQuery ODBC Driver 9. In the screen Parameters for Generic ODBC datasource connection (2/3) change the Connection Pool Mode to Disconnect after each connection 10. For the screen Parameters for Generic ODBC datasource connection (2/3) there are no changes to be made and the creation of the connection can be completed with Finish Building a Universe using IDT Now a connection against the BigQuery Public Datasets is created, the creation of a universe can be initiated. Below a sample creation of a single source universe within Information Design Tool is provided. In case items or steps are unclear, it is strongly recommended to read the User Manual of Information Design Tool available on Sample steps to create a new universe against the Google BigQuery Sample Data set are: 34

35 1. Create a new Project within IDT 2. In the New Project Screen enter the name of your project and continue with Finish 3. In the Repository Recourses panel, lookup the connection made in the previous steps and select this. Using the right mouse button the context menu for the connection will appear. Select Create 35

36 Relation Connection Shortcut 4. The Select a Local Project screen will appear. Select the project as defined in step 2 36

37 5. A confirmation box will inform you that the connection is successfully added to the project 6. The connection shortcut is added to the project 7. Create a new Data Foundation in the project 37

38 8. Enter a name for the Data Foundation 9. Select to create a single source Data Foundation Note: Multisource- Enabled data foundation is not (yet) supported as the configuration of the Data Federator Engine is not yet enabled for querying Google Big Query 38

39 10. Select the Connection that is to be used by the Data Foundation 11. After Clicking Finish the Data Foundation is created and will be opened in the workscreen of IDT 39

40 12. Unfold the Connection to retrieve the available tables from Google BigQuery 13. Double click on a table to have it added to the Data Foundation Master View 14. Repeat this step for every table you want to add 15. If required, define the joins between the tables 16. Once the data foundation is completed save the Data Foundation and a Business Layer can be created 40

41 17. Add a new Business Layer to the project 18. As Google BigQuery is an relation database, select a Relational Data Foundation as the data source 41

42 19. Enter a name for the Business Layer Note: this name will be used for the final Universe presented to end users 20. In the Select Data Foundation screen, hit the button with the 42

43 21. Select the Data Foundation that you want to use for the Business Layer 22. If preferred, disable the option to Automatically create folders and objects, otherwise continy with Finish the generate the Business Layer 43

44 23. The Business Layer will be opened in the workspace of IDT 24. Validate the Business Layer generated by IDT 25. Enrich the universe where needed / required. 44

45 26. Once completed, Publish the universe into the BI4.0 system 45

46 27. Check if the universe is available within the CMC and authorized for use by the appropriate users 46

47 Building a Universe (UNV) on Google BigQuery using UDT To create a UNV (Legacy Universe) against Google BigQuery, via the SIMBA BigQuery ODBC the normal steps for building a universe can be obtained. This chapter will describe sample steps for creating a universe against the public data sets available at Google BigQuery. Some steps may be different when using own data sets within your Google BigQuery deployment. As mentioned before, this document is solely providing a sample creation of the universe and it s connection within Universe Design Tool. In case items or steps are unclear, it is strongly recommended to read the User Manual of Universe Design Tool available on Creating Connection using UDT To create a UNV (legacy Universe) against Google BigQuery, via the SIMBA BigQuery ODBC the normal steps for building a universe can be obtained. In case the connection is already created, you can skip these steps and continue directly to the next section : 0 47

48 Building a Universe using UDT 1. Start Universe Design Tool and logon with an authorized user (authorization to create and publish new connections) 2. If the Quick Design Wizard appears (depending on user settings), select cancel 3. Within Universe Designer, Select Tools -> Connections (or the direct button within the toolbar) 4. Select to create a new connection 48

49 5. Create a secured connection and provide a connection name. If required, select a folder where to create the connection. 6. In the middleware selection screen, select Generic -> Generic ODBC Datasource -> ODBC Drivers 49

50 7. In the screen Login Parameters (3/5), specify the Data Source Name 8. In the screen Configure Parameters (4/5), set the Connection Pool Mode to: Disconnect after each transaction 50

51 9. In the final screen, no changes are required. And close the connection wizard with Finish 10. Now the connection is created within the BI40 platform 51

52 Building a Universe using UDT Now a connection against the BigQuery Public Datasets is created, the creation of a universe can be initiated. Below a sample creation of a single source universe within Universe Design Tool is provided. In case items or steps are unclear, it is strongly recommended to read the User Manual of Universe Design Tool available on Sample steps to create a new universe against the Google BigQuery Sample Data set are: 1. Create a new universe 2. For the Universe Parameters, provide a name for the universe and select the connection required and finish with OK 52

53 3. Double Click in the Structure Pane to get the table browser and unfold to the required table 4. Double Click the required table to insert it into the Structure Pane 53

54 5. Drop the Table into the Object Pane to allow UDT to create objects for each attribute within the table 6. Enrich the universe where needed / required 7. Once completed, save and export the universe 54

55 55

56 Reporting Examples Now the universe(s) against Google BigQuery are generated, several SAP BusinessObjects BI4. Clients are capable of reporting against the Google BigQuery data source. Web Intelligence Rich Client Report: Data Panel: 56

57 Query Panel: 57

58 Query SQL: SELECT Table 1."year", Table 1."month", Table 1."state", sum(table 1."born_alive_alive"), sum(table 1."born_alive_dead"), sum(table 1."born_dead"), sum(table 1."ever_born") FROM "samples"."natality" Table 1 WHERE Table 1."year" values for Year:','N','Natality\Year',Multi,Free,Persistent,,User:0) GROUP BY Table 1."year", Table 1."month", Table 1."state" Crystal Reports 2011 This sample is build using a UNV universe. Crystal Reports does not support UNX universes. Report: 58

59 Query: 59

60 Query SQL: Lumira 1.x This sample is build using a UNX universe. Free Hand SQL is not (yet) available within Lumira for Generic ODBC Data Sources. This sample would also be executable using SAP Predictive Analysis. Lumira Visualization Lumira Data View 60

61 Lumira Query Panel 61

62 Lumira Data View Explorer 4.0 This sample is build using a UNX universe with a restriction on Year and Month to prevent Explorer from loading too much data. Exploring Information Space 62

63 Exploration View: Information Space Facets Dashboards 4.0 The sample is build using a direct data provider via the UNX universe Dashboard 63

64 Query Panel 64

65 Crystal Report for Enterprise 4.0 TO BE VALIDATED 65

66 Known Limitations The prototype learned some limitations while using a Universe against Google BigQuery, these are: Attribute Truncation to 18 Characters Workaround: At design time of the universe, attributes will be truncated to a length of max length of 18 characters while using Universe Design Tool or Information Design Tool. As Google Big Query has no defined length for attributes, this information is not provided to the Designer Tools, as a default behavior UDT & IDT will truncate the length to 18 Characters. This will remove the capability of viewing the table content from UDT and IDT and will fail to validate the universe model as well as any default created end user object (dimension, measure or detail object) impacted by this Manually edit the SQL definition of any object within the Business layer / Universe Objects that have a technical name of over 18 characters. Multi Source Universe: Workaround: A multi source universe is not supported when using the Simba BigQuery ODBC driver If various Google BigQuery Projects data sets are required within a single report you can create single source universes for the individual projects and merge data on report level Match Pattern: A query with condition match pattern fails since the SQL operator Like is not supported by Google BiqQuery. Potential workaround: Rewriting the SQL; using maybe regexp_match function. Default List of Values Solution: Default List of Values will fail since Select distinct is not supported by Google BiqQuery. Install the latest version of Simba BigQuery ODBC Driver. Simba Technologies added DISTINCT support within the ODBC driver as of version Multiple Table Join A 3-table join is expected to fail as this is not supported by Google BiqQuery. Potential workaround: Rewriting the query using simple joins between sub-queries. 66

67 Table Count Rows The Table Count rows command fails in the data foundation. Potential workaround Rewriting the SQL Any currently unknown limitation that will be shared with the author(s) of this document will be added in updates in this document and shared with the larger audience. 67

68 Prototype Environment This document is written utilizing the following components out of the SAP BusinessObjects BI Suite SAP BusinessObjects BI4.0 SP5 Enterprise Platform installation running on Windows 2008 R2 x64 SAP BusinessObjects Explorer 4.0 SP5 running on Windows 2008 R2 x64 SAP BusinessObjects Information Design Tool 4.0 SP5 SAP BusinessObjects Universe Design Tool 4.0 SP5 SAP BusinessObjects Web Intelligence Rich Client 4.0 SP5 SAP BusinessObjects Web Intelligence Java Client via Internet Explorer 9 and Google Chrome version 26 SAP Crystal Reports 2011 SP5 68

69 Copyright 2014 SAP SE SE or an SAP SE affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE. The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE and its affiliated companies ( SAP SE Group ) for informational purposes only, without representation or warranty of any kind, and SAP SE Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP SE and other SAP SE products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE in Germany and other countries. Please see for additional trademark information and notices. 69