Setting up Caravel with IBM Big SQL

Transcription

1 Setting up Caravel with IBM Big SQL Rajesh Kartha, IBM Introduction: Airbnb recently open sourced their visualization/dashboarding tool called Caravel (inception name Panoramix). This tool is under Apache license and uses Python (2.7 or above), Flask and D3 for visualization. Underneath it uses the Python OR Mapper SQLAlchemy to generate the respective SQLs based on the database and its respective dialects. Caravel helps one create slices of the data and assemble them on dashboards, which are shareable. Please check out the Gallery for its capabilities. It can connect to any database as long as there exist a corresponding SQLAlchemy dialect. IBMDB Dialect: There exists Python IBMDB SQLAlchemy dialect for IBM DB2 and IBM Informix and since IBM Big SQL uses the same connectivity methods, Caravel can be used to query IBM Big SQL using this dialect. Please refer the Reference section below for urls and other related info. The following sections describe setting up Caravel and the IBMDB SQLAlchemy dialect for IBM Big SQL connectivity. If Caravel if already installed one could skip to the next section on IBMDB and SA plugin install. Caravel Install: The current version at the time of writing is Caravel Caravel can be installed on any single server/node that has connectivity to the Big SQL head node. 1) Install the required rpms: yum upgrade python-setuptools yum install gcc libffi-devel python-devel python-pip python-wheel openssldevel 2) Install Caravel and the required python packages (will take some time): pip install caravel 3) Create an Admin user: fabmanager create-admin --app caravel

2 4) Initialize the Caravel database: caravel db upgrade 5) Create default roles and permissions: caravel init

3 6) Load the sample data, slices and dashboards: caravel load_examples 7) Start the Caravel server and navigate to to ensure the application is up and running: caravel runserver d

4 IBMDB and SA plugin install: 1) Install the IBMDB package: easy_install ibm_db 2) Install the IBMDB SqlAlchemy package: easy_install ibm_db_sa 3) Update one the.py file for the IBMDB SA plugin: The base.py file in IBM DB SA plug need couple of fixes to run SQL against BigSQL. This is located under the site-packages of python2.7 at: /usr/lib/python2.7/site-packages/ibm_db_sa py2.7.egg/ibm_db_sa/base.py

5 Create a backup copy of the base.py file and edit the base.py to change the line: def get_select_precolumns(self, select): to def get_select_precolumns(self, select, **kwargs): and def limit_clause(self, select): to def limit_clause(self, select, **kw): The diff between the new and the original files will look like this: Important: Without these fixes, queries from Caravel via the IBMDB SQLAlchemy dialect fails with the following error: A ticket #213 has been opened on the IBMDB GitHub site. 4) Stop and Start Caravel: Ctrl+C to stop the Caravel server and run caravel runserver d to start it

6 Connect to Big SQL: 1) Login to Caravel, navigate to 2) Click on Sources->Databases 3) Provide a Database Name and the SQLAlchemy url for Big SQL (IBMDB), which looks like: ibm_db_sa://<user>:<password>@<host>:32051/bigsql Example: ibm_db_sa://bigsql:xxxxxxxxxx@bdavm1610.svl.ibm.com:32051/bigsql

7 Click on TEST CONNECTION to ensure everything works well and click SAVE. 4) Add a new Table/View to query: 5) Enter table name, Select Database and click on Save:

8 6) From the list of tables View, click on the Edit Record link to view the available cols etc. for the table: 7) Click on the List Table Column header, depending on the column types some of them are already pre-selected for some of the aggregate type operations. One can always make changes:

9 8) Create Slices of data and Save them: Here are some sample slices created using the Free airline dataset, with the BigSQL Hadoop table declared on top. Slice that indicates Airlines and Origins

10 Another slice that indicates Origins with stacked Destination counts : 9) Assemble the slices on dashboards: Needless to say, please refer the samples that comes with Caravel to learn more creating slices and dashboards.

11 Current Limitations: - Could not easily query tables from a different schema. A workaround for now may be to connect as a username same as schema name. - For some of the charts a column of the type DATE/DATETIME/TIMESTAMP is required. - Saving a slice without any GROUP BY cols specified throws an error References: Caravel: Github: Google Groups: SQLAlchemy: Python and DB2: c/t html Install: Potential Issues: Upgrade: As new releases of Caravel become available run pip install --upgrade caravel to upgrade. Then run: caravel db upgrade caravel init caravel load_examples caravel runserver -d Update of PIP: Pip commands showed that the instance had an older version of pip: Solution: pip install --upgrade pip Older version of pytz: Using the previous version of Caravel, I ran into an error: pkg_resources.distributionnotfound: The 'pytz>=0a' distribution was not found and is required by Babel Solution: pip install --upgrade pytz