Learning about software development with Kibana dashboards Jesus M. Gonzalez-Barahona jgb@bitergia.com @jgbarah Bitergia / LibreSoft (URJC) Lightning Talks, FOSDEM Brussels (Belgium), January 31st 2016 Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 1 / 29
c 2016 Bitergia Some rights reserved. This presentation is distributed under the Attribution-ShareAlike 3.0 license, by Creative Commons, available at http://creativecommons.org/licenses/by-sa/3.0/ Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 2 / 29
Structure of the presentation 1 A bit of context 2 From the team who built MetricsGrimoire... 3 Your very own dashboard for your pet GitHub repo 4 Enjoy 5 Some details 6 Bonus track Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 3 / 29
A bit of context Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 4 / 29
Me and my circumstances Uni Rey Juan Carlos: LibreSoft research team Understanding free, open source software development Data analytics approach Bitergia: From research to the real world Understanding free, open source software development Data analytics approach http://gsyc.es/~jgb http://bitergia.com Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 5 / 29
The company The software development analytics company dashboards reports consultancy... http://bitergia.com Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 6 / 29
From the team who built MetricsGrimoire... Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 7 / 29
MetricsGrimoire: the current state of affairs Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 8 / 29
MetricsGrimoire: supported data sources Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 9 / 29
GrimoireLab: the next big thing Perceval: retrieving data ElasticSearch: storing data Python / Pandas scripts: enrich, analyze, customize the data Kibiter: Kibana fork to interact with the data http://grimoirelab.github.io http://blog.bitergia.com Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 10 / 29
Your very own dashboard for your pet GitHub repo Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 11 / 29
One of my pet repos: MetricsGrimoire/Bicho https://github.com/metricsgrimoire/bicho Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 12 / 29
Set up the environment Create and activate a new Python virtualenv: $ virtualenv -p /usr/bin/python3 gitlab $ gitlab/bin/activate Install dependencies (gitlab) $ pip install beautifulsoup4 (gitlab) $ pip install python-dateutil (gitlab) $ pip install requests (gitlab) $ pip install six Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 13 / 29
Install GrimoireLab tools Install Perceval from its GrimoireLab git repo: (gitlab) $ git clone \ https://github.com/grimoirelab/perceval.git (gitlab) $ cd perceval (gitlab) $ git checkout fosdem16 (gitlab) $ python3 setup.py install (gitlab) $ cd.. Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 14 / 29
Install GrimoireLab tools Clone GrimoireELK GrimoireLab git repo, checkout branch fosdem16 (gitlab) $ git clone \ https://github.com/grimoirelab/grimoireelk.git (gitlab) $ cd GrimoireELK (gitlab) $ git checkout fosdem16 (gitlab) $ cd.. Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 15 / 29
Prepare the repository to analyze We will use MetricsGrimoire/Bicho: (gitlab) $ git clone \ https://github.com/metricsgrimoire/bicho (gitlab) $ cd Bicho (gitlab) $ git log --raw --numstat \ --pretty=fuller --decorate=full --parents \ -M -C -c --remotes=origin --all \ > /tmp/bicho-gitlog.log Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 16 / 29
Upload info to ElasticSearch First, upload info in the git log: (gitlab) $ cd../grimoireelk/util (gitlab) $ python3./p2o.py -e http://localhost:9200 \ --no_inc --debug git /tmp/bicho-gitlog.log (gitlab) $ python3./p2o.py -e http://localhost:9200 \ --no_inc --debug --enrich_only git /tmp/bicho-gitlog.log... 2016-01-31 00:47:26,960 Deleted and created index http://localhost:9200/git tmp_bicho-gitlog.log_enrich Index name: git tmp_bicho-gitlog.log_enrich Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 17 / 29
Upload info to ElasticSearch Second, upload info from GitHub issues & pull requests (may take a while & requires a GitHub token) (gitlab) $ python3./p2o.py -e http://localhost:9200 \ --no_inc --debug github --owner metricsgrimoire \ --repository bicho --token XXXX (gitlab) $ python3./p2o.py -e http://localhost:9200 \ --no_inc --debug --enrich_only github \ --owner metricsgrimoire --repository bicho --token XXXX... 2016-01-31 00:45:37,269 Deleted and created index http://localhost:9200/github_https: github.com_metricsgrimo Index name: github_https: github.com_metricsgrimoire_bicho_enrich Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 18 / 29
Produce a Kibana dashboard We upload template dashboards to ElasticSearch, and build our dashboards with them (notice the index names) (gitlab) $ python3./kidash.py -e http://localhost:9200 \ -g --import../dashboards/git-activity.json (gitlab) $ python3./kidash.py -e http://localhost:9200 \ -g --import../dashboards/github-pr-bubbles-geomap.json (gitlab) $ python3./e2k.py -g -e http://localhost:9200 \ -d "Git-Activity" -i git tmp_bicho-gitlog.log_enrich (gitlab) $ python3./e2k.py -g -e http://localhost:9200 \ -d "PRBubblesGeoMap" -i \ github_https: github.com_metricsgrimoire_bicho_enrich Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 19 / 29
Enjoy Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 20 / 29
See the dashboards in your browser http://localhost:5601 Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 21 / 29
Select the produced dashboards Template dashboards: Git Activity and PRBubblesGeoMap Real dashboards: the other two Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 22 / 29
Watch the results of your work! Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 23 / 29
Watch the results of your work! Play with them! Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 24 / 29
Some details Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 25 / 29
Deploying ElasticSearch, Kibana Download and uncompress ElasticSearch, from their website $ cd elasticsearch-1.7.1/ # [Configure, if needed] $ bin/elasticsearch Download and uncompress Kibana, from their website $ cd kibana-4.1.1-linux-x64 # [Configure, if needed] $ bin/kibana Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 26 / 29
Get a GitHub token Your photo Settings Personal access tokens Generate new token Take note of it, or you won t see it again! Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 27 / 29
Bonus track Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 28 / 29
Preview: our new Kibana-based dashboards http://s.bitergia.com/db-fosdem16 Jesus Gonzalez-Barahona (Bitergia) GrimoireLab: What s Next? Jan 2016 29 / 29