Big Data Exercises. Fall 2016 Week 0 ETH Zurich

Transcription

1 Big Data Exercises Fall 2016 Week 0 ETH Zurich 1. Jupyter Basics Welcome to this Jupyter notebook. Jupyter is a web-based open-source tool based on Python that allows you to run python (and other types of) code, visualize results and discuss results, and organize everything into notebooks like this one. We use the notebook server on Microsoft Azure, but you can also install your own. A notebook is organized in cells. Cells of this notebook contain Python code (but other cell types exists). To run a cell, select it, then press ctrl+enter. Try it out! In [1]: print("hello World") Hello World By default, the last expression is printed. Like this: In [2]: maxx = 10 [x * x for x in range(maxx)] Out[2]: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] You can also edit the text. Just double-click on a cell. It's made with markdown code. After you are done editing, press ctrl+enter We will do about half of the exercises with Jupyter. You will learn most things as we go. The notebook you are seing is your copy stored on your account (including the output of the cells). Write your answers inline and save regularly. (Also save a copy somewhere else from time to time this service is still beta.) Read more: Jupyter ( Learn Python ( Python documentation ( 2. Bash Scripts Code blocks by default are executed using a python interpreter (for a python notebook, such as this one). Other languages can be used with annotations. For instance, a code block can be converted into a bash code block using %%bash at the beginning: In [3]: %%bash echo "Test File" > test_file cat test_file Test File Note: do not expect files you write into this file system to be durable. The sandboxed environment may be reset and data lost when notebook is closed. The notebook files themselves are durable though. Bash commands can be also inlined using exclamation mark (!) infront of the bash line

2 In [7]: print("first I modify the file.")!echo "Hello World" > test_file print("then I check its content:") print()!cat test_file First I modify the file. Then I check its content: Hello World 3. Extensions As part of this course you will use Jupyter to interact with various systems and interfaces (e.g. SQL, Map Reduce, Spark). To use these, you will need to install certain Python and Jupyter extensions. Note, that as like the file system the extensions will not be durable and need to be rerun when the notebook server is restarted Setting up a SQL connection As part of preparation for the next week's exercise, let's setup a connection to a sample relational database (hosted from this course's Azure account) First set the access variables (make sure you execute the next code block by running ctrl+enter) In [8]: server='ethbigdata2017' user='student' password='bigdata17' database='' Next, install the PyMSSQL ( python package. Run the following cell and make sure that the output confirms that the installation was successfull. In [9]:!pip install pymssql==2.1.2 Collecting pymssql==2.1.2 Using cached pymssql tar.gz Building wheels for collected packages: pymssql Running setup.py bdist_wheel for pymssql... done Stored in directory: /home/nbuser/.cache/pip/wheels/6d/25/5d/3c673e8cdcbaffd5b ed62a09 13c380814da16b00f0 Successfully built pymssql Installing collected packages: pymssql Found existing installation: pymssql Uninstalling pymssql-2.1.1: Successfully uninstalled pymssql Successfully installed pymssql Running a SQL Query Then we run a first query against our server (following this tutorial ( f=255&mspperror= ) from the Azure website). Make sure that running the following cell successfully prints some rows.

3 In [10]: import pymssql import os os.environ['tdsver'] = '7.3' conn = pymssql.connect(server=server + '.database.windows.net', user=user + '@' + server, password=password, database=database) cursor = conn.cursor(as_dict=true) cursor.execute('select TOP 10 Id, DisplayName FROM Users') row = cursor.fetchone() while row: print(row) row = cursor.fetchone() {'Id': -1, 'DisplayName': 'Community'} {'Id': 1, 'DisplayName': 'Geoff Dalgas'} {'Id': 2, 'DisplayName': 'Kasra Rahjerdi'} {'Id': 3, 'DisplayName': 'Adam'} {'Id': 4, 'DisplayName': 'Arie Litovsky'} {'Id': 5, 'DisplayName': 'Brian Nickel'} {'Id': 6, 'DisplayName': 'Jeremy T'} {'Id': 7, 'DisplayName': 'Tom Medley'} {'Id': 8, 'DisplayName': 'LessPop_MoreFizz'} {'Id': 9, 'DisplayName': 'Nick Craver'} 3.3. Inlining SQL It is also possible to inline SQL code. To do this install the following Jupyter extension: In [11]:!git clone \ cd ipython-sql && LC_CTYPE="C.UTF-8" python setup.py -q install fatal: destination path 'ipython-sql' already exists and is not an empty directory. **At this point, we need to restart the notebook kernel!** To do that, go to Kernel in the menu at the top, then click on Restart. You need to run the cell with the connection details again after that. We can now load the extension and establish a connection to our database from above. Run the following cell and make sure the output says "Connected: <connection string>". In [12]: %load_ext sql connection_string = 'mssql+pymssql://{user}@{server}:{password}@{server}.database.windows.net:1433/{database }'.format( server=server, user=user, password=password, database=database) print(connection_string) %sql $connection_string mssql+pymssql://student@ethbigdata2017:bigdata17@ethbigdata2017.database.windows.net:1433/beer. stackexchange.com Out[12]: 'Connected: student@ethbigdata2017@' Now we can use the %sql and %%sql magic words to run SQL directly. %%sql makes a cell a SQL cell. A SQL cell can run an arbitrary number of SQL statements and displays the result of the last one of them.

4 In [13]: %%sql SELECT TOP 10 Id, DisplayName FROM Users Out[13]: Id DisplayName -1 Community 1 Geoff Dalgas 2 Kasra Rahjerdi 3 Adam 4 Arie Litovsky 5 Brian Nickel 6 Jeremy T 7 Tom Medley 8 LessPop_MoreFizz 9 Nick Craver The %sql magic words lets us run SQL statements in a regular cell. Again, the result of the last statement is displayed. In [14]: print("run a query!") %sql SELECT TOP 10 Id, DisplayName FROM Users; Run a query! Out[14]: Id DisplayName -1 Community 1 Geoff Dalgas 2 Kasra Rahjerdi 3 Adam 4 Arie Litovsky 5 Brian Nickel 6 Jeremy T 7 Tom Medley 8 LessPop_MoreFizz 9 Nick Craver 3.4. Plotting results Matplotlib can also be used to plot results. Next is a plot of a sample query that finds the number of created users per year.

5 In [15]: %matplotlib inline import matplotlib.pyplot as plt result = %sql SELECT YEAR(CreationDate) as CreationYear, COUNT(*) as YearCount \ FROM Users \ GROUP BY YEAR(CreationDate) \ ORDER BY YEAR(CreationDate) ASC; # Print the result in tabular form print(result) # Convert the result to a Pandas data frame df = result.dataframe() # Extract x and y values for a plot x = df['creationyear'].tolist() y = df['yearcount'].tolist() # Plot the distribution of registrations per year fig, ax = plt.subplots() ax.bar(range(len(df.index)), y, tick_label=[int(i) for i in x], align='center') ax.set_xlabel('creation Year') ax.set_ylabel('number of Users') CreationYear YearCount Out[15]: <matplotlib.text.text at 0x7fe5f9bf7d30>