processing data from the web
|
|
- Evan Blankenship
- 6 years ago
- Views:
Transcription
1 processing data from the web 1 CTA Tables general transit feed specification stop identification and name finding trips for a given stop 2 CTA Tables in MySQL files in GTFS feed are tables in database filling a table with a Python script storing the connections MCS 275 Lecture 39 Programming Tools and File Management Jan Verschelde, 17 April 2017 Programming Tools (MCS 275) processing gtfs data L April / 39
2 processing data from the web 1 CTA Tables general transit feed specification stop identification and name finding trips for a given stop 2 CTA Tables in MySQL files in GTFS feed are tables in database filling a table with a Python script storing the connections Programming Tools (MCS 275) processing gtfs data L April / 39
3 GTFS of our CTA We can download the schedules of the CTA: GTFS = General Transit Feed Specification is an open format for packaging scheduled service data. AGTFSfeedisaseriesoftextfileswithdataonlinesseparatedby commas (csv format). Each file is a table in a relational database. Programming Tools (MCS 275) processing gtfs data L April / 39
4 some tables stops.txt: stoplocationsforbusortrain routes.txt: routelistwithuniqueidentifiers trips.txt: informationabouteachtripbyavehicle stop_times.txt: scheduledarrivalanddeparturetimesfor each stop on each trip. Programming Tools (MCS 275) processing gtfs data L April / 39
5 processing data from the web 1 CTA Tables general transit feed specification stop identification and name finding trips for a given stop 2 CTA Tables in MySQL files in GTFS feed are tables in database filling a table with a Python script storing the connections Programming Tools (MCS 275) processing gtfs data L April / 39
6 finding a stop name $ python3 ctastopname.py opening CTA/stops.txt... give a stop id : 3021 skipping line has name "California & Augusta" The script looks for the line 3021,3021,"California & Augusta", , \ ,0,,1 In a Terminal window, we can type $ cat stops.txt grep ",3021," 3021,3021,"California & Augusta", , \ ,0,,1 $ Programming Tools (MCS 275) processing gtfs data L April / 39
7 ctastopname.py FILENAME = CTA/stops.txt print( opening, FILENAME,... ) DATAFILE = open(filename, r ) STOPID = int(input( give a stop id : )) COUNT = 0 STOPNAME = None while True: LINE = DATAFILE.readline() if LINE == : break L = LINE.split(, ) try: if int(l[0]) == STOPID: STOPNAME = L[2] break except: print( skipping line, COUNT) COUNT = COUNT + 1 print(stopid, has name, STOPNAME) Programming Tools (MCS 275) processing gtfs data L April / 39
8 processing data from the web 1 CTA Tables general transit feed specification stop identification and name finding trips for a given stop 2 CTA Tables in MySQL files in GTFS feed are tables in database filling a table with a Python script storing the connections Programming Tools (MCS 275) processing gtfs data L April / 39
9 finding head signs Given an identification of a stop, we look for all CTA vehicles that make a stop there. $ python3 ctastoptimes.py opening CTA/stop_times.txt... give a stop id : 3021 skipping line 0 adding "63rd Pl/Kedzie" adding "Jackson" [ "63rd Pl/Kedzie", "Jackson" ] We scan the lines in stop_times.txt for where the given stop identification occurs. Programming Tools (MCS 275) processing gtfs data L April / 39
10 ctastoptimes.py FILENAME = CTA/stop_times.txt print( opening, FILENAME,... ) DATAFILE = open(filename, r ) STOPID = int(input( give a stop id : )) COUNT = 0 TIMES = [] while True: LINE = DATAFILE.readline() if LINE == : break L = LINE.split(, ) try: if int(l[3]) == STOPID: if L[5] not in TIMES: print( adding, L[5]) TIMES.append(L[5]) except: print( skipping line, COUNT) COUNT = COUNT + 1 print(times) Programming Tools (MCS 275) processing gtfs data L April / 39
11 processing data from the web 1 CTA Tables general transit feed specification stop identification and name finding trips for a given stop 2 CTA Tables in MySQL files in GTFS feed are tables in database filling a table with a Python script storing the connections Programming Tools (MCS 275) processing gtfs data L April / 39
12 GTFS of our CTA We can download the schedules of the CTA: GTFS = General Transit Feed Specification is an open format for packaging scheduled service data. AGTFSfeedisaseriesoftextfileswithdataonlinesseparatedby commas (csv format). Each file is a table in a relational database. We call our database CTA and will add tables reading the information from stops.txt. Programming Tools (MCS 275) processing gtfs data L April / 39
13 fields in stops.txt The first line in stops.txt lists: 1 stop_id: type INT 2 stop_code: type INT 3 stop_name: type CHAR(80) 4 stop_desc: type VARCHAR(80) 5 stop_lat: type FLOAT 6 stop_lon: type FLOAT 7 location_type: type INT 8 parent_station: type INT 9 wheelchair_boarding: type SMALLINT Programming Tools (MCS 275) processing gtfs data L April / 39
14 creating database and table Note: depending on the setup of mysql,wemayhave to execute mysql as superuser (use sudo on Mac OS X). To make a database CTA, we run mysqladmin: $ mysqladmin create CTA Then we start mysql, tocreateatableinthedatabase: mysql> use CTA; Database changed mysql> create table stops -> (id INT, code INT, name CHAR(80), -> ndesc VARCHAR(128), -> lat FLOAT, lon FLOAT, -> tp INT, ps INT, wb SMALLINT); Query OK, 0 rows affected (0.01 sec) Programming Tools (MCS 275) processing gtfs data L April / 39
15 explain mysql> explain stops; Field Type Null Key Default Extra id int(11) YES NULL code int(11) YES NULL name char(80) YES NULL ndesc varchar(128) YES NULL lat float YES NULL lon float YES NULL tp int(11) YES NULL ps int(11) YES NULL wb smallint(6) YES NULL rows in set (0.00 sec) Programming Tools (MCS 275) processing gtfs data L April / 39
16 manual insertion The first data line in stops.txt contains 1,1,"Jackson & Austin Terminal", "Jackson & Austin Terminal, Northeastbound, Bus Terminal", , ,0,,1 mysql> insert into stops values -> (1,1,"Jackson & Austin Terminal", -> "Jackson & Austin Terminal, Northeastbound, Bus Term -> , ,0,0,1); Query OK, 1 row affected (0.00 sec) mysql> select name from stops where id = 1; name Jackson & Austin Terminal row in set (0.00 sec) Programming Tools (MCS 275) processing gtfs data L April / 39
17 deleting rows To delete a row, given its id: mysql> delete from stops where id = 1; Query OK, 1 row affected (0.65 sec) mysql> select * from stops; Empty set (0.01 sec) If the where clause is omitted, then all rows in the table are deleted. Programming Tools (MCS 275) processing gtfs data L April / 39
18 processing data from the web 1 CTA Tables general transit feed specification stop identification and name finding trips for a given stop 2 CTA Tables in MySQL files in GTFS feed are tables in database filling a table with a Python script storing the connections Programming Tools (MCS 275) processing gtfs data L April / 39
19 filling a table Typing 12,165 is rather tedious... After filling the table stops of the database we query the table for a name: mysql> select name from stops where id = 3021; name California & Augusta row in set (0.00 sec) Our data is on file: FILENAME = CTA/stops.txt Programming Tools (MCS 275) processing gtfs data L April / 39
20 fillstops() in dbctafillstops.py import pymysql def fillstops(printonly=true): """ Opens the file with name FILENAME, reads every line and inserts the data into the table stops. """ if printonly: crs = None else: cta = pymysql.connect(db= CTA ) crs = cta.cursor() print( opening, FILENAME,... ) Programming Tools (MCS 275) processing gtfs data L April / 39
21 fillstops() continued datafile = open(filename, r ) line = datafile.readline() # skip the first line while True: line = datafile.readline() if(line == ): break insert_data(crs, line, printonly) if not printonly: cta.commit() crs.close() datafile.close() For the changes to take effect, we must do a commit(). With rollback(),wecancancelthecurrenttransaction, provided the database and tables support transactions. Programming Tools (MCS 275) processing gtfs data L April / 39
22 extracting data "Jackson & Austin Terminal, Northeastbound, Bus Terminal", Commas in the string! def extract(line): """ Returns a list of 9 elements extracted from the string line. Missing data are replaced by 0. """ result = [] strd = line.split( \" ) # extract strings first (name, desc) = (strd[1], strd[3]) data = strd[0].split(, ) result.append( 0 if data[0] == else data[0]) result.append( 0 if data[1] == else data[1]) result.append(name) result.append(desc) Programming Tools (MCS 275) processing gtfs data L April / 39
23 function extract(line) continued... Extracting latitute, longitude, and last 3 integers: data = strd[4].split( \n ) # remove newline data = data[0].split(, ) for k in range(1, len(data)): result.append( 0 if data[k] == else data[k]) while len(result) < 9: result.append( 0 ) return result Missing data are replaced by 0. The list on return will always have nine items. Programming Tools (MCS 275) processing gtfs data L April / 39
24 inserting data def insert_data(cur, line, printonly=true): """ Inserts the data in the string line, using the cursor c, if printonly is False. """ data = extract(line) dbc = INSERT INTO stops VALUES ( dbc += data[0] +, + data[1] +, dbc += \" + data[2] + \", # name is a string dbc += \" + data[3] + \", # description for k in range(4, 8): dbc += data[k] +, dbc += data[8] + ) print(repr(dbc)) # print raw string if not printonly: cur.execute(dbc) Programming Tools (MCS 275) processing gtfs data L April / 39
25 querying the table mysql> select id from stops -> where name = "California & Augusta"; id rows in set (0.00 sec) mysql> select name from stops where id = 17154; name California & Augusta row in set (0.01 sec) Programming Tools (MCS 275) processing gtfs data L April / 39
26 querying with Python $ python3 dbctastopquery.py give a stop id : has name California & Augusta $ $ python3 dbctastopquery.py give a stop id : 0 0 has name None $ Programming Tools (MCS 275) processing gtfs data L April / 39
27 the main program import pymysql def main(): """ Connects to the database, prompts the user for a stop id and the queries the stops table. """ cta = pymysql.connect(db= CTA ) crs = cta.cursor() stop = int(input( give a stop id : )) name = get_stop_name(crs, stop) print(stop, has name, name) cta.close() Programming Tools (MCS 275) processing gtfs data L April / 39
28 executing the query def get_stop_name(crs, stopid): """ Given a cursor crs to the CTA database, queries the stops table for the stop id. Returns None if the stop id has not been found, otherwise returns the stop name. """ sel = SELECT name FROM stops whe = WHERE id = %d % stopid query = sel + whe returned = crs.execute(query) if returned == 0: return None else: tpl = crs.fetchone() return tpl[0] Programming Tools (MCS 275) processing gtfs data L April / 39
29 processing data from the web 1 CTA Tables general transit feed specification stop identification and name finding trips for a given stop 2 CTA Tables in MySQL files in GTFS feed are tables in database filling a table with a Python script storing the connections Programming Tools (MCS 275) processing gtfs data L April / 39
30 fields in stop_times.txt The first line in stop_times.txt lists: 1 trip_id: type INT 2 arrival_time: type TIME 3 departure_time: type TIME 4 stop_id: type INT 5 stop_sequence: type INT 6 stop_headsign: type VARCHAR(80) 7 pickup_type: type INT 8 shape_dist_traveled: type INT Programming Tools (MCS 275) processing gtfs data L April / 39
31 adding a new table mysql> create table stop_times -> (id BIGINT, arrival TIME, departure TIME, -> stop INT, seq INT, head VARCHAR(80), -> ptp INT, sdt INT); Query OK, 0 rows affected (0.02 sec) Note the types TIME and VARCHAR. Programming Tools (MCS 275) processing gtfs data L April / 39
32 explain mysql> explain stop_times; Field Type Null Key Default Extra id bigint(20) YES NULL arrival time YES NULL departure time YES NULL stop int(11) YES NULL seq int(11) YES NULL head varchar(80) YES NULL ptp int(11) YES NULL sdt int(11) YES NULL rows in set (0.00 sec) Programming Tools (MCS 275) processing gtfs data L April / 39
33 manual insertion mysql> insert into stop_times values ( -> ,"12:09:14","12:09:14",6531,29, -> "Midway Orange Line",0,18625); Query OK, 1 row affected (0.00 sec) mysql> select departure, head from stop_times; departure head :09:14 Midway Orange Line row in set (0.00 sec) Programming Tools (MCS 275) processing gtfs data L April / 39
34 filling the table On Mac OS X laptop: $ python3 dbctafillstoptimes.py opening CTA/stop_times.txt... dbctafillstoptimes.py:26: Warning: Out of range value for column id at row 1 c.execute(d) Redo on a fast Linux Workstation: # time python3 dbctafillstoptimes.py opening CTA/stop_times.txt... dbctafillstoptimes.py:26: Warning: Out of range value for column id at row 1 c.execute(d) real user sys 5m32.433s 1m11.921s 0m17.735s Programming Tools (MCS 275) processing gtfs data L April / 39
35 about the complexity While running dbctafillstoptimes.py, the memory consumption of Python and mysql was of the same magnitude, about 300Mb. mysql> select count(*) from stop_times; count(*) row in set (1.46 sec) Programming Tools (MCS 275) processing gtfs data L April / 39
36 inserting data def insert_data(crs, scsv, printonly=true): """ Inserts the data in the comma separated string scsv using the cursor crs. """ data = scsv.split(, ) cmd = INSERT INTO stop_times VALUES ( cmd += ( 0, if data[0] == else data[0] +, ) cmd += \" + data[1] + \" +, cmd += \" + data[2] + \" +, cmd += data[3] +, + data[4] +, cmd += data[5] +, + data[6] +, wrk = data[7] # must cut off the \n data7 = wrk[0:len(wrk)-1] + ) cmd += ( 0) if data[7] == else data7) print(repr(cmd)) if not printonly: crs.execute(cmd) Programming Tools (MCS 275) processing gtfs data L April / 39
37 querying stop_times mysql> select head from stop_times -> where stop = 3021 and -> arrival < "05:30:00"; head rd Pl/Kedzie 63rd Pl/Kedzie 63rd Pl/Kedzie 63rd Pl/Kedzie rows in set (0.94 sec) Programming Tools (MCS 275) processing gtfs data L April / 39
38 an involved query mysql> select name, departure, head -> from stops, stop_times -> where stops.id = > and stops.id = stop_times.stop -> and stop_times.departure < "05:30:00"; name departure head California & Augusta 04:43:49 63rd Pl/Kedzie California & Augusta 05:03:49 63rd Pl/Kedzie California & Augusta 05:19:49 63rd Pl/Kedzie California & Augusta 05:12:49 63rd Pl/Kedzie rows in set (0.57 sec) Programming Tools (MCS 275) processing gtfs data L April / 39
39 Exercises 1 Modify ctastopname.py so the user is prompted for a string instead of a number. The modified script prints all id s and corresponding names that have the given string as substring. Use the in operator. 2 The file stops.txt contains the latitude and longitude of each stop. Use these coordinates to plot (either with pylab, pyplot, or a Tkinter canvas) the blue line from O Hare to Forest Park. Use a proper scaling so your plot resembles what we see on a map. 3 Write a Python script to return the name of stop, given its id, using the table stops. 4 Design a GUI with Tkinter to query the stop name: one entry field for the stop id, another for the name of the stop, and one button in the middle to execute the query. Note that the GUI allows to query given the stop id or given the stop name. Programming Tools (MCS 275) processing gtfs data L April / 39
processing data with a database
processing data with a database 1 MySQL and MySQLdb MySQL: an open source database running MySQL for database creation MySQLdb: an interface to MySQL for Python 2 CTA Tables in MySQL files in GTFS feed
More informationmaking connections general transit feed specification stop names and stop times storing the connections in a dictionary
making connections 1 CTA Tables general transit feed specification stop names and stop times storing the connections in a dictionary 2 CTA Schedules finding connections between stops sparse matrices in
More informationField required - The field column must be included in your feed, and a value must be
This document explains the types of files that comprise a GTFS transit feed and defines the fields used in all of those files. 1. Term Definitions (#term-definitions) 2. Feed Files (#feed-files) 3. File
More informationpygtfs Documentation Release Yaron de Leeuw
pygtfs Documentation Release 0.1.2 Yaron de Leeuw January 26, 2014 Contents 1 Get it 1 2 Basic usage 3 3 gtfs2db 5 4 Detailed refernce 7 5 Contents: 9 5.1 The schedule module...........................................
More informationHands-on GTFS. Omaha, NE October 29, U.S. Department of Transportation Federal Transit Administration
Hands-on GTFS Omaha, NE October 29, 2017 Presenters & Let s meet you! Marcy Jaffe Technical Support for GTFS Builder Toolkit Available by phone/email Amelia Bostic Greenway Public Transportation No experience
More informationPackage SIRItoGTFS. May 21, 2018
Package SIRItoGTFS May 21, 2018 Type Package Title Compare SIRI Datasets to GTFS Tables Version 0.2.4 Date 2018-05-21 Allows the user to compare SIRI (Service Interface for Real Time Information) data
More informationThis lab will introduce you to MySQL. Begin by logging into the class web server via SSH Secure Shell Client
Lab 2.0 - MySQL CISC3140, Fall 2011 DUE: Oct. 6th (Part 1 only) Part 1 1. Getting started This lab will introduce you to MySQL. Begin by logging into the class web server via SSH Secure Shell Client host
More informationMidterm Exam II MCS 275 Programming Tools 14 March 2017
NAME: answers The exam is closed book, no notes and no computer. All your answers to the questions below must be submitted on paper. Write your name on this sheet and submit it with your answers. Please
More informationAssignment 6: SQL III Solution
Data Modelling and Databases Exercise dates: April 12/April 13, 2018 Ce Zhang, Gustavo Alonso Last update: April 16, 2018 Spring Semester 2018 Head TA: Ingo Müller Assignment 6: SQL III Solution This assignment
More informationWeb Interfaces for Database Servers
Web Interfaces for Database Servers 1 CGI, MySQLdb, and Sockets glueing the connections with Python functions of the server: connect, count, and main development of the code for the client 2 Displaying
More informationData Modelling and Databases Exercise dates: March 22/March 23, 2018 Ce Zhang, Gustavo Alonso Last update: March 26, 2018.
Data Modelling and Databases Exercise dates: March 22/March 23, 2018 Ce Zhang, Gustavo Alonso Last update: March 26, 2018 Spring Semester 2018 Head TA: Ingo Müller Assignment 4: SQL This assignment will
More informationAssignment 6: SQL III
Data Modelling and Databases Exercise dates: April 12/April 13, 2018 Ce Zhang, Gustavo Alonso Last update: April 16, 2018 Spring Semester 2018 Head TA: Ingo Müller Assignment 6: SQL III This assignment
More informationDraft. Students Table. FName LName StudentID College Year. Justin Ennen Science Senior. Dan Bass Management Junior
Chapter 6 Introduction to SQL 6.1 What is a SQL? When would I use it? SQL stands for Structured Query Language. It is a language used mainly for talking to database servers. It s main feature divisions
More informationOperating systems fundamentals - B07
Operating systems fundamentals - B07 David Kendall Northumbria University David Kendall (Northumbria University) Operating systems fundamentals - B07 1 / 33 What is SQL? Structured Query Language Used
More informationGraphical User Interfaces
to visualize Graphical User Interfaces 1 2 to visualize MCS 507 Lecture 12 Mathematical, Statistical and Scientific Software Jan Verschelde, 19 September 2011 Graphical User Interfaces to visualize 1 2
More informationAssignment 5: SQL II Solution
Data Modelling and Databases Exercise dates: March 29/March 30, 2018 Ce Zhang, Gustavo Alonso Last update: April 12, 2018 Spring Semester 2018 Head TA: Ingo Müller Assignment 5: SQL II Solution This assignment
More informationExam. Question: Total Points: Score:
FS 2016 Data Modelling and Databases Date: June 9, 2016 ETH Zurich Systems Group Prof. Gustavo Alonso Exam Name: Question: 1 2 3 4 5 6 7 8 9 10 11 Total Points: 15 20 15 10 10 15 10 15 10 10 20 150 Score:
More informationMore MySQL ELEVEN Walkthrough examples Walkthrough 1: Bulk loading SESSION
SESSION ELEVEN 11.1 Walkthrough examples More MySQL This session is designed to introduce you to some more advanced features of MySQL, including loading your own database. There are a few files you need
More informationUser Interfaces. MCS 507 Lecture 11 Mathematical, Statistical and Scientific Software Jan Verschelde, 16 September Command Line Interfaces
User 1 2 MCS 507 Lecture 11 Mathematical, Statistical and Scientific Software Jan Verschelde, 16 September 2011 User 1 2 command line interfaces Many programs run without dialogue with user, as $ executable
More information1 INTRODUCTION TO EASIK 2 TABLE OF CONTENTS
1 INTRODUCTION TO EASIK EASIK is a Java based development tool for database schemas based on EA sketches. EASIK allows graphical modeling of EA sketches and views. Sketches and their views can be converted
More informationUser Interfaces. getting arguments of the command line a command line interface to store points fitting points with polyfit of numpy
User Interfaces 1 Command Line Interfaces getting arguments of the command line a command line interface to store points fitting points with polyfit of numpy 2 Encapsulation by Object Oriented Programming
More informationWeb Clients and Crawlers
Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning Files looking for strings between double quotes parsing URLs for the server location
More informationSQL: Data De ni on. B0B36DBS, BD6B36DBS: Database Systems. h p://www.ksi.m.cuni.cz/~svoboda/courses/172-b0b36dbs/ Lecture 3
B0B36DBS, BD6B36DBS: Database Systems h p://www.ksi.m.cuni.cz/~svoboda/courses/172-b0b36dbs/ Lecture 3 SQL: Data De ni on Mar n Svoboda mar n.svoboda@fel.cvut.cz 13. 3. 2018 Czech Technical University
More informationTHE DEFINITIVE GUIDE TO GTFS-REALTIME. Quentin Zervaas
THE DEFINITIVE GUIDE TO GTFS-REALTIME Quentin Zervaas The Denitive Guide to GTFS-realtime How to consume and produce real-time public transportation data with the GTFS-rt specication. Quentin Zervaas About
More informationCPSC 217 Midterm (Python 3 version)
CPSC 217 Midterm (Python 3 version) Duration: 50 minutes 6 March 2009 This exam has 61 questions and 11 pages. This exam is closed book. No notes, books, calculators or electronic devices, or other assistance
More informationReview for Second Midterm Exam
Review for Second Midterm Exam 1 Policies & Material 2 Questions modular design working with files object-oriented programming testing, exceptions, complexity GUI design and implementation MCS 260 Lecture
More informationMySQL: an application
Data Types and other stuff you should know in order to amaze and dazzle your friends at parties after you finally give up that dream of being a magician and stop making ridiculous balloon animals and begin
More informationWelcome to MCS 275. Course Content Prerequisites & Expectations. Scripting in Python from OOP to LAMP example: Factorization in Primes
Welcome to MCS 275 1 About the Course Course Content Prerequisites & Expectations 2 Introduction to Programming Scripting in Python from OOP to LAMP example: Factorization in Primes 3 Summary MCS 275 Lecture
More informationData Modelling and Databases Exercise dates: March 20/March 27, 2017 Ce Zhang, Gustavo Alonso Last update: February 17, 2018.
Data Modelling and Databases Exercise dates: March 20/March 27, 2017 Ce Zhang, Gustavo Alonso Last update: February 17, 2018 Spring Semester 2018 Head TA: Ingo Müller Datasets set-up This assignment will
More informationCOMP 4/6262: Programming UNIX
COMP 4/6262: Programming UNIX Lecture 12 shells, shell programming: passing arguments, if, debug March 13, 2006 Outline shells shell programming passing arguments (KW Ch.7) exit status if (KW Ch.8) test
More informationWhat is SQL? Toolkit for this guide. Learning SQL Using phpmyadmin
http://www.php-editors.com/articles/sql_phpmyadmin.php 1 of 8 Members Login User Name: Article: Learning SQL using phpmyadmin Password: Remember Me? register now! Main Menu PHP Tools PHP Help Request PHP
More informationProvider: MySQLAB Web page:
Provider: MySQLAB Web page: www.mysql.com Installation of MySQL. Installation of MySQL. Download the mysql-3.3.5-win.zip and mysql++-.7.--win3-vc++.zip files from the mysql.com site. Unzip mysql-3.3.5-win.zip
More informationMySQL Creating a Database Lecture 3
MySQL Creating a Database Lecture 3 Robb T Koether Hampden-Sydney College Mon, Jan 23, 2012 Robb T Koether (Hampden-Sydney College) MySQL Creating a DatabaseLecture 3 Mon, Jan 23, 2012 1 / 31 1 Multiple
More informationCSCI-UA: Database Design & Web Implementation. Professor Evan Sandhaus Lecture #23: SQLite
CSCI-UA:0060-02 Database Design & Web Implementation Professor Evan Sandhaus sandhaus@cs.nyu.edu evan@nytimes.com Lecture #23: SQLite Database Design and Web Implementation Administrivia! Homework HW 8
More informationSTOP DROWNING IN DATA. START MAKING SENSE! An Introduction To SQLite Databases. (Data for this tutorial at
STOP DROWNING IN DATA. START MAKING SENSE! Or An Introduction To SQLite Databases (Data for this tutorial at www.peteraldhous.com/data) You may have previously used spreadsheets to organize and analyze
More informationConnecting People and Events: Multi-Modal Routing and Dynamic User-Generated Content
2009 Connecting People and Events: Multi-Modal Routing and Dynamic User-Generated Content Honors Thesis Project to develop a multi-modal routing algorithm for The Ohio State University s campus and greater
More informationRandom Walks & Cellular Automata
Random Walks & Cellular Automata 1 Particle Movements basic version of the simulation vectorized implementation 2 Cellular Automata pictures of matrices an animation of matrix plots the game of life of
More informationETH Zurich Spring Semester Systems Group Lecturer(s): Gustavo Alonso, Ce Zhang Date: March 20/March 27, 2017.
Data Modelling and Databases (DMDB) ETH Zurich Spring Semester 2017 Systems Group Lecturer(s): Gustavo Alonso, Ce Zhang Date: March Assistant(s): Claude Barthels, Eleftherios Sidirourgos, Last update:
More informationCS 1110 SPRING 2016: GETTING STARTED (Jan 27-28) First Name: Last Name: NetID:
CS 1110 SPRING 2016: GETTING STARTED (Jan 27-28) http://www.cs.cornell.edu/courses/cs1110/2016sp/labs/lab01/lab01.pdf First Name: Last Name: NetID: Goals. Learning a computer language is a lot like learning
More informationWeb Clients and Crawlers
Web Clients and Crawlers 1 Web Clients alternatives to web browsers opening a web page and copying its content 2 Scanning files looking for strings between double quotes parsing URLs for the server location
More informationInformation Systems Engineering. SQL Structured Query Language DDL Data Definition (sub)language
Information Systems Engineering SQL Structured Query Language DDL Data Definition (sub)language 1 SQL Standard Language for the Definition, Querying and Manipulation of Relational Databases on DBMSs Its
More informationMySQL by Examples for Beginners
yet another insignificant programming notes... HOME MySQL by Examples for Beginners Read "How to Install MySQL and Get Started" on how to install, customize, and get started with MySQL. 1. Summary of MySQL
More informationMTC 511 Regional Real-time Transit System
MTC 511 Regional Real-time Transit System Extensible Markup Language (XML) Document Type Definitions (DTDs) for Java Message Service (JMS) Implementation (DRAFT) February 12, 2009 May 5, 2009 1 Introduction
More informationWHAT IS A DATABASE? There are at least six commonly known database types: flat, hierarchical, network, relational, dimensional, and object.
1 WHAT IS A DATABASE? A database is any organized collection of data that fulfills some purpose. As weather researchers, you will often have to access and evaluate large amounts of weather data, and this
More informationModel Question Paper. Credits: 4 Marks: 140
Model Question Paper Subject Code: BT0075 Subject Name: RDBMS and MySQL Credits: 4 Marks: 140 Part A (One mark questions) 1. MySQL Server works in A. client/server B. specification gap embedded systems
More informationSQL Functionality SQL. Creating Relation Schemas. Creating Relation Schemas
SQL SQL Functionality stands for Structured Query Language sometimes pronounced sequel a very-high-level (declarative) language user specifies what is wanted, not how to find it number of standards original
More informationWelcome to MCS 360. content expectations. using g++ input and output streams the namespace std. Euclid s algorithm the while and do-while statements
Welcome to MCS 360 1 About the Course content expectations 2 our first C++ program using g++ input and output streams the namespace std 3 Greatest Common Divisor Euclid s algorithm the while and do-while
More informationIBM DB2 UDB V7.1 Family Fundamentals.
IBM 000-512 DB2 UDB V7.1 Family Fundamentals http://killexams.com/exam-detail/000-512 Answer: E QUESTION: 98 Given the following: A table containing a list of all seats on an airplane. A seat consists
More informationIn this exercise, you will import orders table from MySQL database. into HDFS. Get acquainted with some of basic commands of Sqoop
Practice Using Sqoop Data Files: ~/labs/sql/retail_db.sql MySQL database: retail_db In this exercise, you will import orders table from MySQL database into HDFS. Get acquainted with some of basic commands
More informationIs the 370 the worst bus in Sydney?
Is the 370 the worst bus in Sydney? 11 October, 2016 Questions:» Bus privitisation? Better or worse?» Is the 370 is the worst bus route in Sydney? (or are they all that bad?) Transport for NSW
More informationTUTORIAL FOR IMPORTING OTTAWA FIRE HYDRANT PARKING VIOLATION DATA INTO MYSQL
TUTORIAL FOR IMPORTING OTTAWA FIRE HYDRANT PARKING VIOLATION DATA INTO MYSQL We have spent the first part of the course learning Excel: importing files, cleaning, sorting, filtering, pivot tables and exporting
More informationKaivos User Guide Getting a database account 2
Contents Kaivos User Guide 1 1. Getting a database account 2 2. MySQL client programs at CSC 2 2.1 Connecting your database..................................... 2 2.2 Setting default values for MySQL connection..........................
More informationSQL Structured Query Language Introduction
SQL Structured Query Language Introduction Rifat Shahriyar Dept of CSE, BUET Tables In relational database systems data are represented using tables (relations). A query issued against the database also
More informationCSE 115. Introduction to Computer Science I
CSE 115 Introduction to Computer Science I Road map Review HTTP Web API's JSON in Python Examples Python Web Server import bottle @bottle.route("/") def any_name(): response = "" response
More informationGraphical User Interfaces
Graphical User Interfaces 1 User Interfaces GUIs in Python with Tkinter object oriented GUI programming 2 Mixing Colors specification of the GUI the widget Scale 3 Simulating a Bouncing Ball layout of
More informationFinal Exam, Version 1 CSci 127: Introduction to Computer Science Hunter College, City University of New York
Final Exam, Version 1 CSci 127: Introduction to Computer Science Hunter College, City University of New York 17 May 2018 1. (a) What will the following Python code print: i. a = "Jan&Feb&Mar&Apr&May&Jun"
More informationuser specifies what is wanted, not how to find it
SQL stands for Structured Query Language sometimes pronounced sequel a very-high-level (declarative) language user specifies what is wanted, not how to find it number of standards original ANSI SQL updated
More informationUsing MySQL on the Winthrop Linux Systems
Using MySQL on the Winthrop Linux Systems by Dr. Kent Foster adapted for CSCI 297 Scripting Languages by Dr. Dannelly updated March 2017 I. Creating your MySQL password: Your mysql account username has
More informationOutline. gzip and gunzip data compression archiving files and pipes in Unix. format conversions encrypting text
Outline 1 Compressing Files gzip and gunzip data compression archiving files and pipes in Unix 2 File Methods in Python format conversions encrypting text 3 Using Buffers counting and replacing words using
More informationCreating Your First MySQL Database. Scott Seighman Sales Consultant Oracle
Creating Your First MySQL Database Scott Seighman Sales Consultant Oracle Agenda Installation Review Accessing the MySQL Server > Command Line > GUI Tools/MySQL Workbench 5.2 Creating Your First Database
More informationLecture Programming in C++ PART 1. By Assistant Professor Dr. Ali Kattan
Lecture 08-1 Programming in C++ PART 1 By Assistant Professor Dr. Ali Kattan 1 The Conditional Operator The conditional operator is similar to the if..else statement but has a shorter format. This is useful
More informationCS 327E Lecture 2. Shirley Cohen. January 27, 2016
CS 327E Lecture 2 Shirley Cohen January 27, 2016 Agenda Announcements Homework for today Reading Quiz Concept Questions Homework for next time Announcements Lecture slides and notes will be posted on the
More informationLab # 1. You will be using MySQL as a database management system during the labs. The goal of this first lab is to familiarize you with MySQL.
DDB Spring 2006 Lab # 1 You will be using MySQL as a database management system during the labs. The goal of this first lab is to familiarize you with MySQL. The reason you are using MySQL is twofolds.
More informationRandom Walks & Cellular Automata
Random Walks & Cellular Automata 1 Particle Movements basic version of the simulation vectorized implementation 2 Cellular Automata pictures of matrices an animation of matrix plots the game of life of
More informationMySQL Installation Guide (OS X)
Step1- Install MySQL MySQL Installation Guide (OS X) Go to MySQL download page (http://dev.mysql.com/downloads/mysql/). Download the DMG archive version. Select the correct installer based on your system.
More informationUNIX II:grep, awk, sed. October 30, 2017
UNIX II:grep, awk, sed October 30, 2017 File searching and manipulation In many cases, you might have a file in which you need to find specific entries (want to find each case of NaN in your datafile for
More informationTUTORIAL FOR IMPORTING OTTAWA FIRE HYDRANT PARKING VIOLATION DATA INTO MYSQL
TUTORIAL FOR IMPORTING OTTAWA FIRE HYDRANT PARKING VIOLATION DATA INTO MYSQL We have spent the first part of the course learning Excel: importing files, cleaning, sorting, filtering, pivot tables and exporting
More informationFrom Chrome or Firefox browser, Open Google.com/MyMaps (You must log-in to a Google account)
DETAILED INSTRUCTIONS FOR GTFS BUILDER CREATING SHAPES DISPLAYED IN GOOGLE TRANSIT (see last page for quick guide/summary or short videos found here /supportcenter/builder-apps/gtfs-builder/advanced-topics
More informationMultiple Choice (Questions 1 14) 28 Points Select all correct answers (multiple correct answers are possible)
Name Closed notes, book and neighbor. If you have any questions ask them. Notes: Segment of code necessary C++ statements to perform the action described not a complete program Program a complete C++ program
More informationShell scripting and system variables. HORT Lecture 5 Instructor: Kranthi Varala
Shell scripting and system variables HORT 59000 Lecture 5 Instructor: Kranthi Varala Text editors Programs built to assist creation and manipulation of text files, typically scripts. nano : easy-to-learn,
More informationInsertions, Deletions, and Updates
Insertions, Deletions, and Updates Lecture 5 Robb T. Koether Hampden-Sydney College Wed, Jan 24, 2018 Robb T. Koether (Hampden-Sydney College) Insertions, Deletions, and Updates Wed, Jan 24, 2018 1 / 17
More informationCPTS 111, Fall 2011, Sections 6&7 Exam 3 Review
CPTS 111, Fall 2011, Sections 6&7 Exam 3 Review File processing Files are opened with the open() command. We can open files for reading or writing. The open() command takes two arguments, the file name
More informationCSC A20H3 S 2011 Test 1 Duration 90 minutes Aids allowed: none. Student Number:
CSC A20H3 S 2011 Test 1 Duration 90 minutes Aids allowed: none Last Name: Lecture Section: L0101 Student Number: First Name: Instructor: Bretscher Do not turn this page until you have received the signal
More informationRunning Cython and Vectorization
Running Cython and Vectorization 1 Getting Started with Cython overview hello world with Cython 2 Numerical Integration experimental setup adding type declarations cdef functions & calling external functions
More informationWorld Premium Points of Interest Getting Started Guide
World Premium Points of Interest Getting Started Guide Version: 2.0 NOTICE: Copyright Pitney Bowes 2017. All Rights Reserved. 1 Table of Contents INTRODUCTION... 3 1. Preface... 3 2. Data Characteristics...
More informationMysql Tutorial Show Table Like Name Not >>>CLICK HERE<<<
Mysql Tutorial Show Table Like Name Not SHOW TABLES LIKE '%shop%' And the command above is not working as Table name and next SHOW CREATE TABLEcommand user889349 Apr 18. If you do not want to see entire
More informationFinal Exam, Version 2 CSci 127: Introduction to Computer Science Hunter College, City University of New York
Final Exam, Version 2 CSci 127: Introduction to Computer Science Hunter College, City University of New York 13 December 2017 Exam Rules Show all your work. Your grade will be based on the work shown.
More informationFrom MySQL to PostgreSQL
From MySQL to PostgreSQL PostgreSQL Conference Europe 2013 Dimitri Fontaine dimitri@2ndquadrant.fr @tapoueh October, 31 2013 Dimitri Fontaine dimitri@2ndquadrant.fr @tapouehfrom MySQL to PostgreSQL October,
More informationGOSAT Tools Installation and Operation Manual
GOSAT Tools Installation and Operation Manual May 2018 NIES GOSAT Project Table of Contents 1. Introduction... 1 1.1 Overview... 2 1.2 System Requirements... 3 2. Installing... 4 2.1 Location Data of Validation
More informationLoad data into Table from external files, using two methods:
Load data into Table from external files, using two methods: 1) SQL Loader 2) External tables I) SQL Loader. Source Table Name : SYS.DBA_USERS Target Table Name : SYS.MY_DBA_USERS 1) We need to have the
More informationCSci 1113 Lab Exercise 6 (Week 7): Arrays & Strings
CSci 1113 Lab Exercise 6 (Week 7): Arrays & Strings Strings Representing textual information using sequences of characters is common throughout computing. Names, sentences, text, prompts, etc. all need
More informationMySQL User Conference and Expo 2010 Optimizing Stored Routines
MySQL User Conference and Expo 2010 Optimizing Stored Routines 1 Welcome, thanks for attending! Roland Bouman; Leiden, Netherlands Ex MySQL AB, Sun Microsystems Web and BI Developer Co-author of Pentaho
More informationWorld Premium Points of Interest Getting Started Guide
World Premium Points of Interest Getting Started Guide Version: 2.3 NOTICE: Copyright Pitney Bowes 2019. All Rights Reserved. 1 Table of Contents INTRODUCTION... 3 1. Preface... 3 2. Data Characteristics...
More informationData Types in MySQL CSCU9Q5. MySQL. Data Types. Consequences of Data Types. Common Data Types. Storage size Character String Date and Time.
- Database P&A Data Types in MySQL MySQL Data Types Data types define the way data in a field can be manipulated For example, you can multiply two numbers but not two strings We have seen data types mentioned
More informationGood Luck! CSC207, Fall 2012: Quiz 1 Duration 25 minutes Aids allowed: none. Student Number:
CSC207, Fall 2012: Quiz 1 Duration 25 minutes Aids allowed: none Student Number: Last Name: Lecture Section: L0101 First Name: Instructor: Horton Please fill out the identification section above as well
More informationXQ: An XML Query Language Language Reference Manual
XQ: An XML Query Language Language Reference Manual Kin Ng kn2006@columbia.edu 1. Introduction XQ is a query language for XML documents. This language enables programmers to express queries in a few simple
More informationExact Numeric Data Types
SQL Server Notes for FYP SQL data type is an attribute that specifies type of data of any object. Each column, variable and expression has related data type in SQL. You would use these data types while
More informationLoop structures and booleans
Loop structures and booleans Michael Mandel Lecture 7 Methods in Computational Linguistics I The City University of New York, Graduate Center https://github.com/ling78100/lectureexamples/blob/master/lecture07final.ipynb
More informationIntro to Database Commands
Intro to Database Commands SQL (Structured Query Language) Allows you to create and delete tables Four basic commands select insert delete update You can use clauses to narrow/format your result sets or
More informationsqoop Automatic database import Aaron Kimball Cloudera Inc. June 18, 2009
sqoop Automatic database import Aaron Kimball Cloudera Inc. June 18, 2009 The problem Structured data already captured in databases should be used with unstructured data in Hadoop Tedious glue code necessary
More informationMultithreaded Servers
Multithreaded Servers 1 Serving Multiple Clients avoid to block clients with waiting using sockets and threads 2 Waiting for Data from 3 Clients running a simple multithreaded server code for client and
More informationMySQL Schema Best Practices
MySQL Schema Best Practices 2 Agenda Introduction 3 4 Introduction - Sample Schema Key Considerations 5 Data Types 6 Data Types [root@plive-2017-demo plive_2017]# ls -alh action*.ibd -rw-r-----. 1 mysql
More informationMysql Information Schema Update Time Null >>>CLICK HERE<<< doctrine:schema:update --dump-sql ALTER TABLE categorie
Mysql Information Schema Update Time Null I want to update a MySQL database schema (with MySQL code) but I am unfortunately not sure 'name' VARCHAR(64) NOT NULL 'password' VARCHAR(64) NOT NULL fieldname
More informationMastering Modern Linux by Paul S. Wang Appendix: Pattern Processing with awk
Mastering Modern Linux by Paul S. Wang Appendix: Pattern Processing with awk The awk program is a powerful yet simple filter. It processes its input one line at a time, applying user-specified awk pattern
More informationApplicaton Instrumentaton for MySQL What Why and How
Applicaton Instrumentaton for MySQL What Why and How Peter Zaitsev, CEO Percona Inc 18/04/12 Agenda Importance of Instrumentation of Application What needs to be Instrumented How can you do it Secret Agenda
More informationBasic SQL. Dr Fawaz Alarfaj. ACKNOWLEDGEMENT Slides are adopted from: Elmasri & Navathe, Fundamentals of Database Systems MySQL Documentation
Basic SQL Dr Fawaz Alarfaj Al Imam Mohammed Ibn Saud Islamic University ACKNOWLEDGEMENT Slides are adopted from: Elmasri & Navathe, Fundamentals of Database Systems MySQL Documentation MIDTERM EXAM 2 Basic
More informationCS Programming Languages: Python
CS 3101-1 - Programming Languages: Python Lecture 5: Exceptions / Daniel Bauer (bauer@cs.columbia.edu) October 08 2014 Daniel Bauer CS3101-1 Python - 05 - Exceptions / 1/35 Contents Exceptions Daniel Bauer
More information1 Truth. 2 Conditional Statements. Expressions That Can Evaluate to Boolean Values. Williams College Lecture 4 Brent Heeringa, Bill Jannen
1 Truth Last lecture we learned about the int, float, and string types. Another very important object type in Python is the boolean type. The two reserved keywords True and False are values with type boolean.
More informationAdvanced MySQL Query Tuning
Advanced MySQL Query Tuning Alexander Rubin August 6, 2014 About Me My name is Alexander Rubin Working with MySQL for over 10 years Started at MySQL AB, then Sun Microsystems, then Oracle (MySQL Consulting)
More information2
2 3 4 5 6 Open a terminal window and type python If on Windows open a Python IDE like IDLE At the prompt type hello world! >>> 'hello world!' 'hello world!' Python is an interpreted language The interpreter
More information