CDO-1 Certificate Program: Foundations for Chief Data Officers. Data Wrangling Lab. David /WEI DAI. Sept 26-29, 2016 (c)

Transcription

1 CDO-1 Certificate Program: Foundations for Chief Data Officers Data Wrangling Lab David /WEI DAI Sept 26-29, 2016 (c)

2 Agenda Basic Python Program MongoDB Lab Clean Data Lab Sept 26-29, 2016 (c)

3 A Tutorial on the Python Programming Language Sept 26-29, 2016 (c)

4 Why do we choose Python? C or C++ Java Perl Scheme Fortran Python Matlab Modern, interpreted, object-oriented, full featured high level programming language Portable(Unix/Linux,MacOS X,Windows) Open source, intellectual property rights held by the Python Software Foundation Python versions: 2.x and 3.x 3.x is not backwards compatible with 2.x This course uses 3.x version Fast program development Simple syntax Easy to write well readable code Large standard library Lots of third party libraries Numpy, Scipy, Biopython Matplotlib Sept 26-29, 2016 (c) 2016 icdo@ualr 4

5 Python Program Platform Open a browser and access the website: Password: python Sept 26-29, 2016 (c) 2016 icdo@ualr 5

6 Hello World At the prompt type hello world! Sept 26-29, 2016 (c)

7 The print and string Statement Elements separated by commas print with a space between them Strings are immutable + is overloaded to do concatenation >>> print('hello') hello >>> print('hello', David') hello David >>> x = 'hello' >>> x = x + ' America' >>> print(x) 'hello America' Sept 26-29, 2016 (c) 2016 icdo@ualr 7

8 Substrings and Methods >>> s = '012345' >>> print(s[3]) '3' >>> print(s[1:4]) '123' >>> print(s[2:]) '2345' >>> print(s[:4]) '0123' >>> print(s[-2]) '4' len(string) returns the number of characters in the String str(object) returns a String representation of the Object >>> print(len(s)) 6 >>> print(str(10.3)) '10.3' Sept 26-29, 2016 (c) 2016 icdo@ualr 8

9 Relational operators == equal!=, <> not equal > greater than >= greater than or equal < less than <= less than or equal Logical operators and or notnot and or Sept 26-29, 2016 (c)

10 Variables Are not declared, just assigned The variable is created the first time you assign it a value Assignment is = and comparison is == Sept 26-29, 2016 (c) 2016 icdo@ualr 10

11 Lists Ordered collection of data Data can be of different types Lists are mutable Issues with shared references and mutability Same subset operations as Strings >>> x = [1,'hello', (3 + 2j)] >>> print(x) [1, 'hello', (3+2j)] >>> print(x[2]) (3+2j) >>> print(x[0:2]) [1, 'hello'] Sept 26-29, 2016 (c) 2016 icdo@ualr 11

12 Lists: Modifying Content x[i] = a reassigns the ith element to the value a Since x and y point to the same list object, both are changed The method append also modifies the list >>> x = [1,2,3] >>> y = x >>> x[1] = 15 >>>print( x) [1, 15, 3] >>> print(y) [1, 15, 3] >>> x.append(12) >>> print(y) [1, 15, 3, 12] Sept 26-29, 2016 (c) 2016 icdo@ualr 12

13 Lists: Modifying Contents The method append modifies the list and returns None List addition (+) returns a new list >>> x = [1,2,3] >>> y = x >>> z = x.append(12) >>> print(z == None) True >>> print(y) [1, 2, 3, 12] >>> x = x + [9,10] >>> print(x) [1, 2, 3, 12, 9, 10] >>> print(y) [1, 2, 3, 12] >>> Sept 26-29, 2016 (c) 2016 icdo@ualr 13

14 If ELSE Statements if expression: statement(s) else: statement(s) Sept 26-29, 2016 (c)

15 For Loops Similar to perl for loops, iterating through a list of values forloop1.py for x in [1,6,12,3] : print(x) forloop2.py for x in range(4) : print(x) range(n) generates a list of numbers [0,1,, n-1] Sept 26-29, 2016 (c) 2016 icdo@ualr 15

16 Functions are first class objects Can be assigned to a variable Can be passed as a parameter Can be returned from a function Functions are treated like any other variable in Python, the def statement simply assigns a function to a variable Sept 26-29, 2016 (c) 2016 icdo@ualr 16

17 Function Basics def min(x,y) : if x > y : return x else : return y >>> mix(2,5) 5 functionbasics.py Sept 26-29, 2016 (c) 2016 icdo@ualr 17

18 Python for graph Matplotlib is a python 2D plotting library which produces high quality figures Read demos is ready at plot_demo.ipy file. Sept 26-29, 2016 (c) 2016 icdo@ualr 18

19 MongoDB LAB Sept 26-29, 2016 (c)

20 MongoDB Express User Interface username: mongotest Password: mongotest Sept 26-29, 2016 (c)

21 MongoDB Express MongoDB Express is Web-based MongoDB admin interface You can create, review, export, delete data through the platform Sept 26-29, 2016 (c)

22 MongoDB Express Lab Export cities.json Add a new city name which you like to MongoDB Query or find the new city name Delete the new city name Sept 26-29, 2016 (c) 2016 icdo@ualr 22

23 Clean Data Lab Sept 26-29, 2016 (c)

24 Courses Data in MongoDB Sept 26-29, 2016 (c)

25 Connect to MongoDB Sept 26-29, 2016 (c)

26 CRUD Operation for MongoDB Sept 26-29, 2016 (c)

27 Basic Python-MongoDB Lab Write codes to add a new course {"courseid": "71XX", <--Change XX "subject": "information science", "title": "data quality algorithm", <--Change course name "hours": 3 <--Change hours } Write codes to search your courses query = {"title": "data quality algorithm" } <--Change title name projection = {"hours": 3 <--Change hours Sept 26-29, 2016 (c) 2016 icdo@ualr 27

28 Basic Python-MongoDB lab (cont.) A challenge project Write codes to add your name at teachers list Sept 26-29, 2016 (c) 2016 icdo@ualr 28

29 Clean Data lab (cont.) Teachers, Courses, and Students are MDM data so that the data is accurate and trust. student_course_report and teacher_course_report contain incorrect data, but teacherid, studentid,and courseid are correct. Teachersinfo teacher_course_report Sept 26-29, 2016 (c)

30 Clean Data lab (cont.) teacher_course_report Sept 26-29, 2016 (c)

31 Clean Data lab (cont.) Write codes to clean student_course_report Tips: studentsinfo coursesinfo student_course_report Sept 26-29, 2016 (c)

32 Clean Data lab (cont.) A challenge project Write codes to clean t_s_c_report. coursesinfo studentsinfo Sept 26-29, 2016 (c) 2016 icdo@ualr Teachersinfo 32

33 THANK YOU Sept 26-29, 2016 (c)

34 Reference index.html n/current/ ual/ /views/edoc_download.php/594 4/pdf O'higgins, Niall. MongoDB and Python: Patterns and processes for the popular document-oriented database. " O'Reilly Media, Inc.", sg=afqjcnh5nwz_paanbl7jcde6pn7sfuvxyw&sig2=sgxl0riqfl8gbxqd7mfura Sept 26-29, 2016 (c)