COMP20008 Elements of Data Processing. Week 1: Lecture 2. Data format and storage

Transcription

1 COMP20008 Elements of Data Processing Week 1: Lecture 2 Data format and storage

2 Announcements Lecture recordings Lecture Capture: Current Technical Issue. There are currently long delays in processing recorded lectures. We apologise for the inconvenience.

3 Announcements Assessable content Includes material from lectures, workshops and assignments Study guide describing in detail what concepts to focus on, will be released towards end of semester

4 Announcements Student representatives Nelson Chen Joanna Lee Next week s workshop Is available on the LMS

5 Today Where is the data? How is data stored and in what formats? RDB, HTML, CSV, XML, JSON, RDF. Question: Why do we have different data formats and why do we wish to transform between different formats?

6 Relational databases (INFO20003) It is good to have structure for data! Easier to analyse, easier to query Easier to store Easier to clean, maintain consistency and security, especially with multiple users Relational databases, the classic method of storing structured data (banking, sales, airlines ) Data stored in tables, each row is a data item and columns describe attributes of the data item Can query the data using a high level language such as SQL

7 Examples from Silberschatz et al Database System Concepts Attributes

8 Sample relational database

9 SQL create table branch (branch_name char(15) not null, branch_city char(30), assets integer, primary key (branch_name))

10 SQL select account.balance from depositor, account where depositor.customer_id = and depositor.account_number=account.account_number

11 Database system structure

12 Database systems In INFO20003 subject you will cover topics like SQL Specification of integrity constraints Data modelling and relational database management systems Transactions and concurrency control Storage management Web-based databases. Highly relevant to data wrangling! Useful to do INFO20003 as part of a data science specialisation

13 Challenges Once data is into a relational database, it is easier to wrangle. But maybe hard to load it there in the first place Unstructured data: text, HTML, sequences, graphs

14 NoSQL Not only SQL databases Want a highly scalable and elastic system, distributed over many servers Can flexibly store different data types (documents, sequences, graphs, objects) Sacrifice some of the nice properties of relational databases Query language Consistency and integrity guarantees Examples Key-value storage, document database, graph database Google BigTable, Amazon SimpleDB, Apache CouchDB, MongoDB,.. We will revisit these in more detail when we cover distributed & cloud topic in a few weeks time

15 Spreadsheets: CSV Huge amounts of data lives in spreadsheets Businesses Hospitals. Microsoft (Excel), OpenOffice (Calc), Google docs CSV (comma separated values) also very popular These are human readable, versus binary XLS format (Excel) CSVs lack the formatting information of an XLS file Python libraries csv xlrd, openpyxl

16 Text data (documents..) Specifying patterns in text regular expressions Good for computing statistics, checking integrity, filtering, substitutions. Specifying patterns in text. matches any character ^ matches start of string $ matches end of string * zero or more repetitions + one or more repetitions the or operator - [] a set of characters, e.g. [abcd] or [a-za-z] regex101.com

17 Exercises (3 minutes) Write regular expressions to specify each of the following Two occurrences of letter e followed immediately by one n and then at least one t An h or an e or an x, followed by at least one `a, followed by an r Any 3 characters, possibly followed by a repeated sequence of the character x, followed by a c or a d

18 address regular expression One attempt See also Python library

19 Table data As well as relational databases and csv files, table data is abundant on the Web Encoded in HTML Google fusion tables

20 HTML Hypertext Markup language Marked up with elements, delineated by start and end tags. Elements correspond to logical units, such as a heading, paragraph or itemised list. Tags: Keywords contained in pairs of angle brackets. Not case sensitive. Browser determines how to display/present the logical units Not all elements need both start and end tags. Some elements can have attributes. Ordering of attributes is not significant.

21 HTML Example <div class="icon section5"> <hh2><a href="about/index.html">about the Melbourne School of Engineering</ a></h2> <ul> <li><a href="about/dean_welcome.html">dean's Welcome</a></li> <li><a href="about/staff.html">leadership & Professional Staff</a></li> <li><a href="about/contact.html">contact Us</a></li> <li><a href=" Computer Resources</a></li> <li><a href="intranet/index.html">for Staff (intranet)</a></li> <li><a href="casual_staff/index.html">for Casual Staff</a></li> <li><a href="intranet/review/prof_staff.html">professional Staff Review</a></li> <li><a href="/about/safety/index.html">environment, Health & Safety</a></li> <li><a href="/about/committees/index.html">committees</a></li> </ul>

22 XML Extensible Markup Language Allows new elements to be defined Applications may generate and process XML Enables data exchange between different platforms Facilitates better encoding of semantics <CATALOG> <CD> <TITLE>Empire Burlesque</TITLE> <ARTIST>Bob Dylan</ARTIST> <COUNTRY>USA</COUNTRY> <COMPANY>Columbia</COMPANY> <PRICE CURRENCY="USD"> 10.90</PRICE> <YEAR>1985</YEAR> </CD> <CD> <TITLE>Hide your heart</title> <ARTIST>Bonnie Tyler</ARTIST> <COUNTRY>UK</COUNTRY> <COMPANY>CBS Records</COMPANY> <PRICE CURRENCY="USD">9.90</PRICE> <YEAR>1988</YEAR> </CD> </CATALOG>

23 JSON: JavaScript Object Notation { } "Catalog": [ ] { "CD": { "title": "Empire Burlesque", }, "artist": "Bon Dylan", "Country": "USA" "price": { "Currency": "USD", "value": }, "year": 1985 } { "CD": { } "title": "Hide your heart", "artist": "Bonnie Taylor", "Country": "UK", "price": { "currency": "USD", "value": 9.90 }, "year": 1988} } JSON is simpler and more compact/ lightweight than XML. Easy to parse. Common JSON application read and display data from a webserver using javascript. json_http.asp XML comes with a large family of other standards for querying and transforming (XQuery, XML Schema, XPATH, XSLT, namespaces, )

24 Jason format (from json.org)

25 JASON format (json.org)

26 Exercise Represent the following information in JSON <Person> <FirstName>Homer</FirstName> <LastName>Simpson</LastName> <Relatives> <Relative>Grandpa</Relative> <Relative>Marge</Relative> <Relative>Lisa</Relative> <Relative>Bart</Relative> </Relatives> <FavouriteBeer>Duff</FavouriteBeer> </Person>

27 Python libraries json ElementTree html.parser

28 Other forms of data Sequences Graphs Can be represented in multiple ways

29 Sequence data: biology Biological sequences (DNA, proteins) >gi sp P01013 OVAX_CHICK GENE X PROTEIN (OVALBUMIN-RELATED) QIKDLLVSSSTDLDTTLVLVNAIYFKGMWKTAFNAEDTREMPFHVTKQESKPVQMM CMNNSFNVATLPAEKMKILELPFASGDLSMLVLLPDEVSDLERIEKTINFEKLTEWT NPNTMEKRRVKVYLPQMKIEEKYNLTSVLMALGMTDLFIPSANLTGISSAESLKISQ AVHGAFMELSEDGIEMAGSTGVIEDIKHSPESEQFRADHPFLFLIKHNPTNTIVYFG RYWSP

30 Graphs: Social networks

31 Protein-Protein Interactions

32 The Internet Graph (

33 Graph Data - RDF RDF= Resource Description Framework Used for storing semantic data (relationships between concepts and objects) Used on the Semantic Web E.g. Freebase, Google knowledge graph

34 Graphs RDF (resource description framework) [materials from w3.org]

35 Serialisation of RDF Example Graph This graph can be serialised as XML (don t worry about syntax!) <?xml version="1.0"?> <rdf:rdf xmlns:rdf=" xmlns:contact=" contact#"> <contact:person rdf:about=" contact#me"> <contact:fullname>eric Miller</contact:fullName> <contact:mailbox rdf:resource="mailto:em@w3.org"/> <contact:personaltitle>dr.</contact:personaltitle> </contact:person>

36 Freebase A large database that connects entities together as a graph The basis of the Google Knowledge graph that is used to improve search. search/knowledge.html

37 RDF Triple Store An alternative format for storing RDF type data triple store < < 2000/10/swap/pim/contact#fullName> "Eric Miller". < < 2000/10/swap/pim/contact#mailbox> <mailto:e.miller123(at)example>. < < 2000/10/swap/pim/contact#personalTitle> "Dr.". < < 1999/02/22-rdf-syntax-ns#type> < pim/contact#person>.

38 Graphs: Matrix Representation A B A B C D A B C D C D Source A 1 in the matrix iff there is an edge from node X to node Y. Or use a relational table Destination A C D C B B

39 Next week Workshop for next week Available on the LMS Useful Unix tools Directory navigation and file manipulation, redirection, pipes, awk, sed, regex, grep... Look at Section 1a before you attend your workshop Lectures next week Data quality and data cleaning (lasting ~2 weeks)

40 Further reading Further reading Relational databases Pages of XML JSON RDF