Big Data 11. Data Models

Transcription

1 Ghislain Fourny Big Data 11. Data Models pinkyone / 123RF Stock Photo

2 CSV (Comma separated values) This is syntax ID,Last name,first name,theory, 1,Einstein,Albert,"General, Special Relativity" 2,Gödel,Kurt,"""Incompleteness"" Theorem" 2

3 CSV (Comma separated values) This is syntax ID,Last name,first name,theory, 1,Einstein,Albert,"General, Special Relativity" 2,Gödel,Kurt,"""Incompleteness"" Theorem" ID Last name First name Theory 1 Einstein Albert General, Special Relativity 2 Gödel Kurt "Incompleteness" Theorem This is a data model 3

4 Syntax vs. Data Models Physical view Syntax ID,Last name,first name,theory, 1,Einstein,Albert,"General, Special Relativity" 2,Gödel,Kurt,"""Incompleteness"" Theorem" 4

5 Syntax vs. Data Models Logical view Data Model ID Last name First name Theory 1 Einstein Albert General, Special Relativity 2 Gödel Kurt "Incompleteness" Theorem Physical view Syntax ID,Last name,first name,theory, 1,Einstein,Albert,"General, Special Relativity" 2,Gödel,Kurt,"""Incompleteness"" Theorem" 5

6 Syntax vs. Data Models Physical view Syntax <a> <d e="f"/> <c>this is <b>text</b>.</c> </a> 6

7 Syntax vs. Data Models a Logical view Data Model e = f d This is c b. text Physical view Syntax <a> <d e="f"/> <c>this is <b>text</b>.</c> </a> 7

8 Edge vs. Node labeling foo bar Labels are on the edges 8

9 Edge vs. Node labeling foo foo foobar bar bar Labels are on the edges Labels are on the nodes 9

10 XML Data models Information Set (Infoset) 10

11 XML Data models Information Set (Infoset) Post Schema-Validation Infoset (PSVI) 11

12 XML Data models Information Set (Infoset) Post Schema-Validation Infoset (PSVI) XQuery and XPath Data Model (XDM) 12

13 JSON Data Models "original" (implicit) JSON Data Model 13

14 JSON Data Models "original" (implicit) JSON Data Model JSON Schema Data Model 14

15 JSON Data Models "original" (implicit) JSON Data Model JSON Schema Data Model JSONiq Data Model (JDM) y/html/section-jsoniq-data-model.html 15

16 HTML/XML Data model Document Object Model (DOM) 16

17 grigory_bruev / 123RF Stock Photo XML Information Set 17

18 Information Set <?xml version="1.0" encoding="utf-8"?> <dc:metadata xmlns:dc=" <title xml:lang="en" year="2008" >Systems Group</title> <publisher>eth Zurich</publisher> </dc:metadata> 18

19 Information Set 19

20 The 11 XML Information Items Document Element Attribute Processing Instruction Character Comment Namespace Unexpanded Entity Reference DTD Unparsed Entity Notation 20 20

21 The 11 XML Information Items Document Element Attribute Processing Instruction Character Comment Namespace Unexpanded Entity Reference DTD Unparsed Entity Notation 21 21

22 Document Information Items Document Information Item doc [children] Element Information Item metadata [document element] Element Information Item metadata [notations] <empty> [unparsed entities] <empty> [base URI ] file:///users/bigdata/documents/info.xml [character encoding scheme] UTF-8 [standalone] <no value> [version]

23 Element Information Items Element Information Item metadata metadata [namespace name] [local name] metadata [prefix] dc [children] Element Information Items title, publisher [attributes] <empty> [namespace attributes] Attribute Information Item xmlns:dc [in-scope namespaces] Namespace Information Items dc->systems, xml->ns [base URI] file:///users/bigdata/documents/info.xml [parent] Document Information Item doc 23

24 Attribute Information Items Attribute Information Item xmlns:dc [namespace name] [local name] dc [prefix] xmlns [normalized value] [specified] true [attribute type] <no value> [references] unknown [owner element] Element Information Item metadata xmlns:dc 24

25 Namespace Information Items Namespace Information Item dc->systems [prefix] dc [namespace name] dc->systems Namespace Information Item xml->ns xml->ns [prefix] xml [namespace name] 25

26 XML Infoset - the tree doc xmlns:dc metadata xml->ns dc->systems xml->ns dc->systems title publisher dc->systems xml->ns lang year ETH Zurich Systems Group 26

27 Post-Schema-Validation Infoset Infoset + Types Post-Schema-Validation Infoset (PSVI) 27

28 Weerapat Kiatdumrong / 123RF Stock Photo XPath and XQuery Data Model 28

29 XDM: Sequences of Items (,,,,, ) 29

30 XDM: Sequence of one item 30

31 XDM: Sequence of one item = ( ) 31

32 XDM: Sequences are flat ((, ), ) 32

33 XDM: Sequences are flat ((, ), )=(,, ) 33

34 XDM: Items Atomic Node 34

35 XDM: Seven Kinds of XML Nodes Document node Element node Attribute node Text node Comment node Processing instruction node Namespace node 35

36 XDM: Seven Kinds of XML Nodes Infoset XDM 36

37 XDM vs. Infoset Infoset XDM xs:untyped 37

38 XDM: New Items in 3.0 and 3.1 Functions 38

39 XDM: New Items in 3.0 and 3.1 lorem ipsum dolor sit amet Functions Maps 39

40 XDM: New Items in 3.0 and 3.1 lorem ipsum dolor sit amet Functions Maps Arrays 40

41 XDM and Querying for let order by if + any else = then every while where return exit with Expression 41

42 Types 42

43 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: 43

44 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: - Distinction between atomic types and structured types 44

45 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: - Distinction between atomic types and structured types - Same categories of atomic types 45

46 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: - Distinction between atomic types and structured types - Same categories of atomic types - Lists and maps as structured types 46

47 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: - Distinction between atomic types and structured types - Same categories of atomic types - Lists and maps as structured types - Sequence type cardinalities 47

48 Types (General) Atomic Types vs. Structured Types 48

49 Atomic Types 49

50 Atomic Types Strings 50

51 Strings (Character sequences with monoid structure) "foo" "Zurich" "Ilsebill salzte nach." f o o Z u r i c h I l s e b i l l s a l z t e n a c h. 51

52 Atomic Types Strings Numbers 52

53 Interval-based integer types (exist as signed and unsigned) 8-bit (Java's byte) 16-bit (Java's short) 32-bit (Java's int) 64-bit (Java's long) 53

54 Arbitrary precision decimals (and integers) Any precision and scale

55 Float and Double IEEE 754 standard 32 bits 64 bits ca. 7 digits ca. 15 digits to to single precision double precision 55

56 Atomic Types Strings Numbers Booleans 56

57 Booleans TRUE t true y yes on 1 FALSE f false n no off 0 57

58 Atomic Types Strings Numbers Booleans Dates and Times 58

59 Dates and times Date Time Timestamp Duration 59

60 Dates (Gregorian calendar) Year + Month + Day 2017 August 1st (AD) 60

61 Times Hours + Minutes + Seconds 10 : 31 :

62 Timestamps Year + Month + Day + Hours + Minutes + Seconds 2017 August 1 st 10 : 31 : (AD) 62

63 Atomic Types Strings Numbers Booleans Dates and Times Time Intervals 63

64 Duration kinds Year Month Day Hour Minute Second Example: 2 years and 4 months 64

65 Duration kinds Year Month Day Hour Minute Second Example: 2 years and 4 months Example: 3 hours and 14 minutes 65

66 Atomic Types Strings Numbers Booleans Dates and Times Time Intervals Binaries 66

67 Atomic Types Strings Numbers Booleans Dates and Times Time Intervals Binaries Null 67

68 Lexical space vs. value space Value space Lexical space 68

69 Lexical space vs. value space "1" "01"... Value space Lexical space 69

70 Lexical space vs. value space "4" "04" "100b"... Value space Lexical space 70

71 Subtypes Supertype's value space Subtype's value space 71

72 Structured Types Data Structure Associative Arrays (a.k.a. maps) Ordered Lists Examples JSON Object, Protobuf Message, Set of XML Attributes JSON Array, XML Element, Protobuf repeated field 72

73 Structured Types Data Structure Associative Arrays (a.k.a. maps) Ordered Lists Examples JSON Object, Protobuf Message, Set of XML Attributes JSON Array, XML Element, Protobuf repeated field 73

74 Cardinality How many? Common sign Common adjective 74

75 Cardinality One How many? Common sign Common adjective required 75

76 Cardinality One How many? Common sign Zero or more * Common adjective required 76

77 Cardinality One How many? Common sign Zero or more * Common adjective required Zero or one? optional 77

78 Cardinality One How many? Common sign Zero or more * Common adjective required Zero or one? optional One or more + 78

79 wklzzz / 123RF Stock Photo JSON Data Model 79

80 JSON Values Strings Objects Numbers Booleans Arrays Null Atomic values Structured values 80

81 JSON Values Strings Numbers Booleans Null Objects String-to-Value map Arrays List of values Atomic values Structured values 81

82 JSON Values Strings Numbers Booleans Null Objects String-to-Value map Recursion Arrays List of values Atomic values Structured values 82

83 Tree-based visual model { } "foo" : true, "bar" : [ { "foobar" : "foo" }, null ] 83

84 Tree-based visual model { } "foo" : true, "bar" : [ { "foobar" : "foo" }, null ] object 84

85 Tree-based visual model { } "foo" : true, "bar" : [ { "foobar" : "foo" }, null ] foo true object bar array 85

86 Tree-based visual model { } "foo" : true, "bar" : [ { "foobar" : "foo" }, null ] foo true object object bar array null foobar foo 86

87 wklzzz / 123RF Stock Photo Protocol Buffers 87

88 Messages message Person { required string last_name = 1; repeated string first_name = 2; optional Title title = 3; optional Person boss = 4; } 88

89 Scalar types double, float int32, int64 and variants bool string bytes 89

90 Enums enum Title { MR = 1; MS = 2; MRS = 3; } 90

91 In C++ person.boss().first_name() 91

92 Validation Burak Cakmak / 123RF Stock Photo 92

93 Validation: The Pipeline Document 93

94 Validation: The Pipeline Document Well- Formedness 94

95 Validation: The Pipeline Document Well- Formedness Validation 95

96 On the oxygen Cheat Sheet Validity Well- Formedness 96

97 Validation vs. Annotation Validation 97

98 Validation vs. Annotation Validation Annotation 98

99 Validation 99

100 Validation 100

101 Validation 101

102 Validation 102

103 Validation 103

104 DTD Validation 104

105 Document Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a>& <a>& &&<d&e="f"/>& &&<c>this&is&<b>text</b>.</c>& </a>& & 105

106 Document Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a>& <a>& &&<d&e="f"/>& &&<c>this&is&<b>text</b>.</c>& </a>& & 106

107 Document Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& ]>& <a>& &&<d&e="f"/>& &&<c>this&is&<b>text</b>.</c>& </a>& & 107

108 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& ]>& <a>& &&<d&e="f"/>& &&<c>this&is&<b>text</b>.</c>& </a>& & 108

109 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& ]>& <a/>& & Empty Content 109

110 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(#PCDATA)>& ]>& <a>& &&This&is&text.& </a>& & Simple Content 110

111 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(foo,&bar)>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&bar&EMPTY>& ]>& <a>& &&<foo/>& &&<bar/>& </a>& & Complex Content 111

112 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(foo+,&bar*,&foobar?)>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&bar&EMPTY>& <!ELEMENT&foobar&EMPTY>& ]>& <a>& &&<foo/>& &&<foo/>& &&<foo/>& &&<foobar/>& </a>& Complex Content 112

113 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(&bar*& &(foo foobar)+)?>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&foobar&EMPTY>& ]>& <a>& &&<foo/>& &&<foobar/>& &&<foo/>& &&<foobar/>& </a>& & Complex Content 113

114 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(#PCDATA& &foo)*>& <!ELEMENT&foo&EMPTY>& ]>& <a>& &&<foo/>lorum<foo/>ipsum<foo/>& </a>& & Mixed Content 114

115 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&ANY>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&bar&EMPTY>& ]>& <a>& &&<foo/>& &&Lorem& &&<bar/>& &&Ipsum& &&<bar/>& &&<bar/>& </a>& & Mixed Content 115

116 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&#REQUIRED>& ]>& <a&foo="this&is&a&"value""></a>& & 116

117 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&#IMPLIED>& ]>& <a&foo="this&is&a&"value""></a>& & 117

118 Attribute-List Declaration <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE a [ <!ELEMENT a EMPTY> <!ATTLIST a foo CDATA #IMPLIED> ]> <a></a> 118

119 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&"bar">& ]>& <a&foo="this&is&a&"value""></a>& & 119

120 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&"bar">& ]>& <a></a>& & 120

121 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&#FIXED&"bar">& ]>& <a&foo="bar"></a>& & 121

122 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&NMTOKEN&#REQUIRED>& ]>& <a&foo="123atoken456"></a>& & 122

123 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&NMTOKENS&#REQUIRED>& ]>& <a&foo="123atoken&456"></a>& & 123

124 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&root&[& <!ELEMENT&root&(foo+,&bar,&barlist)>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&bar&EMPTY>& <!ELEMENT&barlist&EMPTY>& <!ATTLIST&foo&myid&ID&#REQUIRED>& <!ATTLIST&bar&ref&IDREF&#REQUIRED>& <!ATTLIST&barlist&ref&IDREFS&#REQUIRED>& ]>& <root>& &&<foo&myid="foobar"/>& &&<foo&myid="foobar2"/>& &&<bar&ref="foobar"/>& &&<barlist&ref="foobar&foobar2"/>& </root>& & 124

125 DTD Example: External Subset <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&root&SYSTEM&"schema.dtd">& <a&foo="bar"/>& 125

126 Warning: DTDs and Namespaces <!ELEMENT(eth((date,(president,(Rektor)>( <!ATTLIST(eth(xmlns(CDATA(#FIXED( " ((((((((((((((xmlns:xmldb(cdata(#fixed( " ((((((((((((((date(cdata(#implied( ((((((((((((((xmldb:date(cdata(#implied>( <!ELEMENT(date((#PCDATA)>( <!ELEMENT(president((#PCDATA)>( <!ATTLIST(president(number(CDATA(#IMPLIED>( <!ELEMENT(Rektor((#PCDATA)>( ( 126

127 Notations and Unparsed Entities <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE foo [ <!ENTITY presentation SYSTEM "/Users/bigdata/desktop/presentation.pptx" NDATA pptx> <!NOTATION pptx PUBLIC "powerpoint"> <!ELEMENT foo EMPTY> <!ATTLIST foo foo ENTITY #REQUIRED> ]> <foo foo="presentation"></foo> 127

128 XML Schema 128

129 Empty Schema <?xml&version="1.0"&encoding="utf98"?>& <xs:schema& &&xmlns:xs=" </xs:schema>& & 129

130 Simple Scenario <?xml&version="1.0"&encoding="utf98"?>& <xs:schema& &&xmlns:xs=" &&<xs:element&name="foo"&type="xs:string"/>& </xs:schema>& & & schema.xsd <?xml&version="1.0"&encoding="utf98"?>& <foo& &&xmlns:xsi=" &&xsi:nonamespaceschemalocation="schema.xsd">& &&This&is&text.& </foo>& & file.xml 130

131 Simple Scenario <?xml&version="1.0"&encoding="utf98"?>& <xs:schema& &&xmlns:xs=" &&<xs:element&name="foo"&type="xs:integer"/>& </xs:schema>& & & schema.xsd <?xml&version="1.0"&encoding="utf98"?>& <foo& &&xmlns:xsi=" &&xsi:nonamespaceschemalocation="schema.xsd">& &&142857& </foo>& & file.xml 131

132 Simple Types: Built-in Strings Numbers Booleans string anyuri QName decimal integer float double long int short byte positiveinteger nonnegativeinteger... unsignedlong unsignedint... boolean 132

133 Simple Types: Built-in Dates and Times Time Intervals Binaries Null - datetime time date gyearmonth gmonthday gyear gmonth gday datetimestamp duration yearmonthduration daytimeduration hexbinary base64binary 133

134 Dates T10:15:00Z 01:15:00-08:00 134

135 Durations P1Y2MT3H 135

136 User-defined types 136

137 User-defined types Restriction 137

138 User-defined types Restriction Union Not atomic 138

139 User-defined types Restriction Union Not atomic List Not atomic 139

140 Restriction <?xml&version="1.0"&encoding="utf98"?>& <xs:schema& &&xmlns:xs=" &&<xs:simpletype&name="myfixedlengthstring">& &&&&<xs:restriction&base="xs:string">& &&&&&&<xs:length&value="3"/>& &&&&</xs:restriction>& &&</xs:simpletype>& &&<xs:element&name="foo"&type="myfixedlengthstring"/>& </xs:schema>& schema.xsd <?xml&version="1.0"&encoding="utf98"?>& <foo& &&xmlns:xsi=" &&xsi:nonamespaceschemalocation="schema.xsd">zrh</foo>& & & file.xml 140

141 Restriction <xs:simpletype,name="myfixedlengthstring">,,,<xs:restriction,base="xs:string">,,,,,<xs:length,value="3"/>,,,</xs:restriction>, </xs:simpletype>,, <foo>zrh</foo>,,, 141

142 List <xs:simpletype,name="mylist">,,,<xs:list,itemtype="xs:string"/>, </xs:simpletype>,, <foo>foo,bar,foobar</foo>,, 142

143 Union <xs:simpletype,name="myunion">,,,<xs:union,membertypes="xs:integer,xs:boolean"/>, </xs:simpletype>,, <foo>true</foo>,,,, 143

144 Complex Types 144

145 Complex Types Empty <foo/> 145

146 Complex Types Empty <foo/> Simple Content <foo>text</foo> 146

147 Complex Types Empty <foo/> Simple Content Complex Content <foo>text</foo> <foo> <a/> <b/> </foo> 147

148 Complex Types Empty <foo/> Simple Content Complex Content Mixed Content <foo>text</foo> <foo> <a/> <b/> </foo> <foo> Text<a/>Text<b/> </foo> 148

149 Complex content <xs:complextype-name="complexcontent">- --<xs:sequence>- ----<xs:element-name="twotofour"-type="xs:string"-minoccurs="2"-maxoccurs="4"/>- ----<xs:element-name="zeroorone"-type="xs:boolean"-minoccurs="0"-maxoccurs="1"/>- --</xs:sequence>- </xs:complextype>- - <foo>- --<twotofour>foobar</twotofour>- --<twotofour>foobar</twotofour>- --<twotofour>foobar</twotofour>- --<zeroorone>true</zeroorone>- </foo>

150 Complex content <xs:complextype-name="complexcontent">- --<xs:sequence>- ----<xs:element-name="twotofour"-type="xs:string"-minoccurs="2"-maxoccurs="4"/>- ----<xs:element-name="zeroorone"-type="xs:boolean"-minoccurs="0"-maxoccurs="1"/>- --</xs:sequence>- </xs:complextype>- - <foo>- --<twotofour>foobar</twotofour>- --<twotofour>foobar</twotofour>- --<twotofour>foobar</twotofour>- --<zeroorone>true</zeroorone>- </foo>

151 Empty content <xs:complextype-name="emptytype">- --<xs:sequence/>- </xs:complextype>- - <foo/>

152 Simple content <xs:complextype-name="datecountry">- --<xs:simplecontent>- ----<xs:extension-base="xs:date"> <xs:attribute-name="country"-type="xs:string"/>- ----</xs:extension>- --</xs:simplecontent>- </xs:complextype>- - <foo-country="switzerland">2014d12d02</foo>

153 Mixed content <xs:complextype-name="mixedcontent"-mixed="true">- --<xs:sequence>- ----<xs:element-name="b"-type="xs:string"-minoccurs="0"-maxoccurs="unbounded"/>- --</xs:sequence>- </xs:complextype>- - <foo>some-text-and-some-<b>bold</b>-text.</foo>

154 Simple type on attributes <xs:complextype-name="withattribute">- --<xs:sequence/>- --<xs:attribute-name="country" type="xs:string" default="switzerland"/>- </xs:complextype>- - <foo-country="switzerland"/>

155 Named Types <xs:complextype-name="empty">---- <xs:sequence/>- </xs:complextype>- - <xs:element-name="c"-type="empty">- </xs:element>

156 Anonymous Types <xs:element*name="c">* **<xs:complextype>* ****<xs:sequence/>* **</xs:complextype>* </xs:element>* * 156

157 No namespaces <?xml&version="1.0"&encoding="utf98"?>& <xs:schema& &&xmlns:xs=" &&<xs:element&name="foo"&type="xs:string"/>& </xs:schema>& & & <?xml&version="1.0"&encoding="utf98"?>& <foo& &&xmlns:xsi=" &&xsi:nonamespaceschemalocation="schema.xsd">& &&This&is&text.& </foo>& & 157

158 With namespaces <?xml&version="1.0"&encoding="utf98"?>& <xs:schema& &&xmlns:xs=" &&<xs:element&name="foo"&type="xs:string"/>& </xs:schema>& & <?xml&version="1.0"&encoding="utf98"?>& <big:foo& &&xmlns:xsi=" &&xsi:schemalocation="! &&xmlns:big=" &&This&is&text.& </big:foo>& & & 158

159 Keys <xs:schema* ****xmlns:xs=" **<xs:element*name="root">* ****<xs:complextype>* ******..* ****</xs:complextype>* ****<xs:key*name="foodid">* ******<xs:selector*xpath="foo"/>* ****</xs:key>* **</xs:element>* </xs:schema>* * <?xml*version="1.0"*encoding="utfd8"?>* <root* **xmlns:xsi=" **xsi:nonamespaceschemalocation="schema.xsd">* **<foo*id="foo"/>* **<foo*id="bar"/>* **<foo*id="foobar"/>* </root>* * 159

160 Keys <xs:schema* ****xmlns:xs=" **<xs:element*name="root">* ****<xs:complextype>* ******..* ****</xs:complextype>* ****<xs:key*name="foodid">* What must be unique ******<xs:selector*xpath="foo"/>* ****</xs:key>* **</xs:element>* </xs:schema>* * <?xml*version="1.0"*encoding="utfd8"?>* <root* **xmlns:xsi=" **xsi:nonamespaceschemalocation="schema.xsd">* **<foo*id="foo"/>* **<foo*id="bar"/>* **<foo*id="foobar"/>* </root>* * 160

161 Keys <xs:schema* ****xmlns:xs=" **<xs:element*name="root">* ****<xs:complextype>* ******..* ****</xs:complextype>* ****<xs:key*name="foodid">* ******<xs:selector*xpath="foo"/>* ****</xs:key>* **</xs:element>* </xs:schema>* * <?xml*version="1.0"*encoding="utfd8"?>* <root* What must be unique What makes it unique **xmlns:xsi=" **xsi:nonamespaceschemalocation="schema.xsd">* **<foo*id="foo"/>* **<foo*id="bar"/>* **<foo*id="foobar"/>* </root>* * 161

162 Bonus material: The Schema of Schemas!!<xs:schema!!!!!xmlns:xs=" 162

163 Avro 163

164 Avro is language neutral 164

165 Interoperability 165

166 Interoperability Avro DataFile 166

167 Interoperability Avro DataFile 167

168 Unlike Protocol Buffer and Thrift... Schema (Language independent) 168

169 Unlike Protocol Buffer and Thrift... Schema (Language independent) Compilation 169

170 Unlike Protocol Buffer and Thrift... Schema (Language independent) Compilation Code 170

171 Unlike Protocol Buffer and Thrift... Schema (Language independent) Compilation More Code More Code Code More Code More Code 171

172 Unlike Protocol Buffer and Thrift... Schema (Language independent) Compilation More Code More Code Code More Code More Code Data 172

173 Avro processes schemas not known at compile time More Code More Code Code More Code More Code Schema Data + Known at runtime 173

174 Avro Schema Schema This is actually JSON 174

175 Avro Schema Schema Avro IDL But there is also another syntax for users that like C 175

176 Avro DataFile Avro DataFile An Avro DataFile is encoded in binary format 176

177 Avro DataFile DataFile JSONbased encoder But there is also another debuggingfriendly syntax 177

178 Avro types Atomic Types 178

179 Avro types Atomic Types vs. Structured Types 179

180 Avro atomic types Category Absence of value Boolean Number String Binary Types null boolean int long float double string enum bytes fixed (list of 8-bit unsigned bytes) 180

181 Avro atomic types { "type" : "null" } { "type" : "string" } { "type" : "boolean" } Primitive types 181

182 Avro atomic types { "type" : "null" } { "type" : "string" } { "type" : "boolean" } { } { } "type" : "enum", "name" : "cuttlery", "symbols" : [ "KNIFE", "FORK", "SPOON" ] "type" : "fixed", "name" : "MD5", "size" : 16 Primitive types Non-primitive types 182

183 Avro structured types Category Arrays Objects Types array map record 183

184 Avro arrays 1 foo 2 bar 3 foobar 4 foo 5 foobar 6 bar {a: 1, b: "foo"} 2 {a: 2, b: "foo"} 3 {a: 3, b: "bar"} 184

185 Avro arrays 1 foo 2 bar 3 foobar 4 foo 5 foobar 6 bar {a: 1, b: "foo"} 2 {a: 2, b: "foo"} 3 {a: 3, b: "bar"} { } "type" : "array", "items" : "string" 185

186 Avro arrays 1 foo 2 bar 3 foobar 4 foo 5 foobar 6 bar {a: 1, b: "foo"} 2 {a: 2, b: "foo"} 3 {a: 3, b: "bar"} { } "type" : "array", "items" : "string" All items in the array must have the same type. 186

187 Avro maps keys foo bar foobar values true false true 187

188 Avro maps keys foo bar foobar values true false true Must be strings 188

189 Avro maps keys foo bar foobar values true false true Must be same type 189

190 Avro maps keys foo bar foobar values true false true { } "type" : "map", "name" : "flags", "values" : "boolean" Must be same type 190

191 Avro records fields values name ETH canton ZH students 20,

192 Avro records { } fields name canton values ETH ZH students 20,000 "type" : "map", "name" : "university", "fields" : [ { "name" : "name", "type" : "string" }, { "name" : "canton", "type" : "cantonal-code" }, { "name" : "students", "type" : "long" } ] 192

193 Mappings Generic 193

194 Mappings Generic Specific 194

195 Mappings Generic Specific Reflexive 195

196 Avro DataFile Metadata (including schema) 196

197 Avro DataFile Metadata (including schema) Block 197

198 Avro DataFile Metadata (including schema) Block Block 198

199 Avro DataFile Metadata (including schema) Block Block Block 199

200 Avro DataFile Metadata (including schema) Block Block Block Block 200

201 Avro DataFile Metadata (including schema) Block Block Block Block Block 201

202 Avro DataFile Metadata (including schema) Specifies sync marker Block sync marker Block Block Block Block 202

203 Avro DataFiles are splittable 203

204 Avro DataFiles are splittable 204

205 Avro DataFiles are splittable 205

206 Schema resolution Old Schema 206

207 Schema resolution Old Schema Avro DataFile 207

208 Schema resolution Old Schema New Schema Avro DataFile 208

209 Schema resolution Old Schema New Schema Avro DataFile New field 209

210 Schema resolution Old Schema New Schema Avro DataFile New field Default value is taken 210

211 Schema resolution Old Schema New Schema Avro DataFile Removed field 211

212 Schema resolution Old Schema New Schema Avro DataFile Removed field Ignored (projection) 212

213 Schema resolution New Schema Old Schema Avro DataFile New field 213

214 Schema resolution New Schema Old Schema Avro DataFile New field Ignored (projection) 214

215 Schema resolution New Schema Old Schema Avro DataFile Removed field 215

216 Schema resolution New Schema Old Schema Avro DataFile Removed field Default value (error if none) 216

217 Parquet 217

218 YET ANOTHER format?

219 YET ANOTHER format? Don't we already have enough?

220 But this time, it's different

221 Row storage 221

222 Columnar storage 222

223 But... But I thought we were talking about heterogeneous collections of trees?

224 Mapping trees to tables... If our heterogeneous collection is really homogeneous and our trees are flat, this is easy 224

225 Mapping trees to tables... Otherwise, there are ways. Edge shredding Ad-hoc mappings Tree-encoding Dremel 225

226 Parquet atomic types Category Boolean Number String Binary Date Types boolean int32 int64 int96 double DECIMAL(precision, scale) UTF8 ENUM binary fixed_len_byte_array DATE 226

227 Parquet structured types Category Arrays Objects Types LIST MAP 227

228 Parquet schema definition Protocol buffer schemas Avro schemas Thrift schemas 228