Big Data Fall Data Models

Transcription

1 Ghislain Fourny Big Data Fall Data Models pinkyone / 123RF Stock Photo

2 CSV (Comma separated values) This is syntax ID,Last name,first name,theory, 1,Einstein,Albert,"General, Special Relativity" 2,Gödel,Kurt,"""Incompleteness"" Theorem" 2

3 CSV (Comma separated values) This is syntax ID,Last name,first name,theory, 1,Einstein,Albert,"General, Special Relativity" 2,Gödel,Kurt,"""Incompleteness"" Theorem" ID Last name First name Theory 1 Einstein Albert General, Special Relativity 2 Gödel Kurt "Incompleteness" Theorem This is a data model 3

4 Syntax vs. Data Models Physical view Syntax ID,Last name,first name,theory, 1,Einstein,Albert,"General, Special Relativity" 2,Gödel,Kurt,"""Incompleteness"" Theorem" 4

5 Syntax vs. Data Models Logical view Data Model ID Last name First name Theory 1 Einstein Albert General, Special Relativity 2 Gödel Kurt "Incompleteness" Theorem Physical view Syntax ID,Last name,first name,theory, 1,Einstein,Albert,"General, Special Relativity" 2,Gödel,Kurt,"""Incompleteness"" Theorem" 5

6 Syntax vs. Data Models Physical view Syntax <a> <d e="f"/> <c>this is <b>text</b>.</c> </a> 6

7 Syntax vs. Data Models a Logical view Data Model e = f d This is c b. text Physical view Syntax <a> <d e="f"/> <c>this is <b>text</b>.</c> </a> 7

8 Syntax vs. Data Models Physical view Syntax { "foo" : [ 1, 2, 3 ], "bar" : { "a" : 1, "bar" : 2 } } 8

9 Syntax vs. Data Models Object foo bar Logical view Data Model Array a Object bar Physical view Syntax { "foo" : [ 1, 2, 3 ], "bar" : { "a" : 1, "bar" : 2 } } 9

10 Edge vs. Node labeling foo bar Labels are on the edges 10

11 Edge vs. Node labeling Object foobar foobar Object e = f foo bar foo bar Labels are on the edges Labels are on the nodes 11

12 XML Data models Information Set (Infoset) 12

13 XML Data models Information Set (Infoset) Post Schema-Validation Infoset (PSVI) 13

14 XML Data models Information Set (Infoset) Post Schema-Validation Infoset (PSVI) XQuery and XPath Data Model (XDM) 14

15 JSON Data Models "original" (implicit) JSON Data Model 15

16 JSON Data Models "original" (implicit) JSON Data Model JSON Schema Data Model 16

17 JSON Data Models "original" (implicit) JSON Data Model JSON Schema Data Model JSONiq Data Model (JDM) y/html/section-jsoniq-data-model.html 17

18 HTML/XML Data model Document Object Model (DOM) 18

19 grigory_bruev / 123RF Stock Photo XML Information Set 19

20 Information Set <?xml version="1.0" encoding="utf-8"?> <dc:metadata xmlns:dc=" <title xml:lang="en" year="2008" >Systems Group</title> <publisher>eth Zurich</publisher> </dc:metadata> 20

21 Information Set 21

22 The 11 XML Information Items Document Element Attribute Processing Instruction Character Comment Namespace Unexpanded Entity Reference DTD Unparsed Entity Notation 22 22

23 The 11 XML Information Items Document Element Attribute Processing Instruction Character Comment Namespace Unexpanded Entity Reference DTD Unparsed Entity Notation 23 23

24 Document Information Items Document Information Item doc [children] Element Information Item metadata [document element] Element Information Item metadata [notations] <empty> [unparsed entities] <empty> [base URI ] file:///users/bigdata/documents/info.xml [character encoding scheme] UTF-8 [standalone] <no value> [version]

25 Element Information Items Element Information Item dc:metadata [namespace name] [local name] metadata [prefix] dc [children] Element Information Items title, publisher [attributes] <empty> [namespace attributes] Attribute Information Item xmlns:dc [in-scope namespaces] Namespace Information Items dc->systems, xml->ns [base URI] file:///users/bigdata/documents/info.xml [parent] Document Information Item doc 25

26 Attribute Information Items Attribute Information Item year [namespace name] none [local name] year [prefix] none [normalized value] 2008 [specified] true [attribute type] <no value> [references] unknown [owner element] Element Information Item dc:title 26

27 Namespace Information Items Namespace Information Item dc->systems [prefix] dc [namespace name] dc->systems Namespace Information Item xml->ns xml->ns [prefix] xml [namespace name] 27

28 XML Infoset - the tree doc xmlns:dc dc:metadata xml->ns dc->systems xml->ns dc->systems dc:title dc:publisher dc->systems xml->ns en xml:lang Systems Group year 2008 ETH Zurich 28

29 Types 29

30 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: 30

31 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: - Distinction between atomic types and structured types 31

32 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: - Distinction between atomic types and structured types - Same categories of atomic types 32

33 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: - Distinction between atomic types and structured types - Same categories of atomic types - Lists and maps as structured types 33

34 Type Systems Almost all type systems (Java, SQL, PSVI, JDM, Protocol buffers, Avro, Parquet, and so on) share the following properties: - Distinction between atomic types and structured types - Same categories of atomic types - Lists and maps as structured types - Sequence type cardinalities 34

35 Types (General) Atomic Types vs. Structured Types 35

36 Atomic Types 36

37 Atomic Types Strings 37

38 Strings (Character sequences with monoid structure) "foo" "Zurich" "Ilsebill salzte nach." f o o Z u r i c h I l s e b i l l s a l z t e n a c h. 38

39 Atomic Types Strings Numbers 39

40 Interval-based integer types (exist as signed and unsigned) 8-bit (Java's byte) 16-bit (Java's short) 32-bit (Java's int) 64-bit (Java's long) 40

41 Arbitrary precision decimals (and integers) Any precision and scale

42 Float and Double IEEE 754 standard 32 bits 64 bits ca. 7 digits ca. 15 digits to to single precision double precision 42

43 Atomic Types Strings Numbers Booleans 43

44 Booleans TRUE t true y yes on 1 FALSE f false n no off 0 44

45 Atomic Types Strings Numbers Booleans Dates and Times 45

46 Dates and times Date Time Timestamp Duration 46

47 Dates (Gregorian calendar) Year + Month + Day 2017 August 1st (AD) 47

48 Times Hours + Minutes + Seconds 10 : 31 :

49 Timestamps Year + Month + Day + Hours + Minutes + Seconds 2017 August 1 st 10 : 31 : (AD) 49

50 Atomic Types Strings Numbers Booleans Dates and Times Time Intervals 50

51 Duration kinds Year Month Day Hour Minute Second Example: 2 years and 4 months 51

52 Duration kinds Year Month Day Hour Minute Second Example: 2 years and 4 months Example: 3 hours and 14 minutes 52

53 Atomic Types Strings Numbers Booleans Dates and Times Time Intervals Binaries 53

54 Atomic Types Strings Numbers Booleans Dates and Times Time Intervals Binaries Null 54

55 Lexical space vs. value space Value space Lexical space 55

56 Lexical space vs. value space "1" "01"... Value space Lexical space 56

57 Lexical space vs. value space "4" "04" "100b"... Value space Lexical space 57

58 Subtypes Supertype's value space Subtype's value space 58

59 Structured Types Data Structure Associative Arrays (a.k.a. maps) Ordered Lists Examples JSON Object, Protobuf Message, Set of XML Attributes JSON Array, XML Element, Protobuf repeated field 59

60 Structured Types Data Structure Associative Arrays (a.k.a. maps) Ordered Lists Examples JSON Object, Set of XML Attributes Protobuf Message, JSON Array, XML Element, Protobuf repeated field 60

61 Cardinality How many? Common sign Common adjective 61

62 Cardinality One How many? Common sign Common adjective required 62

63 Cardinality One How many? Common sign Common adjective required Zero or more * repeated 63

64 Cardinality One How many? Common sign Common adjective required Zero or more * repeated Zero or one? optional 64

65 Cardinality One How many? Common sign Common adjective required Zero or more * repeated Zero or one? optional One or more + 65

66 wklzzz / 123RF Stock Photo JSON Data Model 66

67 JSON Values Strings Objects Numbers Booleans Arrays Null Atomic values Structured values 67

68 JSON Values Strings Numbers Booleans Null Objects String-to-Value map Arrays List of values Atomic values Structured values 68

69 JSON Values Strings Numbers Booleans Null Objects String-to-Value map Recursion Arrays List of values Atomic values Structured values 69

70 Tree-based visual model { } "foo" : true, "bar" : [ { "foobar" : "foo" }, null ] 70

71 Tree-based visual model object { } "foo" : true, "bar" : [ { "foobar" : "foo" }, null ] 71

72 Tree-based visual model object { } "foo" : true, "bar" : [ { "foobar" : "foo" }, null ] true foo bar array 72

73 Tree-based visual model object { } "foo" : true, "bar" : [ { "foobar" : "foo" }, null ] true foo object bar array null 73

74 Tree-based visual model object { } "foo" : true, "bar" : [ { "foobar" : "foo" }, null ] true foo object bar array null foobar foo 74

75 wklzzz / 123RF Stock Photo Protocol Buffers 75

76 Messages message Person { required string last_name = 1; repeated string first_name = 2; optional Title title = 3; optional Person boss = 4; } 76

77 Scalar types double, float int32, int64 and variants bool string bytes 77

78 Enums enum Title { MR = 1; MS = 2; MRS = 3; } 78

79 In C++ person.boss().first_name() 79

80 This is a schema! message Person { required string last_name = 1; repeated string first_name = 2; optional Title title = 3; optional Person boss = 4; } 80

81 JSON/XML vs. Protobufs No schema (heterogeneous) With schema (homogeneous) 81

82 Without schema... { "a" : 1, "b" : [ "foo", true, null, { "foo" : "bar" } ], "c" : { "d" : { "foo" : null }, "e" : [ 1, 2, [ 3, 4 ] ], "f" : 3.14 } } 82

83 With schema... { } "a" : 1, "b" : true, "c" : [ { "foo" : "bar1", "bar" : [ 1, 2 ] }, { "foo" : "bar2", "bar" : [ 3, 4, 5 ] }, { "foo" : "bar3" } ] 83

84 Validation Burak Cakmak / 123RF Stock Photo 84

85 Validation: The Pipeline Document 85

86 Validation: The Pipeline Document Well- Formedness 86

87 Validation: The Pipeline Document Well- Formedness Validation 87

88 On the oxygen Cheat Sheet Validity Well- Formedness 88

89 Validation vs. Annotation Validation 89

90 Validation vs. Annotation Validation Annotation 90

91 Validation 91

92 Validation 92

93 Validation 93

94 Validation 94

95 Validation 95

96 Post-Schema-Validation Infoset (with XML Schema) Infoset + Types Post-Schema-Validation Infoset (PSVI) 96

97 DTD Validation 97

98 Document Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a>& <a>& &&<d&e="f"/>& &&<c>this&is&<b>text</b>.</c>& </a>& & 98

99 Document Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a>& <a>& &&<d&e="f"/>& &&<c>this&is&<b>text</b>.</c>& </a>& & 99

100 Document Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& ]>& <a>& &&<d&e="f"/>& &&<c>this&is&<b>text</b>.</c>& </a>& & 100

101 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& ]>& <a>& &&<d&e="f"/>& &&<c>this&is&<b>text</b>.</c>& </a>& & 101

102 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& ]>& <a/>& & Empty Content 102

103 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(#PCDATA)>& ]>& <a>& &&This&is&text.& </a>& & Simple Content 103

104 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(foo,&bar)>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&bar&EMPTY>& ]>& <a>& &&<foo/>& &&<bar/>& </a>& & Complex Content 104

105 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(foo+,&bar*,&foobar?)>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&bar&EMPTY>& <!ELEMENT&foobar&EMPTY>& ]>& <a>& &&<foo/>& &&<foo/>& &&<foo/>& &&<foobar/>& </a>& Complex Content 105

106 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(&bar*& &(foo foobar)+)?>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&foobar&EMPTY>& ]>& <a>& &&<foo/>& &&<foobar/>& &&<foo/>& &&<foobar/>& </a>& & Complex Content 106

107 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&(#PCDATA& &foo)*>& <!ELEMENT&foo&EMPTY>& ]>& <a>& &&<foo/>lorum<foo/>ipsum<foo/>& </a>& & Mixed Content 107

108 Element Type Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&ANY>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&bar&EMPTY>& ]>& <a>& &&<foo/>& &&Lorem& &&<bar/>& &&Ipsum& &&<bar/>& &&<bar/>& </a>& & Mixed Content 108

109 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&#REQUIRED>& ]>& <a&foo="this&is&a&"value""></a>& & 109

110 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&#IMPLIED>& ]>& <a&foo="this&is&a&"value""></a>& & 110

111 Attribute-List Declaration <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE a [ <!ELEMENT a EMPTY> <!ATTLIST a foo CDATA #IMPLIED> ]> <a></a> 111

112 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&"bar">& ]>& <a&foo="this&is&a&"value""></a>& & 112

113 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&"bar">& ]>& <a></a>& & 113

114 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&CDATA&#FIXED&"bar">& ]>& <a&foo="bar"></a>& & 114

115 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&NMTOKEN&#REQUIRED>& ]>& <a&foo="123atoken456"></a>& & 115

116 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&a&[& <!ELEMENT&a&EMPTY>& <!ATTLIST&a&foo&NMTOKENS&#REQUIRED>& ]>& <a&foo="123atoken&456"></a>& & 116

117 Attribute-List Declaration <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&root&[& <!ELEMENT&root&(foo+,&bar,&barlist)>& <!ELEMENT&foo&EMPTY>& <!ELEMENT&bar&EMPTY>& <!ELEMENT&barlist&EMPTY>& <!ATTLIST&foo&myid&ID&#REQUIRED>& <!ATTLIST&bar&ref&IDREF&#REQUIRED>& <!ATTLIST&barlist&ref&IDREFS&#REQUIRED>& ]>& <root>& &&<foo&myid="foobar"/>& &&<foo&myid="foobar2"/>& &&<bar&ref="foobar"/>& &&<barlist&ref="foobar&foobar2"/>& </root>& & 117

118 DTD Example: External Subset <?xml&version="1.0"&encoding="utf98"?>& <!DOCTYPE&root&SYSTEM&"schema.dtd">& <a&foo="bar"/>& 118

119 Warning: DTDs and Namespaces <!ELEMENT eth (date, president, Rektor)> <!ATTLIST eth xmlns CDATA #FIXED " xmlns:xmldb CDATA #FIXED " date CDATA #IMPLIED xmldb:date CDATA #IMPLIED> <!ELEMENT date (#PCDATA)> <!ELEMENT president (#PCDATA)> <!ATTLIST president number CDATA #IMPLIED> <!ELEMENT Rektor (#PCDATA)> 119

120 XML Schema 120

121 Empty Schema <?xml&version="1.0"&encoding="utf98"?>& <xs:schema& &&xmlns:xs=" </xs:schema>& & 121

122 Simple Scenario <?xml version="1.0" encoding="utf-8"?> <xs:schema xmlns:xs=" <xs:element name="foo" type="xs:string"/> </xs:schema> <?xml version="1.0" encoding="utf-8"?> <foo xmlns:xsi=" xsi:nonamespaceschemalocation="schema.xsd"> This is text. </foo> schema.xsd Schema Instance file.xml 122

123 Simple Scenario <?xml version="1.0" encoding="utf-8"?> <xs:schema xmlns:xs=" <xs:element name="foo" type="xs:string"/> </xs:schema> <?xml version="1.0" encoding="utf-8"?> <foo xmlns:xsi=" xsi:nonamespaceschemalocation="schema.xsd"> This is text. </foo> schema.xsd Schema Instance file.xml 123

124 Simple Scenario <?xml version="1.0" encoding="utf-8"?> <xs:schema xmlns:xs=" <xs:element name="foo" type="xs:string"/> </xs:schema> <?xml version="1.0" encoding="utf-8"?> <foo xmlns:xsi=" xsi:nonamespaceschemalocation="schema.xsd"> This is text. </foo> Schema Instance 124

125 Simple Scenario <?xml version="1.0" encoding="utf-8"?> <xs:schema xmlns:xs=" <xs:element name="foo" type="xs:integer"/> </xs:schema> <?xml version="1.0" encoding="utf-8"?> <foo xmlns:xsi=" xsi:nonamespaceschemalocation="schema.xsd"> </foo> Schema Instance 125

126 Simple Types: Built-in Strings Numbers Booleans string anyuri QName decimal integer float double long int short byte positiveinteger nonnegativeinteger... unsignedlong unsignedint... boolean 126

127 Simple Types: Built-in Dates and Times Time Intervals Binaries Null - datetime time date gyearmonth gmonthday gyear gmonth gday datetimestamp duration yearmonthduration daytimeduration hexbinary base64binary 127

128 Dates T10:15:00Z 01:15:00-08:00 128

129 Durations P1Y2MT3H 129

130 User-defined types 130

131 User-defined types Restriction 131

132 User-defined types Restriction Union Not atomic 132

133 User-defined types Restriction Union Not atomic List Not atomic 133

134 Restriction <?xml&version="1.0"&encoding="utf98"?>& <xs:schema& &&xmlns:xs=" &&<xs:simpletype&name="myfixedlengthstring">& &&&&<xs:restriction&base="xs:string">& &&&&&&<xs:length&value="3"/>& &&&&</xs:restriction>& &&</xs:simpletype>& &&<xs:element&name="foo"&type="myfixedlengthstring"/>& </xs:schema>& <?xml&version="1.0"&encoding="utf98"?>& <foo& &&xmlns:xsi=" &&xsi:nonamespaceschemalocation="schema.xsd">zrh</foo>& & & Schema Instance 134

135 Restriction <xs:simpletype,name="myfixedlengthstring">,,,<xs:restriction,base="xs:string">,,,,,<xs:length,value="3"/>,,,</xs:restriction>, </xs:simpletype>,, <foo>zrh</foo>,,, Schema Instance 135

136 List <xs:simpletype,name="mylist">,,,<xs:list,itemtype="xs:string"/>, </xs:simpletype>,, <foo>foo,bar,foobar</foo>,, Schema Instance 136

137 Union <xs:simpletype,name="myunion">,,,<xs:union,membertypes="xs:integer,xs:boolean"/>, </xs:simpletype>,, Schema <foo>true</foo>,,,, Instance 137

138 Complex Types 138

139 Complex Types Empty <foo/> 139

140 Complex Types Empty <foo/> Simple Content <foo>text</foo> 140

141 Complex Types Empty <foo/> Simple Content Complex Content <foo>text</foo> <foo> <a/> <b/> </foo> 141

142 Complex Types Empty <foo/> Simple Content Complex Content Mixed Content <foo>text</foo> <foo> <a/> <b/> </foo> <foo> Text<a/>Text<b/> </foo> 142

143 Complex content <xs:complextype-name="complexcontent">- --<xs:sequence>- ----<xs:element-name="twotofour"-type="xs:string"-minoccurs="2"-maxoccurs="4"/>- ----<xs:element-name="zeroorone"-type="xs:boolean"-minoccurs="0"-maxoccurs="1"/>- --</xs:sequence>- </xs:complextype>- - <foo>- --<twotofour>foobar</twotofour>- --<twotofour>foobar</twotofour>- --<twotofour>foobar</twotofour>- --<zeroorone>true</zeroorone>- </foo> Schema Instance 143

144 Complex content <xs:complextype-name="complexcontent">- --<xs:sequence>- ----<xs:element-name="twotofour"-type="xs:string"-minoccurs="2"-maxoccurs="4"/>- ----<xs:element-name="zeroorone"-type="xs:boolean"-minoccurs="0"-maxoccurs="1"/>- --</xs:sequence>- </xs:complextype>- - <foo>- --<twotofour>foobar</twotofour>- --<twotofour>foobar</twotofour>- --<twotofour>foobar</twotofour>- --<zeroorone>true</zeroorone>- </foo> Schema Instance 144

145 Empty content <xs:complextype-name="emptytype">- --<xs:sequence/>- </xs:complextype>- - <foo/>- - - Schema Instance 145

146 Simple content <xs:complextype-name="datecountry">- --<xs:simplecontent>- ----<xs:extension-base="xs:date"> <xs:attribute-name="country"-type="xs:string"/>- ----</xs:extension>- --</xs:simplecontent>- </xs:complextype>- - <foo-country="switzerland">2014d12d02</foo>- - - Schema Instance 146

147 Mixed content <xs:complextype-name="mixedcontent"-mixed="true">- --<xs:sequence>- ----<xs:element-name="b"-type="xs:string"-minoccurs="0"-maxoccurs="unbounded"/>- --</xs:sequence>- </xs:complextype>- - <foo>some-text-and-some-<b>bold</b>-text.</foo> Schema Instance 147

148 Simple type on attributes <xs:complextype-name="withattribute">- --<xs:sequence/>- --<xs:attribute-name="country" type="xs:string" default="switzerland"/>- </xs:complextype>- - <foo-country="switzerland"/>- - - Schema Instance 148

149 Named Types <xs:complextype name="empty"> <xs:sequence/> </xs:complextype> <xs:element name="c" type="empty"> </xs:element> Schema Instance <c/> 149

150 Anonymous Types <xs:element name="c"> <xs:complextype> <xs:sequence/> </xs:complextype> </xs:element> Schema Instance <c/> 150

151 No namespaces <?xml&version="1.0"&encoding="utf98"?>& <xs:schema& &&xmlns:xs=" &&<xs:element&name="foo"&type="xs:string"/>& </xs:schema>& & & <?xml&version="1.0"&encoding="utf98"?>& <foo& &&xmlns:xsi=" &&xsi:nonamespaceschemalocation="schema.xsd">& &&This&is&text.& </foo>& & Schema Instance 151

152 With namespaces <?xml version="1.0" encoding="utf-8"?> <xs:schema xmlns:xs=" targetnamespace=" xmlns:big=" <xs:element name="foo" type="xs:string"/> </xs:schema> Schema <?xml version="1.0" encoding="utf-8"?> Instance <big:foo xmlns:xsi=" xsi:schemalocation=" schema.xsd" xmlns:big=" This is text. </big:foo> 152

153 Warning: named types with namespaces <xs:complextype name="empty"> </xs:complextype> implicitly in the target namespace always unprefixed <xs:element name="c" type="big:empty"> </xs:element> Schema Instance <big:c/> 153

154 Multiple namespaces <?xml version="1.0" encoding="utf-8"?> <big:foo xmlns:xsi=" xsi:schemalocation=" schema-big.xsd schema-medium.xsd schema-small.xsd " xmlns:big=" xmlns:medium=" xmlns:small=" <medium:page> <small:paragraph/> <small:paragraph/> <small:paragraph/> </medium:page> <medium:page> <small:paragraph/> <small:paragraph/> <small:paragraph/> </medium:page> </big:foo> 154

155 Small schema <?xml version="1.0" encoding="utf-8"?> <xs:schema xmlns:xs=" xmlns:small=" targetnamespace=" <xs:element name="paragraph" type="small:empty-string"/> <xs:simpletype name="empty-string> <xs:restriction base="xs:string"> <xs:length value="0"/> </xs:restriction> </xs:simpletype> </xs:schema> 155

156 Medium schema <?xml version="1.0" encoding="utf-8"?> <xs:schema xmlns:xs=" xmlns:small=" xmlns:medium=" targetnamespace=" <xs:import namespace=" schemalocation="schema-small.xsd"/> <xs:element name="page" type="medium:paragraphs"/> <xs:complextype name="paragraphs"> <xs:sequence> <xs:element ref="small:paragraph" maxoccurs="unbounded"> </xs:sequence> </xs:simpletype> </xs:schema> 156

157 Big schema <?xml version="1.0" encoding="utf-8"?> <xs:schema xmlns:xs=" xmlns:big=" xmlns:medium=" targetnamespace=" <xs:import namespace=" schemalocation="schema-medium.xsd"/> <xs:element name="foo" type="big:pages"/> <xs:complextype name="pages"> <xs:sequence> <xs:element ref="medium:page" maxoccurs="unbounded"> </xs:sequence> </xs:simpletype> </xs:schema> 157

158 Keys <xs:schema* ****xmlns:xs=" **<xs:element*name="root">* ****<xs:complextype>* ******..* ****</xs:complextype>* ****<xs:key*name="foodid">* ******<xs:selector*xpath="foo"/>* ****</xs:key>* **</xs:element>* </xs:schema>* * <?xml*version="1.0"*encoding="utfd8"?>* <root* **xmlns:xsi=" **xsi:nonamespaceschemalocation="schema.xsd">* **<foo*id="foo"/>* **<foo*id="bar"/>* **<foo*id="foobar"/>* </root>* * 158

159 Keys <xs:schema* ****xmlns:xs=" **<xs:element*name="root">* ****<xs:complextype>* ******..* ****</xs:complextype>* ****<xs:key*name="foodid">* What must be unique ******<xs:selector*xpath="foo"/>* ****</xs:key>* **</xs:element>* </xs:schema>* * <?xml*version="1.0"*encoding="utfd8"?>* <root* **xmlns:xsi=" **xsi:nonamespaceschemalocation="schema.xsd">* **<foo*id="foo"/>* **<foo*id="bar"/>* **<foo*id="foobar"/>* </root>* * 159

160 Keys <xs:schema* ****xmlns:xs=" **<xs:element*name="root">* ****<xs:complextype>* ******..* ****</xs:complextype>* ****<xs:key*name="foodid">* ******<xs:selector*xpath="foo"/>* ****</xs:key>* **</xs:element>* </xs:schema>* * <?xml*version="1.0"*encoding="utfd8"?>* <root* What must be unique What makes it unique **xmlns:xsi=" **xsi:nonamespaceschemalocation="schema.xsd">* **<foo*id="foo"/>* **<foo*id="bar"/>* **<foo*id="foobar"/>* </root>* * 160

161 Bonus material: The Schema of Schemas!!<xs:schema!!!!!xmlns:xs=" 161

162 Avro 162

163 Avro is language neutral 163

164 Interoperability Avro DataFile 164

165 Avro atomic types Category Absence of value Boolean Number String Binary Types null boolean int long float double string enum bytes fixed (list of 8-bit unsigned bytes) 165

166 Avro structured types Category Arrays Objects Types array map record 166

167 Parquet 167

168 Row storage 168

169 Columnar storage 169

170 Columnar storage Row group Row group 170

171 Columnar storage 171

172 Parquet atomic types Category Boolean Number String Binary Date Types boolean int32 int64 int96 double DECIMAL(precision, scale) UTF8 ENUM binary fixed_len_byte_array DATE 172

173 Parquet structured types Category Arrays Objects Types LIST MAP 173

174 Weerapat Kiatdumrong / 123RF Stock Photo XPath and XQuery Data Model (XDM) JSONiq Data Model (JDM) 174

175 XDM/JDM: Sequences of Items (,,,,, ) 175

176 XDM/JDM: Sequence of one item 176

177 XDM/JDM: Sequence of one item = ( ) 177

178 XDM/JDM: Sequences are flat ((, ), ) 178

179 XDM/JDM: Sequences are flat ((, ), )=(,, ) 179

180 XDM: Items Atomic Node 180

181 XDM: Seven Kinds of XML Nodes Document node Element node Attribute node Text node Comment node Processing instruction node Namespace node 181

182 XDM: Seven Kinds of XML Nodes Infoset XDM 182

183 XDM vs. Infoset Infoset XDM xs:untyped 183

184 JSONiq Data Model (JDM): Objects and Arrays lorem ipsum dolor sit amet Atomic Objects Arrays 184

185 Advanced XDM/JDM: Function items Functions 185

186 XDM/JDM and Querying for let order by if + any else = then every while where return exit with Expression 186