B4M36DS2, BE4M36DS2: Database Systems 2

Transcription

1 B4M36DS2, BE4M36DS2: Database Systems 2 h p:// Lecture 2 Data Formats Mar n Svoboda mar n.svoboda@fel.cvut.cz Charles University in Prague, Faculty of Mathema cs and Physics Czech Technical University in Prague, Faculty of Electrical Engineering

2 Lecture Outline Data formats XML Extensible Markup Language JSON JavaScript Object Nota on BSON Binary JSON RDF Resource Descrip on Framework CSV Comma-Separated Values Protocol Buffers B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

3 XML Extensible Markup Language

4 Introduc on XML = Extensible Markup Language Representa on and interchange of semi-structured data + a family of related technologies, languages, specifica ons, Derived from SGML, developed by W3C, started in 1996 Design goals Simplicity, generality and usability across the Internet File extension: *.xml, content type: Versions: 1.0 and 1.1 W3C recommenda on h p:// B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

5 Example B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

6 Document Structure Document Prolog: XML version + some other stuff Exactly one root element Contains other nested elements and/or other content <?xml version = " version "...?>... element Example B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

7 Constructs Element Marked using opening and closing tags or just an abbreviated tag in case of empty elements Each element can be associated with a set of a ributes < name attribute > element content < / name > < name attribute / > Examples B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

8 Constructs Types of element content Empty content Text content Element content Sequence of nested elements Mixed content Elements arbitrarily interleaved with text values text element element text B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

9 Constructs A ribute Name-value pair name = " value " Escaping sequences (predefined en es) Used within values of a ributes or text content of elements E.g.: for for for B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

10 XML Conclusion XML constructs Basic: element, a ribute, text Addi onal: comment, processing instruc on, Schema languages DTD, XSD (XML Schema), RELAX NG, Schematron Query languages XPath, XQuery, XSLT XML formats = par cular languages XSD, XSLT, XHTML, DocBook, epub, SVG, RSS, SOAP, B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

11 JSON JavaScript Object Nota on

12 Introduc on JSON = JavaScript Object Nota on Open standard for data interchange Design goals Simplicity: text-based, easy to read and write Universality: object and array data structures Supported by majority of modern programming languages Based conven ons of the C-family of languages (C, C++, C#, Java, JavaScript, Perl, Python, ) Derived from JavaScript (but language independent) Started in 2002 File extension: *.json Content type: h p:// B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

13 Example B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

14 Data Structure Object Unordered collec on of name-value pairs (proper es) Correspond to structures such as objects, records, structs, dic onaries, hash tables, keyed lists, associa ve arrays, Values can be of different types, names should be unique { string : value }, Examples B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

15 Data Structure Array Ordered collec on of values Correspond to structures such as arrays, vectors, lists, sequences, Values can be of different types, duplicate values are allowed [ value ], Examples B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

16 Data Structure Value Unicode string Enclosed with double quotes Backslash escaping sequences Example: Number Decimal integers or floats Examples:,, Nested object Nested array Boolean value:, Missing informa on: string number object array true false null B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

17 JSON Conclusion JSON constructs Collec ons: object, array Scalar values: string, number, boolean, null Schema languages JSON Schema Query languages JSONiq, JMESPath, JAQL, B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

18 BSON Binary JSON

19 Introduc on BSON = Binary JSON Binary-encoded serializa on of JSON documents Extends the set of basic data types of values offered by JSON (such as a string, ) with a few new specific ones Design characteris cs: lightweight, traversable, efficient Used by MongoDB Document NoSQL database for JSON documents Data storage and network transfer format File extension: *.bson h p://bsonspec.org/ B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

20 Example JSON BSON B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

21 Document Structure Document = serializa on of one JSON object or array JSON object is serialized directly JSON array is first transformed to a JSON object Property names derived from numbers of posi ons E.g.: Structure Document size (total number of bytes) Sequence of elements (encoded JSON proper es) Termina ng hexadecimal byte int32 element 00 B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

22 Document Structure Element = serializa on of one JSON property name int32 byte name 10 name 12 name 03 name 04 name double int32 int64 document document 08 name A name 09 name 11 name int64 int64 B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

23 Document Structure Element = serializa on of one JSON property Structure Type selector (string) (double), (32-bit integer), (64-bit integer) (object), (array) (boolean) (null) (date me), ( mestamp) Property name Unicode string terminated by byte 00 Property value B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

24 RDF Resource Descrip on Framework

25 Introduc on RDF = Resource Descrip on Framework Language for represen ng informa on about resources in the World Wide Web + a family of related technologies, languages, specifica ons, Used in the context of the Seman c Web, Linked Data, Developed by W3C Started in 1997 Versions: 1.0 and 1.1 W3C recommenda ons h ps:// Concepts and Abstract Syntax h ps:// Seman cs B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

26 Statements Resource Any real-world en ty Referents = resources iden fied by IRI E.g. physical things, documents, abstract concepts, Values = resources for literals E.g. numbers, strings, Statement about resources = one RDF triple Three components: subject, predicate, and object Examples B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

27 Statements Triple components Subject Describes a resource the given statement is about IRI or blank node iden fier Predicate Describes the property or characteris c of the subject IRI Object Describes the value of that property IRI or blank node iden fier or literal Although triples are inspired by natural languages, they have nothing to do with processing of natural languages B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

28 Example B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

29 Iden fiers and Literals IRI = Interna onalized Resource Iden fier Absolute (not rela ve) IRIs with op onal fragment iden fiers RFC 3987 Unicode characters Examples URLs are o en used in prac ce informa on about given resources are then intended to be published / retrieved via standard HTTP B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

30 Iden fiers and Literals Literals Plain values E.g.:, Typed values E.g.:, XML Schema simple data types are adopted and used Strings with language tags E.g.: Types and language tags cannot be mutually combined B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

31 Iden fiers and Literals Blank node iden fiers Blank nodes (anonymous resources) Allow to express statements about resources without explicitly naming (iden fying) them Blank node iden fiers only have local scope of validity E.g. within a given file, query expression, Par cular syntax depends on a serializa on format E.g.: B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

32 Data Model Directed labeled mul graph Ver ces One vertex for each IRI or literal value Edges One edge for each individual triple Edges are directed Property names (predicate IRIs) are used as edge labels B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

33 Example B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

34 Serializa on Available approaches N-Triples nota on h ps:// Turtle nota on (Terse RDF Triple Language) h ps:// RDF/XML nota on XML syntax for RDF h ps:// JSON-LD nota on JSON-based serializa on for Linked Data h ps:// B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

35 N-Triples Nota on RDF N-Triples nota on = A line-based syntax for an RDF graph Simple, line-based, plain text format File extension: *.rdf h ps:// Example Already presented Document Statements are terminated by dots, delimited by EOL triple 0D 0A triple B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

36 N-Triples Nota on Statement Individual triple components are delimited by spaces subject predicate object. Triple components: subject, predicate, object IRI reference blank node id IRI reference IRI reference blank node id literal B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

37 N-Triples Nota on IRI reference IRIs are enclosed in angle brackets < IRI > Blank node iden fier _ : label Literal Literals are enclosed in double quotes " value " IRI reference language tag B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

38 Turtle Nota on Turtle = Terse RDF Triple Language Compact text format, various abbrevia ons for common usage pa erns File extension: *. l Content type: h ps:// Example B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

39 Turtle Nota on Document Contains a sequence of triples and/or declara ons Prefix declara ons Prefixed names can then be used instead of full IRI references Groups of triples Individual groups are terminated by prefix : IRI reference. triples B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

40 Turtle Nota on Triples Triples sharing the same subject and object or at least the same subject can be grouped together object list for a shared subject and predicate predicate-object list for a shared subject Brackets can be used to define blank nodes subject predicate object., ; [ predicate object ], predicate object ;, ; B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

41 Turtle Nota on Triple components: subject, predicate, object IRI reference prefixed name blank node id IRI reference prefixed name IRI reference prefixed name blank node id literal [ predicate-object list ] IRI reference / prefixed name < IRI > prefix : local name B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

42 Turtle Nota on Literal Tradi onal literals + new abbreviated forms of numeric and boolean literals " value " true false integer / decimal / double IRI reference prefixed name language tag B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

43 Example Example revisited B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

44 RDF Conclusion RDF statements Subject, predicate, and object components Schema languages RDFS (RDF Schema) OWL (Web Ontology Language) Query languages SPARQL (SPARQL Protocol and RDF Query Language) B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

45 CSV Comma-Separated Values

46 Introduc on CSV = Comma-Separated Values Unfortunately not fully standardized Different field separators (commas, semicolons) Different escaping sequences No encoding informa on RFC 4180, RFC 7111 File extension: *.csv Content type: B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

47 Example B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

48 Document Structure Document Op onal header + list of records name, name 0D 0A record 0D 0A record Record Comma separated list of fields value, value B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

49 Protocol Buffers

50 Introduc on Protocol Buffers Extensible mechanism for serializing structured data Used in communica on protocols, data storage, Design goals Language-neutral, pla orm-neutral Small, fast, simple Developed (and widely used) by Google Started in 2008 internally and 2011 publicly Versions: proto2, proto3 File extension: *.proto h ps://developers.google.com/protocol-buffers/ Real-world usage: RiakKV, HBase B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

51 Introduc on Intended usage Schema crea on automa c source code genera on sending messages between applica ons Components Interface descrip on language Source code generator ( compiler) Supported languages Official: C++, C#, Java, Python, Ruby 3rd party: Perl, PHP, Scala, Binary serializa on format Compact, not self-describing B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

52 Example B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

53 Schema Structure Schema One schema may contain mul ple message descrip ons Other constructs are allowed as well, e.g. enumera ons... message enum Message Represents a small logical record of informa on Defines a set of uniquely numbered fields Nested messages or enumera ons are allowed too message name { field message enum } B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

54 Schema Structure Field Describes one data value repeated type name = tag ; Rule allowed number of value occurrences Default = 0 or 1 value = 0 or more values (i.e. an arbitrary number) The order of individual values is preserved B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

55 Schema Structure Field Type Atomic:,,,,,, Mappings to data types of par cular programming languages as well as default values are introduced Composed: messages, enumera ons, Name name of a given field Tag internal integer iden fier Used to iden fy individual fields of a message in a binary format Frequently used fields should be assigned lower tags Since lower number of bytes will then be needed B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

56 Schema Structure Enumera on Descrip on of a predefined list of values The first item is considered to be the default value and its value must be equal to 0 enum name { item = constant ; } A few other constructs are available too (e.g. maps) B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats

57

58 Lecture Conclusion Data formats Tree: XML, JSON Graph: RDF Rela onal: CSV Binary serializa ons BSON, Protocol Buffers B4M36DS2, BE4M36DS2: Database Systems 2 Lecture 2: Data Formats