Big Data Exercises. Fall 2016 Week 9 ETH Zurich

Transcription

1 Big Data Exercises Fall 2016 Week 9 ETH Zurich Introduction This exercise will cover XQuery. You will be using oxygen ( AN XML/JSON development IDE, in order to do this exercise. You should have received a license key for it by mail. Before starting, make sure oxygen is installed and working on your computer. Download the XML/XSD files book_catalogue.xml ( export=download&id=0b_c6n15v_sc7mmlscmfdewlkdws), airport-flight-passenger.xml ( airport-flightpassenger.xsd ( in order to complete the exercise. 1. XQuery Basics FLWOR For - Iterates over a sequence of nodes. Let - Binds a sequence to a variable. Where - Filters the nodes / specifies criteria for the result. Order by - Specifies sort order. Return - Specifies what need to be returned. Questions Run the following queries on book_catalogue.xml using oxygen: 1. Find book titles ordered lexicographically where book price is greater than Find books written in English which belong to category "CHILDREN" ordered by the year of publication. 3. Find books ordered by categories which are published after 2010 and costs less than Find all authors who have written books in the "BUSINESS" category before 2014 and sort their names lexicographically.

2 Solutions 1. for $x in doc("book_catalogue.xml")//book where $x/price>15 order by $x/title return $x/title Instead of using the where clause, we can also select books by their price in the xpath expression: for $x in doc("book_catalogue.xml")//book[price>15] order by $x/title return $x/title for $x in doc("book_catalogue.xml")//book where and order by $x/year return $x for $x in doc("book_catalogue.xml")//book where $x/year>2010 and $x/price<10 order by return $x for $x in doc("book_catalogue.xml")//book where $x/year<2014 and order by $x/author return $x/author Above query returns authors that have written several books multiple times. We can avoid this by first creating a list of xs:string of the names of the authors we are interested in and then select the corresponding author nodes, taking the first node for each name: let $doc := doc("book_catalogue.xml") let $books := $doc//book[year < 2014 let $authors := $doc//author[data() = $books//author] for $a in distinct-values($authors) order by $a return ($authors[data() = $a])[1]

3 Conditional Expressions If-Then-Else expressions are allowed in XQuery. Question Run the following query on book_catalogue.xml using oxygen: 1. List all the book titles. If the book belongs to category CHILDREN, put the title within <child></child> tag. Else put the title within <adult></adult> tag. Solution for $x in doc("book_catalogue.xml")//book return if then <child>{data($x/title)}</child> else <adult>{data($x/title)}</adult> 2. Namespaces in XQuery Consider the following document: <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE eth SYSTEM "eth.dtd"> <eth xmlns=" xmlns:db=" date=" " db:date=" "> <date> </date> <president number="1">empty</president> <Rektor>Name 2</Rektor> </eth>

4 Question Correct the following XQuery expressions so that they refer to the correct namespace, in order to determine the value of the date sub-element and of the two date attributes: 1. /eth/date 2. Hint: you can bind a prefix pre to a URI ( with the following prolog declaration: declare namespace pre = " ( You can then use the prefix in path expressions. Solution To refer to the child element: declare namespace def = " /def:eth/def:date To refer to the date attribute in no namespace: declare namespace def = " /def:eth/@date To refer to the date attribute which in the dbisns namespace: declare namespace def = " declare namespace dbisns =" /def:eth/@dbisns:date Note that the expressions that query the attributes seem "not to work" in oxygen. We get the message "Your query returned an empty sequence." This is a missleading message. Actually, the query returns an attribute -- outside of any element! While this can perfectly exists in XQuery, (our version of) oxygen seems unable to display it. You can do the following to get the value of the attribute: declare namespace def = " /def:eth/@date/data()

5 3. XQuery Comparison Operators General comparison operators: =,!=, <, <=, >, >= Value comparison operators: eq, ne, lt, le, gt, ge The value comparison operators (eq, lt, etc.) are designed to compare single values (i.e. sequences of one value each). The "general comparison" operators (=, <, etc.) are designed to compare sequences of more than one values. "foo" eq ("foo","bar") throws an error. "foo" = ("foo","bar") returns true. With a general comparison operator, the expression will return true if any of the items on the left compare successfully with any of the items on the right. This is sometimes called "existential quantification." Questions 1. Find a variable binding for $x so that $x=1 and $x=2. Can one infer that, in XQuery, 1=2? 2. Find variable bindings for $x, $y and $z so that $x > $y and $y > $z, but $x > $z is not true. 3. Find a variable binding for $x so that neither $x eq $x nor $x = $x is true. Explain why. Solutions $x := (1,2) (1,2) = 1 (1,2) = 2 = is not transitive, so one can not infer 1=2. $x := 1, $y := (0, 3), $y := 2 1 > (0,3) and (0,3) > 2 but 1 > 2 is false. > is not transitive either. $x := () because () eq () returns () and () = () returns false

6 4. Atomization and Effective Boolean Value Atomization Atomization is the process of extracting the typed value of an item. This process is implied under certain circumstances. Some of the XQuery operators, such as arithmetic and comparison operators, depend on this process. For example, when you apply arithmetic operators directly to nodes, the typed value of a node is first retrieved by implicitly invoking the data function. This passes the atomic value as an operand to the arithmetic operator. Question Try to guess the outcome of the following XQuery expression: fn:sum( for $x in doc("book_catalogue.xml")//book where $x/@category eq "BUSINESS" return fn:sum($x/price) ) Solution Returns the total price of all books in the BUSINESS category. Note that the atomization in this case happens during the call to the inner fn:sum() and then the outer fn:sum() actually sums up the values. This makes the inner fn:sum() redundant. Instead, the return statement could return the sequence of elements without calling the inner fn:sum() in which case atomization as well as the summation will occur at the outer fn:sum(). Effective Boolean Value The effective Boolean value (EBV) of a sequence is computed implicitly during the processing of expressions that require Boolean values. Question Find the EBV (true/false) for the following descriptions of values: 1. An empty sequence 2. A sequence whose first item is a node 3. A single value of type xs:boolean 4. A single value of type xs:string 5. A single value of any numeric type

7 Solutions 1. false 2. true 3. false - if the xs:boolean value is false; true - if the xs:boolean value is true 4. false - if the length of the value is zero; true - if the length if the value is greater than zero 5. false - if the value is NaN or is numerically equal to zero; true - if the value is not numerically equal to zero 5. XQuery Functions XQuery is built on XPath expressions. XQuery 1.0 and XPath 2.0 share the same data model and support the same functions and operators. Here is the link ( to the documentation. A call to a function can appear where an expression may appear. Examples: In an element: for $x in doc("book_catalogue.xml")/bookstore/book where $x/@category="children" return <name>{upper-case($x/title)}</name> The above XQuery expression returns all books in the CHILDREN category in upper case. In the predicate of a path expression: for $x in doc("book_catalogue.xml")/bookstore/book[substring(title,1,4)='alg o'] return $x/title The above XQuery expression returns title elements for the books whose titles begin with "ÂAlgo". If you cannot find the XQuery function you need, you can write your own:

8 declare function prefix:function_name($parameter as datatype) as returndatatype {...function code here... }; Use the declare function keyword The name of the function must be prefixed The data type of the parameters are mostly the same as the data types defined in XML Schema The body of the function must be surrounded by curly braces Question Consider that the prices of books in the book_catalogue.xml file are given in USD. Find the price of the book titled "If Animals Kissed Good Night" in Euros (consider 1USD = 0.92EUR). Write a function that takes as input the price in USD and gives as output the price in EUR. Solution declare function local:convertprice($p as xs:decimal?) as xs:decimal? { $p * 0.92 }; for $x in doc("book_catalogue.xml")/bookstore/book[title='if Animals Kissed Good Night'] return <conv>{local:convertprice($x/price)}</conv> Note that use of xs:decimal? here is not necessary and instead we could use xs:decimal as well. But in cases where we need to handle empty sequences as input or output, xs:decimal? can be necessary. Note that book_catalogue.xml is not validated against a schema. In the next section you will run XQuery expressions on XML document validated against a schema.

9 6. More XQuery Expressions Consider the XML document airport-flight-passenger.xml validated against airport-flight-passenger.xsd distributed together with this exercise (download links in Introduction section). Write the XQuery expressions for solving the following problems. Your results should be well-formed XML. 1. Give the list of direct flights on which have "North Pole" (airport name) as the source airport. 2. Retrieve the busiest airport on (based on the number of departures and arrivals). 3. Identify all the flight destinations of Passenger "Santa Claus". 4. Consider the case of combined flights (one or more intermediate stops). As an example, flying from London to Zurich on might mean taking two separate flights: London-Amsterdam and Amsterdam-Zurich, both on the same date. Retrieve all flight possibilities from "North pole" to "South pole" on with one or two intermediate stops.

10 Solutions <Flights> { let $root := doc("airport-flight-passenger.xml") for $a in $root//airport[name/text() eq 'North Pole'] return $root//flight[(source eq $a/@airid) and (date/text() eq ' ')] } </Flights> 2. We first count the number of inbound and outbound flights for each airport. Then we order the airports based on these counts in descending order. Finally we find the maximum count and the airport(s) corresponding to this maximum count. <BusiestAirports> { let $doc := doc("airport-flight-passenger.xml") let $results := for $a in $doc//airport let $c as xs:integer := count($doc//flight[source eq $a/@airid or destination eq $a/@air Id]) order by $c descending, $a/name ascending return <result> {$a} <count>{$c}</count> </result> let $maxcount := max($results//count) return $results[xs:integer(count) eq $maxcount]/airport } </BusiestAirports> 3. We perform a join between passengers and reservations for Santa Claus. Then we look for the destination of each flight in the reservations, and retrieve the airport with the corresponding code. Finally, we make the list of airports unique and format the result.

11 <destinations> { let $doc := doc("airport-flight-passenger.xml") let $dest := for $p in $doc//passenger[name eq 'Santa Claus'], $r in $doc//reservation where $r/passref eq $p/passportnumber ion return eq $doc//flight[$r/flightref ]/name for $d in distinct-values($dest) return <destination>{$d}</destination> } </destinations> We can achieve the same effect by a sequence of let clauses: <destinations> { let $doc := doc("airport-flight-passenger.xml") let $passref := $doc//passenger[name eq 'Santa Claus']/passportnumber let $flightref := $doc//reservation[passref = $passref]/flightref let $destid := $doc//flight[@flightid = $flightref]/destination for $destination in $doc//airport[@airid = $destid]/name/text() return <destination>{$destination}</destination> } </destinations> 4. We perform joins between flights and append connections with one stop to connections with two stops.

12 let $doc := doc("airport-flight-passenger.xml") return <Possibilities> { for $f1 in $doc//flight[source eq 'NPL'], $f2 in $doc//flight[destination eq 'SPL'] where $f1/destination eq $f2/source and $f1/arrival lt $f2/departure and $f1/seats > 0 and $f2/seats > 0 return <OneIntermediateStop>{$f1}{$f2}</OneIntermediateStop>, for $f1 in $doc//flight[source eq 'NPL'], $f3 in $doc//flight[destination eq 'SPL'], $f2 in $doc//flight where $f1/destination eq $f2/source and $f2/destination eq $f3/source and $f1/arrival lt $f2/departure and $f2/arrival lt $f3/departure and $f1/seats > 0 and $f2/seats > 0 and $f3/seats > 0 return <TwoIntermediateStops>{$f1}{$f2}{$f3}</TwoIntermediateStops> } </Possibilities> 7. Analogy with SQL Have you noticed that the XQuery FLWOR expressions are similar to SQL SELECT statements? Of course, the for expression in XQuery identifies XML nodes while the FROM clause in SQL identifies tables from which rows are chosen. But this does not restrict us from trying to map the sub-expressions of XQuery FLWOR with the analogous syntax elements of SQL's select. Question Find the SQL syntax elements for the following XQuery expressions: for, where, order by, return. Solutions 1. FROM clause 2. WHERE clause 3. ORDER BY clause 4. SELECT clause