CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

Transcription

1 Hands-on Session NoSQL DB Donato Summa THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION 1

2 Summary Elasticsearch How to get Elasticsearch up and running ES data organization ES configuration ES mapping Practice ES plugins 2

3 Elasticsearch (ES) Essentially it is a search engine You can use it on top of a relational database to create indices and speed up the SQL queries Or you can index data from your NoSQL data store to add search capabilities there Moreover you can also store your data directly in ES so you can use it as a NoSQL data store with powerful search capabilities 3

4 Elasticsearch (ES) ES is open-source and can be deployed on a distributed architecture It is built on top of Apache Lucene open-source search engine library that allows you to implement search functionality in your own Java application ES takes this Lucene function and extends it to provide storing indexing searching (faster and easier) BTW your application doesn t need to be written in Java to work with ES because you can send data over HTTP in JSON to index, search, and manage your ES cluster Within the Sandbox an R library is normally used 4

5 Elasticsearch cluster? In order to be elastic ES is clustered by default You call it a cluster even if you run it on a single server You can always add more servers to increase capacity or fault tolerance OR You can easily remove servers from the cluster to reduce costs if you have lower load 5

6 Indexing and inverted indexing An index is a data structure which you create along with your data and which is meant to allow faster searches You can add indices to fields in most databases, and there are several ways to do it Lucene does it with inverted indexing, which means it creates a data structure where it keeps a list of where each word belongs 6

7 Indexing and inverted indexing Imagine that you have a huge number of posts on your site searching through all of them for the word elections can take a long time, and you don t want your users to wait If you search for blog posts that have an elections tag, it s much faster to look at the index rather than looking at each word of each blog post, because you only have to look at the place where the tag is elections, and you ll get all the corresponding blog posts. 7

8 How to get Elasticsearch up and running Prerequisite : JRE 1.7 or above already installed Download the file elasticsearch tar.gz from : Unzipping it in your home you will get the folder : /home/your_name/elasticsearch Open a terminal and type the following commands : cd /home/your_name/elasticsearch /bin/elasticsearch Open a browser and type the following URL in the address bar: 8

9 How to get Elasticsearch up and running You should see something like this : { } "status" : 200, "name" : "Infinity", "cluster_name" : "elasticsearch", "version" : { "number" : "1.5.2", "build_hash" : "62ff9868b4c8a0c45860bebb259e ab1c", "build_timestamp" : " T09:21:06Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" The base configuration is done! 9

10 JSON (JavaScript Object Notation) JSON is a format for expressing data structures and exchange data. ( self-describing and easy to understand Easy to read and write for humans Easy to analyze and produce for computers Language independent (Java, C, R, Python, ) A JSON object typically contains keys and values, where values can be strings, numbers, true/false, null, another object, or an array 10

11 JSON 11

12 JSON Example: { } "employees": [ ] {"firstname":"john", "lastname":"doe"}, {"firstname":"anna", "lastname":"smith"}, {"firstname":"peter", "lastname":"jones"} 12

13 YAML (YAML Ain t Markup Language) In ES YAML ( is also supported for the same purpose. To activate YAML, add the format =yamlparameter to the HTTP request. Although JSON is typically used for HTTP communication, the configuration files are usually written in YAML. 13

14 Examining the ES starting log [INFO ][node ] [Justice] version[1.5.2], pid[2669], build[62ff986/ t09:21:06z] [INFO ][node ] [Justice] initializing... [INFO ][plugins ] [Justice] loaded [], sites [] [INFO ][node ] [Justice] initialized [INFO ][node ] [Justice] starting... [INFO ][transport ] [Justice] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/ :9300]} [INFO ][discovery ] [Justice] elasticsearch/osgmwepbqdiawglvcwp0vg [INFO ][cluster.service ] [Justice] new_master [Justice][oSgmWepbQdiAWGLVCWp0Vg][summa- VirtualBox][inet[/ :9300]], reason: zen-disco-join (elected_as_master) [INFO ][http ] [Justice] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/ :9200]} [INFO ][node [INFO ][gateway ] [Justice] started ] [Justice] recovered [0] indices into cluster_state

15 Examining the ES starting log Justice ES gives your node a random name which you can modify from the configuration the actual version of ES :9300 Port 9300 is used by default for inter-node communication, if you use the native Java API instead of the REST API, this is the point where you need to connect. 15

16 Examining the ES starting log elected_as_master each cluster has a master node, responsible for knowing which nodes are in the cluster and where all the shards are located. Each time the master is unavailable, a new one is elected. In this case, you started the first node in the cluster, so this is your master :9200 Port 9200 is used for HTTPcommunication by default. You have to use this address for REST API connections recovered [0] indices When you start your node, the gateway looks on the disk to see if any data is saved so it can restore it. In this case, there s no index to restore 16

17 ES data organization To understand how data is organized in ES, you have to look at it from two perspectives: Logical layout = what your search application needs to be aware of Physical layout = how ES handles your data in the background 17

18 ES data organization - Logical layout Documents = the unit used for indexing and searching (you can think of it like a row in a RDBMS). Types = documents are grouped into types (which contain documents in a way similar to how tables contain rows) Index = one or multiple types live in an index (similar to a database in the SQLworld) 18

19 ES data organization - Logical layout Relational DB Databases Tables Rows Columns Elasticsearch Indices Types Documents Fields 19

20 ES data organization - Physical layout ES divides each index into shards, which can migrate between servers (nodes) that make up a cluster. applications don t care about this because they work with ES in much the same way, whether it s one or more servers. But when you re administering the cluster, you care because the way you configure the physical layout determines its performance, scalability, and availability. 20

21 ES configuration We will create and configure an ES cluster formed by just 1 server containing just 1 node with 1 or a few indexes. The main configuration file of ES can be found in the config/ directory of ES installation directory In your elasticsearch.yml configuration file change : o the cluster name (uncomment and change the cluster.name line) o the node name (uncomment and change the node.name line) Restart ES in order to apply the modifications 21

22 ES configuration The elasticsearch.yml configuration file has different sections and within each section it is possible to change the default value of lots of ES parameters such as: o node.master : allow this node to be eligible as a master node (enabled by default) o node.data : allow this node to store data (enabled by default) o index.number_of_shards : set the number of shards (splits) of an index (5 by default): o index.number_of_replicas : set the number of replicas (additional copies) of an index (1 by default): 22

23 ES configuration o path.conf : path to directory containing configuration o plugin.mandatory : if a plugin listed here is not installed for current node, the node will not start. o and many more.... but for our testing purposes we can use the default setting for all of them 23

24 Almost ready to start... ES is document-oriented, meaning the smallest unit of data you index (store) or search for (retrieve) is a document A document is normally a JSON representation of your data. JSON over HTTP is the most widely used way to communicate with ES (we will use this method) 24

25 Almost ready to start... curl is a command-line tool for transferring data over HTTP. We will use the curl command to make HTTPrequests with this sintax: PUT or curl X GET or POST 25

26 Let s start Open a terminal and run the following command : curl -XPUT 'localhost:9200/my_index/car/1' -d '{ "brand": "Ford", "model": "Focus", "engine": "1.6 litre", "power": "90 hp" }' You have just : created the my_index index created the car type (automapping) indexed the document with id =1 26

27 Getting the current mapping (type) This command shows the fields from your document and the field types that ES detected for each field. curl -XGET 'localhost:9200/my_index/car/_mapping?pretty' { "model" : { "my_index" : { "mappings" : { "car" : { "properties" : { "brand" : { "type" : "string" }, "engine" : { "type" : "string" }, "type" : "string" }, "power" : { "type" : "string" } } } } } } 27

28 How to create a new mapping (a new Type) Simply run the following command : curl -XPUT 'localhost:9200/my_index/_mapping/new-car' -d '{ "new-car" : { "properties" : { "new-brand": { "type" : "string" } } } }' You have just created the new-car type 28

29 How to extend an existing mapping (a Type) In order to extend the new-car mapping : curl -XPUT 'localhost:9200/my_index/_mapping/new-car' -d '{ "new-car" : { "properties" : { "new-model": { "type" : "string" } } } }' curl -XGET 'localhost:9200/my_index/new-car/_mapping?pretty' 29

30 Extending an existing mapping (type) As we saw, when you put a mapping over an existing one, ES merges the two Unfortunately, not all merges work For example you can t change an existing field s data type (in general you can t change the way a field is indexed) You can define a new mapping after you create the index but before inserting any document into that type ( ES is flexible tool but it is not magical! ) 30

31 Extending an existing mapping (type) If you need to modify an existing field in a mapping the only way is to re-index all the data in a new mapping, this involves the following steps: 1) Remove all data from the existing type (removing all data also removes the current mapping) 2) Put in the new mapping 3) Index all the data again 31

32 Design a mapping (type) In ES a field can be one of the core types or it can be a more complex type derived from core types, such as an array 32

33 Exercise Design a mapping Index some brand new data Update indexed data Delete indexed data Retrieve data by executing some queries 33

34 Plugins A default installation of ES comes with no plugins installed Plugins are a powerful way to extend or enhance the functionality that ES provides out of the box. ES plugins are split into two categories: site plugins and code plugins A site plugin is one that provides no additional functionality, it simply provides a web page served by ES A code plugin is any plugin that includes JVM code that ES executes; this can include plugins that add features to ES as well as plugins used to better handle language-specific characteristics of text during analysis (there are even plugins that replace internal parts of ES) It s possible for a code plugin to also include basic HTML, image, and Java-Script files that ES can serve to provide an interface as well. 39

35 Plugins An example of a plugin like this is the Marvel plugin which: includes Java code that collects and stores metrics provide also a site portion which includes an interface to display analytic information about data in ES In order to use a plugin, you first need to install it Plugins come in many forms, but they are most frequently.zip files You can choose to manually extract a.zip file to the plugins directory, or you can use the bin/plugin tool to install it by downloading it either from the internet or from a local zip file 40

36 Plugins You can list the installed plugins using the following command :./plugin --list or you can see them during ES startup : [datetime][info][plugins] [Spellcheck] loaded [marvel], sites [marvel] 41

37 Marvel Download marvel zip from : manual_download.html Create the folder mydownloads in the folder : /home/your_name/elasticsearch-1.5.2/ Move marvel zip into mydownloads Run the following command : bin/plugin -i marvel -u file:///home/your_name/ elasticsearch-1.5.2/mydownloads/marvel zip Restart ES and go to the url : 42

38 Head Download elasticsearch-head-master.zip from : Move elasticsearch-head-master.zip into mydownloads folder Run the following command : bin/plugin -i head u file:///home/your_name/ elasticsearch-1.5.2/mydownloads/ elasticsearch-head-master.zip Restart ES and go to the url : 43