Importing into Neo4j - No witty subtitle - Dave July 27

Size: px

Start display at page:

Download "Importing into Neo4j - No witty subtitle - Dave July 27"

Caitlin Campbell
6 years ago
Views:

1 Importing into Neo4j - No witty subtitle - Dave July 27

2 Life as a Field Engineer 2

3 Ask the right questions 3

Model the Problem Simple relationship has a name, but no properties Employee Various Properties :ATTENDED Event Various Properties :ATTENDED Employee

4 Model the Problem Simple relationship has a name, but no properties Employee Various Properties :ATTENDED Event Various Properties :ATTENDED Employee Various Properties Relationship entity has a name and also various properties Expense Report Various Properties :REFERENCES :SUBMITTED SubmitDate ReportNbr 4

5 User stories

6 Derive questions Which peopleclaimed expenses for the same event?

7 From user story to model MATCH (e:employee)-[:attended]->(ev:event)<-[:attended]- (e1:employee) WITH e,ev,e1 MATCH (ev)-[:expensed_on]->(er:expense_report) RETURN e,ev,e1,er;? Which people claimed expenses for the same event? person ATTENDED event person SUBMITTED exense_report (person)-[:attended]->(event)<-[:attended]-(colleague)

8 Get some representative data 8

9 Loading your Data

10 Load CSV More than just a basic data ingestion mechanism It's an ETL Power Tool 10

11 Load CSV ETL Power Tool Combines multiple aspects in a single operation Supports loading / ingesting CSV data from an URI (file://, ftp://) Direct mapping of input data into complex graph/domain structure Data conversion Supports complex computations Create or merge data, relationships and structure 11

12 Load CSV Nodes Indexes - Relationships Do multiple passes to create nodes and relationships instead of large, combined statements LOAD CSV WITH HEADERS FROM file:///path/to/file.csv AS line MERGE (a:person {id:line.id}) ON CREATE SET a.name=line.name; CREATE INDEX on :Person(id); CREATE INDEX on :Movie(id); LOAD CSV WITH HEADERS FROM file:///path/to/file.csv AS line MATCH (m:movie {id:line.movieid}) MATCH (a:person {id:line.personid}) CREATE (a)- [:ACTED_IN {roles:[line.role]}]- >(m); 12

13 Load CSV Periodic Commit ALWAYS prefix your LOAD CSV with USING PERIODIC COMMIT. The number given is the number of import rows after which a commit of the imported data happens. Depending on the complexity of your import operation, you might create from 100 elements per 1000 rows (if you have a lot of duplicates) up to 100,000 when you have complex operations that generate up to 100 nodes and relationships per row of input. That s why a commit size of 1000 might be a safe bet. 13

14 Load CSV More Tips Rather than a long merge/create statement that attempts to create multiple entities in one pass, favor short, simple statements and do multiple passes over the input as needed If you load your CSV file over the network make sure the network is fast enough to sustain the ingestion rate you d like to have. Otherwise: If possible download it, and use a file:// URL. Column names are case sensitive. Misspelled column names result in null values. 14

15 Load CSV Considerations Make sure you have sufficient RAM Use file:///path/to/file.csv on OSX and Unix, use file:c:/path/to/file.csv on Windows Check correct delimiters and columns Columns are case sensitive Empty columns are treated as null Default data type is String. Use toint or tofloat to convert Change the delimiter if needed with AS line FIELDTERMINATOR ; Create necessary indexes and constraints upfront Use the Neo4j- Shell for larger imports 15

16 CSV

17 CSV Files for Northwind Data

18 CSV Files for Northwind Data

19 Step- by- step Creating the Graph 1.Import Nodes 2.Create Indexes 3.Import Relationships

20 LOADing the Data // Create customers USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row CREATE (:Customer {companyname: row.companyname, customerid: row.customerid, fax: row.fax, phone: row.phone}); // Create products USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row CREATE (:Product {productname: row.productname, productid: row.productid, unitprice: tofloat(row.unitprice)});

21 LOADing the Data // Create suppliers USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row CREATE (:Supplier {companyname: row.companyname, supplierid: row.supplierid}); // Create employees USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row CREATE (:Employee {employeeid:row.employeeid, firstname: row.firstname, lastname: row.lastname, title: row.title});

22 LOADing the Data // Create categories USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row CREATE (:Category {categoryid: row.categoryid, categoryname: row.categoryname, description: row.description}); // Create orders USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row MERGE (order:order {orderid: row.orderid}) ON CREATE SET order.shipname = row.shipname;

23 Creating the Indexes CREATE INDEX ON :Product(productID); CREATE INDEX ON :Product(productName); CREATE INDEX ON :Category(categoryID); CREATE INDEX ON :Employee(employeeID); CREATE INDEX ON :Supplier(supplierID); CREATE INDEX ON :Customer(customerID); CREATE INDEX ON :Customer(customerName);

24 Creating the Relationships USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row MATCH (order:order {orderid: row.orderid}) MATCH (customer:customer {customerid: row.customerid}) MERGE (customer)-[:purchased]->(order); USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row MATCH (product:product {productid: row.productid}) MATCH (supplier:supplier {supplierid: row.supplierid}) MERGE (supplier)-[:supplies]->(product);

25 Creating the Relationships USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row MATCH (order:order {orderid: row.orderid}) MATCH (product:product {productid: row.productid}) MERGE (order)-[pu:product]->(product) ON CREATE SET pu.unitprice = tofloat(row.unitprice), pu.quantity = tofloat(row.quantity); USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row MATCH (order:order {orderid: row.orderid}) MATCH (employee:employee {employeeid: row.employeeid}) MERGE (employee)-[:sold]->(order);

26 Creating the Relationships USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row MATCH (product:product {productid: row.productid}) MATCH (category:category {categoryid: row.categoryid}) MERGE (product)-[:part_of]->(category); USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM " AS row MATCH (employee:employee {employeeid: row.employeeid}) MATCH (manager:employee {employeeid: row.reportsto}) MERGE (employee)-[:reports_to]->(manager);

27 Neo4j- Import

28 Neo4j- Import: Handling the initial firehose of data 28

29 Neo4j- Import Create your initial database Allows you to load large amounts of data into a Neo4j database Allows you to specify nodes and relationships in separate files Supports loading / ingesting CSV data from your file system Does not support data transformations 29

30 Neo4j- Import Notes: Fields are comma separated by default but a different delimiter can be specified. All files must use the same delimiter. Multiple data sources can be used for both nodes and relationships. A data source can optionally be provided using multiple files. A header which provides information on the data fields must be on the first row of each data source. Fields without corresponding information in the header will not be read. UTF- 8 encoding is used. 30

31 Neo4j- Import Sample Script./bin/neo4j- import - - into /Users/davidfauth/testDB - - nodes /Users/davidfauth/neo4j- atlanta- meetup/employee.csv - - nodes /Users/davidfauth/neo4j- atlanta- meetup/locations.csv - - nodes /Users/davidfauth/neo4j- atlanta- meetup/events.csv - - nodes /Users/davidfauth/neo4j- atlanta- meetup/expense_report.csv - - relationships /Users/davidfauth/neo4j- atlanta- meetup/events_rels.csv - - relationships /Users/davidfauth/neo4j- atlanta- meetup/exp_rep_rels.csv - - relationships /Users/davidfauth/neo4j- atlanta- meetup/expense_report_rels.csv - - bad- tolerance

32 Transactional REST Endpoint Pass Cypher Statements over REST The Neo4j transactional HTTP endpoint allows you to execute a series of Cypher statements within the scope of a transaction. Can use the same transaction for multiple HTTP requests 32

33 Building a Neo4j Application #3 #1 #2

34 Additional Resources Max DeMarzi ( Wikipedia into Neo4j with Graphipedia Scaling concurrent writes in Neo4j Online payment risk management with Neo4j Mark Needham ( Loading data REST API vs Batch Import The Batch Inserter and the sunk cost fallacy RikVan Bruggen ( Import: summarised Using `LOAD CSV` to import data from a Google Spreadsheet Food networks, countries, diets, health and Load CSV Some Neo4j import tweaks, what and where Spreadsheet method: plenty! Michael Hunger ( `LOAD CSV` into Neo4j quickly and successfully Use `LOAD CSV` to import Git history into Neo4j On Importing Data into Neo4j (blog) 34

35 Importing into Neo4j - No witty subtitle - Thank You

Submit the MS Access Database file that contains the forms created in this lab.

Submit the MS Access Database file that contains the forms created in this lab. A. Lab # : BSBA BIS245A-5B B. Lab 5B of 7: Completing Forms C. Lab Overview--Scenario/Summary TCO(s): 5. Given a physical database containing tables and relationships, create forms which demonstrate effective