Traversing Graph Databases with Gremlin. Marko A. Rodriguez Graph Systems Architect.

Transcription

1 Traversing Graph Databases with Gremlin Marko A. Rodriguez Graph Systems Architect NoSQL New York City Meetup May 16, 2011 Gremlin G =(V, E) May 10, 2011

2 Thank You Sponsors Short Slideshow + Live Demo = This Presentation

3 TinkerPop Productions 1 1 TinkerPop ( Marko A. Rodriguez ( Peter Neubauer ( Joshua Shinavier ( Pavel Yaskevich ( Darrick Wiebe ( com/), Stephen Mallette ( Alex Averbuch ( com/in/alexaverbuch)

4 TinkerPop Graph Stack RESTful Server Traversal Language Traversal Engine Graph-to-Object Mapper Generic Interface Graph Database TinkerGraph

5 TinkerPop Graph Stack Focus Traversal Language Traversal Engine Generic Interface TinkerGraph

6 Blueprints > Pipes > Groovy > Gremlin To understand Gremlin, its important to understand Groovy, Blueprints, and Pipes. Blueprints Pipes

7 Gremlin is a Domain Specific Language Gremlin 0.7+ uses Groovy as its host language. 2 Gremlin 0.7+ takes advantage of Groovy s meta-programming, dynamic typing, and closure properties. Groovy is a superset of Java and as such, natively exposes the full JDK to Gremlin. 2 Groovy available at

8 Gremlin is for Property Graphs name = "lop" lang = "java" name = "marko" age = 29 weight = knows weight = 0.4 created weight = 1.0 knows 4 3 created 11 weight = 0.4 name = "josh" age = 32 weight = 0.2 created 12 6 name = "peter" age = 35 name = "vadas" age = weight = 1.0 created 5 name = "ripple" lang = "java" A graph is composed of vertices (nodes), edges (links), and properties (keys/values).

9 Gremlin is for Blueprints-enabled Graph Databases Blueprints can be seen as the JDBC for property graph databases. 3 Provides a collection of interfaces for graph database providers to implement. Provides tests to ensure the operational semantics of any implementation are correct. Provides numerous helper utilities to make working with graph databases easy. Blueprints 3 Blueprints is available at

10 A Blueprints Detour - Implementations TinkerGraph

11 A Blueprints Detour - Ouplementations JUNG Java Universal Network/Graph Framework

12 Gremlin Compiles Down to Pipes Pipes is a data flow framework for evaluating lazy graph traversals. 4 A Pipe extends Iterator, Iterable and can be chained together to create processing pipelines. A Pipe can be embedded into another Pipe to allow for nested processing. Pipes 4 Pipes is available at

13 A Pipes Detour - Chained Iterators This Pipeline takes objects of type A and turns them into objects of type D. A A A A A Pipe1 B Pipe2 C Pipe3 D Pipeline D D D D Pipe<A,D> pipeline = new Pipeline<A,D>(Pipe1<A,B>, Pipe2<B,C>, Pipe3<C,D>)

14 A Pipes Detour - Simple Example What are the names of the people that marko knows? knows B name=peter A knows C name=pavel name=marko created created D name=gremlin

15 A Pipes Detour - Simple Example Pipe<Vertex,Edge> pipe1 = new OutEdgesPipe("knows"); Pipe<Edge,Vertex> pipe2 = new InVertexPipe(); Pipe<Vertex,String> pipe3 = new PropertyPipe<String>("name"); Pipe<Vertex,String> pipeline = new Pipeline(pipe1,pipe2,pipe3); pipeline.setstarts(new SingleIterator<Vertex>(graph.getVertex("A")); InVertexPipe() OutEdgesPipe("knows") knows B PropertyPipe("name") name=peter A knows C name=pavel name=marko created created D name=gremlin HINT: The benefit of Gremlin is that this Java verbosity is reduced to g.v( A ).oute( knows ).inv.name.

16 A Pipes Detour - Pipes Library [ FILTERS ] AndFilterPipe BackFilterPipe CollectionFilterPipe ComparisonFilterPipe DuplicateFilterPipe FutureFilterPipe ObjectFilterPipe OrFilterPipe RandomFilterPipe RangeFilterPipe [ GRAPHS ] OutEdgesPipe InEdgesPipe OutVertexPipe InVertexPip IdFilterPipe IdPipe LabelFilterPipe LabelPipe PropertyFilterPipe PropertyPipe [ SIDEEFFECTS ] AggregatorPipe GroupCountPipe CountPipe SideEffectCapPipe [ UTILITIES ] GatherPipe PathPipe ScatterPipe Pipeline...

17 A Pipes Detour - Creating Pipes public class NumCharsPipe extends AbstractPipe<String,Integer> { public Integer processnextstart() { String word = this.starts.next(); return word.length(); } } When extending the base class AbstractPipe<S,E> all that is required is an implementation of processnextstart().

18 Now onto Gremlin proper...

19 The Gremlin Architecture Neo4j OrientDB DEX

20 The Many Ways of Using Gremlin Gremlin has a REPL to be run from the shell. Gremlin can be natively integrated into any Groovy class. Gremlin can be interacted with indirectly through Java, via Groovy. Gremlin has a JSR 223 ScriptEngine as well.

21 Pipe = Step: 3 Generic Steps In general a Pipe is a step in Gremlin. Gremlin is founded on a collection of atomic steps. The syntax is generally seen as step.step.step. There are 3 categories of steps. transform: map the input to some output. S T oute, inv, paths, copysplit, fairmerge,... filter: either output the input or not. S (S ) back, except, uniqueobject, andfilter, orfilter,... sideeffect: output the input and yield a side-effect. S S aggregate, groupcount,...

22 Abstract/Derived/Inferred Adjacency oute inv oute loop(2){..} back(3) {it.salary} } friend_of_a_friend_who_earns_less_than_friend_at_work oute, inv, etc. is low-level graph speak (the domain is the graph). codeveloper is high-level domain speak (the domain is software development). 5 5 In this way, Gremlin can be seen as a DSL (domain-specific language) for creating DSL s for your graph applications. Gremlin s domain is the graph. Build languages for your domain on top of Gremlin (e.g. software development domain ).

23 Developer's FOAF Import Graph Friend-Of-A-Friend Graph Developer Imports Graph Friends at Work Graph Developer Created Graph Software Imports Graph Friendship Graph Employment Graph You need not make derived graphs explicit. You can, at runtime, compute them. Moreover, generate them locally, not globally (e.g. ``Marko's friends from work relations"). Explicit Graph This concept is related to automated reasoning and whether reasoned relations are inserted into the explicit graph or computed at query time.

24 Integrating the JDK (Java API) Groovy is the host language for Gremlin. available in a path expression. Thus, the JDK is natively gremlin> v.out( friend ){it.name.matches( M.* )}.name ==>Mattias ==>Marko ==>Matt... gremlin> x.out( rated ).transform{jsonparser.get(it.uri).stars}.mean()... gremlin> y.oute( rated ).filter{textanalysis.iselated(it.review)}.inv...

25 Time for a demo marko$ gremlin \,,,/ (o o) -----oooo-(_)-oooo----- gremlin> Gremlin G =(V, E)

26 Graph Bootcamp June th in Chicago, Illinois. August X-Y th in Denver, Colorado. Book a private gig for your organization.