COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Size: px

Start display at page:

MargaretMargaret Greene
5 years ago
Views:

1 Building and Operating High Performance MarkLogic Apps James Clippinger, VP, Strategic Accounts, MarkLogic Erin Miller, Manager, Performance Engineering, MarkLogic COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

The Problem: Debugging Performance in Prod SLIDE: 3

4 Performance Success Development: understand MarkLogic Server and how it works Agile dev, agile performance testing Firefights: Avoid when possible (see above bullet points). Use methodology when necessary (ie, don t take the humidifier apart without the manual) SLIDE: 4 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

7 Ingest: Resource Costs CPU: Merges operate as background I/O; show up as nice % CPU: index creation/updates (ingest + merges) CPU: transforms Memory: increases as content is loaded (tree, termlist, triples, reverse) - Preload mapped data: all memory mapped file data pages loaded at end of merge Network: NAS impact; mlcp fastload avoids network hops but can only be used for new content (not updates) and when forest topology won t change. How big is your cluster? SLIDE: 7 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

8 Ingest: Understanding Merges and IOPS Merge: multiple stands combined, reclaims disk space, optimizes indexes Runs automatically based on the merge algorithm, or on demand Storage: need total indexed forest size + 64GB/forest (8.0-1 and lower, or 96GB/forest 9.0 and greater) - DA/HA changes everything - DB replication. What if replication pauses? All unshipped changes retained on the master. 2x 4x indexed data size - DB restores: 2x indexed data + 64 or 96 GB per forests IOPS: Spikey I/O when in-memory stand flushed to disk - 512KB sequential writes, journal writes <= 512KB SLIDE: 8 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

10 Index Resolution and Filtering 1. NARROW DOCUMENT SET WITH INDEXES 2. FILTER RESULTS TO REMOVE FALSE POSITIVES 1 INITIAL RESULT SET 2 FILTERED RESULTS Quick brown fox SOURCE DATA The clown spotted the quick brown monkey and the slow brown fox in the quick clown car more indexes enabled = fewer results FINAL RESULTS FINAL RESULTS SLIDE: 10 COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

12 Queries: Filtered Searches Remember: Context is everything First: Tune indexes to reduce false positives - Resolve queries from the indexes - Use filtering only if you expect to have false positives and your application can t tolerate false positives SLIDE: 12 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Queries: Understanding Distributed Caches SLIDE: 13

14 Queries: Caches and their Contents List cache Compressed tree cache Expanded tree cache Triples and Triples value cach Not caches, but still in memory: memory mapped files - Range indexes - Lexicons SLIDE: 14 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

15 Queries: Making Best use of your Caches Cache sizes optimized at install-time - What if I add memory? - Increase cache sizes - What if I separate into D- and E-nodes? - Create two groups with two different cache sizes - E-nodes should have more ETC, less CTC and List Cache - D-nodes should have more CTC and List, less ETC - But don t forget about reindexing SLIDE: 15 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

16 Queries: Case Study Use case: I need to update a config file once a minute based upon the content in the db Config file keeps track of different document types: - News - Tweets - Stock Prices - Articles SLIDE: 16 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Queries: Case Study: Attempt One SLIDE: 17 COPYRIGHT

Queries: Case Study: Attempt One Use the Query Console built-in profiler, or grow your

Queries: Case Study: Remediation, Step Two Now the count is taking a really long time.

21 Queries: Case Study: Remediation at Scale with Load Now it works great in dev, even with 100M docs. But with load, performance is horrible SLIDE: 21 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

23 Queries: Case Study: Remediation at Scale But I m only doing one document update, my one small config doc. - But you re in the context of an update transaction, so EVERY doc referenced in the transaction will be locked. - Do you have other cts:search statements in your module? - To fix this, do the update in a separate transaction. - Be intentional about update transactions SLIDE: 23 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

24 Queries: Case Study: Remediation at Scale What other techniques might we use to solve this problem? - Element in doc with content-type (ie, <content-type>tweet</content-type> ) - But you still need to find those docs at runtime (remember list cache, CTC, ETC?) - Collections - No need to have custom code that creates collection info on ingest just use the mlcp option output_collections - Get benefits of range indexes - Envelope pattern and canonical format (see Data Hub presentation) SLIDE: 24 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

SQL and Semantics SLIDE: 25 13 June 2017

26 SQL and Semantics Memory, Memory, Memory! Analytics workloads Separate e- and d-nodes Use at least 64 Gb RAM E-Nodes for high-performance applications SLIDE: 26 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

27 SQL and Semantics Debug Trace Events Turn on diagnostic profiling and then: - SPARQL Cost Analysis - SPARQL Execution XDMP:MEM_CANCELED - Your query was using too much memory try to tune to return fewer triples/rows SLIDE: 27 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

28 System Resources: Wrap up Use Meters to understand: - Cache hits/misses, document locks, CPU, disk, memory utilization Be intentional about: - Transformations - Updates - Filtered Results Separate analytical workloads (Semantics, SQL) with E- D-nodes SLIDE: 28 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

29 Themes in Recent Performance Issues

30 THEME Monitoring In Context Meters is your best friend, but it can t live your life for you Know what your system should be doing Know if it s reasonable to expect certain performance SLIDE: 30 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

31 Monitoring In Context Resources (most problematic to least) - Disk Bandwidth - RAM Capacity - CPU Capacity - Disk Capacity - Network Bandwidth OS bugs MarkLogic bugs SLIDE: 31 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

32 THEME Speed Costs Money Consider the expense of code choices Every index in production should have demonstrable business value SLIDE: 32 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

33 THEME Load Testing Best way to reduce performance risk Real load against production (or equivalent) hardware Push load until bottleneck found SLIDE: 33 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

34 Recent Performance Cases

35 Case: Intermittent Slowness System was not meeting SLAs for a few minutes at start of each hour Meters showed CPU maxed with iowait during slowness SLIDE: 35 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

36 Case: General Slow Queries Queries running slower than expected Disk bandwidth normal Low user CPU, some iowait No swapping SLIDE: 36 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

37 Case: Unusual CPU Activity Very high system CPU utilization and low user CPU utilization under load Only seen on larger AWS instances SLIDE: 37 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

38 Case: High Memory Consumption RAM utilization very high, eventually leading to swapping Caches configured to default values SLIDE: 38 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

39 Case: Everything s Okay Until. High-volume production system running great, meeting all SLAs DBA runs report in Query Console after overnight data load System response time spikes for a few minutes while report is running SLIDE: 39 COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

40 Questions?

Optimizing the Performance of Your MarkLogic Applications

Optimizing the Performance of Your MarkLogic Applications JAMES CLIPPINGER Vice President, Strategic Accounts, MarkLogic @jclip ERIN MILLER Manager, Performance Engineering, MarkLogic 4 June 2018 MARKLOGIC