Huge Codebases Application Monitoring with Hystrix

Transcription

1 Huge Codebases Application Monitoring with Hystrix 30 Jan Roman Mohr Red Hat FOSDEM

2 About Me Roman Mohr Software Engineer at Red Hat Member of the SLA team in ovirt Mail: Github: IRC: #ovirt irc.oftc.net 2

3 ovirt ovirt is a powerful virtual machine manager for up to datacenter-class deployments, and provides an awesome KVM management interface for multi-node virtualization. 3

4 ovirt Artitecture Host 1 VMs Host 2 VMs ovirt-engine... Host n VMs Storage 4

5 ovirt-engine The Beast Take 1 5

6 Architecture of ovirt-engine REST GWT Commands DAL DB Broker VDSM 6

7 Git Statistics of ovirt-engine Branch: master Generated: :02:53 (in 370 seconds) Generator: GitStats (version ), git version 2.4.3, gnuplot 5.0 patchlevel 0 Report Period: :43:09 to :18:53 Age: 4928 days, 1449 active days (29.40%) Total Files: Total Lines of Code: ( added, removed) Total Commits: (average 13.9 commits per active day) Authors: 174 (average commits per author) 7

8 Issues we have Many developers A lot of code No second level cache REST performance problems The product runs at the user/customer site Test coverage Hard to configure and run the application 8

9 Where to start? Try to get a high level overview of the architecture Data-mine your Source Control Greg Young* Gather code metrics (JArchitect, Sonar) Monitor your application before you change something * How to get productive in a project in 24h 9

10 Java and Application Monitoring JBug, Jan

11 Profiler? You can see where your application spends its time Easy to get started. Just connect to the JVM in question and browse the CPU profiling graph. Some profilers even support JDBC, JPA, But: In general no application logic specific insights Many are closed source Not easy to collect data 11

12 XRebel for Monitoring? Easy to integrate. Just start an additional Java agent Every HTTP servlet now contains an additional popup where you can access application metrics. But: Closed source Development only 12

13 NewRelic for Monitoring? Excellent visualization Supports multi-host applications Knows a lot about Java Examples: Closed source Production only License based business model 13

14 Hystrix Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable. 14

15 Hystrix Hystrix also provides metrics! 15

16 Hystrix Dashboard 16

17 Hystrix Dashboard Request shape Error percentage Requests/s Circuit breaker status Statistics 17

18 Hystrix Dashboard Successful Rejected (Short circuit) Timeouts (thread isolation) Bad request (exception) Failed executions (exception) Rejected (max. concurrent invocations) 18

19 Hystrix Dashboard Easy to run $ git clone $ cd Hystrix/hystrix-dashboard $../gradlew jettyrun > Running at Easy to integrate Drop the WAR from maven central in your container Add the WAR as dependency and serve the resources folder on an endpoint. Add the hystrix-metrics-event-stream servlet to your application 19

20 Hello World Hystrix Command Setter setter = Setter.withGroupKey( HystrixCommandGroupKey.Factory.asKey("helloWorld") ).andcommandkey( HystrixCommandKey.Factory.asKey("helloWorld") ); HystrixCommand<String> helloworldcommand = new HystrixCommand<String>(setter) protected String run() throws Exception { return "Hello world!"; } }; return helloworldcommand.execute(); 20

21 ovirt-engine The Problem Take 2 JBug, Jan

22 Problem description We have a datacenter with 1000 VMs. We query the /api/vms endpoint which returns all VMs. We need 2.5 seconds to fetch them with no additional load. We have a datacenter with 2000 VMs. We query the /api/vms endpoint which returns all VMs. We need 5 seconds to fetch them with no additional load. 22

23 Solution: Let's curl a little bit 1000 VMs, 1 request $> time bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed k k k 0 --:--:-- 0:00:02 --:--: k real 0m2.547s user0m0.008s sys 0m0.011s 2000 VMs, 1 request $> time bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed M M k 0 --:--:-- 0:00:04 --:--: k real 0m4.900s user0m0.009s sys 0m0.014s 23

24 Solution: Let's curl a little bit 2000 VMs, 10 parallel requests $> time seq 1 10 parallel -j 10 bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed M M k 0 --:--:-- 0:00:28 --:--: k [...] M M k 0 --:--:-- 0:00:28 --:--: k real 0m29.590s user0m0.212s sys 0m0.438s 24

25 Solution: We can guess That's because our database is so slow. The database can cache everything, it is because our REST application code is so slow. That's because we are keeping the database busy with status updates of Hosts and VMs. That's because our architecture is not smart enough, it is just an ordinary monolith. That must be solved with streaming and eventbuses. 25

26 Solution: Let's curl a little bit more Let us execute the following scenario: 100 Vms, 10 parallel requests, 100 requests total $> seq parallel -j 10 bash rest.sh vms > /dev/null 26

27 Find the Error 27

28 Find the Error 28

29 With the Fix 1000 VMs, 1 request $> time bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed k k k 0 --:--:-- --:--:-- --:--: k real 0m1.008s user0m0.091s sys 0m0.042s 2000 VMs, 1 request $> time bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed M M k 0 --:--:-- 0:00:01 --:--: k real 0m2.218s user 0m0.079s sys 0m0.062s 29

30 With the Fix 2000 VMs, 10 parallel requests time seq 1 10 parallel -j 10 bash rest.sh vms > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed M M k 0 --:--:-- 0:00:09 --:--: k [...] M M k 0 --:--:-- 0:00:09 --:--: k real 0m10.228s user0m0.211s sys 0m0.436s Much better but still too slow. We will see later how to avoid being overwhelmed by too much expensive calls. 30

31 With the Fix 31

32 Collect data from user systems Collect as much streaming data as you want: $> curl -H "Accept: application/json" -H "Content-type: application/json" -X GET --user ping: data: {"type":"hystrixcommand","name":"getvmsinit","group":"getvmsinit", [...] } data: {"type":"hystrixcommand","name":"vdshostdevlistbycaps", [...] } Import it later in your favourite analysis tool or send the data directly to it by using Hystrix plugins. 32

33 Archaius: config.properties How to limit concurrent invocations of a command protected by Hystrix: Default Value 10 Default Property hystrix.command.default.execution.isolation.semaphore.maxconcurrentrequests Instance Property hystrix.command.hystrixcommandkey.execution.isolation.semaphore.maxconcurrentrequests How to Set Instance Default HystrixCommandProperties.Setter().withExecutionIsolationSemaphoreMaxConcurrentRequests(int value) This will override the default configuration of the SearchVM command: hystrix.command.searchvm.execution.isolation.semaphore.maxconcurrentrequests=10 Just add it to your config.properties file or set it as system property. 33

34 Protected VMs Endpoint 34

35 Great OSS Monitoring Ecosystem Hystrix Servo Dropwizard Metrics Cockpit Thermostat Grafana, Graphite Hawkular Prometheus... 35

36 Conclusion Hystrix provides an easy way to monitor and protect your monolith or your microservices An interesting open source ecosystem around monitoring exists From some closed source projects we can learn a lot about usability 36

37 Questions? 37

38 Thank you! Mail: Github: IRC: #ovirt irc.oftc.net 38