Implementing Microservices Tracing with Spring Cloud and Zipkin

Transcription

1 Implementing Microservices Tracing with Spring Cloud and Zipkin Marcin Pivotal

2 About me Spring Cloud developer at Pivotal Working mostly on Spring Cloud Sleuth Spring Cloud Contract Spring Cloud Pipelines Blog: 2

3 Agenda What is distributed tracing? How to correlate logs with Spring Cloud Sleuth? How to visualize latency with Spring Cloud Sleuth and Zipkin? 3

4 4

5 An ordinary system... 5

6 UI calls backend UI -> BACKEND 6

7 Everything is awesome CLICK 7 200

8 Until it s not CLICK 8 500

9 9

10 Time to debug 10

11 It doesn t look like this 11

12 More like this 12

13 On which server / instance was the exception thrown? 13

14 SSH and grep for ERROR to find it? 14

15 How to find all logs from all servers that correspond to that business action? 15

16 The answer: distributed tracing Span Trace Baggage Logs (annotations) Tags (binary annotations) 16

17 The answer: distributed tracing Span Trace Baggage Logs (annotations) Tags (binary annotations) 17

18 Span The basic unit of work (e.g. sending RPC) Spans are started and stopped They keep track of their timing information Once you create a span, you must stop it at some point in the future Has a parent and can have multiple children All spans have unique span ids Spans in a single hierarchy share a trace id 18

19 Trace A set of spans forming a tree-like structure. For example, if you are running a bookstore then Trace could be retrieving a list of available books Assuming that to retrieve the books you have to send 3 requests to 3 services then you could have at least 3 spans (1 for each hop) forming 1 trace 19

20 Baggage (from Sleuth 1.2.0) Key value pairs that get propagated between network boundaries once set is accessible in every application for the duration of trace works for HTTP and messaging based communication WARNING: if size of the baggage is too large then your latency can get greater 20

21 Trace Id = X Span Id = B Server Received Trace Id = X Span Id = A Trace Id = X Span Id = C Trace Id = X Span Id = B Client Send Trace Id = X Span Id = D Client Send Trace Id = X Span Id = D Server Received Trace Id = X Span Id = E REQUEST SERVICE 3 REQUEST REQUEST RESPONSE Trace Id = X Span Id = D Client Received No Trace Id No Span Id SERVICE 1 Trace Id = X Span Id = D Server Sent Trace Id = X Span Id = E SERVICE 2 RESPONSE Trace Id = X Span Id = F Client Send RESPONSE Trace Id = X Span Id = B Client Received Trace Id = X Span Id = F Server Received Trace Id = X Span Id = G REQUEST Trace Id = X Span Id = A SERVICE 4 RESPONSE Trace Id = X Span Id = B Server Sent Trace Id = X Span Id = C Trace Id = X Span Id = F Client Received Trace Id = X Span Id = F Server Sent Trace Id = X Span Id = G

22 Span Id = A Parent Id = null 22 Span Id = B Parent Id = A Span Id = D Parent Id = C Span Id = E Parent Id = D Span Id = F Parent Id = C Span Id = G Parent Id = F Span Id = C Parent Id = B

23 Is it that simple? 23

24 Is context propagation simple? How do you pass tracing information (incl. Trace ID) between: different libraries? thread pools? asynchronous communication?? 24

25 Log correlation with Spring Cloud Sleuth We take care of passing tracing information between threads / libraries / contexts for: Hystrix RxJava Rest Template Feign Messaging with Spring Integration Zuul... If you don t do anything unexpected there s nothing you need to do to make Sleuth work. Check the docs for more info. 25

26 Spring Cloud Sleuth logging format We set a logging format for you... 26

27 Now let s aggregate the logs! Instead of SSHing to the machines to grep logs lets aggregate them! With Cloud Foundry s (CF) Loggregator the logs from different instances are streamed into a single place You can harvest your logs with Logstash Forwarder / FileBeat You can use ELK stack to stream and visualize the logs 27

28 Spring Cloud Sleuth with Maven <dependencymanagement> <dependencies> <dependency> <groupid>org.springframework.cloud</groupid> <artifactid>spring-cloud-dependencies</artifactid> <version>camden.sr6</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencymanagement> 28 <dependency> <groupid>org.springframework.cloud</groupid> <artifactid>spring-cloud-starter-sleuth</artifactid> </dependency>

29 REQUEST SERVICE 3 REQUEST REQUEST Hello from service3 SERVICE 1 /start RESPONSE RESPONSE SERVICE 2 RESPONSE REQUEST Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4] SERVICE 4 RESPONSE Hello from service4 29

30 REQUEST REQUEST REQUEST SERVICE 1 /readtimeout BOOM! 30 SERVICE 2 BOOM! BOOM!

31 Log correlation with Spring Cloud Sleuth DEMO 31

32 32

33 33

34 34

35 Great! We ve found the exception! But meanwhile... 35

36 The system is slow... CLICK

37 One of the services is slow... 37

38 Which one? How to measure that? 38

39 Let s log events! 39 Client Send (CS) - The client has made a request Server Received (SR) - The server side got the request and will start processing Server Send (SS) - Annotated upon completion of request processing Client Received (CR) - The client has successfully received the response from the server side

40 CS 0 ms CR 450 ms 40 SR 100 ms SS 300 ms

41 CS 0 ms SR 100 ms Conclusions CR 450 ms 41 The request started at T=0ms It took 450 ms for the client to receive a response Server side received the request at T=100 ms The request got processed on the server side in 200 ms SS 300 ms

42 CS 0 ms SR 100 ms Conclusions CR 450 ms Why is there a delay between sending and receiving messages?!!11!one!?!1! 42 SS 300 ms

43 43

44 Distributed tracing - terminology Span Trace Baggage Logs (annotations) Tags (binary annotations) 44

45 Logs Represents an event in time associated with a span Every span has zero or more logs Each log is a timestamped event name Event should be the stable name of some notable moment in the lifetime of a span 45 For instance, a span representing a browser page load might add an event for each of the Performance.timing moments (check

46 46

47 CS 0 ms Main logs Client Send (CS) The client has made a request - the span was started Server Received (SR) The server side got the request and will start processing it 47 SR timestamp - CS timestamp = NETWORK LATENCY SR 100 ms

48 CS 0 ms SR 100 ms Main logs CR 450 ms Server Send (SS) Annotated upon completion of request processing 48 SS timestamp - SR timestamp = SERVER SIDE PROCESSING TIME Client Received (CR) The client has successfully received the response from the server side CR timestamp - CS timestamp = TIME NEEDED TO RECEIVE RESPONSE CR timestamp - SS timestamp = NETWORK LATENCY SS 300 ms

49 Tag Key-value pair 49 Every span may have zero or more key/value Tags They do not have timestamps and simply annotate the spans. Example of default tags in Sleuth message/payload-size http.method commandkey for Hystrix

50 How to visualise latency in a distributed system? 50

51 The answer is: ZIPKIN 51

52 How does Zipkin work? SPANS SENT TO COLLECTORS UI QUERIES FOR TRACE INFO VIA API APP SPANS SENT TO COLLECTORS APP 52 STORE IN DB

53 How does Zipkin look like? 53

54 Spring Cloud Sleuth and Zipkin integration We take care of passing tracing information between threads / libraries / contexts Upon closing of a Span we will send it to Zipkin either via HTTP (spring-cloud-sleuth-zipkin) or via Spring Cloud Stream (spring-cloud-sleuth-stream) You can run Zipkin Spring Cloud Stream Collector as a Spring Boot app (spring-cloud-sleuth-zipkin-stream) you can add the dependency to Zipkin UI! 54

55 Spring Cloud Sleuth Zipkin with Maven <dependencymanagement> <dependencies> <dependency> <groupid>org.springframework.cloud</groupid> <artifactid>spring-cloud-dependencies</artifactid> <groupid>org.springframework.cloud</groupid> <version>camden.sr6</version> <artifactid>spring-cloud-starter-zipkin</artifactid> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencymanagement> 55 <dependency> </dependency>

56 Hold it! If I have billion services that emit gazillion spans - won t I kill Zipkin? 56

57 Sampling to the rescue! By default Spring Cloud Sleuth sends only 10% of requests to Zipkin You can change that by changing the property spring.sleuth.sampler.percentage (for 100% pass 1.0) Or register a custom org.springframework.cloud.sleuth.sampler implementation 57

58 REQUEST REQUEST REQUEST SZCZECIN SERVICE /szczecin RESPONSE REQUEST SERVICE 1 /foo RESPONSE REQUEST RESPONSE 58 /bar SERVICE 2 /start RESPONSE RESPONSE SERVICE 3 SERVICE 4 /baz

59 DEMO 59

60 Traced call 60

61 Traced call 1st request 61 Service1 Service2 calling Service2 calling Service3 Service2 calling Service4

62 Traced call Events RPC call Tags 62

63 Traced call - error click! 63

64 Traced call - error 64

65 Baggage Setting a baggage item Retrieving a baggage item 65

66 Manipulating spans via annotations (from Sleuth 1.2.0) New span Continue span 66

67 Summary Log correlation allows you to match logs for a given trace Distributed tracing allows you to quickly see latency issues in your system Zipkin is a great tool to visualize the latency graph and system dependencies Spring Cloud Sleuth integrates with Zipkin and grants you log correlation With you ll be able to propagate any information via baggage With you ll be able to use annotations to create / continue spans and add logs and tags 67

68 Zipkin for Brewery A test app for Spring Cloud end to end tests Source code: Around 10 applications involved Zipkin deployed to PCF for Brewery Sample app: 68

69 Zipkin for Brewery 69

70 Zipkin for Brewery 70

71 71

72 Links Code for this presentation (clone and run getreadyforconference.sh - NOTE: you need Vagrant!) : Sleuth samples: Sleuth s documentation: Repo with Spring Boot Zipkin server: Zipkin deployed to PCF for Brewery Sample app: Pivotal Web Services trial : PCF on your laptop : 72

73 Learn More. Stay Connected. Read the docs Twitter: twitter.com/springcentral Check the samples YouTube: spring.io/video Talk to us on Gitter LinkedIn: spring.io/linkedin Google Plus: spring.io/gplus 73

74 mgrzejszczak 74