- From Theory to Practice -

Transcription

1 - From Theory to Practice -

2

3 Thomas Balthazar Stella Cotton Corey Donohoe Yannick Schutz

4 What is distributed tracing?

5 Tracing requests across distributed system boundaries

6 A Simple Use Case

7 Web Request Monolithic Web Process Authentication User Orders Items

8 Web Request Auth Process New app Ecommerce Process User Ruby app Orders Items

9 Web Request Auth Process Ecommerce Process Original Ruby app Orders Recommendations Billing Python????

10 Microservices! (j/k)

11 Services (Micro or otherwise)

12 Web Request Auth Process Ecommerce Process Orders Recommendations Billing

13 <insert container joke>

14 Why do we need distributed tracing?

15 Internal services look like external APIs

16 Web Request Why is this slow???? Auth Process Ecommerce Process Orders Recommendations Billing Blame data science?

17 You can t tell a coherent macro story about your application by monitoring individual processes Ben Seligman

18 People are bad guessers

19 How do you tell the story?

20 Distributed Tracing!

21 Distributed tracing commoditizes knowledge - Adrian Cole

22 What s Stopping You?

23 Outside Your Language s Wheelhouse

24 Domain Specific Vocabulary

25 Fractured Ecosystem

26 Theory -> Practice

27 The Basics

28 Black Box Tracing

29 Black Box Tracing

30 Why might this not work for you?

31 Need lots of data Delayed results Can t guarantee causality

32 def my_cool_system service_1 service_2 end def service_1 Rails.logger "Service 1" execute_async_job end def execute_async_job Rails.logger "Async Job" end def service_2 Rails.logger "Service 2" end Aggregated Log :01:01 Service :01:02 Async Job :01:03 Service 2

33 Simulate latency def my_cool_system service_1 service_2 end def service_1 Rails.logger "Service 1" execute_async_job end def execute_async_job sleep 15 Rails.logger "Async Job" end def service_2 Rails.logger "Service 2" end Aggregated Log :01:01 Service :01:02 Service :01:17 Async Job Latency

34 White Box Tracing

35 Metadata Propagation

36 Realtime Analysis

37 History Lesson

38 Dapper

39 Zipkin

40 Distributed Tracing

41 So, you want to trace your distributed system? Key design insights from years of practical experience Raja R. Sambasivan, Rodrigo Fonseca, Ilari Shafer, Gregory R. Ganger

42 Tracing Your Applications

43 Main Components: Tracer Transport Collector Storage UI

44 Tracing Requests

45 Tracer: Lives inside your apps, does the tracing

46 Trace: The story of a request s journey through your system

47 Web Request Auth Process A trace tells this whole story Ecommerce Process Orders Recommendations Billing

48 Span: Each chapter in that story

49 Web Request Auth Ecommerce A span Orders Recommendations Billing

50 Web Request Trace id 123 Auth Trace id 123 Ecommerce Trace id 123 Trace id 123 Trace id 123 Orders Recommendations Billing

51 Trace id 123 Parent id nil Span id 1 Web Request Auth Ecommerce Orders Recommendations Billing

52 Trace id 123 Parent id nil Span id 1 Web Request Trace id 123 Parent id 1 Span id 2 Auth Ecommerce Orders Recommendations Billing

53 Trace id 123 Parent id nil Span id 1 Web Request Trace id 123 Parent id 1 Span id 2 Auth Ecommerce Trace id 123 Parent id 2 Span id 3 Orders Recommendations Billing

54 A Trace is many Parent - Child Relationships

55 Directed Acyclic Graph

56 Trace id 123 Parent id nil Span id 1 Web Request Trace id 123 Parent id 1 Span id 2 Auth Ecommerce Trace id 123 Parent id 2 Span id 3 Orders Recommendations Billing

57 Annotations: Gives us richer insights into our spans

58 Client Start 01:01:01 Server Receive 01:01:02 Server Send 01:01:03 Client Receive 01:01:04

59 Trace id 123 Parent id nil Span id 1 Web Request Trace id 123 Parent id 1 Span id 2 Auth Ecommerce Trace id 123 Parent id 2 Span id 3 Orders Recommendations Billing

60 Auth Trace id 123 Parent id 1 Span id 2 Client Send Server Receive Ecommerce

61 Auth Client Send Server Receive Server Send Ecommerce

62 Auth Client Send Client Receive Server Receive Server Send Ecommerce

63 Transporting the Data

64 Web Request Auth Transport Trace id 123 Parent id nil Span id 1 Collector Ecommerce Storage Orders Recommendations Billing

65 Propagates ids in band

66 Reports out of band

67 Viewing the Data

68 Web Request Auth Ecommerce Orders Recommendations Billing

69

70

71 All happening inside the resource allocation & provisioning

72 Blocking execution Parallel execution

73 A widening gap here could indicate queueing

74 Tracing Incoming Requests

75 Middleware

76 def call(env) trace end end Trace some stuff def trace(env, &block) end # tracing code

77 def call(env) trace end end def trace(env, &block) span = Span.new("authentication", generate_span_id) span.record(server_recv) Received a request status, headers, body = yield Execute the app ensure span.record(server_send) Sending back end to the client Non-pseudocode version:

78 # config/initializers/tracing.rb Rails.application.config.middleware.use TracingRackMiddleware, { # some configuration } Use our middleware!

79 # config/initializers/tracing.rb Rails.application.config.middleware.use TracingRackMiddleware, { service_name: "SERVICE_DOMAIN_NAME", service_port: 443, sample_rate: ENV.fetch("ZIPKIN_SAMPLE_RATE", 0.1).to_f, json_api_host: ENV["ZIPKIN_HOST"] } Sample a portion of requests

80 Tracing Outgoing Requests

81 More Middleware!

82 Faraday

83 def call(env) trace!(env) do end end def trace!(env, &block) # some tracing end Execute our http client

84 def call(env) trace!(env) do end end def trace!(env, &block) env = set_headers(env) Manipulate the headers span = Span.new("external_call", 1234) span.record(trace::annotation::client_send) status, headers, body = yield env ensure span.record(trace::annotation::client_recv) end Using client instead of server

85 Each of these colors represents an instrumented application

86 def call(env) trace!(env) do end end def trace!(env, &block) env = set_headers(env) span = Span.new("external_call", 1234) span.record(trace::annotation::client_send) status, headers, body = yield env ensure span.record(trace::annotation::client_recv) end Client Send Client Receive

87 def self.client Faraday.new(url: base_url) do connection connection.use TracingFaradayMiddleware connection.adapter Faraday.default_adapter end end Add our middleware

88 Checklist

89 Buy, Build, or Adopt

90 Buy?

91 Lightstep TraceView and more?

92 Adopt an OSS Solution?

93 Zipkin Jaeger

94 What about Open Tracing?

95 Standardizes Instrumentation

96 Where is OpenTracing at today?

97 Interoperability is Still Messy

98 Language Support

99 Rinse and Repeat

100 Build Your Own?

101 What are other folks doing?

102 End-to-End Tracing: Adoption and Use Cases Jonathan Mace, Brown University

103 15 using Zipkin 9 using internal solutions 1 using other OSS solution 1 using paid solution Jonathan Mace, Brown University

104 Infra Requirements and Limitations

105 Dependency matrix of: - Tracer - Transport Layer - Collection Layer - Storage Layer

106 Installing a Separate Agent

107 Authentication

108 Missing Authentication & Authorization

109 Client Authorization

110 Basic auth via htpsswd

111 by Corey Donohoe

112 Browser Authentication

113 bit.ly s Oauth2 proxy

114 by Corey Donohoe

115 Giving people access

116 Sensitive Data

117 Custom Instrumentation

118 What happens when data leaks?

119 Is Everyone On Board?

120 Get it on the Roadmap

121 Open PRs

122 Evaluating Distributed Tracing Solutions: Should you buy, build or adopt? What are your infrastructure requirements and limitations? How is it authenticated? Do you have sensitive data? What will you do if it leaks? Is everyone on board?

123 OMG, this is so much information

124 Try out Docker Zipkin

125 Thank you! Plant Illustrations designed by Natkacheva / Freepik