Logging in the age of

Transcription

1 Logging in the age of and the Cloud

2 POLL: what type of infrastructure are you running on? On Premise Colocation Root Server Cloud

3 The (good) old days of logging

4 ssh LOG file tail -f server.log

5 Looks great!

6 @axelfontaine Thanks! boxfuse.com

7 ssh LOG file tail -f server.log

8 Times have changed

9 The new reality Cloud Microservices

10 Cloud

11 moving to the cloud

12 lift & shift (= the naïve approach)

13 Congratulations! You now have: lift & shift (= the naïve approach) Lots of (too much?) trust in your cloud provider + legal trouble due to data privacy laws A more expense Hetzner/OVH Potential data loss when auto-scaling

14 Fundamental changes 1. Security 2. Cost-driven 3. Auto-scaling

15 1. Security

16 understanding the cloud

17 regions

18 availability zones <<IMAGE GERMANY + two small clouds with racks>>

19 building blocks

20 building blocks Storage Compute Network Security

21

22 The hard Truth about Security 1. Always breakable with infinite time & resources 2. Must make it more complicated/expensive to break than it s worth (use defense in depth!) 3. Has a usability cost 4. Almost always about the data

23 the 3 states of data Data at Rest Data in Use Data in Motion

24 Trusting your neighbors is good. But it s even better to put a good lock on the door. Werner Vogels CTO of a large online book shop

25 Data in Motion No excuse in 2017! 100% free and automated With Let s Encrypt and AWS Certificate Manager TLS / SSL

26 Client-side encryption Data in Use & at Rest

27 2. Cost-driven

28 Compute

29 Spare Capacity = Wasted Money (paying for something you don t use)

30 Scaling = Adjusting capacity in response to a metric exceeding a threshold

31 Scaling = Corrective action in response to an alarm

32 scaling metrics for different types of services sync => CPU load async => queue depth cron => time

33 scaling up in out down

34 3. Auto-scaling

35 Auto-Scaling = automated alarms + automated corrective actions (scaling in or out)

36 Load Balancer LOG file LOG file LOG file ssh tail -f server.log ssh tail -f server.log ssh tail -f server.log Scale Out Scale In CPU Load

37 Load Balancer LOG file LOG file LOG file ssh tail -f server.log ssh tail -f server.log ssh tail -f server.log Scale Out Scale In LOG file ssh tail -f server.log CPU Load

38 Load Balancer LOG file LOG file LOG file ssh tail -f server.log ssh tail -f server.log ssh tail -f server.log Scale Out Scale In LOG file ssh tail -f server.log CPU Load

39 Load Balancer LOG file LOG file LOG file ssh tail -f server.log ssh tail -f server.log ssh tail -f server.log Scale Out Scale In LOG file ssh tail -f server.log CPU Load

40 LOG file ssh tail -f server.log Load Balancer LOG file LOG file DATA LOSS ssh tail -f server.log Scale Out Scale In LOG file ssh tail -f server.log CPU Load

41 LOG file Load Balancer log server LOG file LOG file where logs can be aggregated stored and backuped indexed searched

42 Many options: Logstash (ELK) AWS CloudWatch Logs Loggly Papertrail log server where logs can be aggregated stored and backuped indexed searched

43 Microservices

44 POLL: what type of architecture does your software have? Integrated (Monolith) Distributed (Microservices)

45 Why are we logging? Postmortem analysis of user activity and programming errors Powerful debugging tool Should contain answers to important questions: What? Who? Where? When?

46 What? Who? Where? When?

47 What? Who? Where? When? Message, Code, Severity

48 What? Who? Where? When? Message, Code, Severity Account, User, Session, Request

49 What? Who? Where? When? Message, Code, Severity Account, User, Session, Request App, Module, Class

50 What? Who? Where? When? Message, Code, Severity Account, User, Session, Request App, Module, Class Timestamp, Hostname, PID, Thread How can all this information be captured? How can these questions be asked?

51 Capturing log info

52 Logging framework architecture Your Code Logger Appender A Appender B Storage A Storage B

53 Logging framework architecture Your Code Logger File Appender Log Server Appender Text File(s) Log Server

54 logger.info( my log message ); What? Who? Where? When? Message, Code, Severity Account, User, Session, Request App, Module, Class Timestamp, Hostname, PID, Thread

55 logger.info( my log message ); What? Who? Where? When? Message, Code, Severity Account, User, Session, Request App, Module, Class Timestamp, Hostname, PID, Thread

56 logger.info( my log message ); What? Who? Where? When? Message, Code, Severity Account, User, Session, Request App, Module, Class Timestamp, Hostname, PID, Thread

57 Mapped Diagnostic Context (Thread-local temporary key-value store) Your Code MDC Logger Appender A Appender B Storage A Storage B

58 MDC.put( account, company ABC ); MDC.put( user, user123 ); What? Who? Where? When? Message, Code, Severity Account, User, Session, Request App, Module, Class Timestamp, Hostname, PID, Thread

59 MDC.put( account, company ABC ); MDC.put( user, user123 ); logger.info( my log message ); What? Who? Where? When? Message, Code, Severity Account, User, Session, Request App, Module, Class Timestamp, Hostname, PID, Thread

60 MDC.put( account, company ABC ); MDC.put( user, user123 ); Populate when: a request enters the application a message is received from a queue an async or cron task starts And don t forget the clear when done! (Threadpools reuse threads!)

61 Querying the logs

62 grep?

63 Truncation! Compression! Single line messages! No MDC info!

64 Decoupling log storage from log representation

65 MDC Appender FORMAT Storage (formatted) Your Code Logger READ Log Viewer

66 Your Code MDC Logger Appender Storage (raw) READ & FORMAT Log Viewer

67 Structured logging

68 { } "account": "axelfontaine", "image": "axelfontaine/xyz:543", "instance": "i-0d843d5af9b366a69", "level": "INFO", "logger": "com.myapp.task.taskservice", "message": "Successfully killed axelfontaine/demo in prod", "request": "crq-7r2cvpumkreuflmque3xb7jwcx", "session": "cli-crfm2ipabrfujd7ktdyvdvxabx", "thread": "Thread-18710", "timestamp": " T10:20:30.444"

69 Machine-readable logs

70 Machine-readable logs Machine-queryable logs What? Who? Where? When? Message, Code, Severity Account, User, Session, Request App, Module, Class Timestamp, Hostname, PID, Thread

71 AWS CloudWatch Logs

72 { $.account = axelfontaine && $.request = crq }

73 log server

74 Querying across systems

75

76

77 Propagating MDC A B C Create MDC (based on session) and assign unique request ID Copy MDC to HTTP(S) headers Read MDC HTTP(S) headers Copy MDC to HTTP(S) headers Read MDC HTTP(S) headers

78 Propagating MDC A B C Filter Decorator Filter Decorator Filter

79 Standardized keys

80 { } "account": "axelfontaine", "image": "axelfontaine/xyz:543", "instance": "i-0d843d5af9b366a69", "level": "INFO", "logger": "com.myapp.task.taskservice", "message": "Successfully killed axelfontaine/demo in prod", "request": "crq-7r2cvpumkreuflmque3xb7jwcx", "session": "cli-crfm2ipabrfujd7ktdyvdvxabx", "thread": "Thread-18710", "timestamp": " T10:20:30.444"

81 Standardized values

82 { } "account": "axelfontaine", "image": "axelfontaine/xyz:543", "instance": "i-0d843d5af9b366a69", "level": "INFO", "logger": "com.myapp.task.taskservice", "message": "Successfully killed axelfontaine/demo in prod", "request": "crq-7r2cvpumkreuflmque3xb7jwcx", "session": "cli-crfm2ipabrfujd7ktdyvdvxabx", "thread": "Thread-18710", "timestamp": " T10:20:30.444"

83 Summary Send your logs to a centralized service Ensure your logs are structured Use and propagate MDC Standardize keys and values Query your logs to answer the what, who, where, when questions

84 About Axel Fontaine Founder and CEO of Boxfuse Flyway creator Continuous Delivery & Immutable Infrastructure expert Java Champion, JavaOne

85 Evolve your relational database schema reliably across all your instances Supports all popular RDBMS flywaydb.org Open-source (with commercial support available) Millions of users

86 Deploy JVM (Spring Boot, Dropwizard, Tomcat, TomEE,...), Node.js and Go apps effortlessly to AWS Immutable infrastructure with minimal images just 1% of size of regular OS (think Linux x64 kernel + your app) Zero downtime orchestration on AWS (atomic blue/green deployments) boxfuse.com First-class support for centralized, structure and standardized logging with AWS CloudWatch Logs

87 boxfuse.com flywaydb.org