Inside Cloudbleed John Graham-Cumming

Transcription

1 Inside Cloudbleed John Graham-Cumming

2 A little about Cloudflare 2

3 3

4 What was Cloudbleed? 4

5 February 23, 2017 Our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines. 5

6 Don t leak random chunks of memory onto the Internet. 6

7 Don t have those chunks of memory cached by search engines. 7

8 The Night Of 8

9 0011 UTC February 18, 2017 (T-0021) 9

10 0032 UTC February 18, 2017 (T+0000) Received details of bug from Tavis Ormandy 10

11 0040 UTC February 18, 2017 (T+0008) Cross functional team assembled in Room 403 in San Francisco 11

12 0119 UTC February 18, 2017 (T+0047) Obfuscation feature disabled globally Project Zero confirm they no longer see leaking data 12

13 0122 UTC February 18, 2017 (T+0050) London team awake and online San Francisco actively fuzzing parsing code to look for additional problems Manual inspection for additional uses 13

14 0424 UTC February 18, 2017 (T+0352) Automatic HTTPS Rewrites disabled worldwide 14

15 0722 UTC February 18, 2017 (T+0650) Discovered we didn t have a kill switch for Server-Side Excludes At was an ancient feature that predated all the engineers on the team Implemented a global kill switch and redeployed Cloudflare NGINX Lua code globally 15

16 The Technical Cause 16

17 Unclosed HTML attribute at end of page Had to have one of Obfuscation Automatic HTTPS Rewrites Server-Side Excludes Page had to end with something like <script type= <IMG HEIGHT="50px" WIDTH="200px" SRC=" 17

18 18

19 <script type= script_consume_attr := ((unquoted_attr_char)* :>> (space '/' '>')) >{ ddctx("script consume_attr"); fhold; fgoto script_tag_parse; } $lerr{ dd("script consume_attr failed"); fgoto script_consume_attr; }; 19

20 <script type= script_consume_attr := ((unquoted_attr_char)* :>> (space '/' '>')) >{ ddctx("script consume_attr"); fhold; fgoto script_tag_parse; } $lerr{ dd("script consume_attr failed"); fgoto script_consume_attr; }; 20

21 <script type= script_consume_attr := ((unquoted_attr_char)* :>> (space '/' '>')) >{ ddctx("script consume_attr"); fhold; fgoto script_tag_parse; } $lerr{ dd("script consume_attr failed"); fgoto script_consume_attr; }; 21

22 <script type= script_consume_attr := ((unquoted_attr_char)* :>> (space '/' '>')) >{ ddctx("script consume_attr"); fhold; fgoto script_tag_parse; } $lerr{ dd("script consume_attr failed"); fgoto script_consume_attr; }; 22

23 <script type= script_consume_attr := ((unquoted_attr_char)* :>> (space '/' '>')) >{ ddctx("script consume_attr"); fhold; fgoto script_tag_parse; } p--; $lerr{ dd("script consume_attr failed"); fgoto script_consume_attr; }; 23

24 End of buffer check /* generated code */ if ( ++p == pe ) goto _test_eof; 24

25 How this is called ngx_int_t ngx_http_ _parse_ (ngx_http_request_ t *r, ngx_http_ _ctx_t *ctx) { u_char *p = ctx->pos; u_char *pe = ctx->buf->last; u_char *eof = ctx->buf->last_buf? pe : NULL; } 25

26 Irony: we caused this because we were migrating away from the buggy parser 26

27 Old parser only (gdb) p *in->buf $6 = { pos = 0x558a238e94f7 "<script type=\"", last = 0x558a238e9504 "", [...] last_buf = 0, [...] } 27

28 Old and new parsers present (gdb) p *in->buf $8 = { pos = 0x558a2f58be30 "<script type=\"", last = 0x558a2f58be3e "", [...] last_buf = 1, [...] } 28

29 Past the end of the buffer /* #line 877 "ngx_http_ _filter_parser.rl" */ { dd("script consume_attr failed"); {goto st1266;} } goto st0; [...] st1266: if ( ++p == pe ) goto _test_eof1266; 29

30 Good, Bad, Ugly 30

31 Good: stemmed the leak in 47 minutes 31

32 Bad: we had been leaking sensitive stuff Private key used to secure internal connectivity between our machines Some internal authentication secrets But also HTTP headers for requests to our customers websites (including cookies) POST data (passwords, credit card numbers, SSNs could have been present) URI parameters JSON blobs for API calls API authentication secrets, OAuth keys, 32

33 Large Wearable Manufacturer 33

34 Large Dating Website 34

35 Large Ride Hailing Service 35

36 Ugly: sensitive data had been cached by search engines Worked with Google, Bing, Yahoo, Yandex, Baidu, DuckDuckGo, and many others to remove cached data Found a total of 770 unique URIs across 161 unique domains with sensitive, cached data 36

37 Going Public 37

38 1900 UTC February 23,

39 1900 UTC February 23,

40 2300 UTC February 23,

41 February 24,

42 Impact Statistics 42

43 Estimated Instances of Data Leakage September 22, > February 13, ,307 February 13, > February 18, ,034 43

44 Based on data cached by search engines Each leak contained Internal Cloudflare Headers 0.44 Cookies 0.04 Authorization Headers / Tokens No Passwords, credit cards, SSNs 44

45 Impact by customer size Requests per Month Anticipated Leaks B 300B 22,356 33, B 200B 11,427 22,356 50B 100B 5,962 11,427 10B 50B 1,118 5,926 1B 10B 112 1, M 1B M 500M M 250M M 100M M 50M 1 6 <10M < 1 45

46 The Really Ugly Truth 46

47 This had been going on for months September 22, 2016 new parser) January 30, 2017 parser February 13, 2017 new parser Feburary 18, 2017 and leak is stopped Automatic HTTP Rewrites enabled (uses Server-Side Excludes migrated to new Obfuscation partially migrated to Google reports problem to Cloudflare 47

48 This had been going on for months Roughly 180 sites had flaw plus one of these two: September 22, 2016 parser) January 30, 2017 Around 6,500 sites had flaw plus: February 13, 2017 parser February 18, 2017 and leak is stopped Automatic HTTP Rewrites enabled (uses new Server-Side Excludes migrated to new parser Obfuscation partially migrated to new Google reports problem to Cloudflare 48

49 This had been going on for months 49

50 Core War 50

51 51

52 52

53 53

54 54

55 The bad value just seems to come out of nowhere ZOINKS! 55

56 Mysteries Instructions & memory state could not explain the register state Instructions & register state could not explain the memory state RIP does not point to a valid instruction Instruction & register state did not match the signal information 56

57 Infrequent mysteries On average, ~1 mystery core dump a day Scattered over all servers, all colos Per server, 1 in 10 years Can t reproduce Hard to try any potential fix 57

58 Core dump corruption? 58

59 Operating system bug? Bugs in the kernel code that controls virtual memory can lead to mysterious effects, but: No bugs reported upstream Couldn t find any bugs in the relevant code No correlation between the core dumps and relevant metrics 59

60 Hardware? 60

61 61

62 62

63 63

64 Rolled out new BIOS fleet wide on July 29,

65 Thoughts Be as transparent as possible Co-operate with security researchers Start taking notes immediately Over communicate Go back and see if hindsight was 20/20 65

66 Thank you 66