Brian Dodds, Craig Ross Facebook

Transcription

1

2 Hardware Lifecycle at Scale Brian Dodds, Craig Ross Facebook

3 Agenda 1 Facebook s Infrastructure Evolution 2 Hardware Lifecycle 3 Learnings 4 Wrap Up

4 Facebook's Infrastructure Evolution

5 Facebook s Growth M B Intro Acquisition B 200M 200M Acquisition B 900M 500M 1B

6 Facebook s Scale Today Each Day: Billions of photo and video uploads Trillions of user requests Tens of trillions of database queries 100s of trillions of cache queries Huge demands on servers, storage, network, and power

7 Why Build Our Own Hardware? Advantages Faster response to growth demands Optimize end-to-end (Application->Power->Thermal) Highest Operational Efficiency Commodity components Be Open

8 The Facebook Datacenter

9 2010 Infrastructure Evolution 2011 Hardware PRN Open Project Launch 2012 FRC 2013 Hardware Storage LLA 2014 ATN 2015 Hardware Network 2016 Fabric FTW, CLN

10 2010 Hardware Evolution 2011 Freedom Rack & Power Freedom triplet 2012 Windmill 2013 Winterfell Storage Knox Rack & Power Open Rack V Rack & Power Open Rack V Leopard Storage Honey Badger Network Switch Wedge 2015 Storage BluRay 2016 Yosemite GPU Big Sur Network Back Pack Storage Lightning

11 Facebook Datacenters

12 Hardware Lifecycle Scale

13 Hack Design Build Deploy Sustain Decom

14 Hack Design Build Deploy Sustain Decom

15 Hack Design Build Deploy Sustain Decom

16 Hack Design Build Deploy Sustain Decom

17

18 Hack Design Deploy Sustain Decom Build Chassis Level Assembly Rack Assembly (in Region) Data Centers

19 Hack Design Deploy Sustain Decom Build Component Level Manufacturing Chassis + Rack Level Assembly (in Region) Data Centers

20 Hack Design Build Deploy Sustain Decom

21

22 Hack Design Build Deploy Sustain Decom

23 Hack Design Build Deploy Sustain Decom

24 Learnings

25 2010 Hardware Evolution 2011 Freedom Rack & Power Freedom triplet 2012 Windmill 2013 Winterfell Storage Knox Rack & Power Open Rack V Rack & Power Open Rack V Leopard Storage Honey Badger Network Switch Wedge 2015 Storage BluRay 2016 Yosemite GPU Big Sur Network Six Pack Storage Lightning

26 2010 Learnings - Sensors 2011 Freedom Rack & Power Freedom triplet Issues: BMC and PSU monitoring woes 2012 Windmill Learnings: Improve monitoring of critical sensors Winterfell Storage Knox Rack & Power Open Rack V Rack & Power Open Rack V Leopard Storage Honey Badger Network Switch Wedge 2015 Storage BluRay 2016 Yosemite GPU Big Sur Network Six Pack Storage Lightning

27 Learnings Supply Chain/Application Freedom Rack & Power Freedom triplet Issues: Single-sourced epidemic failure. App performance issues. Row Hammer Windmill Learnings: Multi-source components, robust app scale, improve component 2013 Winterfell Storage Knox monitoring. Rack & Power Open Rack V Rack & Power Open Rack V Leopard Storage Honey Badger Network Switch Wedge 2015 Storage BluRay 2016 Yosemite GPU Big Sur Network Six Pack Storage Lightning

28 2010 Learnings DC Tooling 2011 Freedom Rack & Power Freedom triplet Issues: Shipped hardware before all tooling was finished Idle HW Windmill Learnings: Make tooling a firstclass citizen for phase exit Winterfell Storage Knox Rack & Power Open Rack V Rack & Power Open Rack V Leopard Storage Honey Badger Network Switch Wedge 2015 Storage BluRay 2016 Yosemite GPU Big Sur Network Six Pack Storage Lightning

29 Hardware Eventually Fails

30 Monitor Alarm Robust Infrastructure Design Feedback Remediate

31 Monitor Alarm Robust Infrastructure Design Feedback Remediate

32 Monitoring Many servers, components, services, and regions

33 Monitoring Failure Rate

34 Monitoring Error Types

35 Monitoring Filters

36 Monitor Alarm Robust Infrastructure Design Feedback Remediate

37 Alarms Anomaly Detection Gradual Increases And Sudden Spikes Anomaly Within Cohorts

38 Monitor Alarm Robust Infrastructure Design Feedback Remediate

39 Remediation The Journey is 1% Finished Phase 1: Root Cause Analysis Phase 2: Review Remediation Plan Phase 3: Implement Remediation 40% 30% 20% 10% 0% Mar 2014 Jul 2014 Nov 2014 Mar 2015 Jul 2015 Now

40 Monitor Alarm Robust Infrastructure Design Feedback Remediate

41 Design Improvements HDD Slot Temperature vs. Swap Rate Higher temps. More swaps.

42 Wrap Up

43 Key takeaways FB scale is growing. Infrastructure needs to innovate Move fast and adapt with robust HW lifecycle Everything fails minimize impact with tooling

44