CodeTickler: Automated Software Testing as a Service. Cris%an Zamfir, Vitaly Chipounov, George Candea

Transcription

1 CodeTickler: Automated Software Testing as a Service Cris%an Zamfir, Vitaly Chipounov, George Candea

2

3 Wouldn t it be nice to have reliable software?

4 Vision Machines should find corner cases do tricky security testing generate test suites Developers (a.k.a. sentient beings) should build cool new software tell machines how software should behave

5 Outline Status quo Our approach The road ahead

6 Outline Status quo Our approach The road ahead

7 Status Quo Automate the running of tests do not automate test generation Automatically finding bugs static analysis false positives random fuzzing inefficient at finding corner cases white-box fuzzing hard to use, requires models of the environment ZZUF SAGE KLEE

8 Software bugs cost US economy $59B/year * 50% of development budgets go to testing * * US National Institute of Standards and Technology

9 Outline Status quo Our approach The road ahead

10 program binary CodeTickler test report

11 Technology No false positives Higher quality testing Tests lots of types of binaries Source code not required Scales in the cloud CodeTickler No upfront costs

12 rpm {0,350,944,1200,1800} rpm=1200 autoshift (int rpm) if (rpm > 1000) gear = gear+1 rpm = 0.5*rpm if (rpm < 700) gear=0 return False rpm > 1000 True gear = gear+1 rpm = 0.5*rpm rpm < 700 rpm < 700 False True False True return gear=0 return return gear=0 return

13 Symbolic Execution autoshift (int rpm) if (rpm > 1000) gear = gear+1 rpm = 0.5*rpm if (rpm < 700) gear=0 return rpm {0,350,944,1200,1800} rpm=λ Z False λ 1000 False < rpm < 700 λ Z rpm > 1000 λ Z λ<700 λ 1000 λ>1000 True False 0.5*λ 700 λ>1000 < < λ Z True < < < λ Z rpm < 700 True 0.5*λ<700 λ>1000 λ Z < < λ Z

14 Finding Bugs Automatically autoshift (int rpm) if (rpm > 1000) gear = gear+1 rpm = 0.5*rpm if (rpm < 700) gear=0 return False λ 1000 False < rpm < 700 λ Z rpm > 1000 λ Z λ<700 λ 1000 λ>1000 True False 0.5*λ 700 λ>1000 < < λ Z True < < < λ Z rpm < 700 True 0.5*λ<700 λ>1000 λ Z < < λ Z Constraint solver λ=1200

15 Developer: one test CodeTickler: many tests Generated tests contain: program inputs system events thread schedules faults

16 in vitro in vivo real environment mockup environment program (libraries, operating sytem, drivers, etc.) program

17 real environment program environment program environment program program paths 2 system size

18 real environment mockup environment program

19 mockup environment program environment program environment program program paths 2 program size

20 real environment program environment program environment program program

21 The S 2 E Platform d s k s applica ons libraries opera ng system kernel virtual CPU dynamic binary transla on real CPU QEMU VM phys memory LLVM real phys memory drivers virtual devices symbolic execu on KLEE real devices analysis Runs unmodified interface x86 binaries (incl. proprietary/obfuscated/ encrypted binaries) Customized user-defined analyzers virtual machine 2 S E stock Selection analyzers done at runtime Most code runs natively

22 Summary Test any binary program In-vivo analysis No false positives From lab to real world running at any layer of the software stack no modeling of the environment generates tests that developers can run ASPLOS 11, TOCS 12, USENIX 10

23 Outline Status quo Our approach The road ahead

24 Which Software Needs Fixing? Software that runs at the highest privilege level CVE : USB kernelmode drivers in Microsoft Windows 8 allow attackers to execute arbitrary code CVE : Buffer overflow in the macvtap Linux device driver CVE : Stack-based buffer overflow in the Broadcom wireless device driver used in Cisco Linksys. Third-party software Third-party testing is increasingly more important. (Veracode) Closed-source software is more vulnerable to backdoors (Bruce Schneier)

25 Goal Get rid of low-level bugs in device drivers Target binary of closed-source software Open service that everyone can use Started building service for device drivers

26 Device driver Intel 82801AA AC97 Atheros WiFi Broadcom NetLink Gigabit Undisclosed network driver Intel Pro/1000 3Com EtherLink Server 3Com Ethernet Silicom FastEthernet NDIS subsystem RTL8029 Winbond PCI Ethernet Linksys Gigabit Ethernet Ensoniq AudioPCI AMD PCNet Bug type Race condition Kernel crash (Blue screen of death) Kernel crash (Blue screen of death) Kernel crash (Blue screen of death) Kernel crash (Blue screen of death) Kernel crash (Blue screen of death) Kernel crash (Blue screen of death) Kernel crash (Blue screen of death) Kernel crash (Blue screen of death) Multiple kernel crashes and resource leaks Concurrency bug Incorrect use of API Multiple kernel crashes and resource leaks Multiple resource leaks

27 Next Challenges Usability Scale Provenance SaaS We want feedback on these.

28 Usability We want to improve developer productivity Challenges integrate with software development processes produce easy-to-understand test reports integrate with IDEs

29 Scale Re-use test results libraries frameworks Big-data problem CodeTickler GBs of data for each device driver

30 Provenance How to identify when bad guys are using the service?

31 SaaS? binary CodeTickler test results How to securely store program binaries?

32 Conclusion CodeTickler open, automated cloud-based testing service First step get rid of low-level bugs in device drivers Join us Make software testing as easy as tweeting!