The Gorilla Approach to Scaling & Integrating Open Source Forensic Tools: Vassil Roussev

Size: px

Start display at page:

Download "The Gorilla Approach to Scaling & Integrating Open Source Forensic Tools: Vassil Roussev"

Neal Harmon
5 years ago
Views:

1 The Grilla Apprach t Scaling & Integrating Open Surce Frensic Tls: Vassil Russev

2 Trends in Frensic Analysis Data Vlume Data Cmplexity Deadlines Human Resurces time 2

3 The 4-way Scalability Challenge Data Scalability Extensibility Cst Scalability UI Scalability 3

Data Scalability Nw http://accessdata.cm/distributed-prcessing: Impressive Test Results! In testing, AccessData fully prcessed a massive data set, including 62,649,383 items, [.

4 Data Scalability Nw Impressive Test Results! In testing, AccessData fully prcessed a massive data set, including 62,649,383 items, [...] The cmpressed size f this data set was. [ ] Hwever with AccessData s distributed prcessing technlgy, it nly tk 6 days, 5 hurs. 1.28TB zip ~ 3TB raw 3TB / 129hrs = 23.5MB/s T prc = 5.25 T HDDclne (@130MB/s) T prc = 25.5 T SSDclne (@600MB/s) 4

Cst Scalability Nw Cmmercial Integrates licensed cmpnents Oracle, dtsearch, There is a price flr Per-CPU pricing 2x CPUs 2x $$$ User cannt keep up n a fixed budget.

5 Cst Scalability Nw Cmmercial Integrates licensed cmpnents Oracle, dtsearch, There is a price flr Per-CPU pricing 2x CPUs 2x $$$ User cannt keep up n a fixed budget. Open Surce Nminally free, BUT Tls prly integrated Tl chain incmplete Extra cst/resurces Higher tech expertise Develpment/integrati n csts Testing/validatin N budget guarantees. 5

6 Extensibility Nw Cmmercial Sme custm scripting/dll integratin E.g., EnCase Nt a platfrm Hw d yu extend a black bx? and test it? N incentive fr change. Open Surce Extensibility unlimited, BUT integratin is a huge burden N cmmn platfrm N cmmn data stre Only a few languagecentric APIs Integratin = scripting Need a new apprach. 6

7 Analytical UI Nw It s WIMP wrld EnCase, FTK, X- Ways, pyflag, Des NOT scale Mre data mre UI data Offlads prblem t user Des NOT supprt cgnitive prcess

Frensics is a niche market, this is nt sustainable.

8 The 80/20 Rule At least 80% f frensic prcessing is NOT frensic-specific. S far, we act as if the ppsite is true! Frensics is a niche market, this is nt sustainable. Shuld we a) Cntinue n the current path, r b) Lk arund fr slutins frm ther areas? 8

9 Lessns frm the Internet (1) Data scalability: ACID vs. BASE BASE scales much better Think Ggle, Amazn, Facebk, Twitter, etc. ACID is expensive: ~20x slwer Frensics des NOT need ACID By definitin, all prcessing must be repeatable! Cst scalability: Build an cmmn infrastructure; add prprietary cmpnents n tp We can use the same data stres that Big Data cmpanies d. FREE! 9

10 Lessns frm the Internet (2) Extensibility Simple data-exchange prtcls wrk JSON, Thrift, PrtBuff, XML desn t (t much verhead) The ne-size-fits-all mdel is falling apart Perfrmance/scalability Impedance mismatch Schemaless data mdels fit better than relatinal nes Hadp, MngDB, Cassandra, CuchDB, Redis, Hbase, 10

11 Lessns frm the Internet (3) UI extensibility Big data is driving data analytics and visualizatin Plenty f pen tls are available nw The brwser is replacing the desktp; whlesale! Standards wn t need prprietary extensins (Flash, Silverlight) HTML5, CSS, JS, WebGL D we really need hme grwn WIMP interfaces fr frensics? 11

12 A Slutin Sketch 12

Time Analysis Higher-rder services Cre Infrastructure Data Imprt Service Mgmt

13 User Interface Brwser-based GUI Navigatin/Query Visualizatin Case Mgmt Services Hashing Indexing Carving Zer-/first-rder services Hash Filter Data Crrelatin Time Analysis Higher-rder services Cre Infrastructure Data Imprt Service Mgmt Scheduling Data Stre Metadata Stre Raw data flw REST/JSON (meta-)data flw Cntrl flw 13

14 Quick Dem 14

15 Summary We need an pen and scalable frensics infrastructure t facilitate: Develpment, instructin, & field wrk; Research, testing, and validatin. Current appraches d nt wrk: Cmmercial: fragmented, expensive, mypic O/S: fragmented, incmplete, nt ready fr prime time We shuld lk t the Web fr Big Data answers 80% f the frensics is nt unique We share prblems/requirements New, rbust technlgy is freely available Need t adpt web-centric standards 15

16 Thank Yu! Q & A Cntact Vassil Russev vassil@cs.un.edu Cme t New Orleans/DFRWS 11 (Jul 31 Aug 3) dfrws.rg 16

The Current Forensic Workflow

The Current Forensic Workflow Vassil Russev The Current Frensic Wrkflw Frensic Target (3TB) Clne @150MB/s ~5.5 hrs Prcess @10MB/s ~82.5 hrs 129 * We can start wrking n the case after 88 hurs. * http://accessdata.cm/distributed-prcessing