skb2 Shaon Barman, Sarah Chasins, Ras Bodik UC Berkeley Sumit Gulwani Microsoft Research

Transcription

1 skb2 Web Scripting for End Users Shaon Barman, Sarah Chasins, Ras Bodik UC Berkeley Sumit Gulwani Microsoft Research 1

2 Slide 1 skb2 I think the title should be changed since the project has expanded past just compensation events. Probably want something more in line with the goals of the project. Maybe something like "End user scripting for web browsers" Shaon Barman, 6/6/2013

3 skb3 Problem: Repetitive Tasks Web is filled with repetitive tasks Ex: data scraping, form filling So much data, so few APIs A simple sequence of webpage interactions is hard to automate No way to save page state mid navigation Use tools such as Selenium, imacros Few tools for non programmers 2

4 Slide 2 skb3 The phrasing is kinda funny on point 1. I would stress that "users" have repetative tasks on the web. I would add as bullet point 2 saying that its impossibly to bookmark the state of a website after interacting it with it. Shaon Barman, 6/6/2013

5 Solution: End User Scripting Record and replay for the browser Recording of user actions > synthesize a script Input: User demonstration of web interaction Output: Script to reproduce the interaction Accessible to end users 3

6 When can I swim?

7 How it works

8 DOM

9 DOM html head body div div div span span span h5 h5 ul 7

10 DOM

11 DOM

12 DOM div select option option option option (all) (Select Multiple) Building Hours Open Rec Badminton 10

13 Node div select option option option option (all) (Select Multiple) Building Hours Open Rec Badminton 11

14 Node select { iscontenteditable: false, value: 6, childelementcount: 15, Id: ctl00_pc_cs_eventtypes_ddl, } 12

15 mousedown MouseEvent DOM L3 A pointing device button (usually a mouse) is pressed on an element. mouseenter MouseEvent DOM L3 A pointing device is moved onto the element Event that has the listener attached. mouseleave MouseEvent DOM L3 A pointing device is moved off the element that has the listener attached. mousemove MouseEvent DOM L3 A pointing device is moved over an element. mouseout MouseEvent DOM L3 A pointing device is moved off the element that has the listener attached or off one of its children. mouseover MouseEvent DOM L3 A pointing device is moved onto the element that has the listener attached or onto one of its children. mouseup MouseEvent DOM L3 A pointing device button is released over an element. noupdateevent Offline The manifest hadn't changed. obsolete Event Offline The manifest was found to have become a 404 or 410 page, so the application i cache is being bi deleted. dl d offline Event HTML5 offline The browser has lost access to the network. online Event HTML5 offline The browser has gained access to the network (but particular websites might be unreachable). open Event WebSocket A WebSocket connection has been established. open Event Server Sent Events An event source connection has been established. orientationchange Event Screen Orientation The orientation of the device (portrait/landscape) has changed pagehide PageTransitionEvent HTML5 A session history entry is being traversed from. pageshow PageTransitionEvent HTML5 A session history entry is being traversed to. 13

16 Event 14

17 Event click { altkey: false, ctrlkey: false, x: 550, y: 178, charcode: 0, } 15

18 Event Usually fired by user actions Typing, using a mouse JavaScript dispatchevent() 16

19 click System Basics click click RECORD 17

20 click System Basics click click click click click RECORD REPLAY 18

21 click System Basics click dispatchevent() click click dispatchevent() click click dispatchevent() RECORD REPLAY 19

22 Challenges Browser imperfections Ex: User generated and JS generated events treated differently Non determinism in scheduling Ex: Server delays

23 click Browser Imperfections click dispatchevent() click click dispatchevent() click click dispatchevent() RECORD REPLAY 21

24 Solution: Synthesis Calculate difference between execution during record and replay In record time browser, find the difference in DOM state before and after each event Repeat for the replay time browser Check if all effects of the event in the record time browser occur also in replay time browser If unmatched effect, synthesize compensation function Record Time Replay Time { value : } { value : } { value : } { value : } Before Event After Event Before Event After Event 22

25 Why Synthesize? Too many compensation events Cannot write all by hand before users need them Changing web > newly required events End users Spec is available Demonstration Common form makes partial program available 23

26 Sample Compensation Event What do we want to create? when should { we run this correction? what do we do at replay time? "clickoption": } guard: function(node, eventmessage) { return eventmessage.nodename nodename == "option" && eventmessage.type == "click"; }, replay: function(node, eventmessage) { } node.selected = true; 24

27 click { altkey: false, ctrlkey: false, x: 550, y: 178, charcode: 0, } Given The Players div select option option option o (all) (Select Multiple) Event message, em DOM state before em is recorded, rec 0 DOM state after em is recorded, rec 1 DOM state before em is replayed, rep 0 DOM state after em is replayed, rep 1 Building Hours Op Bad 25

28 Program Skeleton when should we run this correction? what do we do at replay time? { compensation_event_name": t { guard: function(node, eventmessage) { return eventmessage[ nodename ] == "option" && eventmessage[ type ] == "click"; }, replay: function(node, eventmessage) } } node[ selected ] = true; 26

29 Program Skeleton compensation_event_name": t { guard: function(node, eventmessage) } { return eventmessage[ nodename ] == "option" && eventmessage[ type ] == "click"; }, replay: function(node, eventmessage) { } node[ selected ] = true; 27

30 Program Skeleton compensation_event_name": t { guard: function(node, eventmessage) { return eventmessage[ nodename ] == em[ nodename ] eventmessage[ type ] == em[ type ] ; }, replay: function(node, eventmessage) } { } node[ selected ] = true; && 28

31 Program Skeleton compensation_event_name": t { guard: function(node, eventmessage) { return eventmessage[ nodename ] == em[ nodename ] eventmessage[ type ] == em[ type ] ; }, replay: function(node, eventmessage) } { } node[ divergingprop1 ] = true; node[ divergingprop2 ] = true;... && 29

32 Sketch function replay(element, eventmessage){ element[divergingprop1] =??; element[divergingprop2] =??; } Sketch by synthesizer designer Holes filled with expressions Prop ::= element[p] eventmessage[p] Val ::= Prop Constant Expr ::= Val Combine(Expr, Expr) SubStr(Expr) IfThenElse(Expr, Expr, Expr) Equals(Expr, Expr) RHS ::= Expr Recorded 30

33 Results of Synthesis element[ value ] = e element[ value ] = Combine(element[ value ], String.fromCharCode(eventMessage[ keycode element[ value ] = Combine(element[ value ], String.fromCharCode(eventMessage[ charcod Live, online synthesis

34 Modeling the Browser Record and replay breaks down Want to understand why? Create a model of the browser Simulate interaction between server and browser Simulate pieces within a browser Analyze orthogonal problems in browser Allows us to visualize a recording

35 Visualization time server script dom event loop user view mousedown mouseup click execute listener request reply execute handler update view

36 Limited Observability time server script dom event loop user view mousedown mouseup click execute listener request reply execute handler update view

37 Replay Challenges Which event to replay? Sometimes browser will automatically raise an event in response to another event When to replay event? May need to wait for a response from the server Can we tell when a site will replay correctly?

38 Questions?

39 Example

40 Example

41 Example

42 Example 6:15 9:00 AM Hearst OpenRec Swim 11:20 2:00 PM Hearst Open Rec Swim 4:15 8:00 PM Hearst Open Rec Swim

43 Example 6:15 9:00 AM Hearst OpenRec Swim 11:20 2:00 PM Hearst Open Rec Swim 4:15 8:00 PM Hearst Open Rec Swim Fandango Movies showing after 7:30 today at my favorite theater Google Maps Commute time in current traffic UPS Package delivery status ESPN Basketball score Local Bus Site Next arrival time of my bus

44 Example Berkeley Calendar Swim schedule for Hearst Pool Fandango Movies showing after 7:30 today at my favorite theater Google Maps Commute time in current traffic UPS Package delivery status ESPN Basketball score Local Bus Site Next arrival time of my bus

45 Challenge: Browser Imperfections Browser handles userraised and JS raised events differently Replay time cloned events fail Must make DOM state match th mousedown { altkey: false, input ctrlkey: { false, charcode: altkey: 116, false, keypress keycode: ctrlkey: { 116, false, char: charcode: t, altkey: 116, false, keycode: ctrlkey: 116, false, } char: charcode: t, 116, keycode: 116, } char: t, } 43

46 Optimizations candidate 1 candidate 2 candidate 3 eventmessage[ a ] eventmessage[ b ] eventmessage[ c ] l la lan l la lan true false true ex 1 ex 2 ex 3 ex 1 ex 2 ex 3 ex 1 ex 2 ex 3 target candidate 1 combine candidate 2 combine Language Design Language Des i JavaScri i