CrowdCode: A Platform for Crowd Development

Transcription

1 CrowdCode: A Platform for Crowd Development Thomas D. LaToza 1, Eric Chiquillo 1, 2, W. Ben Towne 3, Christian M. Adriano 1, André van der Hoek 1 1 University of California, Irvine 2 Zynga 3 Carnegie Mellon University sdcl.ics.uci.edu

2 What if so:ware could be built by a crowd? Decomposing tasks (hours - days) into microtasks (seconds - to minutes) increases parallelism, reducing -me to market. Could 1,000,000 developers build a large applica?on in a day? task microtasks Lowering joining costs exploits the long tail of poten?al contributors. Could a developer join a project, and immediately contribute? 2

3 Could this work? Let s find out 3

4 Microtasking programming Workers work on a single func?on or test at a?me, decomposing tasks to implement a feature or fix a bug into many microtasks that can be done in parallel by the crowd. 4

5 Self- contained microtasks Microtasks are designed to provide self- contained, well- defined tasks, including all informa?on necessary, allowing transient workers to login and immediately begin work. 5

6 Automa-c task genera-on System tracks the state of each ar?fact, determine work to be done, generates microtasks. Func-on state machine Edit code Debug Debug described Write description described written Edit code Run tests described written buggy Edit code Edit code described written buggy Workers login to system and are automa-cally assigned a microtask. 6

7 Key simplifica-ons Work begins with a set of user stories (scenarios) specified by client which do not change. Each user story can be tested by a set of tests of a main() func?on. Func?ons are completely specified by their inputs and outputs. (e.g., a library) Func?ons do not mutate global state or interact with environment (e.g., write output). All bugs can be detected through unit tests. Programs are wriwen in a (basic subset of) Javascript (e.g., no callbacks). Programming tasks are to implement a feature, fix a bug, write tests. All design is done locally and itera?vely (e.g., through refactoring). Workers are mo-vated by pay or reputa?on and not malicious. ==> crowdsourcing the programming of func?onal Javascript libraries 7

8 Demo 8

9 Challenges crowdsourcing so:ware development How does the system generate microtasks? How can microtasks be done concurrently? 9

10 How does the system generate microtasks? Exis?ng approaches to crowdsourcing complex work rely on a fixed sequence of steps. e.g., map reduce approach So_ware development is dynamic, cannot enumerate tasks in advance Each ar?fact has aoributes describing its state. Submi`ng microtask may change a func?on s awributes, genera-ng microtasks. wriwen (no pseudocode) described (has a func?on descrip?on) buggy (fails test) Func-on state machine described Write description Edit code described written Edit code Debug Debug Run tests described written buggy Edit code If code changes, run tests, unless there s pseudocode. Edit code described written buggy 10

11 How can microtasks be done concurrently? So_ware development involves dependencies between ar?facts; as an ar?fact changes, others may also need to change. All ar?facts in the system may be changing concurrently. Only one microtask may be ac?ve per ar?fact, preven?ng merge conflicts. Events propagate changes (signature, tests) across dependencies. Microtasks may check out a readonly copy of global interfaces, but may only commit a change to a single ar?fact. Events received on an ar?fact queue microtasks to be done. Edit function e Edit function a [interface change] Debug function c Queued: Interface change Edit function f Debug function b [new test] Edit function d Queued: debug 11

12 Current status Ran a pilot study, crowdsourcing a small (~500 line?) checkers program Revealed bugs, usability issues, need for data structures. 12

13 Emergent design: aggrega-ng conflic-ng local views Mul?ple func?ons may call the same func?on. What happens when their expecta?ons conflict? ==> edit war, as func?ons repeatedly changed to conform to conflic?ng tests Aggrega?on through discussion threads, bringing in relevant ar?facts Func?on has a discussion thread visible to func?on, tests, callers. Workers are transient, so can t reference workers (e.g., they might leave). to reference ar?facts in discussion, microtask assigned to followup. work by Lucinda Lim 13

14 Conclusions Programming tasks can be decomposed into microtasks at a func-on granularity. Challenges involve ensuring microtasks are self- contained and can be done in parallel. Open ques?ons Decomposi?on at what granularity? Smaller - > more parallel, less entry barrier; larger - > less overhead How much context is necessary? How much background about the system is necessary? Tradeoffs between pulling informa?on (Q&A) vs. pushing (reviews) What s the role of design? Can so_ware be built en?rely modularly? Does this increase the duplica?on and conflicts within the system? 14

15 Ques-ons 15

16 Backup 16

17 17

18 18

19 19

20 20

21 21

22 22

23 Fixing a bug 23