A Sense of Time for Node.js: Timeouts as a Cure for Event Handler Poisoning

Transcription

1 A Sense of Time for Node.js: Timeouts as a Cure for Event Handler Poisoning Anonymous Abstract The software development community has begun to adopt the Event-Driven Architecture (EDA) to provide scalable web services. Though the Event-Driven Architecture can offer better scalability than the One Thread Per Client Architecture, its use exposes service providers to a Denial of Service attack that we call Event Handler Poisoning (EHP). This work is the first to define EHP attacks. After examining EHP vulnerabilities in the popular Node.js EDA framework and open-source npm modules, we explore various solutions to EHPsafety. For a practical defense against EHP attacks, we propose Node.cure, which defends a large class of Node.js applications against all known EHP attacks by making timeouts a first-class member of the JavaScript language and the Node.js framework. Our evaluation shows that Node.cure is effective, broadly applicable, and offers strong security guarantees. First, we show that Node.cure defeats all known language- and frameworklevel EHP attacks, including real reported EHP attacks. Second, we evaluate its costs, finding that Node.js incurs reasonable overheads in micro-benchmarks, and low overheads ranging from 1.02x (real servers) to 1.24x (middleware). Third, we show that Node.cure offers strong security guarantees against EHP attacks for the majority of the Node.js ecosystem. I. INTRODUCTION Web services are the lifeblood of the modern Internet. To handle the demands of millions of clients, service providers replicate their services across many servers. Since every server costs time and money, service providers want to maximize the number of clients each server can handle. Over the past decade, this goal has led the software community to seriously consider a paradigm shift in their software architecture from the One Thread Per Client Architecture (OTPCA) used in Apache to the Event-Driven Architecture (EDA) championed by Node.js 1. Although using the EDA may increase the number of clients each server can handle, it also exposes the service to a new class of Denial of Service (DoS) attack unique to the EDA: Event Handler Poisoning (EHP). The OTPCA and the EDA differ in the resources they dedicate to each client. In the OTPCA, each client is assigned a thread, and the overhead associated with each thread limits the number of concurrent clients the service can handle. In the EDA, a small pool of Event Handler threads is shared by all clients. The per-client overhead is thus smaller in the EDA, improving scalability. Though researchers have investigated application- and language-level security issues that affect many services, we are the first to examine the security implications of using the EDA from a software architecture perspective. In III we describe a 1 The EDA paradigm is also known as the reactor pattern. new Denial of Service attack that can be used against EDAbased services. Our Event Handler Poisoning attack exploits the most important limited resource in the EDA: the Event Handlers themselves. The source of the EDA s scalability is also its Achilles heel. Multiplexing unrelated work onto the same thread reduces overhead, but it also moves the burden of time sharing out of the thread library or operating system and into the application itself. Where OTPCA-based services can rely on preemptive multitasking to ensure that resources are shared fairly, using the EDA requires the service to enforce its own cooperative multitasking [89]. An EHP attack identifies a way to defeat the cooperative multitasking used by an EDA-based service, ensuring Event Handler (s) remain dedicated to the attacker rather than handling all clients fairly. Our analysis identifies many EHP vulnerabilities among previously-reported npm vulnerabilities, and a sample of npm modules suggests that EHP vulnerabilities may be an epidemic ( IV). In V we define criteria for EHP-safety, and use these criteria to evaluate different EHP defenses ( VI). The general solution we propose is to relax but not eliminate the cooperative multitasking constraint of the EDA. Our prototype, Node.cure ( VII), makes timeouts a first-class member of the Node.js framework, and is the first framework we know of that does so. If an Event Handler in Node.cure blocks, Node.cure will emit a TimeoutError. Although cooperative multitasking is ultimately maintained, Node.cure thus informs every Event Handler when it ought to yield. Our findings on EHP-safety have been corroborated by the Node.js community ( IX). We have developed a guide for practitioners on building EHP-proof systems (PR preliminarily approved), updated the Node.js documentation to warn developers about the perils of particular APIs (two PRs merged), and submitted a pull request to improve the safety of these APIs (PR preliminarily approved). In our evaluation of Node.cure ( VIII), we describe the strong security guarantees against EHP that Node.cure offers, and demonstrate this by showing that Node.cure defeats representatives of each of the classes of EHP attack. Our cost evaluation on micro- and application benchmarks shows that Node.cure introduces acceptable monitoring overhead, as low as 0% on real server applications. The first steps toward this paper were presented at a workshop 2. There we introduced the idea of EHP attacks and argued that some previously-reported security vulnerabilities were actually EHP vulnerabilities. We also proposed the C- WCET Principle, which we have evaluated and found wanting 2 We omit the reference to meet the double-blind requirement.

2 ( VI-C). Here we offer the complete package: we refine our previous definition of EHP, increase our survey of EHP vulnerabilities in the wild, and describe a prototype that offers strong security guarantees using timeouts. In summary, our work makes the following contributions: 1) We are the first to define Event Handler Poisoning (EHP) ( III), a DoS attack against the EDA. EHP attacks consume Event Handlers, a limited resource in the EDA. Our definition of EHP accommodates all known EHP attacks. 2) After evaluating the solution space for EHP defenses ( VI), we describe a general antidote for Event Handler Poisoning attacks: first-class timeouts ( VII). We demonstrate its effectiveness with a prototype for the popular Node.js EDA framework. To the best of our knowledge, our prototype, Node.cure, is the first EDA framework to incorporate firstclass timeouts. 3) We evaluate the security guarantees of timeouts (strong), their costs (small), and their effectiveness (quite) in the Node.js ecosystem ( VIII). 4) We released a guide on EHP-safe techniques for new developers in the EDA, which has been accepted. Our PR improving the safety of Node.js APIs has been accepted and our PRs documenting unsafe APIs were merged ( IX). II. BACKGROUND In this section we contrast the EDA with the OTPCA ( II-A). We then explain our choice of EDA framework for study ( II-B), and finish by introducing formative work on Algorithmic Complexity attacks ( II-C). A. Overview of the EDA There are two main architectures for scalable web servers, the established One Thread Per Client Architecture (OTPCA) and the upstart Event-Driven Architecture (EDA). In the OT- PCA style, exemplified by the Apache Web Server [61], each client is assigned a thread or a process, with a large maximum number of concurrent clients. The OTPCA model isolates clients from one another, reducing the risk of interference. However, each additional client incurs memory and context switching overhead. In contrast, by multiplexing multiple client connections onto a single-threaded Event Loop and offering a small Worker Pool for asynchronous tasks, the EDA approach reduces a server s per-client resource use. While the OTPCA and EDA architectures have the same expressive power [71], in practice EDA-based servers can offer greater scaling [84]. Although EDA approaches have been discussed and implemented in the academic and professional communities for decades (e.g. [94], [40], [63], [56], [98], [87]), historically the EDA was only applied in user interface settings like GUIs and web browsers. The EDA is increasingly relevant in systems programming today thanks largely to the explosion of interest in Node.js, an event-driven server-side JavaScript framework. The most common version of the EDA on the server side is the Asymmetric Multi-Process Event-Driven (AMPED) architecture [83], used by all mainstream general-purpose EDA frameworks. The AMPED architecture (hereafter the EDA ) is illustrated in Figure 1. In the EDA the operating system or a framework places events in a queue (e.g. through Linux s poll system call 3 ), and pending events are handled sequentially by an Event Loop. The Event Loop may offload expensive Tasks to the queue of a small, fixed-size Worker Pool, whose Workers execute Tasks and generate Task Done events for the Event Loop when they finish. One common use of the Worker Pool is to provide quasi-asynchronous (blocking but concurrent) access to the file system. Fig. 1. This is the AMPED Event-Driven Architecture. Incoming events from clients A and B are stored in the event queue, and the associated Callbacks (CBs) will be executed sequentially by the Event Loop. The Event Loop may generate new events, to be handled by itself later or as Tasks offloaded to the Worker Pool. Alas, B has performed an EHP attack on this service. His evil input (third in the event queue) will eventually poison the Event Loop, and the events after his input will not be handled ( III-D). Application developers must adopt the EDA paradigm when developing EDA-based software. Developers define the set of possible Event Loop Events for which they supply a set of Callbacks [60]. During the execution of an EDA-based application, events are generated by external activity or internal processing and routed to the appropriate Callbacks. We refer to the Event Loop and the Workers as Event Handlers. Events for the Event Loop are supplied by the operating system or the framework, while events for the Worker Pool (i.e. Tasks) are supplied by the Event Loop. We say that the Event Loop executes a Callback in response to an event, and that a Worker performs a Task. A Callback is executed synchronously on the Event Loop, and a Task is performed synchronously on a Worker, asynchronously from the Event Loop s perspective. In the EDA, each Callback and Task is guaranteed atomicity: once scheduled, it runs to completion. Because there are relatively few Event Handlers, cooperative multitasking [89] is a must. Developers therefore partition the generation of responses into multiple stages (forming its Lifeline [38], a DAG) and regularly yield to other requests by deferring work until the next stage [52]. Unfortunately, in Figure 1 an attacker B carried out an EHP attack, defeating the cooperating multitasking and preventing other requests from being handled. B. Node.js among other EDA frameworks An EDA approach can be (and has been) implemented in most programming languages. Examples of EDA frameworks include Node.js (JavaScript) [17], libuv (C/C++) [11], Event- Machine (Ruby) [5], Vert.x (Java) [28], Zotonic (Erlang) [31], 3 [P]oll...shall identify those file descriptors on which an application can read or write data, or on which certain events have occurred [35]. 2

3 Twisted (Python 4 ) [27], and Microsoft s P# [55]. These frameworks have been used to build a wide variety of industry and open-source services, at companies including LinkedIn [78], PayPal [66], ebay [82], and IBM [68], and for projects like Ubuntu One [33], Apple s Calendar and Contacts Server [32], and IoT frameworks [9], [4]. We selected Node.js for further investigation based on its community interest and maturity. To measure this, we surveyed each framework s GitHub [7] repository for the number of contributors and forks (community interest), and the number of lifetime and recent commits (maturity). As of November 14, 2017, Node.js had the most community interest (1,782 contributors against libuv s 269; 8,775 forks against Vert.x s 1,177), and strong maturity (19,832 lifetime commits to Twisted s 22,161; 2,382 commits in the last six months against Twisted s 1,106). Thus, we selected Node.js (JavaScript) for closer investigation, though as described in III, EHP attacks apply to any EDA-based service. Node.js is a server-side event-driven framework for JavaScript that uses the architecture shown in Figure 1. Node.js executes an application s JavaScript using Google s V8 engine [3], and offers a set of core JavaScript modules with C++ bindings that implement system calls using libuv [11]. Introduced in 2009, Node.js claimed 3.5 million users in 2016 [34], and as of August 2017 its open-source package ecosystem, npm, is the largest of any programming language or framework [13], with 550,000 modules and counting. C. Algorithmic complexity attacks Our work is inspired by Algorithmic Complexity (AC) attacks, a form of Denial of Service attack ([76], [51]). In an AC attack, attackers force victims to take longer than expected to perform an operation by incurring the worst-case complexity of the victim s algorithms. Rather than overwhelming the victim with volume, the attacker turns the victim s vulnerable algorithms against him. AC vulnerabilities can therefore be exploited to launch DoS attacks, when the attacker forces the victim to spend a long time processing his request at the expense of legitimate clients, or when the attacker increases the cost even of legitimate requests. Well-known examples of AC attacks include attacks on hash tables, whose worst-case O(N) behavior can be triggered by attackers [51], and Regular expression Denial of Service (ReDoS) attacks, which exploit the worst-case exponentialtime behavior typical of regular expression (regexp) engines ([50], [88], [93], [70], [86], [95]). As will be made clear in III, our work is not simply the application of AC attacks to the EDA. The first distinction is a matter of perspective: where AC attacks focus on the algorithms the service employs, EHP attacks target the software architecture used by the service. The second distinction is one of definition: AC attacks rely on CPU-bound activities to achieve DoS, while EHP attacks may use either CPU-bound or I/O-bound activities to poison an Event Handler. 4 In addition, Python 3.4 introduced native EDA support. 1 const cb = (req, resp) => { 2 let f = requestedfile(req); 3 if (f.match(/(\/.+)+$/)) { // ReDoS 4 FS.readFile(f, (err, d) => { // ReadDoS 5 resp.end( Finished read ); 6 }); 7 } 8 else { resp.end( Invalid file ); } 9 } 10 const srv = _HTTP.createServer(cb).listen(3000); Fig. 2. Gist of our simple server vulnerable to EHP attacks. Line 3 is vulnerable to ReDoS. Line 4 is vulnerable to ReadDoS. III. EVENT HANDLER POISONING ATTACKS In this section we provide our threat model ( III-A), define Event Handler Poisoning (EHP) attacks more formally ( III-B), and present a taxonomy for EHP attacks ( III-C). To illustrate the problem, we demonstrate simple EHP attacks using ReDoS and ReadDoS ( III-D). A. Threat model Our threat model is straightforward: the attacker crafts evil input that triggers an expensive operation, and convinces the EDA-based victim to feed it to a Vulnerable API ( III-C). An ideal EHP attack is asymmetric, such that a small evil input triggers a large amount of work on the part of the poisoned Event Handler. Depending on how much work the evil input triggers, the attacker can achieve DoS outright, or can repeatedly apply poison in a Slow DoS attack [44]. Servers are in general vulnerable to DDoS attacks, whether OTPCA or EDA, and we exclude these from our model. This model is reasonable because our victim is an EDAbased server; servers exist solely to process client input using APIs. If the server uses a Vulnerable API, and if it invokes it with unsanitized client input, the server can be poisoned. In III-C we show examples of Vulnerable APIs in Node.js and other EDA frameworks, at the language, framework, and application levels. B. What happens in an EHP attack? Total/synchronous complexity and time. Before we can define EHP attacks, we must introduce a few definitions. First, recall the standard EDA formulation illustrated in Figure 1. Each client request is the initial event of a Lifeline [38] partitioned into one or more events and Tasks handled by the appropriate Event Handler. A Lifeline is thus a graph of event Callbacks and Tasks, whose vertices represent a Callback or Task and whose edges indicate events or Task submission. To illustrate a Lifeline, consider the server in Figure 2. The arrival of an HTTP request on port 3000 is the initial event. The corresponding Callback cb (lines 1-9) is handled by the Event Loop, which synchronously invokes an application-level API (line 2) and a language-level regexp API (line 3), and then creates another vertex in the Lifeline graph by submitting an asynchronous file system request. The Worker Pool handles this Task, and on completion it adds another vertex to the Lifeline by emitting a Task Complete event to the Event Loop (Callback defined on lines 4-6). 3

4 We define the total complexity of a Lifeline as the combined complexity (cost) of each vertex (Callbacks and Tasks) as a function of the cumulative input to all vertices. The synchronous complexity of a Lifeline is the greatest individual complexity among all vertices in the Lifeline. Similarly, we define a Lifeline s total time and synchronous time as the cumulative and greatest-individual time taken, respectively, by the vertices in the Lifeline. Since the EDA relies on cooperative multitasking, a Lifeline s synchronous complexity and synchronous time provide theoretical and practical upper bounds on how effective an EHP attack against it might be. We also refer to the synchronous complexity and time of individual Callbacks and Tasks, defined as the individual complexity and time of the corresponding Lifeline vertex. EHP attacks. Remember that each Callback and Task is handled in its turn by one of the Event Handlers, and that the Event Handlers are not dedicated one to each client, but shared among all of the clients. This sharing means that the responsiveness of the server to its clients depends not on OS preemption, but on cooperative multitasking; each Callback and Task must be handled quickly by its Event Handler to give Callbacks and Tasks from other clients a turn. An EHP attack exploits this architecture by poisoning one or more Event Handlers with evil input. After identifying a vulnerable Lifeline, one with a large worst-case synchronous complexity, an attacker submits evil input that triggers this worstcase complexity. The Lifeline s large synchronous complexity leads to large synchronous time, causing the affected Event Handler to block, perhaps indefinitely, while handling the expensive event or Task. While an Event Handler is blocked, it will not continue to process other events or Tasks, and the Lifelines of pending events and Tasks will starve. EHP attacks thus exploit a fundamental property of the EDA, namely the need to correctly implement fair cooperative multitasking. An EHP attack can be carried out against either the Event Loop or the Workers in the Worker Pool. If the Event Loop is poisoned, neither pending nor future events will be handled. On the other hand, for each poisoned Worker the responsiveness of the Worker Pool will degrade, until every Worker is poisoned and all pending and future requests whose Lifelines submit a Task to the Worker Pool will receive no responses. Thus, an attacker s aim is to poison either the Event Loop or enough of the Worker Pool to harm the throughput of the server. You might be surprised that we name the Worker Pool as a potential target for EHP attacks. Remember, however, that in the EDA the Worker Pool is deliberately quite small, as part of the EDA s low-overhead philosophy. For example, in a Node.js process the Worker Pool defaults to 4 Workers, with a maximum of 128 Workers [20]. Though a small Worker Pool is in keeping with the spirit of the EDA, its size makes it a realistic target. In contrast, though typical OTPCAbased servers also maintain a pool of threads (Workers) to bound the total number of concurrent connections, this pool is comparatively enormous. Apache s httpd server has a default connection pool of size , with a maximum around 20,000 [61]. Due to the size of an OTPCA server s Worker Pool, attempts to poison it would presumably be detected by an IDS or addressed through rate limiting before they poisoned enough of the pool to have an effect. By comparison, the small number of requests needed to poison a vulnerable Vuln. APIs Event Loop ( VII-B) Worker Pool ( VII-A) CPU-bound I/O-bound CPU-bound I/O-bound Language Regexp, JSON N/A N/A N/A Framework Crypto, zlib FS Crypto, zlib FS, DNS Application while(1) DB query Regexp [15] DB query Table I. Taxonomy of Vulnerable APIs in Node.js, with examples. An EHP attack through a Vulnerable API poisons the Event Loop or a Worker, and its synchronous time is due to CPU-bound or I/O-bound activity. A Vulnerable API might be part of the language, framework, or application, and might be largely synchronous (Event Loop) or asynchronous (Worker Pool). zlib is the Node.js compression library. N/A: JavaScript has no native Worker Pool nor any I/O APIs. EDA server s Worker Pool is too few to draw attention from network-level defenses, placing the burden of defense on the language, framework, and application developers or, as we propose in VI-D, the runtime system. We believe EHP attacks on the EDA have not previously been explored because the EDA has only recently seen popular adoption by the server-side community. On the client side EHP attacks are not a concern, as misbehaving users will hurt only themselves. On the server side, the stakes are rather higher, especially as the EDA is adopted in safety-critical settings like IoT controllers [9] and robotics [4]. C. What makes an API Vulnerable? Thus far we have relied on the intuitive notion of a Vulnerable API. We turn now to a more precise taxonomy. Using the terms from III-B, a Vulnerable API begins a Lifeline that can have large synchronous time, due either to its large synchronous complexity or to the size of the input. A Vulnerable API poisons the Event Loop or a Worker. Its synchronous time can be attributed to heavy computation or the need for I/O. Note that an API that creates a Lifeline with large total time is not vulnerable so long as each vertex (Callback/Task) has a small synchronous time; it is large synchronous time that leads to EHP. Table I classifies Vulnerable APIs along two axes. Along the first axis, a Vulnerable API affects one of the two types of Event Handlers, and it might be CPU-bound or I/O-bound. Along the second axis, a Vulnerable API can be found in one of three places: the language, the framework, or the application. Table I lists examples of Vulnerable APIs for applications written in JavaScript for the Node.js framework. In our evaluation we provide an exhaustive list of Vulnerable APIs at the language and framework levels ( VIII-A). Although the framework component of Table I is specific to Node.js, the same general classification can be applied to other EDA frameworks. For example, most mainstream languages (Java, Python, Perl, Ruby, etc.) support regular expressions and are vulnerable to ReDoS; and unless care is taken, file system or DNS requests to slow resources will poison Event Handlers in many EDA frameworks. D. Two simple EHP attacks To illustrate EHP attacks, we developed a small vulnerable file server. a simplified version is shown in Figure 2. This server has two EHP vulnerabilities. First, it sanitizes input with a regexp vulnerable to ReDoS. This regular expression ensures that paths are /-delimited, but the inclusive. will also match a /. On the evil input /.../\n (50 / s and a newline that. 4

5 does not match), this input triggers exponential-time behavior in Node.js s regexp engine while it backtracks in search of a division of the / s that will match, poisoning the Event Loop in a CPU-bound EHP attack and blocking all subsequent requests. The second EHP vulnerability might be more surprising. While the sanitization regexp fulfills its primary purpose, preventing readfile from throwing a malformed path error, it permits a client to read an arbitrary file a directory traversal vulnerability. In the EDA, directory traversal vulnerabilities can be parlayed into I/O-bound EHP attacks, ReadDoS, provided the attacker can identify a Slow File 5 or a Large File from which to read. Since line 4 uses the asynchronous framework API FS.readFile, each ReadDoS attack on this server will poison a Worker. In Figure 3 you can see the baseline server throughput (circles) and the throughput while undergoing two attacks (ReDoS in triangles, ReadDoS in squares), averaged over five runs. In each condition we configured a Worker Pool of size 4, and used the ab benchmark [1] to make 2500 legitimate requests per second divided among 80 clients. We measured the instantaneous throughput of the server at 200 ms intervals. Exploiting the ReDoS vulnerability has an immediate effect. After 1 second the attacker submits the evil input, poisoning the Event Loop and blocking all subsequent requests. In contrast, the ReadDoS attack only becomes noticeable after three evil requests because the Worker Pool is not saturated in this example. At seconds 1, 2, 3, and 4 an attacker submits one read request to a Slow File. Each evil input poisons one of the four Workers. The throughput dips after three malicious requests, as the legitimate requests saturate the sole surviving Worker. The fourth evil input entirely poisons the Worker Pool, flatlining the throughput. The architecture-level behavior of the ReDoS attack, an EHP attack on the Event Loop, is illustrated in Figure 1. After client A s benign request is sanitized (CB A1 ), the file read Task goes to the Worker Pool (T ask A1 ), and when the read completes the Callback returns the file content to A (CB A2 ). Then client B s malicious request arrives, and the service s input sanitization triggers ReDoS (CB B1 ). IV. EVENT HANDLER POISONING ATTACKS IN THE WILD With a definition of EHP attacks in hand, we now assess the presence of EHP vulnerabilities in the Node.js ecosystem. We found that the Node.js documentation does not carefully outline the need for cooperative multitasking ( IV-A), which might contribute to the fact that many popular npm modules expose Vulnerable APIs and rely on the application to sanitize input ( IV-B), and that 35% of known npm module vulnerabilities can be used for EHP attacks ( IV-C). A. What do the Node.js docs say about EHP attacks? Users in search of Node.js documentation can look in three places. First, there is the JavaScript language specification [14]. 5 In addition to files exposed on network file systems, /dev/random is a good example of a Slow File. According to the Linux manual, [R]eads from /dev/random may block...users will usually want to open it in nonblocking mode (or perform a read with timeout) [36]; by default, Node.js on Linux does neither. Fig. 3. This figure shows the effect of evil input on the throughput of a simple server based on Figure 2, using either Baseline Node.js (grey) or our proposed runtime, Node.cure (black). For ReDoS (triangles), evil input was injected after three seconds, poisoning the Baseline Event Loop. For ReadDoS (circles), evil input was injected four times at one second intervals beginning after three seconds, eventually poisoning the Baseline Worker Pool. The lines for Node.cure shows its effectiveness against these EHP attacks. Node.cure s throughput on ReDoS dips until the first Callback times out ( VII-B), after which it catches up to pending requests from clients. Because the Worker Pool is not saturated, Node.cure does not suffer while it waits for the timeout of the first Task containing a ReadDoS request for /dev/random ( VII-A), and marks /dev/random s inode Slow. Subsequent ReadDoS requests time out immediately ( VII-D). Second, Node.js maintains documentation for the APIs of its core modules [21]. Third, the nodejs.org website offers a set of guides for new developers [19]. Although there are potential EHP vulnerabilities in an implementation of JavaScript, like ReDoS or large JSON manipulation, language specifications do not always discuss their risks. For example, in MDN s documentation only eval has warnings. The Node.js documentation does not indicate any vulnerabilities in its JavaScript engine. However, it does give hints about avoiding EHP attacks on its framework APIs. Node.js APIs include several modules with Vulnerable APIs, and developers are warned away from the synchronous versions. The asynchronous versions, which could poison the Worker Pool, bear the warning: [These] APIs use libuv s threadpool, which can have surprising and negative performance implications for some applications, and refer to another page reading Because libuv s threadpool has a fixed size, it means that if for whatever reason any of these APIs takes a long time, other (seemingly unrelated) APIs that run in libuv s threadpool will experience degraded performance. Missing from the Node.js new developer guides is a discussion about how to design a Node.js application to avoid EHP vulnerabilities. Later we describe the guide we have prepared on these topics ( IX); it has been preliminarily approved for placement on nodejs.org. B. Do npm modules have Vulnerable APIs? In December 2016 we examined 80 npm modules to determine how they handle potentially computationally expensive operations, i.e. application-level CPU-bound potentially- Vulnerable APIs (Table I). We manually inspected both their documentation and their implementation. Using npm s Best overall sorting scheme, we examined the first 20 (unique) modules returned by npm searches for string, string 5

6 manipulation, math, and algorithm, in hopes of striking a balance between popularity and potential complexity. These modules were downloaded over 200 million times that month. We found that many of these modules had Vulnerable APIs, performing operations with high synchronous complexity and thus leaving callers vulnerable to EHP attack. Although knowing the synchronous complexity of an API is critical to preventing EHP, the documentation for the majority of the APIs did not state their running times. Nearly all of the hundreds of APIs we examined executed synchronously on the Event Loop; though two were asynchronous, one of these synchronously executed an exponential-time algorithm before yielding control. During our analysis of the source code we identified algorithms with complexities including O(1), O(n), O(n 2 ), O(n 3 ), and O(2 n ). In summary, in our survey of 80 popular npm modules we found examples of Vulnerable APIs in the majority of the modules. Module and application developers who use these APIs must weigh the risks of calling these APIs, but this is difficult without knowing the synchronous complexity of the API (usually undocumented) or studying its implementation to learn what input might be evil (impractical). Given the prevalence of Vulnerable APIs in these modules, we did not open bug reports against each of them. We believe the problem of EHP attacks is systemic in the Node.js community. Rather than pursuing ad hoc bug reports and patches, we took fundamental steps to improve the state of affairs, with a detailed guide for new developers ( IX) and a comprehensive defense against EHP in the form of Node.cure ( VII). C. Are there examples of EHP vulnerabilities? We identified two databases describing vulnerabilities in npm modules, one maintained by the Node Security Platform (NSP) [16] and the other by Snyk.io [25]. Our evaluation of the first several hundred vulnerabilities in each suggested that NSP s database (355 entries) is a subset of Snyk.io s database (713 entries), so we performed a detailed analysis only of Snyk.io s database. In this section we summarize the vulnerabilities in the Snyk.io database and analyze them through the lens of EHP attacks. First, we divided the vulnerabilities by category. Each raw Snyk.io database entry includes a title and description, and some have additional material like CWE labels and links to GitHub patches. Since the titles indicate the high-level exploit, e.g. Directory Traversal in servergmf..., we sorted the npm vulnerabilities into 17 common categories and one Other category for the remaining 40. Figure 4 shows the distribution, absorbing categories with fewer than 10 vulnerabilities into Other. A high-level CWE number is given next to each class. Then, we studied each vulnerability to see if it could be employed in an EHP attack. The dark bars in Figure 4 indicate the 186 vulnerabilities (35%) that could be used to launch an EHP under our threat model ( III-A). The 145 EHP-relevant Directory Traversal vulnerabilities are exploitable because they allow arbitrary file reads, which can poison the Worker Pool through ReadDoS. The 32 EHP-relevant Denial of Service vulnerabilities poison the Event Loop; 28 are ReDoS, 3 lead to infinite loops, and 1 allows an attacker to trigger worstcase algorithm behavior in input processing. In Other are 9 Fig. 4. Classification of the 713 npm module vulnerabilities, by category and by usefulness in EHP attacks. Vulnerabilities were iteratively categorized using regexp. We analyzed the contents of the database as of 8 August Arbitrary File Write vulnerabilities that, like ReadDoS, can be used for EHP attacks by writing to Slow Files. V. WHAT DOES IT MEAN TO BE EHP-SAFE? Having defined the EDA ( II), explained EHP attacks ( III), and surveyed their reality ( IV), we now ponder what it means to be EHP-Safe. The fundamental cause of EHP vulnerabilities should be clear: EHP vulnerabilities stem from Vulnerable APIs that fail to honor cooperative multitasking. As categorized in Table I, Vulnerable APIs can poison the Event Loop or the Worker Pool, they can be CPU-bound or I/O-bound, and they occur at any level of the application stack: language (vulnerable constructs), framework (vulnerable core modules), or application (vulnerable code in the application or the library modules on which it depends). To be EHP-safe, an application must use APIs whose synchronous time is bounded. If an application cannot bound the synchronous time of its APIs, it is vulnerable to EHP attack; conversely, if an application can bound the synchronous time of its APIs, then it is EHP-safe. This application-level guarantee must span the application s full stack: the language constructs it uses, the framework APIs it invokes, and the application code itself. In VI, we outline several approaches to EHP-safety. How shall we choose between them? We suggest three criteria: 1) General. A developer must be able to use existing APIs or slight modifications thereof. This ensures that applications are general, that they can solve meaningful problems. 2) Developer-friendly. A developer should not need expertise in topics outside of their application domain. 3) Composable. If a developer uses only EHP-safe APIs, they should be unable to create a Vulnerable API. In a composably EHP-safe system, an API containing while(1); would not be Vulnerable, because it is composed only of non-vulnerable language APIs. In contrast, a noncomposably EHP-safe system has only ad hoc EHP safety, because in addition to making language- and frameworklevel APIs non-vulnerable, application-level APIs composed of them must also be assessed. VI. ANTIDOTES FOR EVENT HANDLER POISONING Here we discuss four schemes by which to guarantee EHPsafety: removing Vulnerable APIs ( VI-A), only calling Vulnerable APIs with safe input ( VI-B), refactoring Vulnerable 6

7 APIs into safe ones ( VI-C), and strictly bounding an Event Handler s synchronous time at runtime ( VI-D). We evaluate these schemes using the criteria from V. A. Remove calls to Vulnerable APIs: The Blacklist Approach To become EHP-safe, an application might eliminate calls to Vulnerable APIs. Because this Blacklist Approach fails the generality criterion, it does not merit further discussion. B. Sanitize input to Vulnerable APIs: The Sanitary Approach Many Vulnerable APIs are only problematic if they are called with unsafe input. Another approach to EHP-safety, then, is the Sanitary Approach: ensure that every Vulnerable API is always called with a safe input. The Sanitary Approach is developer-unfriendly, because it requires developers to identify evil input for the APIs they call. For example, Vulnerable APIs like regexp are themselves commonly used to identify invalid input. If the sanitization regexps are vulnerable to ReDoS, quis custodiet ipsos custodes? C. Refactor Vulnerable APIs: The C-WCET Principle A more promising path to EHP-safety is to refactor Vulnerable APIs into safe ones. Since the goal of EHP-safety is to bound the synchronous time of a Vulnerable API, a good rule of thumb might be to restrict the cost of every Callback/Task vertex in a Lifeline to Constant Worst-Case Execution Time (C-WCET) by partitioning the generation of a response into (ideally constant-time) chunks wherever possible. Such a C- WCET partitioning would bound the synchronous complexity of the Lifeline rooted at this Vulnerable API for all inputs. If all Vulnerable APIs were replaced with APIs following the C-WCET Principle, the application would be EHP-safe. The C-WCET Principle is the logical extreme of cooperative multitasking, extending input sanitization from the realm of data into the realm of the algorithms used to process it. Evaluating the C-WCET Principle. The C-WCET Principle has two benefits. First, by bounding the synchronous complexity of all Callbacks and Tasks, every client request could be handled whether or not its input was evil. Second, the C-WCET Principle can be applied without changing the language specification. For example, as has previously been proposed [49], a non-exponential-time regexp engine could be used in place of Node.js s exponential-time Irregexp engine [48], and the application would be none the wiser. The C-WCET Principle meets the generality criterion; every API can be partitioned towards C-WCET. It is also somewhat developer-friendly: language- and framework-level APIs can be C-WCET partitioned by framework developers, leaving developers responsible for C-WCET partitioning their own APIs. However, since application developers rely heavily on third-party npm modules that might not have been C-WCET-partitioned, they might be called on to C-WCET partition modules outside their own expertise. Critically, like the Blacklist and Sanitary Approaches, the C-WCET Principle is not composable. Since Vulnerable APIs can always be composed from non-vulnerable APIs 6, developers would need 6 As noted in V, an API consisting of a while(1); loop is Vulnerable in the Blacklist, Sanitary, and C-WCET Approaches, even though it only calls the constant-time language APIs while(1) and ;. to take responsibility for applying the C-WCET Principle to every API at the application and library levels. Nevertheless, because the C-WCET Principle has attractive benefits, we developed a Node.C-WCET prototype to evaluate its practicality at the language and framework levels. Node.C- WCET aimed to defeat ReDoS (language-level) and ReadDoS (framework-level) EHP attacks using the C-WCET Principle. To combat ReDoS, wherever possible Node.C-WCET uses the linear-time RE2 regexp engine [22] instead of V8 s exponential-time regexp engine. And to combat ReadDoS, we replaced the file system read API s synchronous-on-a-worker implementation with truly asynchronous kernel AIO 7 so that a Worker would not be poisoned by a Slow File, and further ensured that all reads of Large Files are partitioned. Due to space limitations we omit a detailed description of Node.C- WCET and our experiments. Ultimately, we deemed Node.C-WCET a failure, and rejected the C-WCET Principle as a solution to EHP attacks. From an engineering perspective, we found that Node.C- WCET introduced significant overheads (e.g. Linux kernel AIO requires O_DIRECT), and that the effort to understand and fully refactor all language- and framework-level Vulnerable APIs would be considerable. But more importantly, from a security perspective Node.C-WCET remained unsound in two ways. First, Node.C-WCET was still vulnerable to real ReDoS and ReadDoS attacks from known vulnerabilities, because not all regexps can be evaluated in linear time, and Linux s kernel AIO does not support device files like the Slow File /dev/random. Second, the C-WCET Principle is ad hoc, not composable: even if Node.C-WCET partitioned all lower-level Vulnerable APIs, application-level APIs composed of these APIs could still be Vulnerable. D. Dynamically bound running time: The Timeout Approach The goal of the C-WCET Principle was to bound an API s synchronous complexity as a way to bound its synchronous time. Instead of statically bounding an API s synchronous complexity, what if we could dynamically bound its synchronous time? Then the complexity of each Callback and Task would become irrelevant, because it would be unable to take more than the quantum provided it by the runtime. In this Timeout Approach, the runtime detects and aborts longrunning Callbacks and Tasks by emitting a TimeoutError, thrown from Callbacks and returned from Tasks. The Timeout Approach provides EHP-safety. Like the C- WCET Principle, the Timeout Approach is general. It is more developer-friendly because it only requires developers to handle TimeoutErrors in their own APIs. And where the C-WCET Principle was ad hoc, the Timeout Approach is composable. If the EDA runtime had a sense of time, then a long-running Callback/Task could be interrupted no matter where it was, whether in the language, the framework, or the application. An application that calls only APIs from the language and the framework would always be EHP-safe. Although the Timeout Approach fulfills all of the EHP-safety criteria, like C-WCET Partitioning it requires 7 Unlike Posix asynchronous I/O, which is implemented by libc using a threadpool, true Linux kernel AIO via io_submit gives the kernel responsibility for managing the I/O. 7

8 application-level refactoring. Because the TimeoutErrors emitted by the runtime are not part of the language and framework specification, existing applications would need to be ported. We do not anticipate this to be a heavy burden for well-written applications, which should already be exceptionaware to catch errors like buffer overflow (RangeError) and null pointer exceptions (ReferenceError), and which should test the result of a Task for errors. Though the C-WCET Principle and the Timeout Approach both meet the generality criterion, the Timeout Approach is more developer-friendly, and is the only EDA-centric solution we analyzed that is composable 8. In VII we describe our Node.cure prototype, which applies the Timeout Approach to all levels of the Node.js stack. VII. NODE.CURE: TIMEOUTS FOR NODE.JS Our Node.cure prototype implements the Timeout Approach for Node.js by bounding the synchronous time of every Callback and Task. To do so, Node.cure makes timeouts a first-class member of Node.js, extending the JavaScript specification by introducing a TimeoutError to abort long-running Tasks and Callbacks. A long-running Task poisons its Worker. Such a Task is aborted and fulfilled with a TimeoutError ( VII-A). A long-running Callback poisons the Event Loop; in Node.cure these throw a TimeoutError whether they are in JavaScript or in a Node.js C++ binding ( VII-B). As a result, Node.cure meets the composability criterion of V: the Node.js language and framework APIs are timeout-aware, and an application whose APIs are composed of these APIs will be EHP-safe ( VII-C). Node.cure also has an exploratory Slow Resource Policy to avoid unnecessary timeouts ( VII-D). We describe the Timeout Approach for Tasks first, because it introduces machinery used to apply the Timeout Approach to Callbacks. A. Timeout-aware Tasks Node.cure defends against all EHP attacks that target the Worker Pool by bounding the synchronous time of Tasks. To make Tasks timeout-aware, we modified both the Worker Pool and the Node.js APIs that use it. Timeout-aware Worker Pool. Asynchronous Vulnerable APIs submit long-running Tasks that can poison a Worker on the Worker Pool. We modified the libuv Worker Pool to be timeout-aware, replacing libuv s Workers with Executors that combine a permanent Manager with a disposable Worker. If a Task times out, the Executor creates a new Worker to resume handling Tasks and creates a Hangman to pthread_cancel the poisoned Worker. These roles are illustrated in Figure 5. Here is a slightly more detailed description of our timeoutaware Worker Pool. Node.js s Worker Pool is implemented in libuv, exposing a Task submission API uv_queue_work, which we extended as shown in Table II. The original Workers of the Worker Pool would simply invoke work and then notify the Event Loop to invoke done. This is also the behavior of our timeout-aware Workers on the good path. When a Task takes too long, however, the potentially-poisoned Worker s Manager 8 We discuss other approaches to EHP-safety in IX. Fig. 5. This figure illustrates Node.cure s timeout-aware Worker Pool, including the roles of Event Loop, Executors (Worker Pool and Priority), and Hangman. Grey entities were present in the original Worker Pool, black are new. The Event Loop can synchronously access the Priority Executor, or asynchronously offload Tasks to the Worker Pool. If an Executor s Manager sees its Worker time out, it creates a replacement Worker and passes the Dangling Worker to a Hangman. Callback void work int timed out* void done void killed* Description Perform Task. When Task has timed out. Can request extension. When Task is done. Special error code for timeout. When a timed out Task s thread has been killed. Table II. Summary of the Worker Pool API. work is invoked on the Worker. done is invoked on the Event Loop. The new callbacks, timed_out and killed, are invoked on the Manager and the Hangman, respectively. On a timeout, work, timed_out, and done are invoked, in that order; there is no ordering between the done and killed callbacks, which sometimes requires reference counting for safe memory cleanup. *New callbacks. invokes the Task s timed_out callback. If the submitter does not request an extension, the Manager creates a replacement Worker so that it can continue to process subsequent Tasks, creates a Hangman thread for the poisoned Worker, and notifies the Event Loop that the Task is complete. The Event Loop then invokes its done Callback with a TimeoutError. Concurrently, once the Hangman successfully kills the Worker thread, it invokes the Task s killed callback and returns. One difficulty with concurrently creating a replacement Worker and assigning the poisoned Worker to a Hangman is that the poisoned Worker may not have been canceled before its TimeoutError is delivered to the Event Loop. Such a Dangling Worker may be in a system call like write that modifies externally-visible state. It is therefore critical that if a timeout occurs, a Dangling Worker cannot corrupt shared state. In general this can be dealt with using transactional techniques like undo-logging or write buffering, but we investigate a simpler alternative for the file system in VII-D. In addition to external shared state like the file system, memory safety is also a consideration for Dangling Workers because the Node.js Worker Pool is implemented using threads. Workers cannot use memory allocated by the Task submitter because a Dangling Worker may exist after its Task s done is invoked. Although the new uv_queue_work API is more complex than the previous one, only experts like framework developers will need to use it. This and other libuv-level changes are only visible to the average application-level developer in the form of TimeoutErrors. Timeout-aware Worker Pool Customers. The Node.js APIs affected by this change (i.e. that create Tasks) are from the file system, DNS, encryption, and compression modules. Node.js implements its file system and DNS APIs by relying on libuv s file system and DNS support, which on 8

9 Linux make the appropriate calls to libc. When asynchronous file system and DNS calls time out in libc, we allow the Manager to dispose of its Dangling Worker. We modified the libuv file system and DNS implementations to use buffered memory for memory safety of Dangling Workers. We used pthread_setcancelstate to ensure that threads will not be canceled while in a non-cancelable libc API [8]. Although this requires the Hangman to await the completion of the blocked API, the Manager concurrently replaces its poisoned Worker. It remains to handle Node.js s asynchronous encryption and compression APIs, implemented in the Node.js C++ bindings by invoking APIs from openssl and zlib, respectively. If the Worker Pool notifies these APIs of a timeout, they allow the Task to be canceled and wait for it to be killed before returning, to ensure that the Worker is no longer modifying state in these black-box libraries nor accessing memory that might be released after done is invoked. Finally, when the Event Loop sees that a Task timed out, it invokes the application s Callback with a TimeoutError. B. Timeouts for Callbacks Node.cure defends against EHP attacks that target the Event Loop by bounding the synchronous time of Callbacks. To make Callbacks timeout-aware, we introduce a Timeout- Watchdog that monitors the start and end of each Callback and ensures that no Callback exceeds the timeout threshold. We time out JavaScript instructions using V8 s interrupt mechanism ( VII-B1), and we modify Node.js s C++ bindings to ensure that Callbacks that enter these bindings will also be timed out ( VII-B2). 1) Timeouts for JavaScript: TimeoutWatchdog. Our TimeoutWatchdog instruments every Callback using the experimental Node.js async-hooks module [18]. These hooks cannot be disabled by an application. Before a Callback begins, our TimeoutWatchdog starts a timer. If the Callback completes before the timer expires ( good path ), we erase the timer. If the timer expires, the watchdog signals V8 to interrupt JavaScript execution by throwing a TimeoutError. The watchdog then starts another timer, ensuring that timeouts while handling the previous TimeoutError(s) are also detected. Although a precise TimeoutWatchdog can be implemented to carefully honor the timeout threshold, the resulting communication overhead between the Event Loop and the TimeoutWatchdog can be non-trivial ( VIII-B). A cheaper alternative is a lazy TimeoutWatchdog, which simply wakes up at intervals of the timeout threshold and checks whether the same Callback it last observed is still executing; if so, it emits a TimeoutError. A lazy TimeoutWatchdog reduces the overhead of making a Callback, but decreases the precision of the TimeoutError threshold: it can only guarantee to catch Callbacks that take at least twice the threshold. V8 interrupts. To handle the TimeoutWatchdog s request for a TimeoutError, Node.cure extends the interrupt infrastructure of Node.js s V8 JavaScript engine [3] to support timeouts. In V8, low priority interrupts like a pending garbage collection are checked regularly (e.g. each loop iteration, function call, etc.), but no earlier than after the current JavaScript instruction finishes. High priority interrupts take effect immediately, and can interrupt a running JavaScript instruction. The only high priority interrupt is TerminateExecution, which throws an uncatchable termination exception. Timeouts require the use of a high priority interrupt because they must be able to interrupt long-running individual JavaScript instructions like str.match(regexp) (possible ReDoS). To support a TimeoutError, we modified V8 as follows: (1) We added the definition of a TimeoutError into the Error class hierarchy; (2) We added a TimeoutInterrupt into the list of high-priority interrupts; and (3) We exposed an API by which Node.js can trigger a TimeoutInterrupt. The TimeoutWatchdog calls this API, which interrupts the current JavaScript stack by throwing a TimeoutError. The only JavaScript instructions that V8 instruments to be interruptible are regexp matching and JSON parsing; these are the language-level Vulnerable APIs. Other JavaScript instructions are treated as effectively constant-time, so these interrupts are evaluated e.g. at the end of basic blocks. We agreed with the V8 developers in this 9, and did not attempt to instrument other JavaScript instructions to poll for pending interrupts. 2) Timeouts for the Node.js C++ bindings: The TimeoutWatchdog described in VII-B1 will interrupt any Vulnerable APIs implemented in JavaScript, including language-level APIs like regexp and application-level APIs that contain blocking code like while(1);. It remains to give a sense of time to the Node.js C++ bindings that allow Node.js applications to interface with the broader world. Asynchronous C++ bindings are already timeout aware ( VII-A), because these perform a fixed amount of work to launch a Task and then return. However, Node.js also has synchronous C++ bindings that complete the entire operation before returning. Node.js includes an asynchronous API for the synchronous operations deemed Vulnerable by the developers, e.g. fs.readsync/fs.read; we made the synchronous versions of these APIs timeout-aware. In Node.js, the synchronous versions of these APIs execute on the Event Loop, either by calling libuv APIs without a callback (file system), or by invoking the openssl and zlib APIs directly (encryption and compression). Node.js does not expose a synchronous DNS API. We reviewed the rest of the Node.js APIs and determined that execsync also appeared to be Vulnerable. We did not make it timeout-aware because this API is not intended for use in Callbacks in the server context, per our threat model. Because the Event Loop holds the state of all pending clients, we cannot time it out using pthread_cancel, as this would result in the DoS the attacker desired. Another alternative is to offload the request to the Worker Pool and await its completion, but this would incur high request latencies when the Worker Pool s queue is not empty. Instead, we extended the Worker Pool paradigm with a dedicated Priority Executor whose queue is exposed via a 9 For example, we found that string operations complete in milliseconds even when a string is hundreds of MBs long. 9