Implementation of CMS Central DAQ monitoring services in Node.js

Transcription

1 Implementation of CMS Central DAQ monitoring services in Node.js Project ID: Michail Vougioukas Graduate Computer Science Student at AUEB, Athens, Greece Summer Student at PH-CMD, CERN Supervised by: Dr. Srećko Morović, PH-CMD Dr. Emilio Meschi, PH-CMD Geneva, Switzerland September 2015

2 Abstract This report summarizes my contribution to the CMS Central DAQ monitoring system, in my capacity as a CERN Summer Students Programme participant, from June to September Specifically, my work was focused on rewriting from Apache/PHP to Node.js/Javascript - and optimizing real-time monitoring web services (mostly Elasticsearch-based but also some Oraclebased) for the CMS Data Acquisition (Run II Filterfarm). Moreover, it included an implementation of web server caching, for better scalability when simultaneous web clients use the services. Measurements confirmed that the software developed during this project has indeed a potential to provide scalable services. Special Thanks I would particularly like to thank my supervisor, Srećko Morović, for his guidance, help, knowledge sharing and kindness during our good cooperation, as well as my second supervisor, Emilio Meschi, for his support, advice and ideas on the project. I would also like to thank CERN, for the unique opportunity it offered me at my career's beginning and the Summer Students Team for all its hard work. Last, but not least, I would like to thank my fellow Summer Students and the rest of my colleagues at CERN for the good moments we shared and the precious discussions we had. Table of Contents 1. Introduction 2. System Description Architecture Components and roles 3. Contribution and Novelty Implementation of callbacks in Node.js Implementation of web server cache Cache performance measurements 4. Conclusion 5. References Relevant material Code 6. Annex

3 1. Introduction The CMS DAQ2 Filter Farm includes a file-based monitoring system (FFF) that produces JSON files with metadata describing the raw data, HLT outputs per data stream and the stages of output file merging. These metadata contain interesting, real-time information on the CMS HLT data filtering. For reasons of scalability, low latency, schema flexibility and file format compatibility, the metadata JSON files are indexed with a distributed Elasticsearch NoSQL database, combined with an HTTP interface for querying. Elasticsearch also offers convenient, built-in, advanced functionality, such as aggregations, which can potentially lead to even lower latency, by discarding redundant result set objects as early as possible. Real-time monitoring applications can typically use these metadata by submitting queries to the Elasticsearch service, process the results (serverside) and provide useful insights in human-friendly format, e.g. browser-based visualizations (client-side). Such applications are used, for instance, by the shift crew at the CMS Control Room. One such application, F3Mon, has been redeveloped as the core part of this project. As the client-side part of the application (polling mechanism and GUI) already uses an adequate, standard combination of HTML, Javascript and CSS, as well as the Highcharts JS library, our work was focused mostly on re-developing the server-side part. This included moving from an Apache server to a Node.js environment and replacing all PHP server functions with asynchronous callbacks, written in Javascript. This is expected to scale better with a growing number of users, as all callbacks are asynchronous within the same context (single thread server) and responses can be cached across different requests without blocking. Moreover, using Node.js enabled us to rewrite the Elasticsearch transactions in a more simplified way, as JSON responses by Elasticsearch are conveniently supported by Javascript. Also, by using the elasticsearch.js library, it was made possible to query Elasticsearch asynchronously. In addition to the standard F3Mon functionality implementation explained above, our work also included the implementation of a web server response cache mechanism (enabled by the use of a single context for all requests, a novelty of Node.js that did not exist in the multithreaded PHP server), which is expected to decrease the number of Elasticsearch queries performed, thus reducing the client waiting times and the Elasticsearch server load. Although the cache expiration times are kept small, in order not to impact the application's latency, our measurements show that there is a clear benefit in using cache. As the number of users grows, a larger and larger percentage of requests are served using cached information, rather than fresh retrievals from Elasticsearch. Clearly, this is an indication that scalability is possible and the existence of more users can actually help reducing the average per request waiting times, within the reasonable infrastructure's limitations. Finally, the project also included the redeveloping of parts of SC (Figure 1.1), an application associated with F3Mon, although SC also retrieves data from an Oracle relational database, apart from Elasticsearch. The redeveloped parts have been incorporated with F3Mon in a single Node.js service and they also use the server cache for faster responses and fewer queries to the database. Figure 1.1: Accessing associated SC app through the F3Mon user interface ( Tribe button)

4 2. System description Architecture The system developed during this project (central component of the Figure 2.1) is the server-side part of the Elasticsearch-based F3Mon application. It was developed as a Node.js server, which contrasts with the Apache/PHP that the previous version was built on. Our system also includes the redeveloped SC (Oracle-based app) requests, incorporated in a single Node.js instance with F3Mon. In the immediate future, it is possible to incorporate the entire SC methods set with F3Mon, after converting the remaining PHP requests to Node.js. For the moment, the non-yet-converted SC requests are also served by our system by running their original PHP code with the node-php library. As a consequence, these requests are neither asynchronous (although they still use multithreading), nor can they use the server cache facility yet. Both F3Mon and SC (converted parts) can use the same cache, albeit their different needs should define the cache expiration times accordingly. For example, SC should use longer expiration times to avoid flooding the database with demanding queries. This is a less significant issue for Elasticsearch. Figure 2.1: System and environment diagram The central component corresponds to system developed in this project Components and roles In particular, our Node.js server comprises of the following development units: The collection of asynchronous Node.js callbacks, written in Javascript. There is one callback for each request (method) that the client-side polling mechanisms call. No distinction exists between an F3Mon and an SC callback, apart from their different URN prefixes (/f3mon/api for F3Mon, /sc/api for SC). Naturally, though, they employ different

5 approaches of implementation: F3Mon uses the elasticsearch.js library to query Elasticsearch by using asynchronous callbacks, specifying the index, the mapping and the query body, which is normally loaded from a JSON file and gets updated with request parameter values. To speed up the requests, our most recent version preloads these files in memory objects when the server is started. SC uses the oracledb library to query relational database by using asynchronous callbacks, specifying SQL query, which depends on the request parameter values. Both seek a response in the cache, by using a key built by concatenating the request name, argument names and argument values. All cached entries are formatted and ready to be sent. If an entry is not found or is expired, normal request service flow runs (unless an identical request's handling is ongoing, in which case a special procedure is followed, to avoid processing the same request multiple times in a short time frame). Normal request service flow typically includes one or more Elasticsearch/Oracle queries. Subsequently, a response is built, formatted into a JSON object, stored in the cache using the key and returned to the client-side caller. External libraries that are needed for the server to run, for example oracledb, node-cache, express etc. Node-php is also required until all SC requests are converted to Node.js. Nodephp allows us serving older, PHP-coded requests through our Node.js server. The collection of the client-side files, such as GUI and polling scripts, CSS style files, HTML pages, images, libraries etc. that are downloaded by the client for local execution. The collection of JSON files that are used to store configuration data, cache expiration times and Elasticsearch query structures. The rest of Node.js Javascript code, excluding the callbacks, which includes helper functions, initialization procedures for the server and the cache, logging configuration, library and files loading, etc. The server-side Javascript code and the client-side code make up most of this project's source code. While the client-side code is almost identical to the one also provided by the PHP version, the Node.js callbacks and the rest of the server-side code was written during this project. Changes have also been performed in the JSON-based Elasticsearch queries (stored in static files) in order to accommodate new features and improvements, introduced by our Node.js version. Figure 2.2: Real-time monitoring with F3Mon (view of the client-side GUI)

6 3. Contribution and Novelty Implementation of callbacks in Node.js Although the Node.js server runtime environment is a novelty introduced to the monitoring system by this project, PHP implementations of the services were available from previous versions. As the client-side part assumes certain input and output formats that the PHP versions provided (HTTP GET and JSON over HTTP), our system's development largely followed legacy code styles, flows and granularity. Each PHP server function was rewritten as a new, equivalent Node.js asynchronous callback, so that compatibility with the client-side is preserved with minimal clientside code changes. Nevertheless, numerous adaptations had to be done and challenges had to be addressed during the conversion process. To begin with, PHP and Javascript differ in numerous native functions, as well as in data structures. It was quite often impossible to precisely follow the legacy implementations with the available Javascript tools, therefore we often had to implement from scratch, taking only the requirements into account. Moreover, the replacement of synchronous Elasticsearch/Oracle queries with queries in asynchronous callbacks, often required completely different coding approaches, especially when multiple, interdependent queries had to be executed in order to serve a single request. However, by addressing the challenges above, we had the chance to spot and fix existing code defects, to improve implementations by taking advantage of Javascript characteristics and to update Elasticsearch transaction implementations in accordance with the most recent Elasticsearch Reference (e.g. facets were replaced with aggregations, as the former will be soon deprecated). The extensive use of asynchronous callbacks is also instrumental in reducing server blocking during a request serving, for example by parallelizing data store queries. Therefore, the switching we performed, from Apache/PHP to Node.js/Javascript, has resulted in creating a fully functional, non-blocking web application that can also employ cache because of shared data among requests. The application has been tested under real conditions, as a production version is already running for users. There is, also, a potential to make even more out of the new environment, for example by fully exploiting the asynchronous callback features when converting the remaining SC requests. Some of these requests include particularly expensive Oracle queries that are suitable for caching. Implementation of web server cache After the functionality implementation of our server, a web server cache mechanism was also developed (enabled by the Node.js architecture), using the node-cache library. It is expected to reduce the average client waiting times, as well as the number of Elasticsearch/Oracle queries, by locally caching responses formatted in previous client requests that actually submitted queries. Cached responses are stored in a map-like collection in memory, expire after ttl seconds (expiration leads to their deletion) and are stored as key:value entries, where key is an identifier string created from the called method (Node.js callback) name and parameters, similar to a URL with parameters. For instance, a response for a request to the getdisksstatus method, with parameters (runnumber=2015, sysname=cdaq) would be stored in cache using the key getdisksstatus?runnumber=2015&sysname=cdaq. This format can handle changes in the argument positions in a request's URL but if new arguments are added in a method, the respective key definition must be updated. The value is a response body, formatted as a Javascript object. The keys never include the parameter callback, as it is unique for each request. Similarly, the cached responses never include values for callback, as these refer to a previous request. The current

7 request's callback value is added to the cached response body before the actual response is sent back. Naturally, a new request will retrieve a (non-expired) cached response only if the name of the callback method it calls, as well as its argument values, exactly match the ones of the request that originally generated the cached response. Certain care has been also taken, using the cache mechanism, to avoid multiple, quasi-parallel handling of identical requests that arrived in a short time frame (version 2 cache). TTL values are stored in a JSON file and each Node.js callback is allocated an individual value, based on how frequently it is expected to return updated data. Other factors that affect the values are parent application-based (F3Mon or SC). These include their expected degree of realtime user experience, the frequency on which new data are injected in their respective data store and whether they use Oracle or Elasticsearch infrastructure; for example, the Oracle database should not receive too many queries. As a consequence, callbacks that serve the most intensively real-time parts (such as F3Mon parts directly linked to an ongoing run) have very small TTLs of just 1-2 seconds, while less frequently updated parts (usually belonging to SC) may have TTLs of 60 seconds. Despite the very conservative TTLs, the measurements presented in the following section reveal significant gains from using cache. Cache performance measurements As our motivation for a cache development was better scalability when many users/browser clients use the services simultaneously, we measured the cache mechanism's benefit in terms of number of total requests served using cached responses over the total number of requests served, as a function of the number of simultaneous clients. This is, basically, the percentage of requests that were served without significant server processing and without data store queries. A large such percentage contributes to better service scaling. Notably, each request's normal flow contains at least one data store query, which means that each request served from the cache entails saving at least one query and, usually, more. The flow also includes some processing to populate and format the response, while cached responses are already in output format. As a result, the server time (including processing and querying time, excluding server-client network time) to serve a request with a cached response is, on average, less than 1 millisecond. In contrast, the time for a full-fledged request handling is often orders of magnitude larger, on average, 40 milliseconds (though some requests returning chart data for ongoing runs can easily reach 100 milliseconds). Besides, this average time seems to increase when more users are served, which could potentially indicate scalability issues. Such a sensitivity was not observed in the average server times for the requests served from cache. Relevant measurements are provided in the Annex (chapter 6). In order to calculate the percentage of cache-served requests over some time with a growing number of users, we set up a simple statistics collector on the server, as well as multiple browser clients running the F3Mon application simultaneously, as in the scenario below: Method: We ran the test during an ongoing run, as then the client-side F3Mon polls the server more intensively (about 175 automated requests per minute per client) than in periods without an ongoing run (about 110 requests). So, studying scalability performance during a run should also cover the less intensive phases. Also, no users interacted with the GUI during this test. So, we only considered automated requests generated on fixed frequency by the client-side polling scripts. Users may generate additional requests through GUI actions (clicking buttons, using bars/textfields etc.) but these normally constitute only a tiny fragment of the total and can safely be ignored.

8 Data: We collected time-series data over a period of 8 minutes, for the variables: C : cumulative number of requests served from cache ( cache-reqs ) Q : cumulative number of requests served with full-fledged handling ( no-cache-reqs ) Data points: We collected 4 data points for each variable, for each time frame (one point every 30 seconds), as explained in the table below: Time frame Frame 1 Frame 2 Frame 3 Frame 4 Duration 2 min. 2 min. 2 min. 2 min. Clients added at frame's start Total simultaneous clients Chart 3.1: Cumulative number of requests per type as more clients come Chart 3.2: Evolution of the percentage of requests served from cache as more clients come

9 As we can see in the Chart 3.1, when F3Mon serves only one client, no significant benefits arise from using cache, because the client-side polls for data in wider time intervals than the usual TTL values, which are kept particularly small for F3Mon, as its real-time behavior must be preserved. Nevertheless, when more clients are present, more and more requests are served with cached responses, because it is then much more probable for a cached response to be used within a short amount of time by many clients, before it expires. In a best-case scenario, with k simultaneous clients, k 1 of them would be served with cached responses at any given moment. Noticeably, with 5 or more simultaneous clients, it seems that more requests are served using cache, than being served without using it. This tends to increase quickly as 10 simultaneous clients arrive, on which point 68.16% of the total requests were served using cache (Chart 3.2). To sum up, it is clear that good scalability at many clients is possible to achieve and actually it is a large number of simultaneous clients that adds value to the benefit introduced by using cache, as it manages to constantly keep the cache filled with valuable and up-to-date responses. Even better results could potentially be observed if the version 2 cache improvements are studied in a similar test (they were not included in the system when this test took place), or if the SC application is studied instead (as it uses more lax TTLs). 4. Conclusion As a general conclusion, it is possible to say that the web services implemented in Node.js during this project, combined with the cache mechanism we devised, have a potential to scale quite well when the number of users grows. The results of the pilot test we ran, indeed reveal a very good system behavior when more simultaneous clients arrive. Positively enough, this does not come at a cost of real-time user experience. Furthermore, the services we developed, simplify and update the server implementation and leave the wider system with a single language, Javascript, used in both client and server sides.

10 5. References Relevant material More information on CMS DAQ's relevant infrastructure and the monitoring system design (including previous implementations of the services) can be found in the material below. Parts of this report are based on this material, particularly on the first presentation. Meschi E., Morovic S., et.al., A scalable monitoring for the CMS Filter Farm based on elasticsearch, Presentation at the 21 st International Conference on Computing in High Energy and Nuclear Physics, Okinawa, Japan, (URL: Meschi E., et.al., File-based data flow in the CMS Filter Farm, Presentation at the 21 st International Conference on Computing in High Energy and Nuclear Physics, Okinawa, Japan, (URL: Mommsen R., et.al., A New Event Builder for CMS Run II, Presentation at the 21 st International Conference on Computing in High Energy and Nuclear Physics, Okinawa, Japan, (URL: Code All code written as part of this project is open and has been published at my personal Github (github.com/mvougiou/f3mon/tree/nodejs), forked from the original project at the CMS DAQ's Github (github.com/cmsdaq/f3mon). My personal Github contains the whole version history of this project between June and September A functional version has been merged back into the CMS DAQ's public repository, under the branch nodejs-rpm. Various external libraries have been also used.

11 6. Annex Table 6.1 Metadata Numbers/Percentages of requests per type Recorded service absolute and average times Time Total server Average server Total server Average server Time Number of C+Q (total time spent time per Percentage of time spent time per offset simultaneous C Q requests on nocache-reqs no-cache-req request that is cache-reqs (%) on cachereqs (ms) cache-req (ms) request that is (s) clients received) (ms) (ms) Start N/A Frame Frame Frame Frame Table 6.1: Measurements of the test described in chapter 3 Notes about the table data All absolute values are cumulative quantities. All average values are calculated with micro-averaging, using the absolute cumulative values in the same row and not other average values in the same column. Each row in the percentages (%) column is calculated using the absolute cumulative values in the same row.