DELL EMC ISILON FOR KDB+ ALGORITHMIC TRADING

Transcription

1 WHITE PAPER DELL EMC ISILON FOR KDB+ ALGORITHMIC TRADING A use case for historical, large scale near real-time, and high concurrency storage for algorithmic trading Abstract This whitepaper describes the Dell EMC Isilon Scale-Out NAS Storage solution for Kx Systems Kdb+ database applications used in the financial services industry. It presents an overview of the Kdb+ near real-time and historical data workflows to understand the suitability of the Dell EMC Isilon platform as the storage platform of choice for these workflows. April 2018 Dell 1 Dell EMC EMC Isilon Isilon Storage Storage for Kdb+ for Kdb+

2 TABLE OF CONTENTS Introduction... 3 Document purpose... 3 Solution purpose... 3 The business challenge... 3 Kdb+ overview... 4 Kdb+ workflow introduction... 4 Kdb+ server requirement... 5 Solution overview... 6 I/O profile... 6 Data layout and file systems... 7 Scaling capacity Scaling performance Automatic tiering Sample use case and configuration Customer profile & requirements Sample configuration Performance Tests Conclusion References KX documentation Dell EMC documentation Dell EMC Isilon Storage for Kdb+

3 Introduction Document purpose The document describes the solution overview of the Dell EMC Isilon Scale-Out NAS Storage for Kx Systems Kdb+ applications Solution purpose The purpose of this document is to present an overview on the Kdb+ workflow and to understand the suitability of Isilon Scale-Out NAS platform as the storage platform of choice for these workflows. The business challenge Massive data volumes, low-latency and complex event processing have been recurring themes in recent years. For example, five years ago, the NYSE averaged half a billion trades and quotes per day. Today this is typically ~4 billion trades and quotes per day, with peaks of over 8 billion trades per day in Options traders now receive over terabytes of data per day, while stock exchanges have built huge co-location datacenters where dealers run algorithms directly on the data feeds, reducing latency to an absolute minimum. Kdb+ is the ideal solution for this market. Designed from outset in anticipation of vast increases in data volumes and demands for timecritical applications, the kdb+ database is used globally for high performance, data-intensive applications by a variety of clients, including banks, investment dealers, telecoms, insurance companies and government agencies. Kdb+ is used in most of the Top 100 Financial institutions across the globe. The Kdb+ data set can be classified as 1. Real-time database (RBD) - A real-time database (RBD) stores today's data. Typically, it would be stored in-memory during the day, and written out to the historical database (HDB) at end of day. Storing the RBD in-memory results in extremely fast update and query performance. 2. Historic database (HDB) - A historical database (HDB) holds data before today, and its tables would be stored on disk as being much too large to fit in-memory. Each new day's records would be added to the HDB at the end of day. These datasets can grow anywhere from a few terabytes to multi-petabyte data sets. Typically, the active or real-time data set is a very small percentage of the total storage requirement for Kdb+. Historically, customers have used local DRAM of their client machine (that runs the Kdb+ application) for their real-time data storage needs. Even this model is starting to morph as the data volumes and business models change to carry positions for multiple days. Customers are now exploring an architectural shift to support weeks of model data for their current financial investment positions. As such, real-time is morphing as the lack of volatility in the markets causes firms to shift their investment postures from exclusively intra-day to a range of intra-day to multi-day to multi-week. As a result, a high performance data storage solution is required for the near real-time and historic data set. Some of the requirements for this solution are: Highly scalable solution (Data sets size can scale from a few terabytes to multi-petabytes to tens of petabytes) High Read and Write performance (mostly sequential Read I/O for historic and real-time data) High concurrency for massively parallel queries and concurrent user access High performance interconnect between the Kdb+ clients and the storage system Additional desirable requirements include: Automated tiering: Faster access to recent data Automatic optimization of data allocation across the disks Automated replication for business continuity and disaster recovery Support for multiple storage technologies (i.e. SSD, SAS and SATA disks) by the solution 3 Dell EMC Isilon Storage for Kdb+

4 Kdb+ overview Kdb+ workflow introduction Kdb+ is an application and database framework for ticker data. Ticker data comes directly from various feeds (NYSE, NASDAQ, etc., as well as aggregators such as Thomson-Reuters). The components of a Kdb+ database are as follows: Runs on industry-standard Linux, Solaris, Windows, Mac OS X 64-bit server platform. Has no restrictions on storage architectures use local disks, SANs, and NAS and configure them according to any preferences. Stores databases as ordinary native files. This means that ordinary database administration (e.g. backup, recovery etc.) can be done with standard NAS commands and utilities. Scalable column-oriented database for massive data volumes. Manages real-time and historical data in one platform. Includes a general-purpose programming language, q that has direct support for databases. Direct support provides an enormous advantage over systems that force the user to rely on traditional SQL for data access, or create a dependency on the supplier for pre-written queries. While it is possible to use Kdb+ as the ticker archive platform, most customers keep Kdb+ separate from the rest of their ticker archive data, which is usually stored in a compressed text or binary format, and is accessible by a wider variety of applications. Figure 1: Kdb+ Database, Services, Clients Other trading clients access the Kdb+ servers to use the services such as analyzing, aggregating, filtering, and publishing the ticker analysis data. 4 Dell EMC Isilon Storage for Kdb+

5 A typical Kdb+ workflow is depicted in the Figure 2 flowchart: 1. Data is fed into the compute servers from the Isilon cluster. Based upon the need for real-time, historical, or archive, different nodes types and performance characteristics can be deployed. This would run from the range of Flash based technology like the Isilon F800 to the historical capabilities of hybrid servers (i.e., H600, H500, and H400) and lastly to archive tiers (i.e., A200 and A2000). It is not uncommon to have several thousand concurrent connections for models to be run on any given tier of Isilon. 2. Data is fed into the Isilon via the real-time layer of the cluster. Data is tiered between the Isilon performance tiers based upon custom or generic rules for data aging and usage. Real-time can be days to weeks. Historical can be weeks to decades. Archive can be leveraged for certain data sets or not at all. 3. Based on whether the data is real-time or historic data, the data is processed in a certain manner by the Kdb+ server a) For real-time data, the Kdb+ server will process the data from the local DRAM or from an Isilon F800 all flash array. For this reason, the Kdb+ servers are on high-end server equipment. The next section talks about a sample Kdb+ configuration. b) For historical data, the Kdb+ server transforms the data into Kdb+ format, which is typically organized into a columnar format and writes to high performance storage. In the case of Isilon, this storage can be H or F series nodes (F800, H600, or H500). Figure 2: Typical Kdb+ Workflow Kdb+ server requirement Typical Kdb+ deployments are on high-end server equipment, with a large amount of DRAM (from 512GB to over 2TB) and CPU cores (from 16 to more than 144). As a minimum, it is recommended to have RAM of at least four times expected data size. Thus for 5 GB data per day, the RBD machine should have at least 20 GB RAM. In practice, much larger RAM might be used. The application itself is multi-threaded, and spawns as many threads (Q processes) as there are CPU cores in the system. This model allows for a high degree of parallelization when running queries against a large working set. This also means that the storage housing the data needs to be fast enough to supply the host with a high degree of performance. 5 Dell EMC Isilon Storage for Kdb+

6 Solution overview This section will discuss the I/O and file system requirements for a Kdb+ storage system and the suitability of Isilon Scale-Out NAS solution for this workflow. I/O profile Underneath the covers, Kdb+ makes heavy use of memory mapping files (mmap()) into local memory. At the time of execution, each thread will: 1. Scan its assigned mount points for all files and subdirectories 2. Perform metadata reads on each file (lookup, getattr, access) 3. Perform an mmap() call on each file 4. Await instruction from the parent thread When a file is memory mapped, no actual data is read or memory used until the application actually requests the page. At this point, a page fault occurs, and the page of data (typically 4k) is requested. In the case where the data lives on an NFS mount, a page fault results in an NFS read request against the file handle obtained by the client when the process was spun up. The size of this read request can vary, but typically will be equal to the rsize value set when the NFS mount was established. For sequential workloads, setting this rsize to a large value (128KB) can be beneficial, as the client will effectively perform read ahead on the first 4k page fault, causing subsequent 4k page faults to be served by the client s local DRAM. For more random workloads, setting the rsize to a smaller value (4KB or 8KB) may be beneficial, as read-ahead is likely to be wasted. It has been observed in testing and at customer sites that the majority of Kdb+ workloads: Are highly sequential I/O Are read intensive (90%) Are multi-threaded (from 16 -over 40 Q threads per host) Operate against large datasets (50+ TB of historical data, 5-10% of which is most active) This IO profile is very well suited to an Isilon cluster. The highly concurrent workload with a mostly read profile takes advantage of all the strengths of Isilon. Reads on Isilon do not incur the overhead associated with latches or locks like other storage operating systems. This allows for embarrassingly parallel connects to a single file or file system. This allows Kdb+ jobs to scale out as required to complete the job. 6 Dell EMC Isilon Storage for Kdb+

7 Data layout and file systems The way data is laid out in a Kdb+ environment is somewhat unique. All data for a given date is placed in a single directory. Files range from 100MB to greater than several GB in size, typically files per dated directory. The following is a truncated example: In Kdb+, each file is a full database table. The above example assumes only a single mount point (db), which is non-optimal for the following reasons: 1. A single mount point can be accessed by only one Kdb+ thread (Q process) 2. A single mount point implies either: a) A single disk or LUN b) A single NFS mount point 7 Dell EMC Isilon Storage for Kdb+

8 Both points above imply that a single mount point becomes a bottleneck for Kdb+, and would limit performance to that of a single thread. The solution instead is to spread the data across an array of partitions, with the following guidelines: 1. The number of partitions should be equal to, or divisible by the number of Kdb+ threads (generally one per CPU core) 2. Each partition should contain an equal amount of data, in order to ensure that each thread can complete in an equal amount of time. 3. Each partition should perform equally to all other partitions. Kdb+ provides a simple method to allow the partitioning of Kdb+ data across an array of mount points. A single text file (par.txt) is populated with a list of directories, one per line. If the storage to be used is DAS or SAN attached, each line represents a discrete file system. If a NAS storage is to be used (in the case of NFS mounted file systems), each line can either be a separate file system or simply be a separate subdirectory on the same file system. Kdb+ is intelligent enough to balance new data on ingest to each of these mount points. Dated directories are effectively round-robin assigned to a mount point. A single day worth of data will live on at most one mount point. When configuring par.txt to attach to an Isilon cluster, the following guidelines should also be followed: Leverage as many nodes as possible Each node used should be used an equal number of times (e.g.: if there are three nodes, and six threads, make sure that there are two mount points per node) Each mount point should go to a separate subdirectory If applicable, spread the mount points across more than one client-side forty gigabit Ethernet network interface. A sample par.txt to a 3 node Isilon NAS system would look something like this: /mnt/isi-1/subdir_1 /mnt/isi-2/subdir_2 /mnt/isi-3/subdir_3 /mnt/isi-1/subdir_4 /mnt/isi-2/subdir_5 /mnt/isi-3/subdir_6 8 Dell EMC Isilon Storage for Kdb+

9 Figure 3: Sample par.txt to a 3 node Isilon NAS system Underneath each subdirectory would live an equal portion of the dataset. Note that in the above example; only six mount points are defined, which means this configuration is only valid when running Kdb+ with six or less threads. When a Q process is started with slave threads (as is the case most of the time), each of the partitions in par.txt is handed out to slave threads in a round-robin fashion. If there are more par.txt lines than slaves, each slave will get assigned some number of par.txt entries. Note that this is why it is beneficial for the number of lines in par.txt to be divisible by the number of slave threads. For example, if you have a par.txt file with twelve mount points, and you have four slave threads, then each slave thread will get three mount points, resulting in even performance across all threads. If the balance of mount points among slave threads is not even (e.g.: if we had fourteen mount points in the above four-thread example), then performance between threads would be uneven. The runtime of a query is gated by the speed of the slowest thread. If some threads are responsible for a larger set of data than others, they will slow the entire query down, as idle threads cannot be dynamically reassigned to share another s work. The goal of many Kdb+ queries is to scan through a date-range worth of ticker history (e.g.: the last 21 days), and return a result, such as find all trades for all symbols where bidprice was greater than 50'. This results in a full 'table-scan' of all data in this date-range. Each Q process would perform scans of data under their jurisdiction. When each slave thread is finished reading data, it reports the result to the parent thread, which then collates the data and presents it back to the user or application that initiated the query. Note however that the parent thread is not able to tabulate the results until ALL slave threads have reported back. This means that the overall runtime can be negatively impacted when a single thread underperforms its peers. This is the primary reason why load should be as evenly balanced among threads as possible. 9 Dell EMC Isilon Storage for Kdb+

10 Scaling capacity In order to scale the amount of storage available to a Kdb+ instance, customers have two basic approaches available to them: 1. Extend size of current filesystems (as defined in par.txt) 2. Create additional filesystems (add lines to par.txt) Option #1 is only achievable if the filesystems/mount points defined in par.txt can continue to scale capacity. Traditional architectures typically have per LUN or per volume size limitations, which may make this prohibitive. Figure 4: Traditional DAS architecture On an Isilon cluster, the mount points are merely directories sitting on top of a single scale-out volume, which can grow to be as large as 68PB. An additional consideration when scaling is to ensure that data remains balanced between all partitions. In a traditional disk or LUN based environment, this means that introducing new partitions (adding lines to par.txt) requires a shuffle or rebalance of data. Shuffling data among partitions requires I/O travel through the host, and can represent a significant amount of time and I/O, depending on the amount of data which is required to move. During a shuffle, queries cannot be run against the instance. On an Isilon cluster, reshuffling merely requires moving directories from one folder to another, requiring no I/O to the host, and takes seconds (or less) to complete. Scaling performance At some point, the I/O deliverable to a client may be bottlenecked on the storage to which it is attached. Similarly to scaling capacity, this requires either adding additional or faster spindles to existing partitions, or creating new partitions. The latter of these options would require modification of par.txt, as well as a reshuffle of data. With an Isilon scale-out NAS solution, scale is linear for storage, compute, memory and I/O. Additionally, adding nodes to an Isilon scale-out NAS does not require a manual reshuffle of data, as Isilon uses its auto balance feature to automatically rebalance data across the entire cluster. Automatic tiering As with most data, only a portion of a historical Kdb+ instance will be accessed frequently. Generally, the most recent thirty days worth of trading data is queried frequently, with older data being read periodically. For example: one back history strategy may involve scanning the past thirty days of trade data on a daily basis to validate new and existing algorithms. Another strategy may involve scanning the past 180 days of trade data on a weekly basis, and so on. For this reason, it is most cost effective to develop a tiered approach to storing this data. Isilon provides this functionality inherently within the OneFS file system using SmartPools, moving data from tier to tier without the need to change mount points or directory structure. All data, regardless of storage tier, continues to be accessible to clients, without downtime or application reconfiguration. 10 Dell EMC Isilon Storage for Kdb+

11 Sample use case and configuration Customer profile & requirements A customer profile example is the investment arm of a global financial institution. The requirements for their near real-time and historic Kdb+ NAS storage are: Capacity to scale from multi- terabyte to multi-petabyte (e.g. 40TB to 2PB) High Read performance (e.g. 3GB/s per client thread) High Write performance: (e.g. up to 1GB/s per host) Fast Interconnect between Kdb+ servers and NAS storage: 40 Gigabit Ethernet One primary site and one replication site Another requirement could be to store the ticker data archive in Isilon storage as well. Sample configuration For the requirements listed in the previous section, a sample configuration could be something such as: 2 x 4 node Isilon F800 cluster (one for primary and one for replication site) Read and Write performance - Each F800 will provide up to 7GB/sec in sequential read performance in a Kdb+ environment, resulting in over 15GB/sec in aggregate throughput from a 4 node cluster. As the number of Kdb+ clients increase, the Isilon cluster can be scaled linearly. Also, if ticker data archival is a requirement, a separate Isilon archive node cluster (A200 or A2000) can be used based on current capacity. 11 Dell EMC Isilon Storage for Kdb+

12 Performance Tests To understand the performance profile of the Isilon F800-based solution, we recently took an Isilon F800 all-flash NAS, paired it with Dell PowerEdge server, and Kx Systems KDB+ database and ran the STAC-M3 benchmarks, Antuco and Kanaga. The specific configuration is shown in figure below. Together these tests run the end-to-end tick data analysis from real-time trading scenarios to scaled historical batch analytics. The results below compare time-to-completion for key use cases with those of a competing platform with the closest performance and a comparable configuration. The effect of concurrent IO, high IOPS and aggregate throughput delivered by Isilon F800 are directly reflected in the benchmarked completion times. STAC-M3 Configuration STAC Benchmark Highlighted Results The results showed the sweet spot for Isilon All-Flash is its ability to deliver near real-time performance on large data sets at high concurrency. *Based on the STAC M3 Antuco benchmark test by Dell EMC in February 2018, comparing similarly configured solutions, Dell EMC Isilon F800 vs. Netapp EF550 Flash Array. Actual performance will vary based on configuration, usage, and manufacturing variability 12 Dell EMC Isilon Storage for Kdb+

13 Conclusion With Dell EMC Isilon OneFS, organizations and administrators can scale from 18 terabytes to 68 petabytes within a single file system, single volume, and with a single point of administration. OneFS delivers high-performance, high-throughput, or both, without adding management complexity. The Kdb+ product has been designed in anticipation of vast increases in data volumes. The ability of OneFS to scale to multipetabytes in a single file system -while delivering high performance for the sequential read/write I/Os makes Isilon storage ideal for Kdb+ s near real-time and historic workflows. References KX documentation Kdb++ Whitepaper - Kdb+ FAQ - Dell EMC documentation Dell EMC Isilon OneFS Operating System EMC Isilon Scale-Dell EMC Isilon Overview Learn more about Dell EMC [product name] solutions] Contact a Dell EMC Expert View more resources Join the conversation with #HashTag 13 Dell EMC Isilon Storage for Kdb Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. Reference Number: H17106