Applications of the Embedded In-Memory Technology in the Public Sector

Transcription

1 SEP 33 ALLEGRO v Applications of the Embedded In-Memory Technology in the Public Sector Bitkom Big-Data.AI Summit 2018, Hanau, February 28 - March Eldar Sultanow, Andreas Jung, Martin Blomenhofer, Philippe Masson

2 General Application areas of the Embedded In- Memory Technology in the Public Sector Fraud detection Supervision of private sector regulated activities Sentiment analysis of Internet content for the prioritization of public services Threat detection from external and internal data sources for the prevention of crime, intelligence, and security Prediction for planning purposes of public services Munné, R. (2016). Big Data in the Public Sector, in: J.M. Cavanillas et al. (eds.), New Horizons for a Data-Driven Economy, pp Cham, Switzerland: Springer. DOI: / _11 Seite 2

3 General Application areas of the Embedded In- Memory Technology in the Public Sector (cont.) Opinion Mining Public Opinion Mining for Governmental Decisions Complex analyzes in real-time using up-to-date full data sets (SQL-based) Downstream analyzes can be performed directly with a standard data warehouse solution (SQL) using the original data sets as same input e-participation Perform and analyze votes including multiple parameters such as geographical information Use many in-memory databases (decentral approach, geographical partitioning) at edge locations in the cloud for ingesting the data Federal and state authorities can query and join data (votes, opinions, referendums) efficiently Stylios, G, et al. (2010). Public Opinion Mining for Governmental Decisions, Electronic Journal of e-government, 8(2), pp Weerakkody, V. (2012). Technology Enabled Transformation of the Public Sector: Advances in E-Government, Hershey, PA: IGI Global. Irregularity Detection Evaluate performance, quality of text mining classification algorithms applied to detect irregularities in public sector records Measurements include only minimal footprint (no I/O waits, network waits) Suitable for statistical analyses, since original data set always consistent between runs/environments Handle large public data Large public data relating to overall citizens (range of millions) Queries are generally short (and simple), but occur very frequently Avoid footprint during performing many tests that are less CPU-intensive Santos, B. S., et al. (2015). Comparing Text Mining Algorithms for Predicting Irregularities in Public Accounts, Proceedings of SBSI 2015 Seite 3

4 General Application areas of the Embedded In- Memory Technology in the Public Sector Opinion Mining Public Opinion Mining for Governmental Decisions Complex analyzes in real-time using up-to-date full data sets (SQL-based) Downstream analyzes can be performed directly with a standard data warehouse solution (SQL) using the original data sets as same input e-participation Perform and analyze votes including multiple parameters such as geographical information Use many in-memory databases (decentral approach, geographical partitioning) at edge locations in the cloud for ingesting the data Federal and state authorities can query and join data (votes, opinions, referendums) efficiently Irregularity Detection Evaluate performance, quality of text mining classification algorithms applied to detect irregularities in public sector records Measurements include only minimal footprint (no I/O waits, network waits) Suitable for statistical analyses, since original data set always consistent between runs/environments Handle large public data Large public data relating to overall citizens (range of millions) Queries are generally short (and simple), but occur very frequently Avoid footprint during performing many tests that are less CPU-intensive Seite 4

5 Development background Federal Employment Agency (FEA) Central service provider at the labor market Public body with self-administration Approximately 130,000 employees Central with 10 Regional Directorates 156 agencies for work 616 branch offices 303 Job center (ge) 12 Education and care centers (BTS) 7 Special departments in 1,600 Properties FEA-Information Technology Short profile Headquarters: Nuremberg Employees: 2,400 Interlinked PC: 160,000 Server: 9,200 System landscape 120 individual IT procedures Two highly available central data centers Nationwide 11 regional data centers (RRZ) Output (monthly) -Volumes: 41.5 million s Transfers: 16.5 million transfers (EUR 7.9 billion) Posts: 11 million mails Print pages: 56 million pages ALLEGRO 3.3 million households with a total of 6.4 million people Up to 50 thousand parallel users Approximately 60 developers Approximately 100 former developers Seite 5

6 Initial position test requirements in ALLEGRO Test requirements in ALLEGRO Tests should be executed for every single code commit All tests should be executed at all times (GTS: Global Test Suite) Code commits pass through the quality gate (Gerrit) only with a successful test run Automatic execution of GTS for every commit including the result check (CI build) Sub-project TEST has deviating requirements Current test setup An installation of Oracle XE database is carried out for each machine Tests are executed on the local machine of the developer Each test run is provided with a blank, initialized database An independent life cycle is initiated on the server for each test run A test suite with a fixed sequence of tests is defined for a test run The tests are executed sequentially in currently running VM Seite 6

7 Problem areas Need for optimization during test lead time Data volume very large Complexity high and still growing Test runs take a long time Number of test ~ 18,000, majority are integration tests Continuously increasing number of tests due to further development the runtime increases proportionately Database required for the tests (Schema DDL + test data) grows from release to release Complex integration tests (with their dependencies) Test setup is not trivial, for example, database must not be emptied constantly after every test (certain data must be always initialized) The runtime of GTS is within hourly range Maximum turn around time in the database unchangeable changeable Seite 7

8 Preliminary considerations Safeguarding inventory of previous test requirements Tests should be executed for every single code commit All tests should be executed at all times (GTS: Global Test Suite) Code commits pass through quality gate (Gerrit) only with a successful test run Automatic execution of GTS for every commit including the result check (CI build) Additional test requirements Tests are not interdependent Least possible installation effort of the database Each test VM should have its own, independent, newly created database The database is reset to initialized version after every test Seite 8

9 Solution approach Introduction of a new test system with embedded In-memory database 1 Embedded In-Memory DB versus local Oracle DB (single instance) No installation is required, low maintenance costs due to portability Database duplication using simple file copying (dumps are not necessary) Higher speed for runtime since waiting periods are eliminated: No access to IO hard disk Network IO accesses are no more available (Embedded DB in the same JVM of the tests) 2 ALLEGRO-specific advantages by innovative application of embedded In-Memory-DB concept in the context of process ALLEGRO's new test system architecture (with embedded in-memory DB) enables easy execution of parallel test Provision of database instance contingent with rotation principle Seite 9

10 Which embedded In-Memory DB do we consider? Comparison of embedded In-memory database technologies Oracle Derby HSQLDB H2 Type of Installation preinstalled On-demand Portable - SQL-Standards 1 Supports Oracle Features - 2 In-Memory Support 3 (tables only) (full DB and table-specific) Runtime-embedded - Overheads Networking, disk writes 4 none Supports native SQL functions, triggers - DML only - 1) Support is restricted up to SQL Standard 99 2) Support is restricted with regard to accepted data types, clauses, automatic data conversion, conversion of return values 3) Oracle Database 12c is required; In-Memory table feature requires appropriate license fees 4) Disk writes overhead can be reduced by the use of In-Memory tables Seite 10

11 HSQLDB as the first choice for ALLEGRO Comprehensive analysis in ALLEGRO and exchange with other projects leads to result: HSQLDB is the most suitable target platform Reasons for HSQL DB HSQL, H2 and Derby are the only Java-based embeddable In-memory solutions Derby is opted out, because there is no support for Oracle features at all (as Oracle remains being used in production) No support for schema cleaning in H2 (SQL command TRUNCATE SCHEMA killer argument) This statement would have to be split up into individual TRUNCATE TABLE... statements (which require way more time to execute while preparing the database for the next test) H2 does not support the SQL:2003 standard command MERGE INTO (using a proprietary implementation), whereas HSQLDB does This statement is widely used in our initialization script and within the application server and the conversion would be costly HSQLDB is the only in-memory DB that supports SQL-based procedures and functions Not many used in ALLEGRO, but they are essential and would entail a noticeable conversion effort Only syntax adaptation with some refactoring to SQL standards is required in HSQLDB Some procedures are Oracle specific, but only used for partitioning purposes, which neither work in the Oracle XE instance nor are necessary for tests HSQLDB is the only DB that supports the Oracle Feature for Synonyms Seite 11

12 Architecture of the test system Database-Instances contingent with rotation principle Architecture with containerprinciple Working Databases DB i n Test 7 State: waiting Rotational reuse of DBs Causal test dependencies can still exist with performance gain Test Controller DB i m+2 Test 4 State: waiting Reduce DB initialization time to almost ZERO DB i m+1 Test 3 State: running Modern architecture, which is rediscovered in cloud-environments Test 7 w Test 4 w Test 3 r Test 2 f Test 1 f Scoreboard DB i 0 IDLE Databases DB i m Test 2 Test 1 Test Set State: finished DB Connection: terminated Seite 12

13 Architecture of the test system Database-Instances contingent with rotation principle Time Frame DB 1 Test 3k+1 init Test 3(k+1)+1 init DB 2 Test 3k+2 init Test 3(k+1)+2 DB 3 init Test 3(k+1) init Test 3(k+2) Test sequences can be maintained and parallelization is still possible Valuable initialization time is reduced completely Seite 13

14 Result of measurement of runtime reduction in ALLEGRO Runtime of tests (min-max range in seconds) Oracle 1 Fork 2 Forks 3 Forks 4 Forks 5 Forks 6 Forks 8 Forks Developer-PC: 32 GB Ram, Xeon 4-core, w/hyperthreading Drastic reduction in test runtime Saturation from determined degree of parallelization Seite 14

15 Reasons for introduction in other processes Available technical basis High standardization due to established cloud-process like dockerization Tests (and development) are already standardized according to SERA (framework of FEAs software development process), which forms the basis for cross-procedural introduction High test efficiency and flexibility for all processes All the processes can benefit from high performance and flexibility of the approach introduced. The test runtime can be reduced drastically mainly due to major/mission critical processes having complex test cases and high data volume. Thanks to dump mechanism, the resources are left clean, which in turn reduces the errors related to configuration (It is always possible to achieve a clean status) Synergy can be achieved in the testing: For example, an overall test with ALLEGRO and linked VERBIS- or StEP-container Seite 15

16 Summary Conversion efforts were profitable Advantages Disadvantages Number of tests are now manageable and can continue to increase In future, the process can be expedited due to hardware expansion Execution platform is irrelevant Migration to Cloud is possible Guaranty for developer: Tests do not cause any side effects During development, the DB must be initialized again only for modified scripts Simple implementation of rotation principle Oracle specific features are eliminated/ SQL standards are restricted Migration scripts must be tested on both DBs at all times Total conversion efforts are 60 BT Rotation principle is useful only for specially tailored/customized tests Seite 16

17 Contributing Authors Eldar Sultanow Andreas Jung Capgemini Capgemini Bahnhofstraße 11C Nuremberg, Germany E: Bahnhofstraße 11C Nuremberg, Germany E: Martin Blomenhofer IT Dept. of the Federal Employment Agency Südwestpark Nuremberg, Germany E: Capgemini Philippe Masson Bahnhofstraße 11C Nuremberg, Germany E: Seite 17