A web-based Census of services: an ISTAT evolutionary study

Transcription

1 A web-based Census of services: an ISTAT evolutionary study Annalisa Cesaro, Bernardo Palazzi, Manlio Paterniti, Paola Ranaldi 5/03/2013

2 1. The Italian experience in numbers 2. The Italian experience: the web architecture 3. The census web technologies CENSUS TODAY 4. The Census Management System (CMS): a metadata-driven system 5. CMS: a functional overview 6. CMS: building blocks for a cost effective improvement 7. What is SOA? 8. SOA as a Web Service Architecture SOA PROPOSAL 9. SOA as a Web Service Architecture in Census LAN 10. SOA criticalities 11. SOA-based web solution ISTAT exploration: technical feasibility 12. SOA-based web solution ISTAT exploration: service overhead ISTAT STUDY 13. SOA-based web solution ISTAT exploration: network latency 14. Conclusions

3 The Italian experience in numbers online Census Surveys: 60 thousand forms for 2010 Agricolture Census (3%) around 7000 census operators 8,5 million forms for 2011 Population and Household Census (one third directly filled by citizens) (33%) around census operators 420 thousand forms for 2012 Industry and Services Census (54%) around 3000 census operators

4 The Italian experience: the web architecture Census network internet Online questionnaire (OQ) internet Census Operations Management System (CMS) Documentation System (DOC) Oracle MySQL

5 The census web technologies GUI QO Actions Not-SOA GUI CMS Actions Replicated mappings connection Replicated mappings connection Process for Postal System on its own IT Web site Typo3 Microdata Processing for dissemination on its own Replicated mappings IT connection or BATCH sharing of data Services Services PHP DAOs Entities Replicated mappings DAOs Entities Replicated mappings connection or BATCH sharing of MySQL data Microdata Processing for first evaluations on its own IT Oracle

6 The Census Management System: a metadata-driven system Text are stored as metadata; The questionnaire processing rules are stored in metadata; The questionnaire processing has been modeled as a finite state machine. A triggering event determines a questionnaire state transition. The state transition tables are a key element in CMS database. 1 delivery arrival registration function (function 1) 31 arrival registered Function 1 Function 2 State 1 State 31 State 1 State 2 State 2 State 2 State 1 State 31 Output Delivery Arrival registered

7 The Census Management System: a functional overview A Census Management system: manages census operators: creation of the operators networks (roles and responsabilities); BATCH PROCESSING FOR UPLOADING AND VALIDATING IMPORTED DATA FROM DIFFERENT eventually integrates different data sources for minimizing errors; SOURCES HEAVY PROCESS trace the questionnaires life cycle, by handling the state flow of each questionnaire; monitoring of the survey process by supervising of all the operations performed on each questionnaire; SEPARATE TABLES UPDATED PERIODICALLY HEAVY PROCESS

8 The Census Management System: building blocks for a cost effective improvement BETTER HEAVY PROCESSES ON DEDICATED DB SERVERS FOR PERFORMANCE BETTER EASILY INTEROPERABLE SYSTEMS WELL DEFINED FUNCTIONAL PROCESSES BETTER NONE CODE REPLICATION FOR MAINTENANCE STRONG METADATA - DRIVEN APPROACH A WEB BASED CENSUS OF SERVICES WHICH IS BETTER CONNECTED TO BOTH INSIDE AND OUTSIDE

9 What is SOA? Not-SOA SOA GUI Web Application Front end TECH NEUTRALITY Services NO CODE REPLICATION Service Repository Service Bus Actions INTEROPERABILITY Services Implementation Interface DAOs Entities Busines s Logic Data

10 SOA as a Web Service Architecture A Web Service is a software system designed to support interoperable machine-to-machine intercation over a network. It has an interface described in a machine processable format (specifically WSDL). Other systems interact with the Web Service in a manner prescibed by its description using SOAP messages, typically transported on HTTP with an XML serialization. w3c specification Security Reliability Transaction messaging M E T A D A T A XML

11 SOA as a Web Service Architecture in Census LAN Web Service CLIENT DATA & METADATA REQUEST/RESPONCE SOAP/XML Container SOA Web Service SERVER Front end Web service CLIENT Web Service SERVER Service Repository Service Bus DATA & METADATA DB

12 SOA criticalities serialize XML SOAP message XML SOAP message deserialize Client program WS handle network deserialize XML SOAP message XML SOAP message serialize Distributed objects have a number of architectural challenges and others point out: Problems introduced by latency and unreliability of underlying transport; NET LATENCY The lack of shared memory between the caller and object; EXCEPTION CLASSIFICATION The problems introduced by partial failures scenarios; The challenges of cuncurrent access to remote resources; PERFORMANCE Overhead in terms of serialization/deserialization and service handle ; Great amount of data; TECHNICAL FEASIBILITY IN CENSUS CONTEXT

13 SOA-based web solution ISTAT exploration: technical feasibility SOA Menu item Function 1 GUI Web Service TECHNICAL FEASIBILITY IN CENSUS CONTEXT Actions Services Function 2 Function 3 Function 4 1:1 RELATIONSHIP WITH WEB SERVICE OPERATION WAR file SERIALIZABILITY THROUGH EXTERNAL JAX-WS BINDING DAOs Entities NO CIRCLE REFERENCES IN ENTITIES Oracle WAR file

14 Machine A SOA-based web solution ISTAT exploration: service overhead Front end WAR file TESTS UP TO RECORD EXCHANGED Web Service WAR file Machine B NO NETWORK LATENCY Oracle

15 Machine A SOA-based web solution ISTAT exploration: network latency Front end WAR file TESTS UP TO RECORD EXCHANGED Machine B Web Service WAR file Machine C YES NETWORK LATENCY Oracle

16 Conclusions - First classification of advantages of a SOA-based web architecture in survey context, thanks to the metadata-driven features of actual web-based systems and to the modular functional nature of them; - Piloting applications for: - technical feasibility testing; - first classification of SOA criticalities in census context; - Performance evaluation for design/monitoring/routing purpouses; GOOD RESULTS PRODUCES FURTHER EXPLORATION NEED!

17 Thank you