STEP C : A Demonstration of utilising SOA across multiple

Transcription

1 STEP C : A Demonstration of utilising SOA across multiple Universities Final Report Universities: Contractor: Principal Investigator: De Montfort University, Southampton Solent University Fulcrum Logic Paul Hopkins Date: 11 th March 2010 Executive Summary This project was funded by the JISC Flexible Services Delivery programme to demonstrate how Service Oriented Architecture (SOA) based solutions could be rapidly constructed within an existing University landscape. The project implemented different Enterprise Service buses (ESBs) as the backbone communications channel at two different Universities. It connected 4 quite different applications at each University to the ESB and demonstrated how information could be combined from each application. The applications were deliberately chosen to have different technical challenges (eg already enabled to be web service connected through to having to develop connectors directly to underlying database tables). The project and all of its documents, design, coding etc are all available the intellectual property of third party systems integrators has been transferred so that it can be re used by any UK University. The project demonstrates the ease and also the difficulties of developing and implementing an SOAbased system within a traditional University environment. Table of Contents Introduction... 2 Background... 2 Concept behind the project... 2 Project Execution and Deliverables... 3 Project Timetable... 4 Project Methodology... 4 Data Protection Act issues... 4 Key lessons learned within the project... 5 Next Steps... 6 STEP C Demonstration of utilising SOA across multiple Universities Page 1

2 Appendix... 6 Project Initiation Folder... 6 Project Management Files Folder... 7 STEP C Fulcrum Development Project files... 7 STEP C Project Deliverables... 8 Introduction This project was developed and executed within the overall JISC programme called Flexible Service Delivery, managed by Mr Alex Hawker. It has been executed by De Montfort and Southampton Solent Universities with a contractor Fulcrum Logic. This Final Report on the project is deliberately a short report which describes the overall aspects of the project. There are a mass of electronic documents which describe how the entire project was executed. These files are indexed in the Appendix to this report. All Intellectual Property developed within this project has been transferred from Fulcrum Logic to De Montfort University as part of the formal Agreement between the parties. Then, as an explicit condition of the Grant Letter, all relevant documents and intellectual property etc was transferred to the HE sector. This was intended so that further projects in the FSD programme could have access to any/all of the materials created within this project. The project team explicitly requested Fulcrum s developers to explain WHY they made certain decisions (e.g. Why BizTalk? ) or HOW they designed certain aspects of the project hence all of the design documents are included within the Appendix to this report. Background There has been much discussion about Service Oriented Architecture (SOA) during the past several years in both the HE sector and also within the commercial IT world. There has been a limited amount of take up and practical realisation of SOA within the HE sector apart from within a small number of Universities. The FSD programme was intended to dramatically increase the use of both SOA and Enterprise Architecture concepts across the UK HE community. This particular project, STEP C, was designed to be a fast track demonstration of how best practise SOA principles could be realised within several Universities. The outcomes of this project are intended to be made fully available to any member of the sector wishing to develop their own specific SOA based projects. The project achieved ALL of its goals, resulting in a presentation to the FSD workshop in late January This report is intended to summarise the project achievements and point towards where ALL of the project materials can be found Concept behind the project The project sought to demonstrate in real terms how many different types of applications could be connected to an Enterprise Service Bus to enable information to be transferred between multiple STEP C Demonstration of utilising SOA across multiple Universities Page 2

3 applications to create a common output. A test situation was constructed to observe whether a student was engaged with her University by accessing a series of applications to see when the student had last performed a transaction. In addition, the project wished to demonstrate how the above scenarios could be implemented in a second University which was operating completely different application systems. The situations are shown below: Project Execution and Deliverables The project was designed to be a physical demonstration of how an SOA architectural approach could be used within the HE sector. It was not intended to create a full strength production ready solution. The metaphor behind the project was to measure Student Tracking and Engagement. This scenario would require a report to be created which pulled information from a variety of different applications (e.g. the Student Record system, a finance system, a VLE, the library system, an system etc). The actual application systems selected within both Solent and DMU were chosen in order to demonstrate situations in which there were quite different types of connections required e.g. an application might already be web services enabled already have an API (application programming interface) which could be utilised have no ready made connector available hence requiring a direct link to the underlying database tables The scenarios did not require complex data manipulation rather they just required certain data fields to be retrieved from specific application systems. Therefore the bulk of the development activity was in creating appropriate connectors between the application and the underlying Enterprise Service Bus (ESB) and in developing the necessary process logic within the ESB to meet the needs of the solution (... i.e. Go to the Finance system and retrieve the financial records for each STEP C Demonstration of utilising SOA across multiple Universities Page 3

4 of the students that have been selected then Go to the Library system to retrieve the latest date on which each of these students borrowed a library book etc). Project Timetable The key phases of the project were: August/September 2009 September/October 2009 November 2009 November January 2010 January 2010 Get JISC Approval for the project Invitation to Tender for project contractor Detailed project planning Project execution Sign offs at Universities and presentation to FSD workshop All of the project documents are highlighted in the Appendix of this report but the actual location of the documents is xxxxx Project Methodology The project methodology involved: a) Extracting a set of records (for students) from each of the applications to be modelled. b) Within Fulcrum s development labs, identical data bases were recreated to contain the extracted records. The architects/developers then created and tested their solutions based upon the ESBs which were to be used at each University. c) Within Solent and DMU IT test environments, Fulcrum implemented the relevant ESBs and test environment remotely but utilising relevant University technical support staff. d) After (b) and (c) were completed, a Fulcrum developer visited each University to install the developed software and to link the solutions directly in to full size test applications (e.g. student record, VLE, etc). e) The Fulcrum on site developer tested the application against copies of live data, resolving any issues locally and getting sign off with the relevant analyst for each of the applications concerned. f) The Fulcrum on site developer then created and demonstrated the integrated solution to the local IT department to get sign off that the full solution had been successfully demonstrated. g) The Fulcrum team created and presented the full solution to the FSD Workshop in late January, then packaged up all relevant notes and documents and transferred them to De Montfort University. Data Protection Act issues In addition to executing the above steps for both Universities, the project had to overcome difficulties in developing a solution while using off shore contractors. The key issues involve the Data STEP C Demonstration of utilising SOA across multiple Universities Page 4

5 Protection Act which expressly forbids the export of personal data outside of the European Union without the express permission of the data subject unless the service being provided is of benefit to the data subject. Hence, whilst call centres and applications maintenance can be argued to be of benefit to the data subject, it is not possible to claim that a new application development will be of benefit to any individual student. The project team developed a very pragmatic solution which is partially described in the project methodology above, which kept the project within both the spirit and letter of the Data Protection Act and without impacting adversely on the project deadlines. Essentially, this involved spoofing/anonymising personal data before exporting it to Fulcrum s Global Development Centre and then ensuring that the subsequent installation/testing of the solution was performed completely on campus and without direct support from any staff based offshore. Also, the project commissioned a review, by Dr Tim Watson of the DMU Forensic Computing department, of the security issues which should be considered when implementing an SOA solution. The report is included in the project outputs. Key lessons learned within the project There were many lessons which were learned by the participants in this project which will enable them to work together on further assignments these included utilising fast track project methods, learning how to engage with an off shore development company etc. However, lessons which may be of direct relevance to other HE institutions include 1. Data accuracy, integrity and consistency becomes even more important within an SOA environment than ever. When applications (and their associated data stores) are connected via an ESB then there will be an increasing volume of data being shared across the enterprise. The concepts of which version of data is the Master? will be of increasing importance. In itself, SOA does little to improve data accuracy but any production quality systems will need to ensure that processes are created which clean up the data which will be used. 2. Be wary of claims from certain application vendors that their products are already Web Services enabled. It is likely that they designed the Web Service connectors to satisfy specific, rather than generic, problems. For example, at DMU it was noted that the Agresso Student Record application was WS enabled but that the connector would only return a single student record. It had not been designed to return a list (or all ) student records. Similarly, it is conceivable that a pre existing connector may not return certain data fields that the new SOA application requires. 3. The reaction of existing Business Analysts to the introduction of SOA is important. The current generation of Analysts are extremely familiar with toolsets that enable them to develop applications successfully. Persuading them to switch their analysis and development methods to one based upon SOA principles will be challenging. This appears to be both an educational and also a cultural issue. It is certainly an area which will require substantial investigation especially because there may well be 2,500 staff engaged within the Applications sections of UK Universities ( according to UCISA statistics!). STEP C Demonstration of utilising SOA across multiple Universities Page 5

6 Next Steps The STEP C project, as originally conceived, is now completed after the publication of this report. However, there are several areas which could be examined for further development but just as a Proof of Concept / Demonstrator level. These include a) Replace one of the ESB buses with a different one to show that the application can continue without ANY changes to process logic; b) Add in some additional applications to either Solent or De Montfort; c) Replicate what has been done at a third University; d) Remove one of the applications from the Process Logic and show that the basic system can continue to get information from the other application systems without change; and e) Change the scenario to demonstrate how UPDATES could be achieved within this scenario. Of the above, only the last two concepts would materially increase the value of the learning which can come from the STEP C project already. A more relevant set of opportunities will arise when the STEP C work is used as a foundation for moving forward. Key suggestions for further work may include a) Create a small scenario which may involve just a small number of application systems. Develop as simple problem which can be developed in to a much richer situation involving creating/amending/reading and deleting records. Then consider implementing this solution in several different institutions b) Building upon the STEP C connector demonstration, begin to develop a set of HE Sector standards for connecting applications to a service bus. By taking the same type of application (e.g. a student record, finance, library, VLE etc) from several vendors of the same application, examine the difficulty/ease of creating look up tables so that they can each be converted to utilise a standard set of HE specific names... Appendix This report is specifically designed NOT to contain dozens of pages of relevant and interesting documents. There are a mass of documents created within this project which are stored in electronic format in various locations. See below a list of documents and also pointers to their location. Note that all documents are covered by the transfer of intellectual property from the developers, Fulcrum Logic to De Montfort University and then from De Montfort on to the entire HE sector. Project Initiation Folder Project Files Description a) POC EOI STEP C The raw FSD Programme Expression of Interest form b) STEP C POC Outline DMU and SSU s submitted Expression of Interest c) Presentation to September FSD A very short presentation to an FSD workshop to Workshop outline what DMU and SSU wanted to do d) The ITT published by DMU The official ITT published on the DMU INTEND website e) The Marking Scheme results The internal marks and comments on the STEP C Demonstration of utilising SOA across multiple Universities Page 6

7 Fulcrum proposal (comments on unsuccessful bids are NOT included here) f) HEFCE letter of intent to find STEP C Copy of the official letter of intent for funding Folder Fulcrum s Proposal documents The complete proposal document submitted by Fulcrum for this project Project Management Files Folder Project Files Description a) STEP C POC Initial Outline of project DMU and SSU s submitted Expression of Interest b) STEP C Project Kick off meeting at DMU Minutes of meeting on 27 th Oct 2009 at DMU c) STEP C Project Kick off meeting at SSU Minutes of meeting on 4 th November at SSU d) STEP C Change Request Version 1.2 This document reflects the changes that have been made to the project scope AFTER the original ITT was published and Fulcrum s proposal accepted e) Applications to be considered for STEP C at DMU Diagram showing the DMU applications which would be connected within this project f) Applications to be considered for STEP C at SSU Diagram showing the SSU applications which would be connected within this project g) Change Order Approval for STEP C Formal approval of the change of scope of the project h) STEP C Contact List A project contact list i) Folder containing all Weekly Progress Reports Copy of EVERY weekly report submitted from the Fulcrum project manager to DMU and SSU STEP C Fulcrum Development Project files File Name Description 1. STEP_C Statement of Requirements Specification Version 1.0 Overall STEP C System Requirements Specification 2. STEP C pre Engagement Questionnaire Version 1 (DMU) Initial questions which Fulcrum team had for DMU 3. STEP C DMU WBS (Project Plan) The STEP C Project Plan for DMU 4. Step C DMU High Level Design The High Level Design Spec for DMU 5. Low Level Design DMU Version 1.0 The detailed design specification of the DMU solution 6. STEP C Test Cases Results DMU Results of Testing the DMU cases 7. STEP C Pre Engagement Questionnaire Initial questions which Fulcrum team had for SSU Version 2 (SSU) 8. STEP S SSU WBS (Project Plan) The STEP C project plan for SSU 9. SSU High Level Design Version 1.1 The High Level Design Spec for SSU 10. SSU Low Level Design Version 1.0 The detailed design specification of the SSU solution 11. Step C Test Plan Draft Test Plan for STEP C project 12. Real World SOA A Microsoft report on real World approaches to SOA included here for completeness 13. High Level Benefits of ESB Ver 1.2 Copy of a report produced by Fulcrum during STEP C Demonstration of utilising SOA across multiple Universities Page 7

8 discussions as to whether an ESB should be included within the scope of STEP C project 14. ESB Technology Comparison Document Report which DMU specifically requested from Final Fulcrum after their proposal to explain why they recommended BizTalk as the ESB report compares BizTalk, OpenESB and TIBCO explicitly 15. BizTalk Server 2009 Software Document of hardware and software requirements prerequisites for implementing BizTalk at DMU 16. STEP C Test Cases Results SSU Results of testing the SSU cases 17. Analysis of Microsoft ESB Toolkit 2.0 Evaluation of which parts of standard BizTalk functionality would be used within STEP C STEP C Project Deliverables File name Description a) STEP C Security Review report Review of the risks of implementing an SOA solution within a University environment b) Final Presentation to FSD Workshop Jan 2010 This is the major presentation delivered to the FSD meeting on Jan 25 th 2010 c) DMU Presentation The DMU presentation (zipped) d) SSU Presentation The SSU presentation (zipped) STEP C Demonstration of utilising SOA across multiple Universities Page 8