CwJ Consulting Ltd Technology & Roadmap guide. Author: Christopher Cantle Date: 7 th March 2017 Version: 2.5

Transcription

1 CwJ Consulting Ltd Technology & Roadmap guide Author: Christopher Cantle Date: 7 th March 2017 Version: 2.5 Technology & Roadmap Guide Page 1 of 17

2 Contents 1. Introduction Environment Management Challenges emdash Key Metrics Technology Landscape Architecture emdash environment pre-requisites rd Party Integration Workflow Process Repository (Version Control) Diagnostics emdash Logging Notification SQL Server Logs Windows Event Viewer Compatibility Matrix Deployment Applications Endur/Findur (OpenLink) RightAngle (OpenLink) CubeIntelligence (OpenLink) Symphony (Amphora) emdash Roadmap Cloud Computing Create Environment Oracle Support Scheduling & Notifications Multi-Tenant Support Technology & Roadmap Guide Page 2 of 17

3 1. Introduction Implementation projects are costly, complex and time-consuming. To meet challenging budgets, deadlines and client demands, projects need to avoid unnecessary delays. Development and test environments should be: Available to the project staff when needed to avoid wasted time Deployed consistently and repeatedly Be representative of the Production infrastructure to gain most value out of testing, be it functional or technical. The availability of development and test environments directly impacts the critical path of all implementation projects. In addition to the delivery of the initial implementation, businesses demand that their trading systems are flexible enough to enable them to trade in new markets and with new instruments as opportunities are identified. Having an agile, responsive development process to meet these demands after the initial implementation project has completed is a critical success factor of trading systems implementations. As such, the requirement for efficient, consistent and reliable development/test environments does not end at the completion of the first system implementation. Ongoing support of the core code and tailored configuration is vital to avoid unnecessary delay to delivering functionality to the business. 2. Environment Management Challenges Delivering a high quality set of development and test environments reduces the cost of an implementation project. To achieve this, the approach must consider the following challenges: Quality. Environments must be delivered that meet the needs of the developer/tester that will be consuming it. All code/configuration versions need to be correct Flexibility. Environments need to be spun up and ramped down quickly and efficiently avoiding any delays to the project Capacity. Efficient development/testing requires a lot more environments than you would first expect one per test scenario to run in parallel is ideal Change control. Code-based and database-based customisations both need to be managed and controlled (Source code control tools will not easily manage all database configuration) Security and Access Control. Environments should be accessible by developers and testers for their specific requirements only, rather than granting blanket access to everyone. This mitigates the risk of one developer/tester impacting others Environment complexity. The environment landscape gets more complex following the first release into Production. Fix on fail changes, minor enhancements and major releases need to be supported. Technology & Roadmap Guide Page 3 of 17

4 The environment management challenges are managed through the following offering and capabilities. emdash (tm) Development Operation Services Technical Architecture Consulting Software Provision emdash Technical Architecture Consulting Development Operation Services Software Provision emdash is our development operations toolset that encompasses a wide range of environment management tasks. It streamlines DevOps processes through automation and provides control and audit information for all activities.. Resource pool of SMEs to design and setup technical architectures and troubleshoot production issues Resources, processes and tools to provision and manage development and test environments for clients. Key functions: Parameterise applications to logical environments Create environment Refresh environment from another Deploy code / configuration from source Creation and support of software tools. Includes emdash Can be expanded to encompass client-specific requirements (Exchanges, Interfaces, Regulatory Reporting, etc.) Manage user access Record and report on what component versions are in an environment Record history of change within an environment Technology & Roadmap Guide Page 4 of 17

5 3. emdash emdash is our environment management dashboard service that provides environment management capabilities through a portal interface to manage the whole development landscape. emdash Environment Management Dashboard These capabilities allow complex environment management tasks to be reduced to a simple point and click action, reducing task completion times, environment downtime, required man power, the possibility of deployment mishaps, and human error. These capabilities range in Managed Application Deployments to Production, Disaster Recovery and multiple Testing environments Managed Application Code Upgrades with Deployment Versioning Environment Cloning & Refreshing Application Recovery Application Automated Configuration Disaster Recovery Planning and Deployment Environment and Application Security Application Service Management Environment Housekeeping Cloud Computing Services emdash functionality is modular and is currently managing, configuring and deploying a diverse selection of applications including ETRM/CTRM systems Endur/Findur (OpenLink), RightAngle (OpenLink) and Symphony (Amphora), and Credit Risk system CubeIntelligence (OpenLink/CubeLogic) at multiple top tier global companies. New applications can be added to the toolkit with minimal effort through the development of modules and APIs as required. Once embedded in BaU processes, emdash means that the number of skilled Environment Support resources can be reduced. Common tasks are automated so they are done quickly, repeatedly and efficiently leaving the support team to focus on the more complex tasks. emdash is a web portal interface with integration to multiple 3 rd Party services to allow version management, recovery management, and deployment and configuration of application components. Technology & Roadmap Guide Page 5 of 17

6 The emdash web portal is built on the Microsoft.Net Framework, running under Windows IIS (Web Server) and is delivered through SSL Certificate Encryption (HTTPS up to 256bit). A Master application is created for each managed application (e.g. Endur/Findur, Symphony, RightAngle, CubeIntelligence, etc.) with 3 rd party integration to deploy application components to their required deployment target environment. emdash is designed around a process driven workflow for each managed application (e.g. Endur/Findur, Symphony, RightAngle, CubeIntelligence, etc.). Each workflow is configured and managed to deploy application components to their target destination as instructed in the managed application installation guides. emdash provides 3 rd party integration through embedded C# functions, database scripting, PowerShell scripting and configuration file manipulation. This allows integration with Windows Server (Active Directory, Services and Security), Cloud Computing Services (MS Azure & AWS), MS SQL Server (DS, AS, IS, RS, Security and SQL Agent), Distributed File Systems, and Version Control Repositories. Further details on the emdash 3 rd party integration can be found in the Technology Landscape Section. Technology & Roadmap Guide Page 6 of 17

7 4. Key Metrics Here are some core activities and duties that the Environment Support Team would undertake, along with benchmark timescales for completing these tasks with or without the emdash toolset. Create the first Endur environment With emdash Within 1 day of the infrastructure being available Without emdash 2 weeks Create a new Endur environment Within 1 hour 3 days Refresh Endur environment from a Master Config database Create a new Endur user (including AD user setup) Re-allocate all empowered Endur users to an environment after refresh Code version upgrade FTE Skilled Resources required to manage estate of 20 environments 15 minutes (including Cube deployment) 4-6 hours < 1 minute per user 10 minutes per user <1 minute per environment 15 minutes per environment minutes depending on number of users 6 hours per environment The cost and efficiency benefits of emdash are realized exponentially as the number of environments increases through the delivery lifecycle. As can be seen above, the number of experienced resources needed to manage the suite of environments reduces dramatically by using the emdash toolset. Technology & Roadmap Guide Page 7 of 17

8 5. Technology Landscape 5.1. Architecture emdash web portal interface is hosted on Web Server Internet Information Services 7.0 or above. The management functionality, workflow process, template configuration and deployment integration are control within the portal web service. emdash provides the capability to deploy across physical or virtual (Hyper-V, VMWare, etc.) 32bit and 64bit architecture environments emdash environment pre-requisites To host the emdash suite the following hardware and software components are a minimum requirement. emdash server requirements Windows Server 2008 R2 or above IIS 7.0 or above (Web Server).Net Framework 4.5 or above Remote SQL Server database schema emdash server requirements with locally hosted SQL Server database schema Windows Server 2008 R2 or above IIS 7.0 or above (Web Server).Net Framework 4.5 or above SQL Server 2008 R2 Standard Edition or above Specification 2 CPU Cores 4GB Physical Memory 100GB Disk Storage Specification 4 CPU Cores 8GB Physical Memory 200GB Disk Storage emdash requires the SQL Server features Client Connectivity and Client SDK installed on the emdash server for managing applications that integrate with SSAS and SSIS rd Party Integration A number of APIs are embedded in emdash to integrate with 3 rd party services and applications. These APIs are triggered during a workflow process and feed essential attribute information for an API to complete its task successfully. A subset of the available emdash APIs are listed below, with new APIs being developed based on product roadmap, industry needs, or client requests. Technology & Roadmap Guide Page 8 of 17

9 Application SQL Server Windows Services Cloud Computing Services notifications Citrix / Remote Desktop Services emdash API Database Backup & Restore Database configurations SSAS Cube Backup & Restore SSAS environment and Cube configurations Database Security SSIS Package deployment SSRS Report deployment SQL Server Agent Job deployment Active Directory Windows Services management Windows Security IIS (Web Server) Task Scheduler Create and configure virtual machines on Microsoft Azure and Amazon Web Services (AWS) SMTP User Monitoring Thin client application provision emdash integrates with a wide range of 3 rd Party products and it s the responsibility of the client to provide the correct level of licensing for each active integrated product. Technology & Roadmap Guide Page 9 of 17

10 5.4. Workflow Process emdash functions through an embedded workflow process triggered by a web portal user action. The pre-configured workflow will initiate and proceed to process a set list of defined tasks. Each event in the workflow task list is monitored by a logging and notification service. A task may consist of initiating an API, executing a database script or stopping specific windows services, etc. Below is an example of the emdash workflow Environment Refresh for ENDUR Event Action 1 -- Notification START Logging 3 -- Create Archive directory 4 -- BACKUP TARGET database in Archive 5 -- STOP ENDUR Services 6 -- STOP orphaned ENDUR processes 7 -- Disconnect open Database connections 8 -- RESTORE SOURCE database to TARGET database 9 -- Configure TARGET database to meet TARGET environment configuration Apply and verify AD security Apply and verify database security Configure TARGET application Environment housekeeping START TARGET ENDUR services STOP logging Notification Repository (Version Control) Application releases and deployment configurations are managed through version control in a file repository (e.g. Microsoft TFS, Tortoise SVN, Git, etc.). emdash maintains the deployment version history and associated link to the repository folder version. There are few restraints on the type of repository tool, as emdash has been designed to link to the repository through a virtual drive mapping and folder path. CwJ Consulting use Tortoise SVN by default if the client does not provide an alternative. Technology & Roadmap Guide Page 10 of 17

11 5.6. Diagnostics emdash utilizes a number of diagnostic tools to log workflow process events, process failures and user actions, with integration to SMTP for notifications emdash Logging All portal user activities and triggered workflow logging is captured in the emdash database schema and a dedicated user log file partitioned by current date. Each entry log is time stamped and tagged with INFO or DEBUG followed with additional information of success or error message, workflow creation details, scheduled details, status, environment, application and triggered job details. Further logging details are captured for each workflow step detailing processing details to completion or failure. Workflow deployment job logging details Workflow Step job message details Technology & Roadmap Guide Page 11 of 17

12 Notification emdash workflow events will trigger an notification on activation and completion of the triggered workflow. The notification can be configured by workflow, on an environment by environment, or application by application basis within the emdash environment wide setting parameters. Example notification generated from emdash Technology & Roadmap Guide Page 12 of 17

13 SQL Server Logs Certain SQL Server workflow events or API failures may generate additional logging events in the SQL Server logs. SQL Server logs can be monitored and examined through SQL Server Management Studio for the TARGET database server and instance. SQL Server Logs generated through emdash events or API failures Windows Event Viewer Certain workflow events may generate additional logging in Windows Event Viewer on the emdash Web Server and TARGET environment. Event Viewer is located in Control Panel and Administrative Tools. The diagnostics logs related to workflow events would be located in the Windows Logs bucket under Application, Security and System. A typical diagnostic log contains the information like Level (Information, Warning or Error), Data and Time, Source, User, Line Number and/or Filename, and Error Message. 6. Compatibility Matrix Event Viewer logging from emdash activity emdash V2 emdash V3 Deployment Applications Endur/Findur (OpenLink) Technology & Roadmap Guide Page 13 of 17

14 CubeIntelligence (OpenLink) Symphony (Amphora) RightAngle (OpenLink) Lacima Cloud Computing Services Microsoft Azure Amazon Web Services (AWS) Database Compatibility (for hosting the emdash Database) SQL Server 2014 (DS, AS, RS, IS, SQL Agent) SQL Server 2012 (DS, AS, RS, IS, SQL Agent) SQL Server 2008 R2 (DS, AS, RS, IS, SQL Agent) SQL Server Express Database Services (for managing Application Databases) SQL Server 2014 (DS, AS, RS, IS, SQL Agent) SQL Server 2012 (DS, AS, RS, IS, SQL Agent) SQL Server 2008 R2 (DS, AS, RS, IS, SQL Agent) SQL Server Express Oracle Database 12c Oracle Database 11g Operating Systems Windows Server 2012 R2 Windows Server 2008 R2 Windows 10 Windows 8 Windows 7 Microsoft Components Windows Services Internet Information Services 7 or above (IIS Web Server) Remote Desktop Services (Web) Windows Security (Users, Groups, Policies).Net Framework 4 or above Virtual Environments Hyper-V VMWare Web Browsers Microsoft Edge Internet Explorer 10 or above Apple Safari Google Chrome Miscellaneous Deployment Workflow Real-time Advanced Logging Integrated Reporting Tortoise SVN (File Repository) Microsoft TFS Git Citrix Gateway Microsoft RemoteApp Technology & Roadmap Guide Page 14 of 17

15 6.1. Deployment Applications Endur/Findur (OpenLink) Endur/Findur is a complete front to back office Commodities Trading and Risk Management (CRTM) solution covering the full range of commodities and derivatives. Physical commodity logistics and scheduling tools are built into the solution. With automated processing of both physical transactions and financial derivatives, you get an aggregated physical-to-financial view of positions, risk and P&L on one system RightAngle (OpenLink) RightAngle is a comprehensive Commodities Trading and Risk Management (CTRM) solution, supporting front to back office workflow from physical and financial deal capture to risk position reporting, logistics, settlement and accounting CubeIntelligence (OpenLink) CubeIntelligence is a business intelligence platform designed to serve the advanced risk management reporting needs of companies that trade commodities and capital markets. With OpenLink CubeIntelligence you can integrate and hold huge volumes of information such as trades, cash flows, positions and exposures from across the enterprise. All this data is instantly available for high-speed data manipulation with unrivalled flexibility for ad hoc reporting, as well as out-of-the-box modules for credit and market risk management Symphony (Amphora) Symphony connects the numerous business functions within commodity and energy trading companies into a single user friendly system that uses one common database. It turns transaction data into a business intelligence tool that you can act on to make better business decisions emdash Roadmap emdash is modular and so we can include new technologies and functionality through the development and deployment of new modules when they are available. As such, here is a timeline of the main roadmap functionality that we will deploy through Cloud Computing (Phase 1) Virtualization Create Environment Wizard Support Applications on Oracle Scheduling & Notifications Multi-Tenant Microsoft Azure (Classic and Resource Manager Models) Hyper-V VmWare ESX GUI to create new environment Stop / Start environment Backup databases Environment Refresh Manage Users Recurrance of emdash jobs Notifications Reporting Support multiple customers 7.1. Cloud Computing / Virtualization emdash provides integrated support for Cloud Computing Services Microsoft Azure and Amazon Web Services (AWS). Initial integration will support Creating servers for environments from scratch in the cloud Shutting down an environment when it s not being used to avoid unnecessary costs Technology & Roadmap Guide Page 15 of 17

16 7.2. Create Environment The Create Environment wizard will provide a GUI driven process for creating a new environment and adding applications into it. It enables users to create their own environments using either on-premise virtualization technologies or integrating with the emdash Cloud Computing processes for Azure and AWS Oracle Support This enables emdash workflows to integrate with applications that are on the Oracle database platform as well as SQL Server. This will provide emdash functionality for environment refresh, code upgrade, user management etc. to function on this platform for Endur/Findur running on Oracle (for example) 7.4. Scheduling & Notifications This is to provide enhanced functionality to: 1. Enable users to schedule emdash processes to run either at a particular time This enhances existing functionality to be more user-friendly and integrated into the core product 2. Enhanced notifications for emdash-driven events such as completed or failed jobs. At present this is performed via and the roadmap item is to enhance this and add extra channels (such as using a subscription notification model like Slack) Multi-Tenant Support This is functionality to aid those organisations who support multiple Customers environment management tasks emdash Roadmap emdash is modular and so we can include new technologies and functionality through the development and deployment of new modules when they are available. As such, here is a timeline of the main roadmap functionality that we will deploy through Modular Applications Custom Steps and Workflows Create Environment Wizard Enhanced User Management Ability to model other applications via flexible framework Ability for customers to define their own workflows and steps in Powershell Enhancement to make this more dynamic July 2017 Further integration with Endur to create database users 8.1. Modular Applications This enhancement is to make it easier to add other applications to emdash to benefit from the core workflows of Environment Create/Refresh, Code Upgrade, and User Management 8.2. Custom Steps and Workflows This is giving customers to run their own steps and workflows in Powershell against the metadata held in the emdash database. Examples of where this is useful would be to perform customer-specific functions and housekeeping. Technology & Roadmap Guide Page 16 of 17

17 8.3. Create Environment Wizard We are continuing to enhance the usability of the Create Environment function to allow for dynamic environment configurations in terms of the number of servers, layout of application services, users and so on. This enhancement is a ground-up rebuild of the Environment Wizard UI Enhanced User Management This is further automation of the user management process both integrating more deeply in Endur to avoid manual tasks, as well as providing mechanisms for adding and removing users to/from environments in bulk Technology & Roadmap Guide Page 17 of 17