White Paper: FSA Data Audit

Transcription

1 White Paper: SA Data Audit Background In most insurers the internal model will consume information from a wide range of technology platforms. he prohibitive cost of formal integration of these platforms means that inevitably a significant proportion of the data feeding internal models is held in, passes through or is manipulated by, End User Computing (EUC) applications (such as Microsoft Excel and Access) or similar files such as.csvs. he integrity of these processes is a specific focus for SA Data Audits. he SA has developed a review tool to help assess whether a firm s data management complies with the standards set out in the Solvency II Directive for the purposes of internal model approval. his tool will be used as part of the SA s Internal Model Approval Process (IMAP). It informs firms on what they might do in order to satisfy the standards set out in the Directive and is also based on what is expected to be required from the delegated acts (formerly referred to as the Level 2 implementing measures). he scope of the review is all data (internal and external) that could materially impact the Internal Model. After conducting the review the firm is expected to provide summary findings to the SA and be ready to provide the evidence that formed the basis of the conclusions. he review schedule has five sections as follows: 1. he approach (i.e. matters of policy) to managing data 2. he level of oversight of the implementation of the data policy 3. he level of understanding of the data used in the internal model 4. Data issues that may undermine the integrity of the internal model 5. Unreliable processes that may undermine the integrity of the model he areas of potential risk and expected controls in each of these sections are documented in detail below, alongside the capabilities of the ClusterSeven solution to meet these s where they are applicable to End User Computing applications. Risk 1: he approach to managing data for use in the internal model does not ensure consistency in quality and application of the internal model Control objective: o ensure that data quality is maintained throughout the process of the internal model as required by Solvency II. Data management is the process of collecting data from disparate sources and combining it in a way that the [internal] model can use. Data operations: Any point in the system where you do something with data e.g.: interpretation (e.g.when a user takes a freeform field and interprets it as a structured one) formatting (e.g.changing a date format from to 1/3/2010) alteration (e.g. data cleaning) joining (e.g.contractual data such as annuity policies with observational data such as mortality tables) restructuring (e.g. Excel Pivot function) aggregation (e.g.sumi function in Excel) extraction (e.g. a SQL query downloading a CSV file) derivation (e.g. matrix multiplication) merging (e.g. cut and paste from different sources) translation (a term which has a meaning for one system is translated into a different term that has the same meaning for another system). SA Spreadsheets and Solvency II July hrogmorton Avenue Registered office 10 Argyll Street, London, W1 7Q Registered number Registered in England & Wales.

2 White Paper: SA Data Audit A data policy has been established and implemented. he policy, its associated procedures, and standards include: a definition of the different data sets that are to be covered by the policy; a definition of materiality (which is aligned to the firm s risk appetite where appropriate e.g. when an expert judgement is made to adjust for insufficient observational data); the respective ownership and - responsibility for the data sets, including the system of governance and assurance over data quality; a definition of the standards for maintaining and assessing quality of data, including specific qualitative and quantitative standards for the data sets, based on the criteria of accuracy, completeness and appropriateness; the use of assumptions made in the collection, processing and application of data; the process for carrying out data updates to the internal model, including the frequency of regular updates and the circumstances that trigger additional updates and recalculations of the probability distribution forecast; a high level description of the risk and impact assessment process including the frequency with which the assessment process is conducted, and; the frequency of the review of the data policy, associated procedures, and standards. Central monitoring and associated reporting provide robust evidence of implementation. lexible control models enable data policies to be efficiently tuned for different data streams. ull inventory reports available on demand Materiality of an EUC may be defined by users as a tag within the EUC or by independent technology-based assessment. his metadata can drive the application of different control processes. Expert judgement amendments are automatically caught as manual changes and reported. Client may establish multiple attributes for an EUC to allow for such roles as ownership, testing, auditing, alerting, approval. All such attributes can be reported across the EUC inventory, including the exposure of gaps such as non-existent owners. lexible controls allow a wide range of qualitative and quantitative standards to be automatically applied to any chosen data set. Data and metadata assumptions held in a spreadsheet can be routinely checked for timeliness and consistency. Changes to assumptions can be automatically notified. Where spreadsheet-based updates to the internal model require a specific set of actions to be completed in a specific order these processes can be automatically logged and checked. Anomalies to the expected process can be automatically notified. he Risk of a spreadsheet can be automatically assessed against client-defined rules. Users can extend the documentation of the Risk status by tags inserted into the spreadsheet. hese metadata items can automatically drive the application of different control processes. he frequency and completeness of all spreadsheet review processes is automatically monitored and may be reported as required. I still see audit reports or project plans that recommend replacing spreadsheets and manual processes with I solution his will never happen It is impractical to replace 2 or more fragmented systems with a single system Replacing the spreadsheet operations with I designed ones only compounds the problem and removes any ability of users to address problems. he only solution is to eliminate the worst processes, and to apply appropriate controls to the ones that remain. SA Spreadsheets and Solvency II July hrogmorton Avenue Registered office 10 Argyll Street, London, W1 7Q Registered number Registered in England & Wales.

3 White Paper: SA Data Audit Risk 2: Inadequate oversight of the development and implementation of the data policy increases the risk of poorly informed decision-making and non-compliance with the required quality and standards Control Objective 2.1: o set the tone and provide appropriate oversight of the implementation of the data policy necessary for sound decision making Control Objective 2.2: o ensure appropriate and timely reporting to support required governance and management decision making process and timely detection of issues he data governance structures and processes are operating as defined in the data policy and associated procedures and effective in: providing appropriate oversight in the application of the data policy Central monitoring and associated reporting provide robust evidence of processes operating as per policy. lexible control models enable data policies to be efficiently tuned for different data streams. Summary reports are available to meet all stakeholder needs including Executive, Management, Risk, Compliance and I. In many firms, spreadsheets provide a key area of risk, because they are typically not owned by I, but by other business or control areas, such as the actuarial function. hey may not be subject to the same general I controls as the firm s formal I systems (e.g. change controls, disaster recovery planning, security etc) and firms need to develop a control system around this. ensuring that the data policy, associated procedures, and standards including the responsibilities and accountabilities of the various stakeholders across the firm, the quantity and quality of data metrics reported to management, the data directory, and the risk and impact assessment are kept under regular review; ensuring appropriate assurance is carried out and received for validating the quality of data used in the internal model. Data quality metrics (qualitative and quantitative) defined in the data policy are reported (individually, aggregated or categorised) to appropriate levels of management on a regular basis to enable them to assess the quality of data and take remedial action when there are material issues. he system of reporting should include a deficiency management process whereby exceptions identified as a result of data quality checks and controls, which could have a material impact on the internal model, are escalated to appropriate levels of management and actions taken to address them on a timely basis. Summary reports are available to highlight the frequency of operational issues that will drive escalation for the improvement of underlying business processes. Where required the client can require that anomalies to data or processes are signed off /approved before running the internal model. A wide range of data checks are available and may be reported in detail or summary form to designated management. hese checks include the presence (or absence) of change, tolerance levels (max, min, percentage) and trends over time. Where required the client can require that anomalies to data or processes are signed off /approved before running the internal model. SA Solvency II: IMAP, hematic Review findings ebruary 2011 Audit trail Controls we expect to see Audit trail amper proof record of changes Version control + backup Segregation of duties Code checking / code reading esting Maintainability SA hrogmorton Avenue Registered office 10 Argyll Street, London, W1 7Q Registered number Registered in England & Wales.

4 White Paper: SA Data Audit Risk 3: Lack of a clear understanding of the data used in the internal model, and of its impact and vulnerabilities, can create gaps in ownership and control Control Objective: o ensure that data used in the internal model, its impact and vulnerabilities has been clearly identified and maintained. A directory of all data used in the internal model has been compiled specifying source, usage and characteristics including: storage (e.g. location, multiple copies) across the data flow to internal model how data is used in internal model including any transformation (e.g. aggregation, enrichment, derivation) processes ClusterSeven automatically compiles an inventory of files linked to the feeds of the internal model (presented in spider diagrams and associated reports) and captures and reports associated metadata. his metadata may be enhanced by defining additional spreadsheet attributes that are automatically compiled for reporting as required. Spreadsheet locations, historical versions and version copies can be automatically identified. ClusterSeven can monitor the changes (or lack of changes) within spreadsheets to confirm that the observed activity matches the expected transformation. Reports/alerts can be delivered based on anomalies or business-related changes in the process (e.g the arrival of new transactions). Version control Possible controls Peer (non-independent) review Independent expert review Segregation of production from test version Version control over production version he list is long think in terms of what controls would be applicable to corporate I application. A user-developed application, if business critical, should be no different. SA 2010 or each data set, a risk and impact (sensitivity) assessment has been performed to identify:- whether the impact of poor quality data(individually or in aggregation) on the internal model is material; the points in the data flow from source to internal model where likelihood of data errors is the greatest, and therefore, what specific data quality controls are required; tolerance threshold beyond which a data error could become material (individually or in aggregation). Risk assessments can be tailored to client specific needs and run on an ad hoc (but monitored) basis or on a pre-scheduled basis. Cell and range-level tolerances can be placed on all critical values imported into the internal model Spidering and cell-precedent analysis allow key data nodes to be identified for the application of appropriate controls. Cell and range-level tolerances can be placed on all critical values imported into the internal model. Access control Where controls can be difficult Any firms books and records e.g. those involving legal or contractual records hese typically require access control and an audit trail of changes Any application where changes would have significant economic impact hese require monitoring, access control and frequent independent checking SA hrogmorton Avenue Registered office 10 Argyll Street, London, W1 7Q Registered number Registered in England & Wales.

5 White Paper: SA Data Audit Risk 4: Errors, omissions and inaccuracies in the data can undermine the integrity of the internal model and management decision making. Control Objective: o ensure that data quality (complete, accurate, appropriate, and timely/current) is maintained in the internal model. he management and data quality controls (preventative, detective, and corrective) proportional to the probability and materiality of potential data errors have been identified and implemented effectively. he controls should include (at a minimum): having individuals with sufficient competence to conduct the manual data checks on accuracy, completeness and appropriateness A well-defined and consistent process for refreshing or updating all data items in line with the data policy (timeliness and currency of data). he process must include appropriate change controls (automated or manual) that take into account any material impact (individually or in aggregation) on the internal model. Data input validations (auto/manual) that prevent data having incorrect or inconsistent format or invalid values. Completeness checks such as: Reconciliation of data received against data expected A process to assess if data is available for all relevant model variables and risk modules Risk assessment enables the detection of control vulnerabilities such as inadequate use of passwords on spreadsheet files, VBA modules and lock down of critical cells. Additional reports provide analysis of ACL vulnerabilities (e.g. single name user groups attached to file shares). Once vulnerabilities have been closed and a new baseline established, alerts provide immediate notifications of regression of these controls. It is usually the case that manual checks require much higher user competency than the automated processes delivered by ClusterSeven. he automation provided by ClusterSeven enables well-defined processes to be consistently applied, including a full audit log of changes. In addition the materiality of changes in terms of individual/aggregation, action/inaction, and data/structure may be separately highlighted and reported for specific attention. ClusterSeven enables an automated data validation process to be conducted across millions of data items from spreadsheets and flat files (e.g..csvs) prior to being uplifted into internal models in order to capture inconsistent or invalid data. It is unrealistic to assume that manual checks will be reliable on anything but a small number of data items. ClusterSeven enables automated reconciliations of data against both expected control values and expected processes (such as transaction maturity dates). ClusterSeven enables automated checks on the presence of up-to-date data extracts from third party systems. olerance alerts Controls over accuracy A recognised check over accuracy is to compare the data received with the original source Reasonable checks, or random checks against the primary (i.e. objectively verifiable) data may often be sufficient Internal coherency/ consistency checks based on known properties of the data (e.g. its expected distribution) can also be effective hink about possible worst/bad cases and place appropriate controls SA 2011 Reconciliation Controls over completeness Reconciliation is a recognised check on completeness Reconciliation means a check on records that were received against the records that were expected to be received his can be difficult to achieve, as it requires transparency about what records were requested e.g. by a complex SQL query Reconciliation can also be more difficult with end-user applications SA hrogmorton Avenue Registered office 10 Argyll Street, London, W1 7Q Registered number Registered in England & Wales.

6 White Paper: SA Data Audit Accuracy checks such as Comparison directly against the source (if available). Internal consistency and coherence checks of the received/output data against expected properties of the data such as age-range, standard deviation, number of outliers, and mean. Comparison with other data derived from the same source, or sources which are correlated. Appropriateness checks such as Consistency and reasonableness checks to identify outliers and gaps through comparison against known trends, historic data and external independent sources. A definition and consistent application of the rules that govern the amount and nature of data used in the internal model. A process to assess the data used in internal model for any inconsistencies with the assumptions underlying the actuarial and statistical techniques or made during the collection, processing and application of data. ClusterSeven enables automated checks of values held in spreadsheets and flat files against values held in other locations such as source data. ClusterSeven reports can apply a range of consistency and coherence checks on output data. However, for more complex analysis (e.g. statistical) these calculations may be completed in a spreadsheet and automatically captured and reported using standard ClusterSeven functionality. In addition to formal value reconciliations ClusterSeven enables visual comparisons (e.g. trending) via graphs to confirm expected correlations. Unlike a spreadsheet file or.csv which typically only contains a snapshot of data in time, ClusterSeven retains the full time series of each cell history, enabling full analysis of historic data and trends. Policy definitions applicable to spreadsheets and flat file data sources may be established within ClusterSeven, enabling the consistent application of these rules during the peaks and troughs of other business activity. Inconsistencies in process or data highlighted by ClusterSeven may be surfaced by alerts or reports to trigger workflow for amendment or approval. Data management appeared to be one area where firms still have comparatively more to do to achieve the likely Solvency II requirements. Also firms did not have a documented validation policy that clearly explained all the processes used to validate their internal model. We will be looking at these areas in more detail at a firm-by-firm level during the pre-application phase of IMAP. SA Solvency II: IMAP, hematic Review findings ebruary, 2011 Risk 5: Unreliable I environment, technology or tools can compromise the quality and integrity of the data and its processing within the internal model Control Objective: o ensure that the quality of data and its processing for use in the internal model is maintained 10 hrogmorton Avenue Registered office 10 Argyll Street, London, W1 7Q Registered number Registered in England & Wales.

7 White Paper: SA Data Audit I general computer (IGC) controls over the data environment (for e.g. Mainframes, End User Computing applications such as spreadsheets, etc) that may have material impact on the internal model are established, such as logical access management development and change management (infrastructure, applications, and database); security (network and physical); business continuity; incident management and reporting, and; other operational controls that support the collection (including data feeds), storage, analysis and processing. ClusterSeven provides a complete solution for End User Computing (EUC) applications such as spreadsheets and MS Access databases as well as flat file extracts such as.csv and.da. ClusterSeven provides analysis of access vulnerabilities (e.g. inappropriate ACLs or inadequate application of Excel passwords) and facilitates the application and maintenance of these controls together with the adoption of Information Rights Management. ClusterSeven provides complete lifecycle support for EUC applications including test and audit cycles. N/a ClusterSeven provides the opportunity for disaster recovery and business continuity with respect to monitored files. ClusterSeven provides a full alerting and reporting environment for EUC activity (or inactivity). ClusterSeven provides operational controls appropriate to these processes when conducted within EUCs e.g. confirming that data feeds have been updated. ClusterSeven success stories in insurance to date: Part of our financial close process used to take hours, now it takes minutes inancial controller, Global insurer his is the first time I can show the rest of the business how it all works Head of Actuarial, Lloyds managing agent Conclusions he presence of spreadsheets and flat files (e.g. CSVs) within the data processing chain, between source systems and the internal model, presents significant challenges in meeting the wide-ranging requirements of the SA Data Audit. he SA Data Audit specifically allows manual controls to be established provided that they are applied consistently. However, the practicality and cost constraints of applying experienced staff to these processes means that automation is likely to be a far more effective and robust solution for anything beyond the simplest and smallest of files. We will use data dictionary terms within our spreadsheets and then automatically upload validated information into our corporate BI solution UK General Insurer 10 hrogmorton Avenue Registered office 10 Argyll Street, London, W1 7Q Registered number Registered in England & Wales.