FSA data review stock take Dean Buckner Financial Services Authority March 2012
Agenda FSA data review process Common themes Next steps
FSA data review In three acts Act 1 Review of approx. 25 firms Objective: Determine dependencies in data & Systems work stream. Understand material data flows, agree scope and timing of external review Act 2 Scene 1 Firm s internal audit performs review Scene 2 FSA review of the review Act 3 Selective deep dive
Timings Act 1 September 2011 April 2012 Act 2 March 2012 Q1 2013 Act 3 Q2 2012 Q3 2013
Common themes (Act 1) Data governance operating model Data directory maintenance Data transformation Pervasive use of spreadsheets Data semantics Dependency on IT
Data operating model Needs careful design Design of operating model /= policy design Needs a manager! Some firms have appointed permanent managers Others are using the project manager until bau Others are using their existing governance frameworks and applying it to data risk & controls.
Other issues with data governance Inconsistent definition of data classification, ownership and responsibility Impact and materiality assessment firms are slowly getting to grips with this and there is no consistent approach.
Data directory maintenance Three different approaches. Tight coupling Structured directory, updated automatically Loose coupling Structured directory, semi-automatic update No coupling Freeform, unstructured, updated manually
Directory maintenance trade offs Tight coupling Benefit hardly any maintenance, automatic update Cost a pain to build, dependent on IT Loose coupling Benefit relationships can be accurately represented Cost Highly skilled maintenance No coupling Benefit no dependence on IT, flexibility Cost doesn t reflect reality, staff costs may prove prohibitive, possibly error prone FSA will not be prescriptive
Data transformation
Data is not just moved The idea of data movement is from IT N bytes of data are copied from system A to system B This kind of movement is trivial and uninteresting Typically when data passes from A to B, stuff happens Data sets are joined, merged, mapped Often transformed in exotic and interesting ways Data is operated on
Examples of data transformation Extrapolation, interpolation Extraction of key economic features A bond position is turned into a sensitivity calibration of risk factor stress
Scope of data transformation All material transformations of data outside the IM Kernel are in scope of the review. This includes Testing to confirm that the implementation (e.g. using spreadsheets, ETL, etc) complies with its design specifications Data Quality checks to ensure that the output of the transformation reflects the input data Where the transformation is functional, and its design involves expert judgment, the design or methodology or functional specification of the transformation is out of scope
Data semantics
What is semantics Semantics = fancy word for meaning Data records are made up of symbols which have a syntax and a meaning The same term can have a different meaning for different systems Different terms can have the same meaning for different systems
Meaning and translation Human dictionaries translate a term in one language into a term in another language with the same meaning omnis in Latin has the same meaning as the English quantifier every So different computer systems need translation or mapping tables
Semantic errors Many errors result from changes in the upstream meaning or basis of a term not being appropriately reflected downstream Basis is the hardest data characteristic to document and may well be the most frequent or material cause of error (See war stories below)
War story 1 A firm thought that its bodily injury motor claims estimates were not keeping pace with rapid inflation, so changed its claims diary from 12 months to 6 months Claims would be reviewed every six months, ensuring that case reserves were updated for claims inflation more frequently. The effect of reviewing small claims more often led to smaller claims being settled earlier and the surplus in the case estimates being released more quickly. Staff change meant loss of knowledge about the change so reason for distortion of claims data not understood. Result: significant underestimation of claims reserves. So an action that was intended to be prudent resulted in material under-reserving.
War story 2 Upstream system was sending credit swap positions using a single column Long position positive, short position negative For the new implementation, the one column was changed to two Amount of position now always positive, new column has B (buy) and S (sold) flag. Downstream system was never notified The first column remained the same, so nothing broke, and no preventative alarm was raised. Result: the downstream system thought all the short positions were really long positions. This led to a material mis-estimation of the firm s exposure.
War story 3 Upstream system didn t understand inflation bonds. These pay a fixed coupon plus an inflation factor computed using external data So they were booked as standard bonds, with the coupon adjusted upwards to compensate for the inflation factor This is very common practice for old systems which cannot represent new products without major engineering works. But the downstream system did understand inflation bonds, and assumed the upstream coupon amount was merely the fixed coupon component. So it added on an extra inflation factor which had already been included in the artificially adjusted coupon. Result: more material mis-estimation of exposure.
Controls over basis change Genuinely very difficult Getting computer systems to communicate with one another is one of the great unsolved problems of computer science Common methods include Quantitative change analysis Impact assessment Corporate memory Basic reconciliation or reasonableness checks
Dependency on IT
Impact of major IT implementations Key principle What we review is what you apply for If we review a tactical solution, then that is what you are applying for But what if there is a major strategic solution in the pipeline? Then we still review the tactical, and the strategic solution is model change
What is model change Change policy is key what items, systems, transformations related to data are part of the model change policy? change of platform only (same software and methodology)? Change of software only, no methodology change? Change of methodology?
Next steps
Act 2 Act 2 is marking the completed external reviews as they are returned by the firms Criteria Geographical, legal entity and systems scope should be proportionate [rewrite] Impact of finding should be clear and unequivocal Due dates must be consistent with application process Was there sufficient operational testing (one endto-end flow is probably not enough)
Materiality What materiality criteria were used to determine the scope of the audit and to assess impact of a finding / residual risk? Has audit considered: justification for determining materiality thresholds? consistency of materiality assessment with other policies? identification of future risks? possibility of material error caused by aggregation of errors which are not material singly?
External review format No precise criteria, except that it must include the FSA schedule at the top (as specified on our website). As per the scoping document, Act 2 results submission should include an executive summary, the FSA schedule, followed by details of each finding. Nice to have: Appendix of detailed findings, cross-referenced from main schedule Detailed findings to include rating, observation, clear articulation of impact and consequence, recommendation and management action plan with precise dates. Mitigating factors?
Act 3 Act 3 is optional deep dive following Act 3 is optional deep dive following themes identified by Act 2
Questions & Comments