Effective Audit Trail of Data With PROV-O Scott Henninger, Senior Consultant, MarkLogic

Transcription

1 Effective Audit Trail of Data With PROV-O Scott Henninger, Senior Consultant, MarkLogic COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

2 EFFECTIVE AUDIT TRAIL WITH PROV-O Operationalizing the Metadata Data Governance Quality management Provenance Models Metadata description Provenance Dimensions Technical perspective SLIDE: 2

3 DATA GOVERNANCE Strategy and Execution DATA QUALITY POLICIES ORGANIZATION PROCESSES Shared: Exchange of data between different departments is possible Reliable: Source has competence in the field of interest RISK MANAGEMENT DATA LINEAGE Accurate: All accounting events are correct in value and description REGULATORY COMPLIANCE Current: Data is up-to-date for the world it models SLIDE: 3

4 INGEST PREPARE TRANSFORM PUBLISH DATA GOVERNANCE Information Chain Create: Generate new data entities or update their state Derive: Create value from more contributing data entities Analyze: Inspect data to discover new useful information Report: Submission of summary data as evidence of events 5 DELETE SLIDE: 4

5 DATA GOVERNANCE Provenance Metadata Origin: Proof of the data ownership during the history of the data UPDATES RESPONSIBILITY Timeline: Recorded timestamps of all events the data experienced INFORMATION ORIGIN Process: Transformations that change the data during its lifecycle REPRODUCIBILITY EVOLUTION ACCESS Data provenance documents the inputs, entities, systems, and processes that influence data of interest, providing a historical record of the data and its origins Data lineage includes the data's origins, what happens to it and where it moves over time SLIDE: 5

6 If the benefits of provenance are so well understood, why don t more firms recognize it as a priority? What makes it difficult Location Data is spread across different systems in different organizational silos Ownership Lack of mature data governance makes the challenge of data lineage even more daunting Spreadsheets Business processes run outside of data management processes Unforeseen costs Compliance risk The business gets exposed to difficult contract negotiations which can incur additional data costs Redundant data activities Duplicate controls are performed in different departments several times Accuracy of analytics Impossible to verify why models result in sub-optimal outcomes SLIDE: 6

7 EFFECTIVE AUDIT TRAIL WITH PROV-O Operationalizing the Metadata Data Governance Quality management Provenance Models Metadata description Provenance Dimensions Technical perspective SLIDE: 7

8 PROVENANCE MODEL Metadata Repository LAZY Complex technique for reasoning EAGER Derived directly from output database ETL TRACING PROVENANCE TRACING PROVENANCE ETL ETL* SLIDE: 8

9 PROVENANCE MODEL Provenance Storage Envelope Pattern Provenance stored with data <envelope> <provenance> <sem:triple> <sem:subject>/doc/id_a12a3.xml</sem:subject> <sem:predicate> </sem:predicate> <sem:object>/xform </sem:object> </sem:triple> <sem:triple> <sem:subject>/canonicaltransform </sem:subject> <sem:predicate> </sem:predicate> <sem:object> datatype=" T12:01:42.987</sem:object> </sem:triple>... </provenance> <content> <doc-id>a12a3</docid> <workflowstatus>draft</workflowstatus> <version>2.3</version>... </content> SLIDE: 9 </envelope> Separate Database Large provenance payloads stored with reference to data Content Database uri: /doc/id_a12a3.xml <content> <doc-id>a12a3</docid> <workflowstatus>draft</workflowstatus> <version>2.3</version>... </content> Provenance Database <provenance> <sem:triple> <sem:subject>/doc/id_a12a3.xml </sem:subject> <sem:predicate>wasgeneratedby </sem:predicate> <sem:object>/xform </sem:object> </sem:triple> </provenance>

10 PROVENANCE MODEL PROV Data Model Entity: a trade, order, document, or other kind of entity, physical, digital or conceptual with some fixed aspects wasattributedto wasderivedfrom ENTITY Activity: something that occurs over a period of time and acts upon or with entities, such as creating, consuming, transforming, modifying, etc. AGENT uses wasgeneratedby Agent: the business line responsible for an activity taking place, for the existence of an entity wasassociatedwith startedattime ACTIVITY endedattime W3C standard, circa 2013: xs:datetime xs:datetime SLIDE: 10

11 PROVENANCE MODEL Encoding Specification PROV-O Reason on provenance data Specialized properties Model-based extensions of the standard PROV-XML Types and elements are prov: : < :genesequencing a prov:activity; prov:startedattime " T01:30:00Z"; prov:used :drosophilasample-84; prov:wasassociatedwith :lab-technician-gh-32. :drosophilasample-84 a prov:entity; prov:wasattributedto :lab-technician-fe-56. :lab-technician-gh-32 a prov:agent. <prov:document xmlns:prov=" xmlns:ex=" <prov:entity prov:id="ex:e1"> <prov:type xsi:type="xsd:string">approval </prov:type> </prov:entity> <prov:activity prov:id="ex:a1"> <prov:type xsi:type="xsd:qname">editing</prov:type> </prov:activity> </prov:document> XML SLIDE: 11

12 EFFECTIVE AUDIT TRAIL WITH PROV-O Operationalizing the Metadata Data Governance Quality management Provenance Models Metadata description Provenance Dimensions Technical perspective SLIDE: 12

13 PROVENANCE DIMENSIONS Content, Use, and Management Non mutually exclusive Who endorses the information How a decision is made User consumption of provenance What was considered for that decision MANAGEMENT USE CONTENT SLIDE: 13

14 PROVENANCE DIMENSIONS Content Use Management Non mutually exclusive Who endorses the information How a decision is made User consumption of provenance What was considered for that decision Scenario an investment bank is implementing new regulatory reporting defined by CFTC, that will provide more information on their trading activities (extended type of financial products USE MANAGEMNT and pieces of data) in a shorter time frame (near real-time CONTENT publication) with higher complexity of the rules determining who has the obligation to deliver the information. SLIDE: 14

15 @prefix prov: < PROVENANCE CONTENT Attribution What the provenance is about Sources used to create new result Process that yielded the : < :TransactionReport a prov:entity; prov:generatedattime " T12:12:12 ; prov:wasderivedfrom :TransactionA ; prov:wasgeneratedby :ReportGen. :ReportGen a prov:activity; prov:used prov:transactiona ; prov:used :Venue1 ; prov:wasassociatedwith :Msma. XML :TransactionA a prov:entity prov:wasatttributedto :Murex :Venue1 a prov:entity. :Murex a prov:agent, prov:softwareagent. :Msma a prov:agent, prov:organization ; SLIDE: 15

16 @prefix prov: < PROVENANCE CONTENT Evolution Amendments are incorporated in the trade Different aspects of the same trade linked : < :Transaction1 a prov:entity. :Transaction2 a prov:entity; prov:wasrevisionof :Transaction1. :TransactionReport1 a prov:entity; prov:wasderivedfrom :Transaction1. :TransactionReport2 a prov:entity; prov:wasderivedfrom :Transaction2. ORIGINAL VERSION AMENDED VERSION :PostTradeReport a prov:entity; prov:generatedattime " T12:12:14"; prov:wasderivedfrom :Transaction2; prov:alternateof :TransactieReport2. SLIDE: 16

17 PROVENANCE CONTENT Bitemporal Timelines Two temporal axes to maintain the business valid and the system times WHEN THE EVENT OCCURRED (Valid Time) NOV 19 WHEN THE EVENT OCCURRED (Valid Time) NOV 20 LAG WHEN IT WAS RECORDED (System Time) NOV 21 WHEN IT WAS RECORDED (System Time) { "transaction": { "system-start": " T11:00:00", "valid-start": " T12:00:00", "trader": "12XL9A", "price": 12 } } { "trader": { "system-start": " T11:00:00", "valid-start": " T12:00:00", "id": "12XL9A", "name": "John" } } SLIDE: 17

18 PROVENANCE DIMENSIONS Content T12:12:12 Reporting wasinfluencedby Ingest generatedattime Post Trade Report generated wasderivedfrom used generated wasattributedto Murex alternateof Trade v2 wasrevisionof Trade v1 Transaction System Feedback wasinvalidatedby wasderivedfrom receivedattime Transaction Report T10:22:12 value Software Agent SLIDE: 18

19 PROVENANCE USE Understanding what was the trading and reference data used to generate this transaction report... why is there a difference between the transaction report and the post trade report were there any changes in reference data at the time the correction was sent DATA STEWARD DATA QUALITY SLIDE: 19

20 PROVENANCE USE Compliance which department provided the trade data and when was the booking done what transactions were not reported in the time required, and for what reasons REGULATORY COMPLIANCE are any transactions that should have been reported for new versions of rules... have traders complied with rules... COMPLIANCE OFFICER PENALTY BUSINESS DEPARTMENT SLIDE: 20

21 PROVENANCE USE Debugging... where did an error occur in a specific data field... was the notification for the post trade publisher sent for that specific trade what version of data extraction rules were used when the transaction report was created what is the percent of reportable transactions from the daily volume OPERATIONAL DATA STORE IT OPERATIONS APPROVED PUBLICATION ARRANGEMENT SLIDE: 21

22 PROVENANCE USE Trusting Data Sources Forward-Looking provenance - Anticipating problems given provenance information from other systems Analysis may find some sources/transformations/etls are troublesome -...sometimes in specific contexts, such as high load rates, etc. Look for alternatives when designing future efforts Target troublesome processes for future refactoring efforts SLIDE: 22

23 PROVENANCE MANAGEMENT Publication HOW TO CONSUME PROVENANCE? LINK SEARCH html - HTTP xml - HTTP ACCESS TARGET URI PROVENANCE URI CONTENT XML RESOURCES Access Locate Query rdf - SPARQL BROWSE PROVENANCE SLIDE: 23

24 PROVENANCE MANAGEMENT Dissemination Security: secure HTTP should be used across unsecured networks; authentication should be enforced Access control: provenance information should follow the same access control rules as the resources Bundle: care is needed to ensure that the integrity of provenance is maintained Provenance discovery HTTPS Provenance of provenance PRIVACY WALL SLIDE: 24

25 PROVENANCE DIMENSIONS PUBLICATION UNDERSTANDING COMPLIANCE QUERY DEBUGGING XML FORWARD-LOOKING ACCESS SEMANTIC DOCUMENT USE MANAGEMENT CONTENT SLIDE: 25

26 solid compliance architecture DATA LINEAGE What makes it easy ALL of your data and metadata Complete track of data changes Full query composability Security, publishing, monitoring, etc. Fewer tools and processes to manage SLIDE: 26

27 Questions?