COntext INterchange (COIN) Mediation Technology

Transcription

1 COntext INterchange (COIN) Mediation Technology Technology developed at MIT Sloan School of Management Exploitation by Mediation Technologies, Inc. by Allen Moulton Stuart Madnick Michael Siegel 1 Information Integration Challenge Lack of it costly and problematic Costs U.S. capital facilities industry $15.8 billion/year (NIST) Estimated cost $1 trillion worldwide Achieving it difficult, expensive 30-50% IT $$$ $100 M/yr in a large org. Simple example: US user is interested in products from South Korea and Turkey. South Korean vendors quote their prices in thousands of South Korea won (KRW, $1 is about 1000 KRW), not including taxes. Turkish vendors quote prices with taxes included in millions of Turkish lira (TRL, $1 is about 1,000,000 TRL). US user wants the comparison to be in US Dollars (USD) with taxes and Shipping and Handling included. Currently this usually requires considerable manual intervention and/or custom software. How can this information be integrated with the semantic differences automatically resolved? 2

2 Traditional approach 1 Brute Force Data Conversion For each combination of receivers and sources, implement data conversion software Widely used in ad-hoc integration 5 4 Costly to implement, and maintain 3 Traditional Approach 2 Global Data Standardization All organizations must agree on a global standard E.g., use kg for weight, cm for height, etc Legitimate reasons for not standardizing U.S. agent is used to lbs, ft, inches.. Other difficulties Agreeing means somebody needs to change existing system Difficult to maintain and enforce DoD case , aimed to standardize ~ 1M elements; managed to put 12,000 into repository, 4

3 Traditional Approach 3 Standardize Interchange Format Agreement on data formats for a predefined set of data elements exchanged at the interface between systems Drawbacks multiple standards, simultaneously and over time extensible standards result in different standards different interpretations and implementations semantic interoperability of internal enterprise systems with external standards Interchange standard examples COT EDI OFX IFX SWIFT MTF 5 Mediation Approach sources and receivers are autonomously designed and evolve autonomously reasoning uses declarative descriptions of source and receiver data semantics prepares a plan for combining sources and conversion components to meet receiver semantic requirements 6

4 Role Of $? CONTEXT VARIATIONS: - GEOGRAPHIC ( US vs. UK ) - FUNCTIONAL (CASH MGMT vs. LOANS ) - ORGANIZATIONAL ( CITIBANK vs. CHASE ) Data: Databases Web data 7 COIN Approach Concept: Length Meters Feet Shared Ontologies f() meters Conversion Libraries feet Management Administrator part length 17.0 Source Mediator Transformation Receiver select partlength from catalog where partno= 12AY select partlength/.3048 from catalog where partno= 12AY Source Auto-composition of conversion plan Receiver 8

5 COIN System Architecture SERVER PROCESSES MEDIATOR PROCESSES CLIENT PROCESSES Web Client WWW Gateway HTTPD-Daemon COIN Repository Mediator Optimizer Mediated Query SQL Compiler Datalog Query SQL Query SQL Query (cgi-scripts) HTTPD-Daemon N N Wrapper HTTPD-Daemon Executioner Optimized Query Plan Results ODBC-compliant Apps (e.g Microsoft Excel) ODBC-Driver HTTPD-Daemon Web-site Data Store for Intermediate Results 9 COIN Technology Status Basic research funded by DARPA (about $2M). MIT patented technologies (3 patents) Series of prototypes implemented Tested in variety of application settings Current technology ready to be made into commercial-grade software Research on extending context interchange mediation technology continues 10

6 MIT Patented Technology Patent number 5,913,214: "Data Extraction from World Web Pages" Filed: Aug. 8, 1996; Patent Issued: June 15, Patent number 5,953,716: "Querying Heterogeneous Data Sources Distributed over a Network Using Interchange" Filed: May 30, 1996; Patent Issued: Sept 14, Patent number 6,282,537: "Querying and Retrieving Semi-Structured Data from Heterogeneous Data Sources by Translating Structured Queries" Filed: Apr 6, 1999; Patent Issued: Aug. 28, COIN Application Case Studies Counter-terrorism intelligence integration Financial services information integration Insurance (intra-company context differences) Equity securities analysis Fixed income securities investment Medical supplies and logistics Weather and hotels Airfare, car rental & merged travel agent Disaster relief 12

7 Example: Counter-Terrorism Intelligence (CTI) Problem 3 different heights and weights reported for one individual because: src1 reports in US context using inches and pounds src2 reports in NATO context using meters and kilograms src3 reports in UK context using feet and stones SQL none Stage Naive Datalog SQL Translation Sensitive Datalog Execution Conflict Detection Mediation Result recid srcid person height weight 106 src2 Aleph Faruk src3 Aleph Faruk src3 Aleph Faruk src3 Aleph Faruk src1 Aleph Faruk CTI Domain Ontology place Place Event event Type event Date dateformat gmtoffset Country City geo Coord event Time timeformat gmtoffset country language city language geocoord RefFrame datum geoformat height Person weight person country Code airport lengthunit weightunit romanization System ctrycodestd aptsymtype Legend attribute modifier Entity semantic type 14

8 Detection of Semantic Conflicts Semantic Type weight height Semantic Type weight height Column Source Modifier source context cti_reports_usa(, _country, _countrycode, _city, _airport, _place, _weight weightunit c_usa : lb _eventtype, _eventdate, _eventtime, ) cti_reports_usa(, _country, _height _countrycode, _city, _airport, _place, lengthunit c_usa : in _eventtype, _eventdate, _eventtime, ) Column Source Modifier source context cti_reports_uk(, _country, _countrycode, _city, _airport, _place, _weight weightunit c_uk : stone _eventtype, _eventdate, _eventtime, ) cti_reports_uk(, _country, _countrycode, _city, _airport, _place, _height lengthunit c_uk : ft _eventtype, _eventdate, _eventtime, ) target context c_analyst : kg c_analyst : m target context c_analyst : kg c_analyst : m Conversion Function Conversion Function Semantic Type height Column Source Modifier _height source context cti_reports_nato(, _country, _countrycode, _city, _airport, _place, lengthunit c_nato : cm _eventtype, _eventdate, _eventtime, ) target context c_analyst : m Conversion Function 15 CTI Mediation Solution Solution Mediation finds semantic conflicts and rewrites the query in semantics needed by Analyst receiver (meters and kilograms) SQL c_analyst Stage Naive Datalog SQL Translation Sensitive Datalog Execution Conflict Detection Mediation Result recid srcid person height weight 106 src2 Aleph Faruk src3 Aleph Faruk src3 Aleph Faruk src3 Aleph Faruk src1 Aleph Faruk select cti_reports_usa.recid, cti_reports_usa.srcid, 'Aleph Faruk', (cti_reports_usa.height*unit_conv.unitfactor), (cti_reports_usa.weight*unit_conv2.unitfactor) from (select recid, country, countrycode, city, airport, place, geocoord, 'Aleph Faruk', height, weight, eventtype, eventdate, eventtime, srcid from cti_reports_usa where person='aleph Faruk') cti_reports_usa, (select 'in', 'm', UnitFactor from unit_conv where FrmUnit='in and ToUnit='m') unit_conv, (select 'lb', 'kg', UnitFactor from unit_conv where FrmUnit='lb' and ToUnit='kg') unit_conv2 union select cti_reports_uk.recid, cti_reports_uk.srcid, 'Aleph Faruk', (cti_reports_uk.height*unit_conv3.unitfactor), (cti_reports_uk.weight*unit_conv4.unitfactor) from (select recid, country, countrycode, city, airport, place, geocoord, 'Aleph Faruk', height, weight, eventtype, eventdate, eventtime, srcid from cti_reports_uk where person='aleph Faruk') cti_reports_uk, (select 'ft', 'm', UnitFactor from unit_conv where FrmUnit='ft' and ToUnit='m') unit_conv3, (select 'stone', 'kg', UnitFactor from unit_conv where FrmUnit='stone' and ToUnit='kg') unit_conv4 union select cti_reports_nato.recid, cti_reports_nato.srcid, 'Aleph Faruk', (cti_reports_nato.height*unit_conv5.unitfactor), cti_reports_nato.weight from (select 'cm', 'm', UnitFactor from unit_conv where FrmUnit='cm' and ToUnit='m') unit_conv5, (select recid, country, countrycode, city, airport, place, geocoord, 'Aleph Faruk', height, weight, eventtype, eventdate, eventtime, srcid from cti_reports_nato where person='aleph Faruk') cti_reports_nato 16

9 Fixed Income Securities Fixed Income Investment Portfolio Manager RECEIVER Portfolio Management Application System Domain Ontology Interchange Mediator SOURCES Security Details Security Valuations Portf olio Holdings Receiver Source s Bond Calculation Functions Receiver needs Autonomous sources provide RR AA BB CC DD EE price: price for a security in percent with fraction in 32nds price: price for a security in percent with decimal fraction price: nominal spread to Treasury yield curve in basis points coupon: coupon interest rate for a security in percent 10yr: point on Treasury yield curve in percent price: yield: rate: computes price from yield given coupon interest rate, etc. with price in percent and inputs in factor form 17