CORE Scholar. Wright State University. Amit P. Sheth Wright State University - Main Campus,

Transcription

1 Wright State University CORE Scholar Kno.e.sis Publications The Ohio Center of Excellence in Knowledge- Enabled Computing (Kno.e.sis) Realizing the Relationship Web: Morphing Information Access on the Web from Today's Document- and Entity-Centric Paradigm to a Relationship-Centric Paradigm Amit P. Sheth Wright State University - Main Campus, amit.sheth@wright.edu Follow this and additional works at: Part of the Bioinformatics Commons, Communication Technology and New Media Commons, Databases and Information Systems Commons, OS and Networks Commons, and the Science and Technology Studies Commons Repository Citation Sheth, A. P. (2007). Realizing the Relationship Web: Morphing Information Access on the Web from Today's Document- and Entity- Centric Paradigm to a Relationship-Centric Paradigm.. This Presentation is brought to you for free and open access by the The Ohio Center of Excellence in Knowledge-Enabled Computing (Kno.e.sis) at CORE Scholar. It has been accepted for inclusion in Kno.e.sis Publications by an authorized administrator of CORE Scholar. For more information, please contact corescholar@

2 Realizing the Relationship Web: Morphing information access on the Web from today s document- and entity-centric paradigm to a relationship-centric paradigm ACM Multimedia International Workshop on the Many Faces of Multimedia Semantics September 28, 2007, Augsburg, Germany Amit Sheth Kno.e.sis Center, Wright State University, Dayton, OH This talk also represents work of several members of Kno.e.sis team, esp. the Semantic Discovery and Semantic Sensor Data. Thanks, M. Perry, C. Ramakrishnan, C. Thomas

3 Objects of Interest An object by itself is intensely uninteresting. Grady Booch, Object Oriented Design with Applications, 1991 Keywords Search Entities Integration Relationships Analysis, Insight Entities + Relationships also needed to model & study Events

4 Semantics and Relationships Increasing depth and sophistication in dealing with semantics by dealing with (identifying/searching to analyzing) documents, entities, and relationships. Future Current Past Relationships Entities Documents/Media

5 Data, Information, and Insight What: Which thing or which particular one Who: What or which person or persons Where: At or in what place When: At what time How: In what manner or way; by what means Why: For what purpose, reason, or cause; with what intention, justification, or motive Ramesh Jain

6 Insights require understanding Relationships Object Location Time Relationships What X X X Who X Where X When X How X Why X Ramesh Jain

7 Semantics and Relationships Semantics is derived from relationships. Consider the linguistics perspective. Semantics is the study of meaning. We may distinguish a number of legitimate ways to approach semantics: the relationship between linguistic expressions (e.g. synonymy, antonymy, hyperonymy, etc.): sense; the relationship to linguistic expressions to the "real world": reference. Ontologies use KR language to support modeling of relationships. Quoted part from Jonathan West.

8 Why is this a hard problem? Are objects/entities equivalent/equal(same)? How (well) are they related? Implicit vs explicit; formal/assertional vs social consensus based; powerful (beyond FOL): partial, probilistic and fuzzy match Degrees of relatedness and relevance: semantic similarity, semantic proximity, semantic distance,. [differentiation, disjointedness] related in a context Even is-a link involves different notions: identify, unity, essense (Guarino and Wetley 2002) Semantic ambiguity, also based on incomplete, inconsistent, approximate information/knowledge

9 Issues - Relationships Identifying Relationship (extraction) Expressing (specifying, representing) relationships Discovering and Exploring Relationships Hypothesizing and Validating Relationships Utilizing/exploiting Relationships for Semantic Applications (in document search, querying metadata, inferencing, analysis, insight, discovery)

10 Information Extraction for Metadata Creation WWW, Enterprise Repositories Nexis UPI AP Feeds/ Documents Digital Images Digital Videos Digital Maps... Digital Audios Data Stores Create/extract as much (semantics) metadata automatically as possible; Use ontlogies to improve and enhance extraction EXTRACTORS METADATA

11 Automatic Semantic Metadata Extraction/Annotation

12 Semantic Annotation (Extraction + Enhancement) COMTEX Tagging Value-added Voquette Semantic Tagging Content Enhancement Rich Semantic Metatagging Limited tagging (mostly syntactic) Value-added relevant metatags added by Voquette to existing COMTEX tags: Private companies Type of company Industry affiliation Sector Exchange Company Execs Competitors

13 Semantic Metadata Enhancement

14 Automatic Classification & Metadata Extraction (Web page) Auto Categorization Video with Editorialized Text on the Web Semantic Metadata

15 Ontology-directed Metadata Extraction (Semi-structured data) Web Page Enhanced Metadata Asset Extraction Agent

16 Semantic Extraction/Annotation of Experimental Data ProPreO: Ontology-mediated provenance parent ion m/z fragment ion m/z parent ion charge parent ion abundance fragment ion abundance ms/ms peaklist data Mass Knowledge Spectrometry Enabled Information and (MS) Services Science Data

17 Video metadata and search Technique Who s trying it How it works Wired Tired Scanning the script Blinkx, TVEyes Everything said in a clip is tracked through voice recognition software, closedcaption information, or a combination of the two. Hunts down TV news references to, say, Lindsay Lohan. A mere mention of her name doesn't guarantee Lindsay is in the clip. Identifying what's being shown Google, UCSD's Statistical Visual Computing Lab, VideoMining Algorithms try to figure out what's in the video by monitoring attributes like behavior and movement (VideoMining), faces (Google), and objects (UCSD). Finds friends, public figures, or specific actions like car chases. The tech is stuck in the lab or limited to specialized search tasks. Analyzing links and metadata Dabble, Google Conventional search spiders scan the text around a video and the pages that link to it. The fastest and best way to search for video data. Can't "see" what's in the videos or locate specific action in a long clip. Wired Mag, Aug 2007

18 Semantic Sensor ML Adding Ontological Metadata Domain Ontology Company Person Spatial Ontology Coordinates Coordinate System Temporal Ontology Time Units Timezone Mike Botts, "SensorML and Sensor Knowledge Web Enablement," Enabled Information and Services Science 18 Earth System Science Center, UAB Huntsville

19 Relationships on the Web: early work

20 MREF (Metadata Reference Link -- complementing HREF) Creating logical web through Media Independent Metadata based Correlation

21 Metadata Reference Link (<A MREF >) <A HREF= URL >Document Description</A> physical link between document (components) <A MREF KEYWORDS=<list-of-keywords>; THRESH=<real>>Document Description</A> <A MREF ATTRIBUTES(<list-of-attribute-valuepairs>)>Document Description</A>

22 MREF Metadata Reference Link -- complementing HREF (1996, 1998) Creating logical web through Media Independent Metadata based Correlation ONTOLOGY NAMESPACE ONTOLOGY NAMESPACE METADATA DATA MREF in RDF METADATA DATA

23 MREF (1998) Model for Logical Correlation using Ontological Terms and Metadata Framework for Representing MREFs Serialization (one implementation choice) M R E F R D F X M L K. Shah anda. Sheth, "Logical Information Modeling of Web-accessible Heterogeneous Digital Assets", Proc. of the Forum on Research and Technology Advances in Digital Libraries," (ADL'98), Santa Barbara, CA, May 28-30, 1998, pp

24 Correlation based on Content-based Metadata Some interesting information on dams is available here. information on dams is defined by an MREF defining keywords and metadata (which may be used for a query). water.gif (Data) Metadata Storage water.gif mpeg ppm Content Dependent Metadata height, width and size Content based Metadata Major component(rgb) Blue

25 Domain Specific Correlation Potential locations for a future shopping mall identified by all regions having a population greater than 500 and area greater than 50 sq meters having an urban land cover and moderate relief <A MREF ATTRIBUTES(population < 500; area < 50 & region-type = block & land-cover = urban & relief = moderate )>can be viewed here</a> => media-independent relationships between domain specific metadata: population, area, land cover, relief => correlation between image and structured data at a higher domain specific level as opposed to physical link-chasing in the WWW

26 Repositories and the Media Types Population: Area: Boundaries: Regions (SQL) Land cover: Relief: Boundaries Image Features (IP routines) Map DB Census DB TIGER/Line DB

27

28 Complex Relationships Some relationships may not be manually asserted, but according to statistical analyses of text, experimental data, etc. allow association of provenance data with classes, instances, relationship types and direct relationships or statements

29 A simple relationship? Smoking Causes Cancer

30 Complex Relationships - Cause-Effects & Knowledge discovery VOLCANO ENVIRON. LOCATION PYROCLASTIC FLOW ASH RAIN BUILDING DESTROYS WEATHER COOLS TEMP PLANT PEOPLE LOCATION DESTROYS KILLS

31 Knowledge Discovery - Example Earthquake Sources Nuclear Test Sources Nuclear Test May Cause Earthquakes Is it really true? Complex Relationship: How do you model this?

32 Inter-Ontological Relationships A nuclear test could have caused an earthquake if the earthquake occurred some time after the nuclear test was conducted and in a nearby region. NuclearTest Causes Earthquake <= datedifference( NuclearTest.eventDate, Earthquake.eventDate ) < 30 AND distance( NuclearTest.latitude, NuclearTest.longitude, Earthquake,latitude, Earthquake.longitude ) < 10000

33 Knowledge Discovery exploring relationship For each group of earthquakes with magnitudes in the ranges 5.8-6, 6-7, 7-8, 8-9, and >9 on the Richter scale per year starting from 1900, find number of earthquakes Number of earthquakes with magnitude > 7 almost constant. So nuclear tests probably only cause earthquakes with magnitude < 7

34 Now possible Extracting relationships between MeSH terms from PubMed Biologically active substance complicates causes affects UMLS Semantic Network Lipid causes affects Disease or Syndrome instance_of Fish Oils??????? instance_of Raynaud s Disease MeSH 9284 documents 5 documents 4733 documents PubMed

35 Schema-Driven Extraction of Relationships from Biomedical Text Cartic Ramakrishnan, Krys Kochut, Amit P. Sheth: A Framework for Schema- Driven Relationship Discovery from Unstructured Text. International Semantic Web Conference 2006: [.pdf]

36 Method Parse Sentences in PubMed SS-Tagger (University of Tokyo) SS-Parser (University of Tokyo) Entities (MeSH terms) in sentences occur in modified forms (TOP (S adenomatous (NP (NP (DT An) modifies (JJ excessive) hyperplasia (ADJP (JJ endogenous) (CC or) (JJ exogenous) An excessive ) (NN stimulation) endogenous ) (PP or exogenous (IN by) (NP stimulation (NN estrogen) modifies ) ) ) (VP (VBZ induces) estrogen (NP (NP (JJ adenomatous) (NN hyperplasia) ) (PP (IN of) (NP (DT the) Entities (NN endometrium) can also occur ) as ) ) composites ) ) ) of 2 or more other entities adenomatous hyperplasia and endometrium occur as adenomatous hyperplasia of the endometrium

37 Method Identify entities and Relationships in Parse Tree DT the NP NP JJ excessive ADJP NN stimulation JJ endogenous CC or JJ exogenous IN by PP TOP S NN estrogen VBZ induces VP JJ adenomatous NP NP NN hyperplasia Modifiers Modified entities Composite Entities IN of PP DT the NP NN endometrium

38 Resulting Semantic Web Data in RDF adenomatous hyperplasia hasmodifier An excessive endogenous or exogenous stimulation hasmodifier modified_entity2 haspart haspart modified_entity1 induces composite_entity1 haspart estrogen haspart Modifiers Modified entities Composite Entities endometrium

39 Blazing Semantic Trails in Biomedical Literature Cartic Ramakrishnan, Amit P. Sheth: Blazing Semantic Trails in Text: Extracting Complex Relationships from Biomedical Literature. Tech. Report #TR-RS2007 [.pdf]

40 Relationships -- Blazing the Trails The physician, puzzled by her patient's reactions, strikes the trail established in studying an earlier similar case, and runs rapidly through analogous case histories, with side references to the classics for the pertinent anatomy and histology. The chemist, struggling with the synthesis of an organic compound, has all the chemical literature before him in his laboratory, with trails following the analogies of compounds, and side trails to their physical and chemical behavior. [V. Bush, As We May Think. The Atlantic Monthly, (1): p ]

41 Original documents PMID PMID

42 Semantic Trail

43 Semantic Trails over all types of Data Semantic Trails can be built over a Web of Semantic (Meta)Data extracted (manually, semi-automatically and automatically) and gleaned from Structured data (e.g., NCBI databases) Semi-structured data (e.g., XML based and semantic metadata standards for domain specific data representations and exchanges) Unstructured data (e.g., Pubmed and other biomedical literature) and Various modalities (experimental data, medical images, etc.)

44 Applications Everything's connected, all along the line. Cause and effect. That's the beauty of it. Our job is to trace the connections and reveal them. Jack in Terry Gilliam s 1985 film - Brazil

45 An application in Risk & Compliance Ahmed Yaseer: Watch list FBI Watchlist appears on Watchlist Organization Hamas member of organization Appears on Watchlist FBI Works for Company WorldCom Member of organization Hamas Ahmed Yaseer works for Company WorldCom Company

46 Global Investment Bank Watch Lists Law Enforcement Regulators Public Records World Wide Web content BLOGS, RSS Semi-structured Government Data Un-structure text, Semi-structured Data Establishing New Account User will be able to navigate the ontology using a number of different interfaces Scores the entity based on the content and entity relationships Example of Fraud prevention application used in financial services

47 Hypothesis driven retrieval of Scientific Text

48 Semantic Browser

49 More about the Relationship Web Relationship Web takes you away from which document could have information I need, to what s in the resources that gives me the insight and knowledge I need for decision making. Amit P. Sheth, Cartic Ramakrishnan: Relationship Web: Blazing Semantic Trails between Web Resources. IEEE Internet Computing July 2007 (to appear) [.pdf]

50 Events: 3 Dimensions Spatial, Temporal and Thematic Spatial Temporal Thematic

51 Events and STT dimensions Powerful mechanism to integrate content Describes the Real-World occurrences Can have video, images, text, audio all of the same event Search and Index based on events and STT relations Many relationship types Spatial: What events happened near this event? What entities/organizations are located nearby? Temporal: What events happened before/after/during this event? Thematic: What is happening? Who is involved? Going further Can we use What? Where? When? Who? to answer Why? / How? Use integrated STT analysis to explore cause and effect

52 Example Scenario: Sensor Data Fusion and Analysis High-level Sensor (S-H) Low-level Sensor (S-L) H L A-H E-H A-L E-L How do we determine if A-H = A-L? (Same time? Same place?) How do we determine if E-H = E-L? (Same entity?) How do we determine if E-H or E-L constitutes a threat? 53

53 Data Pyramid Sensor Data Pyramid Relationship Metadata Entity Metadata Feature Metadata Semantics/Understanding /Insight Information Raw Sensor (Phenomenological) Data Data 54

54 Sensor Data Architecture Knowledge Analysis Processes Annotation Processes Object-Event Relations Spatiotemporal Associations Provenance Pathways Semantic Analysis SML-S RDF KB Entity Detection SML-S Feature Extraction O&M Information Entity Metadata Fusion SML-S Ontologies Object-Event Ontology Feature Metadata TML Space-Time Ontology Collection Data Raw Phenomenological Data Sensors (RF, EO, IR, HIS, acoustic) 55

55 Current Research Towards STT Relationship Analysis Modeling Spatial and Temporal data using SW standards (RDF(S)) 1 Upper-level ontology integrating thematic and spatial dimensions Use Temporal RDF 3 to encode temporal properties of relationships Demonstrate expressiveness with various query operators built upon thematic contexts Graph Pattern queries over spatial and temporal RDF data 2 Extended ORDBMS to store and query spatial and temporal RDF User-defined functions for graph pattern queries involving spatial variables and spatial and temporal predicates Implementation of temporal RDFS inferencing 1. Matthew Perry, Farshad Hakimpour, Amit Sheth. "Analyzing Theme, Space and Time: An Ontology-based Approach", Fourteenth International Symposium on Advances in Geographic Information Systems (ACM-GIS '06), Arlington, VA, November 10-11, Matthew Perry, Amit Sheth, Farshad Hakimpour, Prateek Jain. Supporting Complex Thematic, Spatial and Temporal Queries over Semantic Web Data", Second International Conference on Geospatial Semantics (GeoS 07), Mexico City, MX, November 29 30, Claudio Gutiérrez, Carlos A. Hurtado, Alejandro A. Vaisman. Temporal RDF, ESWC 2005:

56 Upper-level Ontology modeling Theme and Space Occurrent Continuant Spatial_Occurrent Named_Place Dynamic_Entity occurred_at located_at Spatial_Region rdfs:subclassof property Occurrent: Events happen and then don t exist Continuant: Named_Place: Spatial_Occurrent: Spatial_Region: occurred_at: Concrete Those Links Records and entities Spatial_Occurents Events Abstract exact with with concrete spatial static Entities location spatial to their spatial persist behavior locations (geometry geographic over (e.g. time (e.g. objects, locations building) a speech) Dynamic_Entity: located_at: Links Those Named_Places entities with dynamic to their coordinate geographic spatial behavior system locations (e.g. info) person)

57 Continuant Occurrent Upper-level Ontology Dynamic_Entity Named_Place located_at occurred_at Spatial_Occurrent Spatial_Region Person City trains_at Speech Soldier Politician assigned_to Military_Unit gives participates_in Military_Event Vehicle on_crew_of Domain Ontology used_in Battle Bombing rdfs:subclassof used for integration rdfs:subclassof relationship type

58 Temporal RDF Graph: Platoon Membership E1:Soldier assigned_to [1, 10] E2:Platoon assigned_to [5, 15] E4:Soldier assigned_to [11, 20] Time interval represents valid time of the relationship E3:Platoon assigned_to [5, 15] E5:Soldier E1 is assigned to E2 from time 1 to 10 and then assigned to E3 from time 11 to 20 Also need to handle inferencing: (x rdf:type Grad_Student):[2004, 2006] AND (x rdf:type Undergrad_Student):[2000, 2004] (x rdf:type Student):[2000, 2006]

59 ORDBMS Implementation: DB Structures Unlike thematic relationships which are explicitly stated in the RDF graph, many spatial and temporal relationships (e.g., distance) are implicit and require additional computation

60 Sample STT Query Scenario (Biochemical Threat Detection): Analysts must examine soldiers symptoms to detect possible biochemical attack Query specifies (1) a relationship between a soldier, a chemical agent and a battle location (graph pattern 1) (2) a relationship between members of an enemy organization and their known locations (graph pattern 2) (3) a spatial filtering condition based on the proximity of the soldier and the enemy group in this context (spatial Constraint)

61 The Machine Factor Formal representation of knowledge RDF(S), OWL, etc. Statistical analysis Similarity Cooccurrence Clustering Intelligent aggregation of knowledge Collaboration/Problem Solving Environments Decision support tools

62 Putting the man back in Semantics The Semantic Web focuses on artificial agents Web 2.0 is made of people (Ross Mayfield) Web 2.0 is about systems that harness collective intelligence. (Tim O Reilly) The relationship web combines the skills of humans and machines

63 Putting the man back in Semantics The Web relationship The 2.0 Web Semantic is about 2.0 web systems is combines made Web that focuses of the people harness skills on of collective artificial (Ross humans Mayfield) intelligence. and agents machines (Tim O Reilly)

64 Going places Formal Powerful Implicit Social, Informal