Semantic Transformation of Large Enterprise Data

Transcription

1 Semantic Transformation of Large Enterprise Data Team (No. 25) Aditya Ramamurthy Rao Amol Anand Tiwari Pandurang Mohan Kamble

2 Enterprise Information Management No In-links Out-links; No ranking paradigm Roles for access Information Vs. Data Search becomes a text based search Cost of information retrieval huge Losses due to lack of information significant

3 Devil Lies in the detail.. Ontology to the rescue. How Cloud computing fits in. Our implementation. Road-blocks and workarounds.

4 What is Ontology? To define briefly... Formal, explicit description of concepts in a domain To define formally... Controlled vocabulary expressed in an ontology representation language. This language has a grammar for using vocabulary terms to express something meaningful within a specified domain of interest.

5 Why Ontology? Ontology Attaches semantics to data. Specifies the relationships between data/concepts. Each enterprise data and thus ontology is different Thus, Machines can interpret meaning of data, helping to build semantically-aware Information systems Data can be Integrated on Web-scale, enabling Semantic Web. Reasoning of data can be automated and complex inferences can be made.

6 Enabling Semantic Web

7 How to build Ontology? Developing an ontology includes: Identifying the domain of Interest Defining the classes that would represent the concepts in our domain. Arranging these classes in a taxonomic hierarchy Defining properties and describing allowed values for these properties Filling in the values for properties for instances of classes.

8 Stanford NER Provides a general implementation of linear chain Conditional Random Field (CRF) sequence models Includes a 3 class (PERSON, ORGANIZATION, LOCATION) named entity recognizer for English by default, trained on the CoNLL 2003 English training data. Ex: <ORGANIZATION>ACET</ORGANIZATION> is a national and international AIDS charity with the following objectives: <ORGANIZATION>AIDS Care Education and Training</ORGANIZATION> <LOCATION>London</LOCATION> N1 2LP.

9 Representing Ontology...

10 Key Problems to Address Automate the generation of Ontologies for Enterprise Information Artifacts that are ever changing. Enable handling of large-scale enterprise data and generate extremely large Ontologies. Enable standards to semantically annotate Enterprise Information Artificats and generate Ontologies with rich interconnectivity amongst the Enterprise entities.

11 Block diagram of the API XSD2OWL Annotated XSDs Pre- Process XSL Transfor mation XSL Style Sheet Post- Process OWL Schemas

12 Mapping XSLT XSD to OWL XSD to OWL OWL Schema EBO XSDs XML to OWL EBO XML XML to OWL OWL Instances

13 Map-Reduce for creating an Ontology Map Function for NER String xml = classifier.classifywithinlinexml(filecontents); String filename = ((FileSplit)context.getInputSplit()).getPath().getName();.. Pattern pattern = Pattern.compile(context.getConfiguration().get("mapred.mapp er.regex")); Matcher matcher = pattern.matcher(xml); while (matcher.find()) { context.write(new Text(matcher.group()), new Text(filename)); } Reduce Function for NER HashMap<String, Integer> index = new HashMap<String, Integer>(); String value = <Occurrence>"; while (val.hasnext()) { Text word = (Text)val.next(); Integer occurence = (Integer)index.get(word.toString()); Integer count = new Integer(0); if (occurence ==null){ count = 1; } else{ count = occurence; count = count +1; } index.put(word.tostring(),count); } Set s = index.keyset(); val = s.iterator(); while(val.hasnext()){ String key_1 = (String)val.next(); value = value + key_1 +":"+index.get(key_1)+", ; } value = value.substring(0,value.length()-1) + </Occurence>"; context.write(key, new Text(value));

14 Map for Ontology transformation String keystring = ((FileSplit)context.getInputSplit()).getPath().getName(); String inputfiletype = keystring.substring(keystring.lastindexof(".")+1, keystring.length()); if(inputfiletype.equalsignorecase("xml")){ String xmlstring = Text.decode(value.getBytes()); Document doc = CommonXMLAPI.getDocumentFromString(xmlString.trim()); XML2OWL xml2owl = new XML2OWL(Constants.XML2OWL_MAPPING_EXCEL, Constants.OWL_OUTPUT_DIRECTORY); String outputowlfilepath = xml2owl.generateowl(doc); key= new Text(outputOWLFilePath); } context.write(key, new Text("")); }

15 XSD to OWL (Sample output)

16 XSD to OWL (Sample output)

17 XML to OWL (Sample output)