An Intelligent Data Broker for Virtual Integration of Heterogeneous Distributed Meteorological Databases

Transcription

1 An Intelligent Data Broker for Virtual Integration of Heterogeneous Distributed Meteorological Databases May 10, 2005 Yamakawa A., T. Kiura & S. Ninomiya National Agricultural Research Center, National Agriculture and Bio-oriented Research rganization

2 Potential for Data Sharing Between DSS for Agriculture Decisions (Clients) Weather Data Data Needed Crop details Soils Topography Variety selection To dam? ptimal land use Spray for disease Irrigate or not? Manage flooding

3 Agricultural applications and databases Agricultural applications need several different types of data sets Databases needed for the applications are usually maintained and managed by different organizations and are located in different places

4 Concept of Agricultural Grid Neural Network Farm Management Regression Model Growth Model1 Meta Database Weather Data 1 The Internet Data Broker Field Data Monitoring Weather Data 2 User who needs Decision Growth Model2

5 Advantages of Grid More powerful DSS by dynamic and flexible integration of distributed resources No duplication of databases and programs Easy updates of databases and programs Highly efficient utilization of resources Reduction of development and maintenance cost

6 Challenges to Realize Grid Database heterogeneity Those databases are heterogeneous in access method, data formats, available items, time resolution etc. Internationalization Security QoS Efficient field data collection

7 Example of Heterogeneity of Meteorological Database Item Air Temperature, Wind, Rain, Radiation, etc. Sub Item Rain Amount, Rain Intensity, Rainy Days, etc. Time Resolution Sub Hourly, Hourly, Daily, Monthly, etc. Summarization Mean, Maximum, Minimum, Total, etc. Equipments

8 Goal of ur Project To provide an environment where heterogeneous distributed databases can be flexibly, seamlessly and virtually integrated for Agricultural Grid

9 ur Initial Challenge Some Data Brokers Meteorological DBs MetBroker 15DB, >13000 stations Map DBs ChizuBroker 3DB Japan NZ World Digital Elevation DBS DEMBroker 2DB Japan 50m, World 1Km Soil DBs SoilBroker

10 Coverage of MetBroker Japan MAFF's database of AMeDAS data hourly Japan Wakayam a Rainfall data hourly USA regon Integrated Plant Protection Center (NorthWest) daily USA Long Term Ecological Research Network (ClimDB) daily USA Georgia Automated Environmental Monitoring Network daily NAA/WM Globa; Surface Summary of Day daily Norway Planteforsk Crop Research Institute hourly New Zealand HortPlus Ltd hourly USA Florida Automated W eather Network min Japan Kanagawa Prefecture Agricultural Research Station Network hourly South Africa South African Sugar Association network daily Korea Seoul National University Plant Disease and Epidem iology Lab hourly Japan Hokkaido Memuro MAMEDAS hourly Japan Chiba and Tottori Prefectural Research Stations hourly Japan FieldServer project min password req. New Zealand, Pacific Antarctica National Climate Database hourly United Kingdom Horticulture Research International daily Taiwan Taiwan Ecological Research Network daily USA W ashington State University Public Agricultural Weather System min

11 Present Coverage

12 Database Broker Service Data Summarization Client Data Secondary Processing Data Request Standa rdized Data Data Brokage Whereabout Meta Database Where, How to use Data contents Database Driver (Wrapper Applications) Data Standardization Data request translated to each DB A-DB B-DB Data acquisition C-DB D-DB

13 Existing Solutions Rice growth forecast service Amedas Soybean blooming time forecast service Web Web Service XML XML Ag experiment Station Weed damage forecast service Wrapper Application Prefecture s data Web service can hide heterogeneity of access method by using XML technology

14 Limitations of Conventional Approach Daily Mean Air Temperature Daily Max Air Temperature Daily Min Air Temperature Temp_max Temp_min Temp_total Temp DB Needed to create a wrapper program which translates the meanings of the terms used in each database. Web service can t hide semantic heterogeneity Hard to add new observation items

15 Problem of Static Translation A Any Daily Air Temperature is acceptable DB 1 B Daily Mean Air Temperature Daily Max Air Temperature Daily Min Air Temperature nly Daily Air Temperature based on 1- minute data is acceptable Daily Air Temperature based on hourly data DB 2 Daily Air Temperature based on 1 minute data For client A, Air Temperatures of DB1 and DB2 are regarded as the same item, But for client B, they are not. It is very hard for conventional solution to meet such requests.

16 Solutions by ntology RDF/WL technology can solve the limitations which the conventional solution has ntology can realize seamlessly and virtually integrated database Intelligent Data Broker is a challenge to provide clients with access to this virtually integrated database

17 Intelligent Data Broker System verview System Implementation How Intelligent Broker works? Single station access Spatial access Access with data summarization Discussion

18 System verview Metadata database Decision-Making Support Services perational Products Simulation Models Detailed Digital Forecast Item Definition WL Station metadata RDF 2. Request 3. Request 1. Register metadata Meteorological databases Inference Engine DB Wrapper Broker 4. Request data DB Wrapper DB Wrapper

19 System Implementation A prototype version Coded in Java language Main part of the inference engine Jena2 ( Composed of one basic vocabulary file and as many item definition files and station metadata files as the number of targeted databases

20 Roles f the RDF/WL files Name File type Content Basic vocabulary WL All standard weather items Vocabulary to describe weather stations Item definition WL Local vocabulary that is used in each database Station metadata RDF Description about all the weather stations included in a particular database

21 How Intelligent Broker works? Single station access Clients request some weather items from a specified weather station Spatial access Clients request some weather items from the weather stations located in a specified geographical area Access with data summarization Clients allow Intelligent Data Broker to summarize data

22 Single station access Basic Vocabulary <owl:class rdf:id="dailymeanairtemperature /> 2. Find the basic vocabulary that is equivalent to the request Database : Item definition <met: DailyMeanAirTemperature rdf:id= Air /> 3. Find the local item equivalent to DailyMeanAirTemperature 1. Request Resolution Daily Summary Mean Item Air Temperature Inference Engine Broker 4. Pass the local item Air DB 5. Query the database using the local item Air

23 Spatial Access Client Broker Temperature Rainfall Radiation

24 Spatial Access Ibaraki Prefec. Complements of sparse observations Field Servers Tsukuba City

25 Spatial access Basic Vocabulary <owl:class rdf:id="dailymeanairtemperature /> Item definition Item definition Item definition Item definition 3. Find the local items equivalent to DailyMeanAirTemperature for each database that includes the stations in the requested area Station metadata Station metadata Station s Station nametadata Station s Station s Station location nametadata Measurement Station s Station s location name items etc. Measurement Station s Station s location name items etc. Measurement Station s location items etc. Measurement items etc. 1. Request NW latitude 35 longitude 134 SE latitude 25 longitude 154 Item Daily Mean Air Temperature Inference Engine Broker 2. Find the stations located in the requested geographical area 4. Pass the local items 5. Query the databases using the local items DB DB DB DB

26 Access with summarization Basic Vocabulary <owl:class rdf:id="dailymeanairtemperature /> <owl:class rdf:id= HourlySampleAirTemperature /> 2. Find the basic item that has the same or shorter time resolution as/than the request Database : Item definition <met:hourlysampleairtemperature rdf:id="ame_time.temperature"> </met:hourlysampleairtemperature> 3. Find the local item that is compatible with the basic item Inference Engine 4. Pass the local item 1. Request Resolution Daily Summary Mean Item Air Temperature Summarization allowed Broker DB 5. Query the database using the local item 6. Summarize the obtained data to requested time resolution

27 Sample of Basic Vocabulary <owl:class rdf:id="dailymaxairtemperature"> <rdfs:subclassf rdf:resource="#maxairtemperature"/> <rdfs:subclassf> <owl:restriction> <owl:allvaluesfrom> <owl:class rdf:about="#dailymaximum"/> </owl:allvaluesfrom> <owl:onproperty> <owl:bjectproperty rdf:about="#summarykind"/> </owl:onproperty> </owl:restriction> </rdfs:subclassf> </owl:class> <owl:class rdf:about="#dailymaximum"> <rdfs:subclassf rdf:resource="#maximum"/> <rdfs:subclassf> <owl:restriction> <owl:allvaluesfrom rdf:resource="#daily"/> <owl:onproperty> <owl:bjectproperty rdf:about="#duration"/> </owl:onproperty> </owl:restriction> </rdfs:subclassf> </owl:class> DailyMaxAirTemperature is a subclass of MaxAirTemperature DailyMaxAirTemperature is recognized as maximum and daily data Sample file:

28 Available Basic Items(1) Item AirTemperature AtmosphericPressure Cloud Evaporation GasConcentration Humidity LeafWetness Pollutant Sub-Item SeaLevelPressure StationPressure CloudAmount CloudForm C2 AbsoluteHumidity DewPointTemperature RelativeHumidity SteamPressure Aerosol Dust Pollen Number of Sub-sub-Item

29 Available Basic Items(2) Item Radiation RainFall SnowFall SoilMoisture SoilTemperature Sunshine Sub-Item LongwaveRadiation SolarRadiation RainAmount RainDuration RainIntensity RainStatus RainyDays SnowAmount SnowDepth SnowDiff SnowType SnowyDays PossibleDuration Number of Sub-sub-Item

30 Available Basic Items(3) Item Sunshine Weather Wind Sub-Item SunshineDuration HorizontalWind VerticalWind Number of Sub-sub-Item

31 Sample of Item Definition Local item name ame_day.temp_max is recognized as maximum and daily data based on every 10 minutes data <met:dailymaxairtemperature rdf:id="ame_day.temp_max"> <met:summarykind rdf:resource= " </met:dailymaxairtemperature> <met:hourlysampleairtemperature rdf:id="ame_time.temperature"> <met:summarykind rdf:resource= " </met:hourlysampleairtemperature> A sample file is available on

32 Sample of Station Metadata <met:metstation rdf:id="01"> <rdfs:label xml:lang= en"> Ag. Res. Inst. Representative bservation Station </rdfs:label> <met:alt>64.0</met:alt> <met:log> </met:log> <met:lat> </met:lat> <met:belongto> <met:metcatalog> <met:metcatalog> <met:metelement>&kngw;#dailyaveragewindvelocity</met:metelement> <met:catalogstart> t15:00: </met:catalogstart> <met:measurementheight>2.0</met:measurementheight> <met:metcatalog> </met:metcatalog>. </met:metstation> This item is defined in Item definition file Description of weather station The details of measured items

33 Discussion We realized two advantages of the RDF/WL technology Secure scalability in building a large scale database by integrating heterogeneous databases Support to develop a system flexibly adjustable to various clients request

34 Future Plan Make this prototype more practical in agricultural decision making Bind various databases, e.g. crop growth databases and soil databases into our system in addition to meteorological databases

35 Thank you for your attention Sample file: