Geospatial Commons Workgroup Meeting with ESRI Thursday April 29, 2010 9:30 11:30 AM Centennial Office Building, Nokomis Conference Room. Workgroup Members in attendance: Chris Cialek, Mark Kotz, Susan Maeder, Chris Pouliot, Nancy Rader, Lesley Kadish, Jim Dickerson, Jessica Deegan, Jessica Fendos, Josh Gumm, Paul Weinberger, John Harrison Guests: Dan Falbo (ESRI), Clive Reece (ESRI) Contents 1. Overview.... 2 2. Discussion on Geoportal Exension related to prioritized needs.... 2 Search... 2 Google Metadata Search... 3 Viewable Catalog... 3 Registry... 4 Metadata and Documentation Viewer.... 4 Map Viewer... 4 Link to Content Provider... 4 Connect User to Web Service... 4 Download Data and Other Resources... 5 Registry Maintenance... 5 Harvesting Tools... 5 Hardware and Network Support Needed... 5 User Profiles for Producers... 5 End User Profiles... 6 Registry of Web Service Users... 6 Metrics of Usage... 6 3. Discussion about Version 10, what s coming.... 6 4. Other questions about Geoportal Extension.... 7
1. Overview. Mark K. described the history of the group, and how we worked with the drive to excellence to get started. Funding didn t materialize at this time Chris C., Develop application to provide single view of apps, services, and data for all state agencies. Further functionality would have to get sorted though to that end. Kotz. Functions prioritized after GIS/LIS conference last year (as group restarted). We reviewed site and other models. We determined going through. Recap concerns from Mn/DOT s implementation: JAVA development was a main issue Documentation lacking during that time of Toolkit 2. Discussion on Geoportal Exension related to prioritized needs. Clive Reece is directly linked to the geoportal team at ESRI. Noted Toolkit (ArcIMS generation) was a soft product, only implemented with training and services. The product has since matured with ArcGIS Server, and is part of core, licensed through ELA. With that, there is support through ESRI, documentation, training, etc. Bottom line, Clive feels we could get 95% of functionality we need out of the box, and for the highest priority functions, all of them. Clive demonstrated several features as connected with our prioritized functions list using the public sandbox of the extension: http://gptogc.esri.com/gpt931/catalog/main/home.page Search Product has keyword search. Spatial search uses where clause with map: anywhere, intersecting, and fully within. Can do both spatial/keyword. Also, additional options: can choose content type, data categories (by ISO), modified date, etc. Keyword search offers and and or Boolean search options. Could choose to search within site, in title, etc. Can configure to have certain words enhanced in search. Mark K asked if we can customize what searches can be made more available on landing page and not in an additional options popup window. Clive: Search engine is Lucene, so options for searching are very in-depth. Nancy R asked about how to do search within title only. Clive - info is hidden but is in documentation. How to use Lucene search text queries. For title this would be title:bla. Clive noted that this could be accomplished through advanced configuration files to bring search like this into standardized interface. Chris C.: Metadata need to be in the right place and indexed correctly. What do we need to know up front in preparing metadata (ours and customers) so that it is easily ported into the system. Clive: Determine who has publishing rights, do outreach, decide on what standard we ll use. Allow many standards or just one? Services may need quick easy form to allow people to write out what the service is. If WMS or ArcGIS Service, why write metadata? Instead just scrape service level documentation (get capabilities doc or REST service endpoint) to populate Dublin core record. There are online validation tools available in administration tab of geoportal extension. Can validate against whatever standard we choose. Chris C: In ArcCatalog we could create metadata fomats that are spin offs of the national standard (MGMG), is that customization available?
Clive: This is available through the web form. Documentation exist for how to go through that. Note that the current MGMG editor is in jeopardy because it is in VB. ArcCatalog editor coming in version 10. Chris C: Will we need to run wholesale conversion to bring into our standard? Clive: Use existing as a starting point (like FGDC) and modify for MGMG. Showing the filed. In bold are required. Can include conditional mandatory elements e.g. if put in point of contact, then several fields mandatory. Through configuration of XML template this is extensible. Chris C: What formats rising to the top? What are other folks doing? Clive: For services, Dublin Core. FGDC has lots more mandatory elements. Jessica F: Asks about clip and ship and data download. Saw availability on Oregon site. Clive: Available in 10 as a sample. Know your users, do they want custom selected areas, vs. FTP bundled files. Harrison: Is the clip and ship still used within templates? Output level support. Clive: Download requires geoprocessing task with clip, zip, and ship functionality. The server has direct access to that data. If no interoperability, server can do shapefile, GML, or geodatabase. Chris P. Cost of interoperability extension? Where would it need to be installed? Dan F. Would be required to be on the host ArcGIS Server, for development purposes also on desktop of local. Cost would be $2500 desktop, $10K server (need to confirm). Google Metadata Search Clive: There is a site map generation tool that can be used to register content to google for crawling. Metadata records are generating unique urls for each record in the database. Nancy R: Useful since this is what Geogateway can t do today. Clive: Can enable the share link function, users can link back to direct metadata, offers social networking ability (via Add this?). Viewable Catalog Clive: Provided example of Oregon Spatial data library, which uses current version. They enabled home page with category links, by embedding key in metadata. Another option in geoportal 10, (GEOS website example) is to enable ability to browse in catalog, by ISO topic categories via tree. Can add in additional topics. Jim D: Noted that the search results catalog interface had several supported results. RSS, ATOM, KMZ, See results through REST. Asked who uses those. Clive: Programmers use primarily. What s happening is the catalog has two service endpoints, OGC service compliant endpoint and a REST API endpoint. Use case, when someone wants to get notified when new catalog, let know when new content comes out. Second use case html and html fragment. Can imbed searches within sharepoint, or other content management system. Third use case, KML and JSON for web programmers. Jim D: What language is the geoportal extension written in? Clive: Java, runs within servlet. Jim D: If custom search page, could interfaces support a custom interface? Clive: Yes, documentation outlines many approaches. Can use different content management system.
Registry Susanne M: MnGeo, MetroGIS, DNR have all struggled to get data into GOS. How would we now manage portal and contribute to another portal? Could new portal harvest the metadata as currently stands? Clive: Is the issue a problem of publishing twice? Susanne M: Yes, would format we currently have allow us for same content to be pushed to a portal? Clive: We will need to think about where will GOS harvest from? Maybe harvested from this catalog? Probably an agency, policy standard, decision to be made. Do agencies want to host their own catalogs that feed into the commons and GOS? Mark K.: Something we ll need to deal with and work through. Chris C: Advantage either way? Clive: No Metadata and Documentation Viewer. Clive: Using onlink tag in metadata, can support looking at metadata through stylized sheet, link to website for further documentation and also link to data. Mark K: Noted that in MGMG just one online link possible. Clive: FGDC offers two. Mark K: In viewer of metadata, what about html files that are created from xml file, possible they are slightly different? Clive: Can link to the htm version from onlink, too. Map Viewer Clive: Interactive map exists in basic form. In the past, map viewer was bundled. But the portal as now offers a very lightweight map viewer based on javascript api with a previewer. Deegan: Can javascript viewer be modified to make a heavier client? Clive: Yes. Can also associate site with a little heavier client. Also noted that the product bundles in a search widget that can be used in all API viewers (Javascript, Flex, Silverlight) Chris C: Challenge for us is to articulate our purpose for an interactive map. Clive: Also note that more functionality requires more training or outreach to users. Mark K: Need to clearly define our needs. Clive: GEOS site does have a viewer. Link to Content Provider Clive: What do you mean by this functionality requirement? Is the requirement met with information within the metadata? Mark K: Requirement was a way to get back to content provider. Connect User to Web Service Clive: Noted that drag-and-drop not possibility unless using Windows and Silverlight widgets. Can provide link to REST service, which can in turn open in Google Earth, ArcGIS Desktop clients.
Download Data and Other Resources Clive: Download could be opening a zip file through a link or could be dynamic zip, clip and ship. Nancy R: Clip zip on the fly, also pulls in metadata record? Clive: Yes, pulls in metadata in the data source. Nancy R: So if shapefile has a separate XML, how does it know? Clive: Runs a python script that uses geoprocessing objects. Mark K: We use shapefile.xml rather than shapefile.shp.xml for our metadata. Clive: May need to edit the python script to look for that. Chris P: Clip zip, ship, is this just available for vector data? What might we need to do to make raster available? Clive: Can extend to raster, but can be quite large. Can use rasters, but a lot of processing power is required. Recommends tiling the rasters and making that available. Registry Maintenance Clive: Logged in with administrative rights to the geoportal, you can look at entire catalog, manage metadata (which can be filtered by many parameters). Can transfer ownership. If needed to support restrictive metadata, you could do that too. Kotz: Can administrators be provided with specific records to administer? Clive: Currently all administrators own everything. Coming in 10, can have specific rights, for groups, such as Scott County only having rights for Scott County owned metadata records. Chris C: Confirming, we are talking single location of metadata, just different ownership roles. Clive: Yes. Clive: Requested clarification of our need for ticklers when stale information will be removed? Mark K: If a record hasn t been touched in a certain amount of time, can we know about that? We could set up an RSS feed to get records older than a certain date? No email push yet. Harvesting Tools Clive: To get information in, if people aren t managing metadata themselves, can use form. Can upload files one at a time through upload function. Rest service through network path. Crawl a web accessible folder (WAF), other procols are CSW catalog, ESRI metadata service, Z39.50 still used, OAI (never used). It s a manual push that could be automated further. Nancy R: Confirming that metadata will be overwritten on update. Clive: Yes. Jim D: Does push use port 80? Do admin functions use web admin? Clive: Yes on port 80, and that it is web form admin. Hardware and Network Support Needed Clive: Asked for clarification on this functionality - What do you need to support? Mark K: Not part of product, just in there to note need for resources to support. User Profiles for Producers Mark K: uploading will not happen anonymously. Profile required, currently only email and last name are required.
End User Profiles Clive requested clarification of quality ratings for user profiles. Jim D: Stars Clive: Ratings are coming in 10, build by end of next week will have a comment link. Anonymous users can view comments, registered users can add comments. In comment screen, add in description, thumbs up/down, plus email/name. Kotz: Does administrator have ability to censor? Clive: Can t blacklist people, but can delete and contact the comment maker. Comments will be searchable as well. Registry of Web Service Users Kotz: This functionality is for the purpose of tracking, say I m using a service and want to know how reliable it is. How can we let folks know if services are changing? Clive: Not available out of the box. Issue is that once someone gets access to a service, then there is no more contact. Metrics of Usage possibly coming in 10 3. Discussion about Version 10, what s coming. John H: When is 10 coming? - July Jim D. Should we wait for 10 to get started? Clive: No, don t wait. There will be new backend database schema for web tables. Will need to have both running simultaneously then migrate. At this timeline, not much customization will have happened. Jim D: What do you mean by migrate? Clive: Will need to harvest from one to the other through CSW. Note that we can also do a single directed search of another catalog. In 10, there will be ability to search multiple catalogs at once. John H: If going to 10, then is 9.3.1 SDE deprecated? Clive: Metadata service is built for MnDOT with SDE, can be harvested still. Can create stand alone schema in their database. SDE can be freed up to manage spatial data alone. Clive described new features (from presentation to petroleum user group conference). o Unify structured (metadata) with unstructured (tagged: eg. Dublin core). OpenSearch is a new search standard. Enable search of Sharepoint, YouTube, Flickr. Additional search adapters out of the box (CSW, Z39.50, WFS, geodata services, image server image catalogs, arcgis server search services). o Federated search o Register top level ArcGIS Services, sync with the services there. o Harvesting, server side. o ID management allow people to log in using a web identity o Silverlight widget for search available. Jessica D: Can the widget be used by others building apps? Clive: Yes. o Desktop extensions for ArcCatalog o Statistics of what s in the catalog, create word clouds
o Twitter as data source. o Expand integration to ArcGIS Online. o Platform Support. Works with most platforms (including MnGeos) Only constraint is that this is a java app. 4. Other questions about Geoportal Extension. Jim D: Directory Server, can you do without it? Clive: Not necessarily. But you do need an LDAP connection. Jim D: Can you use ArcGIS server? Clive: Only if configured with LDAP. Jim D: Does upen ID support that at all? Will there be a problem if all of this is on one box, if two directory servers exist? Clive: Shouldn t be an issue. Depends on your box. Better to be separated, just making service calls to LDAP or active directory. Susan M: Can we get stats on uptime, service checking, etc? Clive: Directs people to FDGC service checker. Geodata.gov uses the service checker. Onlink field gets registered in the service checker. Jim D: Does geoportal know additional published geoprocessing service exists? Clive: Register that service endpoint with the catalog. Then the catalog redirects back to the service. Loosely coupled. Exposed like any other metadata record. Chris C: High level question. Subject of geospatial portals, confused by use of portals. Expects there would be a lot more examples out there (a lot of interest in the product). Is this a growing component of the product line? Clive: Two components: corporate support by putting in core product line is a commitment. Second. SDI s in general. They go through waves of popularity. Has seen a lot of interest in catalogs. In Europe, the INSPIRE process going on. EU-wide set of policies and services. Agency level. EPA, BLM using geoportal. A lot of growth in the defense area, including Corps of Engineers. Meeting adjourned, 11:35 AM Note taker: Jessica Deegan, Met Council