InstantAtlas TM Server NeSS Data Transfer Tool User Guide Author: GeoWise User Support Released: 31/10/2012 Version: 6.5.1
InstantAtlas Server NeSS Data Transfer Tool Table of Contents 1. Introduction... 1 2. Installation... 2 2.1. Prerequisites... 2 3. Overview... 3 3.1. 3.2. Data Type matching... 3 Metadata... 3 3.3. Error logs... 4 4. Workflow... 5 4.1. Choose Source Server... 5 4.2. 4.3. Choose Geographical Areas... 6 Choose Data Page... 8 4.4. 4.5. Choose IAS... 10 Define Geo-Relationships page... 12 4.5.1 Creating a new Geo-Type in IAS from selected NeSS features... 12 4.5.2 Matching features in NeSS to your IAS features... 14 4.6. Define Data Relationships page... 15 4.6.1 4.6.2 Creating a new Indicator in IAS from selected NeSS variables... 15 Link NeSS variables to existing IAS Indicators... 17 4.7. 4.8. Define Date/Time Relationships... 18 Data Synchronization Settings... 19 4.8.1 Standard... 20 4.8.2 Advanced... 21 4.9. Choose Task Schedule... 22 4.10. Completion screen... 23 5. Known Issues... 24 6. 7. Suggested Workflow... 25 Support... 26
InstantAtlas Server NeSS Data Transfer Tool Page 1 1. Introduction The InstantAtlas Server (IAS) NeSS Data Transfer Tool is a wizard based utility that allows the user to download and import indicator data from the Neighbourhood Statistics (NeSS) Data Exchange Service published by the Office of National Statistics (ONS). This tool can be used to import data straight away or can be scheduled to run at particular times using the IAS Data Synchronisation Service. The IAS Data Synchronisation Service is bundled with the IAS Data Transfer Tools installer and is documented in the IAS Data Transfer Tools User Guide. Please contact your support provider if you wish to receive a copy of this document.
InstantAtlas Server NeSS Data Transfer Tool Page 2 2. Installation IAS NeSS Data Transfer Tool can be installed on a local PC or on your IAS server. We recommend tasks to regularly import data are set up to run on your IAS web server rather than on a local PC to ensure that the server is running at the time the task is due to run (as client PC s may be turned off when the task is set to run). 2.1. Prerequisites The IAS NeSS Data Transfer Tool requires version 3.5 or higher of the Microsoft.NET Framework to be installed before you can install the program. Your IAS web server will already have this installed. If you do not have a suitable version of the.net Framework already installed you will be taken to the Microsoft Download Center website to download it. Once this is installed you will be able to install the IAS NeSS Data Transfer Tool. To install IAS NeSS Data Transfer Tool you must be a member of the Administrators group. You should use the version of the NeSS tool that matches the version of IAS that you are importing data in to. For example you should use version 6.5.0 of the NeSS tool against IAS version 6.5.0. From version 6.5.0 of the NeSS tool the welcome screen includes an About link which displays the version information. The version prior to this without the about link is compatible with versions 6.4.x of IAS.
InstantAtlas Server NeSS Data Transfer Tool Page 3 3. Overview NeSS uses data types and metadata that are different from those used by default in IAS. A number of rules have been set up to ensure that appropriate default values apply when data from NeSS is set up to be imported to IAS. 3.1. Data Type matching The table below (Table 1) shows how NeSS data types are mapped to IAS data types by default. Table 1 NeSS to IAS Data Type matching NeSS Data Type Classification Count Hours Kilometers Percentage Percentage Points Rank Rate Ratio Score IAS Data Type Numeric no aggregation Count Numeric no aggregation Numeric no aggregation Ratio Numeric no aggregation Numeric no aggregation Ratio Ratio Numeric no aggregation You should be aware that data types for individual Indicators can be changed during a later stage of the import wizard. 3.2. Metadata Metadata is imported in to IAS in to a number of the Dublin Core standard metadata elements (that are set up by default when installing the system) and a number of custom NeSS metadata elements. Table 2 shows which NeSS metadata elements are matched to the standard Dublin Core elements in IAS. If you have modified the Metadata Standard Elements setting in the Admin Website to remove/modify the Dublin Core elements then these will be recreated when importing data from NeSS as Custom elements. The Custom NeSS elements that are imported in to the system can be hidden from display on the User Website using the Metadata Hidden Elements setting in the Admin Website. The values stored in the Custom NeSS metadata elements should not normally be modified as some of them are used when automatically synchronising download tasks. Table 2 NeSS to IAS Metadata import details IAS metadata element NeSS metadata element Description dc:title ness:title dc:title is populated with the modified ness:title field which can be updated on the Define Data Relationships page. If the Title is not updated this field will be blank. dc:publisher Always populated with Neighbourhood Statistics and a hyper link to the NeSS data download web page. dc:type ness:coinage.unit dc:type is populated with the modified ness:coinage.unit field which can be updated on the Define Data Relationships page. If the Data Type is not updated this field will be blank.
InstantAtlas Server NeSS Data Transfer Tool Page 4 dc:creator ness:creator dc:description ness:description dc:description is populated with the modified ness:description field which can be updated on the Define Data Relationships page. If the Description is not updated this field will be blank. dc:subject ness:subject.category dc:coverage.temporal ness:sourcedatadaterange NeSS start and end dates in the format yyyy-mm-dd yyyy-mm-dd, e.g. 2005-01-01 2005-12-31 ness:title ness:title Original NeSS title. ness:description ness:description Original NeSS description. ness:date ness:date ness:identifier ness:identifier ness:coinage.unit ness:coinage.unit Original NeSS Data Type. ias:ness.downloadurl ias:ness.downloadurl URL used to download the data from NeSS. ias:ness.downloaddate ias:ness.downloaddate Date and time the dataset was downloaded from NeSS in the format yyyy-mm-dd hh:mm. The value in this element will be updated when the dataset is updated/re-downloaded at a later date depending what settings you have in Metadata Action and Metadata Level. ias:ness.sourcedatadat erange ness:sourcedatadaterange NeSS start and end dates in the format yyyy-mm-dd yyyy-mm-dd, e.g. 2005-01-01 2005-12-31. This field is required when scheduling synchronisation tasks and should not be edited. Note only NeSS Topic metadata is imported in to IAS and not NeSS Dataset metadata. On investigation it was found that NeSS Dataset metadata was of limited use in IAS as it consists of duplicate metadata elements. If required, NeSS Dataset metadata can be viewed in the downloaded XML stored in the Transfer Folder as specified on the Choose IAS page of the NeSS wizard. 3.3. Error logs Errors are logged to the following locations: OS Windows Server 2003 Windows Server 2008 Windows XP Windows Vista Windows 7 Location C:\Documents and Settings\All Users\Application Data\GeoWise\InstantAtlas\Server\Logs\ C:\ProgramData\GeoWise\InstantAtlas\Server\Logs\ C:\Documents and Settings\All Users\Application Data\GeoWise\InstantAtlas\Server\Logs\ C:\ProgramData\GeoWise\InstantAtlas\Server\Logs\ C:\ProgramData\GeoWise\InstantAtlas\Server\Logs\ Log files are named using the following format: ness2ias-wizard-<yyyy-mm-dd>.log, e.g. ness2ias-wizard-2010-11-16.log
InstantAtlas Server NeSS Data Transfer Tool Page 5 4. Workflow After successfully installing the software, double-click the desktop icons to launch the NeSS Data Transfer Tool. The wizard will lead you through a number of steps described below: 4.1. Choose Source Server Figure 1 Choose Source Server The Choose Source Server page (Figure 1) has the following options: URL Enter details to connect to NeSS Data Exchange service. This field is pre-populated with the current details, but can be changed if required. Web Proxy o Auto uses your current Internet Explorer proxy settings o No proxy attempts to bypass your Internet Explorer proxy settings
InstantAtlas Server NeSS Data Transfer Tool Page 6 4.2. Choose Geographical Areas Figure 2 Choose Geographical Areas The Choose Geographical Areas page (Figure 2) allows the end user to select the geographic features for which to import data. Note you can only select geographic features from one level in NeSS e.g. Ward level, but you can select multiple features from different groupings in the hierarchy (e.g. you can select a few Wards from Local Authority 1 and a few Wards from Local Authority 2). The Recent option is only present once you have run through the wizard for the first time. This is then populated with the individual geographic feature selections from the tasks that have previously been defined. The recent option makes it very easy to reuse previous geographic feature selections without having to browse the whole NeSS hierarchy. There is also a hyperlink included on this page which links to a page on the www.statistics.gov.uk website which explains in greater details the geographical hierarchies used in NeSS. To select all child features from within a parent feature, e.g. all LSOA s from within an LA boundary, you should select the parent feature then right click on the selected feature and choose Bulk Select Areas from the popup menu, Figure 3.
InstantAtlas Server NeSS Data Transfer Tool Page 7 Figure 3 Bulk selecting features from within a higher level geographical area From the popup menu that appears select the relevant child geographical areas from the drop down and press the find button, Figure 4.
InstantAtlas Server NeSS Data Transfer Tool Page 8 Figure 4 Selecting child features from within a parent Having found the child features from within the selected parent geographical area you should press the Save to recent selections button to save all the displayed child features in to your recent list. 4.3. Choose Data Page The Choose Data page allows the user to browse and select the variables available in NeSS for the geographical areas selected on the Choose Geographical Areas page (section 4.2) for example if you selected Local Authority level features on the Choose Geographical Areas page this page will only show you data that is available at the Local Authority level.
InstantAtlas Server NeSS Data Transfer Tool Page 9 Figure 5 Choose Data The Updated between from and to dates can be modified to narrow down your search to only find data that were modified between your selected dates. Note if you modify the Updated Between start or end dates you must ensure you refresh the NeSS Subject Tree to ensure that you are seeing the data that is available in NeSS for your modified dates. Selecting the Recent Data checkbox overrides the Updated To date field with the current date when the task is run. This is useful when scheduling tasks to ensure that the most recent Instance is always imported. You can select an individual Variable in NeSS then right click on it (Figure 6) to bring up the Available Dates context menu. Selecting this option opens up a window showing all available dates for the selected variable (Figure 7). Figure 6 NeSS Variable context menu
InstantAtlas Server NeSS Data Transfer Tool Page 10 Figure 7 NeSS variable available dates window There are no restrictions on the number of Indicators that can be imported, or the geographic levels at which the data is imported. Users should however be made aware that NeSS does impose a limit of 10,000 data values in a single extraction request for performance reasons. Users of the NeSS Data Transfer Tools can set up multiple data transfer tasks to ensure that no single task exceeds the limit imposed by NeSS. 4.4. Choose IAS On the Choose IAS page (Figure 8) you need to specify the credentials to connect to your destination IAS server.
InstantAtlas Server NeSS Data Transfer Tool Page 11 Figure 8 Choose IAS URL enter the URL of your IAS Web Services, this should be in the following format: http://<servername>/ias_webservices/administrationservice.svc User Enter the username of an IAS Administrator see Table 3 for a summary of the different IAS User Groups and the level of access they have when using the NeSS wizard. If you do not enter a username and password the wizard will attempt to connect using your current Windows credentials. Note if you re scheduling a task to run later (see section 4.9) you must ensure you enter valid username/password credentials rather than leaving the fields blank. Password Enter the password for the above user. Web Proxy o Auto uses your current Internet Explorer proxy settings o No proxy attempts to bypass your Internet Explorer proxy settings Locale this setting controls the Geo-Type feature names that are displayed on the Define Geo-Type Relationships page, the Theme and Indicator names that are displayed on the Define Data Relationships page, and the locale in to which the Indicator names are written when importing a new Indicator from NeSS. Note it is important that you import the data in to an existing Geo-Type that has feature names in the default (en-gb) locale as the Admin Website only runs in the English locale. If you create a new Geo-Type in a non-default locale you will not be able to see it in the Admin Website.
InstantAtlas Server NeSS Data Transfer Tool Page 12 Transfer Folder importing data from NeSS is a two step process. First the data is downloaded from NeSS in to a temporary folder then the data is imported from the temporary folder in to IAS. This Transfer Folder setting specifies the location of this temporary folder. Table 3 User group permission IAS User Group IAS_Administrators IAS_DataManagers IAS_DataSuppliers IAS_DataUsers All other IAS_XXXX groups Permission Full Full None None None Note members of the above groups must also be members of the IAS_Users group to use the NeSS Data Transfer Tool. Note from version 6.5.0 onwards members of the above groups must also be members of one of the ownership groups e.g. IAS_OWNER_ALL. It is strongly recommended that users are members of the IAS_OWNER_ALL group as if the user is a member of a different ownership group you will be able to see all Indicators in the system but if you try and update any that you don't have permission to edit you will get an error at the end of the wizard when the tool tries to import the data. 4.5. Define Geo-Relationships page The Define Geo-Relationships page allows the user to establish relationships between the geographical features selected in NeSS and the Geo-Type features in your IAS system. The wizard allows you to match NeSS features to features in an existing Geo-Type in IAS or create a new Geo-Type in IAS from the features selected in NeSS. 4.5.1 Creating a new Geo-Type in IAS from selected NeSS features To create a new Geo-Type in your destination IAS system select and drag the NeSS area group (those with a folder icon) from the source server in the left hand list and drop it on the server node on the right hand list. When creating a new Geo-Type in your destination IAS system you can change any of the following settings with those of your own (Figure 9), they are populated with default values: Name Unique Code Type (e.g. Point, Line, Polygon)
InstantAtlas Server NeSS Data Transfer Tool Page 13 Figure 9 New Geo-Type Details When creating a new Geo-Type this is shown in the destination IAS panel as a globe icon with a yellow dot, (Figure 10) where the wizard will create a new Geo-Type called Local Authority (NeSS). Figure 10 Creating a new Geo-Type in your IAS system
InstantAtlas Server NeSS Data Transfer Tool Page 14 4.5.2 Matching features in NeSS to your IAS features Where you have existing Geo-Types in your IAS system with features which match those selected in NeSS you should define the relationships between the features in the two systems to ensure data consistency and to avoid the creation of duplicates. To manually match features select and drag each NeSS area from the Source (left hand) list and drop it on to the corresponding IAS Geo-Type feature in the Destination (right hand) list to establish a relationship. Once a relationship has been established between two features the icons of the source and destination features will change, and when either feature is hovered an arrow indicating the relationship is drawn (Figure 11). Figure 11 Manually defining a relationship between geographic features Manual matching is done at the Geo-Feature level, not the Geo-Type level. You are required to create a relationship between each NeSS feature and the corresponding feature in IAS. However you only need to create relationships for the features you wish to import data for so you may have more features in your NeSS selection than you do in your IAS Geo-Type, or vice versa. The functions of the Auto, Clear and Clear All buttons are described below: Auto this button creates relationships based on Geo-Feature Unique ID s. So if you have more than one Geo-Type with the same Geo-Feature ID s it will match to the first ones it finds. Subsets are not displayed in the destination (IAS) panel. If you wish to only import data for a particular Geo-Type subset you must manually define the relationships between the NeSS and IAS geographic features. Clear this button clears the currently selected relationship. Clear All this button clears all relationships.
InstantAtlas Server NeSS Data Transfer Tool Page 15 Note you can only import of data from NeSS in to one IAS Geo-Type at once. So if you define relationships between separate features in NeSS and features in IAS that exist in more than one Geo-Type in IAS you will only be able to transfer data for the first Geo-Type. If you us the Auto button and it finds it could define relationships to multiple IAS Geo-Types for the selected NeSS features you see a warning and only relationships to the first IAS Geo-Type features are created automatically. 4.6. Define Data Relationships page The Define Data Relationships page is used to establish relationships between existing Indicators in IAS and the selected NeSS variables. If you have existing Indicators in your IAS systems that match those in NeSS then you can define the relationships between the two and the data will be imported against the existing Indicators. However if the NeSS Indicators do not already exist on your IAS system they will be created by the wizard, but you must still define a relationship so that the new NeSS variable in imported in to a Theme in IAS. 4.6.1 Creating a new Indicator in IAS from selected NeSS variables You can create a new Indicator in your destination IAS system by dragging a NeSS variable from your source server and dropping it on to a Theme node (those with a folder icon) in the right hand list (Figure 12). Figure 12 Creating new indicators in IAS from NeSS variables When creating a new Indicator in your destination IAS system you can change any of the following settings with those of your own (Figure 13), they are populated with default values: Name
InstantAtlas Server NeSS Data Transfer Tool Page 16 Short Name Description Unique Code Type (e.g. Count, Ratio, etc.) Precision (for Ratio, Mean, Numeric no aggregation Types only) Figure 13 New Indicator Details Once a relationship has been established between two features the icons in the Source and Destination panels change and when either feature is hovered an arrow indicating the relationship is drawn (Figure 14).
InstantAtlas Server NeSS Data Transfer Tool Page 17 Figure 14 Viewing Indicator relationships 4.6.2 Link NeSS variables to existing IAS Indicators It is possible to establish relationships between the selected NeSS variables and existing IAS Indicators. This can be done manually by dragging a NeSS variable from the Source (left hand) list and dropping it on an IAS Indicator node (those with a gauge icon ) in Destination (right hand) list. The functions of the Auto, Clear and Clear All buttons are described below: Auto this button attempts to match NeSS variables to existing IAS Indicators using the following rules (in order): 1. By Unique ID first, then 2. by Name (DataType), then 3. by Name, then 4. contains( Name ) If no matches are found under scenario 1 (e.g. searching by Unique ID) the user is prompted if they would like the wizard to attempt to automatically match by Name. If yes is selected at this prompt scenarios 2-4 are followed in order.
InstantAtlas Server NeSS Data Transfer Tool Page 18 The user is informed if multiple matches are found under scenarios 2-4, in this case no relationships are defined. Additionally if one or more matches are found under scenario 4 these is classed as partial matches and the user is informed to check the relationships that have been defined. Clear this button clears the currently selected relationship Clear All this button clears all relationships Once a relationship has been established between two features the icons in the Source and Destination panels change and when either feature is hovered an arrow indicating the relationship is drawn (Figure 14). 4.7. Define Date/Time Relationships The Define Date & Time Relationships page (Figure 15) is used to establish relationships between the date time periods used in NeSS and those defined in the destination IAS system. Figure 15 Define Date & Time Relationships NeSS uses start and end dates to define time periods for data, however IAS uses start dates and Date/Time Periods, for example Year or Quarter. The Define Date & Time Periods page should be used to ensure that there are suitable relationships defined between NeSS date ranges and IAS Date/Time Periods. The table above (Figure 15) is populated with default mappings, however these can be changed using the drop down arrows on the right hand side. The drop down menu is populated with all the Date/Time Periods currently defined on your destination IAS system.
InstantAtlas Server NeSS Data Transfer Tool Page 19 Note any mappings are not persistent when the wizard is closed so each time you open the wizard the mappings will revert back to their default settings. There is also a hyperlink included on this page which will bring up a window showing the date ranges of all variables selected on the Choose Data page (4.3). This list is not constrained by the dates selected earlier on it shows you all date ranges available for the selected variables. Figure 16 NeSS variable available dates 4.8. Data Synchronization Settings The Standard and Advanced data import settings replicate the settings available in the IAS giving users of the NeSS wizard the same flexibility when importing data from NeSS as you have when importing through the IAS Admin Website. Further information on these settings can be found in the IAS Administration Guide and the Advanced Bulk Data Load Guide. Please contact your support provider if you wish to receive a copy of these documents.
InstantAtlas Server NeSS Data Transfer Tool Page 20 4.8.1 Standard Figure 17 Standard Data Synchronisation Settings The Standard settings (Figure 17) are described below: Ignore Imported Metadata this setting is only relevant when importing data against existing Indicators. It refers to descriptive metadata and determines whether a subsequent load should update any descriptive metadata already present. The default is unchecked, which means that descriptive metadata is always overwritten when loaded. Bypass Staging Area this setting can be used to control if the data uploaded from NeSS is imported in to the Staging Area or not. Replace Existing Data this setting is only relevant when importing data against existing Indicators. If selected any existing data at the same Instance as being imported will be overwritten, if unselected any existing data is not overwritten. See the Known Issues section (section 5) for further information on this setting.
InstantAtlas Server NeSS Data Transfer Tool Page 21 4.8.2 Advanced Figure 18 Advanced Data Synchronisation Settings The Advanced settings (Figure 18) are described below: Metadata Action(existing Indicators only) this setting is only relevant if Ignore Imported Metadata (see section 4.8.1) is set to false and you are importing data against existing Indicators. The setting controls how metadata is treated for existing Indicators and has the following settings: o Full (default): all existing metadata items will be deleted and the supplied metadata will be inserted. o Selective : only those metadata items the names of which match those in the supplied list will be deleted and the supplied metadata will be inserted. Metadata Level (existing Indicators only) this setting is only relevant if Ignore Imported Metadata (see section 4.8.1) is set to false and you are importing data against existing Indicators. The setting allows you to control the level at which metadata values are imported and has the following settings: o Both (default): metadata values for indicators and indicator instances will be processed. o Indicator : only metadata values for indicators will be processed. o Instance : only metadata values for instances will be processed. Import Connection Timeout (minutes) this controls the maximum length of time that the application will wait before generating an error. Increase this limit when importing large XML files.
InstantAtlas Server NeSS Data Transfer Tool Page 22 4.9. Choose Task Schedule The Choose Task Schedule page (Figure 19) allows you to control when the task is run, you have three options: 1. Import the data immediately. 2. Schedule tasks to import at a specific date/time in the future, e.g. out of normal office hours. 3. Schedule tasks to import on a user defined period, e.g. to set up a routine task to download the latest Instance of the Indicator(s) selected on the Choose Data page (section 4.3). Each of these options is described in greater detail below. Figure 19 Choose Task Schedule Transfer Now with this selected the transfer task begins immediately when the Next > button is pressed. Transfer Later this option is only available if the IAS Data Synchronisation Service is installed and running on the PC/Server you are running the NeSS Data Transfer Wizard from (see section 1). o o Transfer at a specific date/time this allows the user to schedule a task to run once at a specific date and time. Transfer every... these settings allow the user specify a particular frequency for the task to run. For example you could set up a task to import the latest Instance
InstantAtlas Server NeSS Data Transfer Tool Page 23 of the Indicators selected on the Choose Data page (see section 4.3) that runs at 2am on the first of every month, as shown in Figure 19. 4.10. Completion screen Figure 20 Completion screen Completion screen shows all success/failure messages. For tasks set to run immediately this screen includes hyperlinks to the raw data imported from NeSS, as shown in Figure 20 above.
InstantAtlas Server NeSS Data Transfer Tool Page 24 5. Known Issues It is not possible to add new Geo-Features to an existing Geo-Type using the NeSS Data Transfer Tool. For example if you have an existing Geo-Type with four features but find data in NeSS for eight features you cannot add the extra four features to your existing Geo-Type through the NeSS tool. To do this you would first need to reload your Geo-Type in IAS, see the IAS Admin Guide for further details on this. If you select the Replace Existing Data checkbox but the Instance from NeSS contains no data values for any Geo-Features in your selected Geo-Type it will not overwrite your existing values with the blank values from NeSS. If you select the Replace Existing Data checkbox and the Instance from NeSS contains one or more data values for any Geo-Features in your selected Geo-Type it will overwrite all existing values in IAS with the values from NeSS, some of which may be blank.
InstantAtlas Server NeSS Data Transfer Tool Page 25 6. Suggested Workflow 1. Download initial data from NeSS covering all historical Instances required either run immediately or schedule to run once at a given date/time (e.g. out of normal office hours). 2. Set up a routine task to download the latest Instance of the Indicator(s) as described below. To import the latest Instance of an Indicator without overwriting previous Instances: 1. Set the Updated Between From date to a date before your most recent Instance. 2. Tick the Recent Data checkbox. 3. Ensure that Replace existing data is not selected. To import a previous Instance of an Indicator for which you already have more recent data without overwriting the existing Instances in IAS: 1. Set the Updated Between From and To dates to exactly match the Instance you wish to import. 2. Ensure that Replace existing data is not selected. Note to make the NeSS Data Transfer Tool run as quickly as possible it only downloads data which is to be imported in to IAS. It does not download Instances where a more recent Instance already exists in IAS which has a date that falls within your Updated Between dates. For example you may have the 2005, 2006, 2008, and 2009 Instances of an annual Indicator in your IAS system, you may wish to import the 2007 Instance to complete the historical list. In this case you must ensure you enter Updated Between dates that exactly match the Instance you wish to import (e.g. 01/01/2007-31/12/2007), you cannot enter 01/01/2007-31/12/2010 as the NeSS wizard will see that you already have a more recent Instance of this Indicator (e.g. 2009) in IAS and it will therefore not download the 2007 Instance.
InstantAtlas Server NeSS Data Transfer Tool Page 26 7. Support You can access additional documentation and troubleshooting resources from My InstantAtlas www.instantatlas.com/mia. If you require further assistance please contact your support provider.