ClientTrack s Guide to CSV Import ClientTrack s HUD CSV 4.1 Import is available is ClientTrack versions 15 and later.

Transcription

1 ClientTrack s Guide to CSV Import ClientTrack s HUD CSV 4.1 Import is available is ClientTrack versions 15 and later. Import Process Overview ClientTrack s HUD CSV 4.1 import is designed to help communities share data for reporting and collaborating to better serve their clients. ClientTrack conforms the 4.1 CSV standard which includes the latest data standards (version 3.1). ClientTrack uses a two-stage import process for CSV Import. Initially data is imported into staging tables. Administrators are able to review the data before committing it to the system. The second stage refers to importing the actual ClientTrack tables used for data collection, access, and reporting. The two-stage process allows administrators to prevent errors before data is fully imported. ClientTrack s CSV 4.1 import uses an asynchronous process to import data. An asynchronous process is a multi-stage process that begins with a request for import which creates an asynchronous task. The asynchronous task processes the next time the service is available. This improves system efficiency. The status of the request can be found on the Asynchronous task form, under Batch Processes in the Global Administration menu group. Before importing, ClientTrack will help you map setup data in ClientTrack to the data you are importing. This improves consistency and prevents duplication of setup data. Preparing for Import Before you import there are a few things you can do and a few you must do in order to successfully import data. Review Files Unlike XML, the CSV standard doesn t have a mechanism for validating data. Here are a few things you can do to determine if the data is consistent with the standard and will import into ClientTrack: Review the files you are importing. Verify the file names consistent with the HUD standard. The files in the standards are and used for import 1 include: o Client.csv 1 Other files like Export.csv are in the standard but are not directly imported into ClientTrack (staging or other tables). The standard also includes files that contain setup data. Setup data, where applicable for import, is mapped to avoid duplication and will not be imported. [1]

2 o Disabilities.csv o EmploymentEducation.csv o Enrollment.csv o EnrollmentCoC.csv o Exit.csv o HealthAndDV.csv o IncomeBenefits.csv o Services.csv Review the specification document. Here are a few ways that the spec document can be useful for reviewing the files: o Each file has its own section that indicates whether the file is required. Not every file is required. For example, for SSVF reporting there are only 9 files required. o The spec document includes the names of the fields and their data types. o The spec also indicates whether a field allows for nulls (i.e. blank, no data). Setup Data The import process facilitates mapping setup data in the import files to ClientTrack setup data. The following setup should already be present in your ClientTrack solution before 2 importing: Organization Create a ClientTrack organization that represents the entity that collected the data you are importing. Please note that the organization used for import is based on your endpoint. Programs The programs referenced in your import files should already exist in your ClientTrack solution. The proper program setup is essential to using the data in ClientTrack for reporting or any other purpose. Grants The funding sources for your import files should already exist and be associated with the appropriate program(s). Services Service code mapping associates ClientTrack service codes with service types in the data standards. You will want to be sure that service codes exist and are mapped appropriately. For programs, services, and grants you will need to grant organization access appropriately, in the very least the endpoint organization (see Setup your Endpoints for more detail) should have 2 ClientTrack s import will alert you to the setup data that must be mapped. This happens in the workflow used for importing. If the appropriate setup data doesn t exist, you can pause the workflow, create the setup data, and resume to remap. [2]

3 Setup your Endpoints An import implies that there is a source system (i.e. where the data comes from) and a destination system (i.e. where the data is being imported). In ClientTrack, an endpoint represents these two system. Endpoints help with mapping and deduplication. Mapping is unique to each system since each system may approach setup data differently. Endpoints are necessary for deduplication. ClientTrack stores unique identifiers for each endpoint to avoid creating new data when it s reimported. Please see the section on deduplication for more details. Endpoints are managed in the Global Administration menu group, under System Integration. Click Add New to add a new endpoint. The endpoints necessary for import represents the source system. The End Point Name is the how users are able to identify the endpoint in dropdown lists and other areas of ClientTrack. Endpoint type should be either Sends Data to this System or Sends and Receives Data from this System The End Point s Local Organization is the organization data will be imported into. Please keep in mind that the organization selected here has data access and reporting implications. [3]

4 Database ID uniquely identifies the database (i.e. the source system). The id will need to be unique and an integer. Exclude Deleted Records? This property applies to exports only and doesn t have an import effect. Importing Data ClientTrack uses a workflow to import data. This workflow is accessible under System Integration>Import in the Global Administration menu group. Click on HMIS 4.1 CSV Import to initiate the workflow. Importing the data The first step in the CSV import allows you to select a name, configuration, an endpoint and your file. The name of the CSV import can be anything you d like. The name identifies the asynchronous task. You must select HMIS CSV 4.1 Import as the configuration. The Endpoint should be the endpoint representing the source system for your import. Click on Choose File to select the file you want to import. ClientTrack expects a zipped file that contain each of the csv files you are importing. Be sure to click upload after selecting the file. After uploading you can save to proceed through the workflow. CSV Import Log The CSV import log is the second step in the import workflow. In this step you will see the status of each task involved in importing. Processing indicates that the data has been imported into the first stage (the staging tables). For example: Enrollment.csv has been staged to integration.[hudcsv4_1_enrollment]. 2,894 line(s) found, 2,893 inserted. In this case, there were 2,893 records inserted into the staging tables. An error indicates a problem. On this step an error usually indicates a problem with the file or field names. [4]

5 No mapping found for the column "VAMC" in the file Enrollment.csv. CSV 4.1 Import This error indicates that import isn t expecting a field named VAMC. In this case, the expected field name is VAMCStation and the file being imported doesn t meet the standard. Program Mapping Programs referenced in the source file will need to be mapped to ClientTrack programs. This prevents duplicates and improves consistency. It s important to note for program and other mapping that if setup data isn t mapped then it won t be imported. HUD CSV Other Benefit Mapping The Other Benefit mapping refers to Data Element A Non-cash Benefits, other source. In the ClientTrack Financial Type column you will see a list of non-cash benefits that are mapped to Other Source. If you don t have an Other Source in the dropdown you will need to add a financial type that is mapped to Other Source HMIS Category. Financial Types are managed in the under System Administration, Other System Codes, Financial Codes, Financial Types in the Global Administration menu group. [5]

6 In the following screenshot you ll see an example of a financial type mapped to Other Source as the HMIS category: Other SSVF Services Mapping SSVF services have a category for other. Service codes mapped to Other will appear on this step for mapping. If you do not see any service codes in the ClientTrack Service Code column you will want to create a service code under Setup Data Management, Service Codes in the Global Administration menu group. Import the Data In the final step you will be prompted to import the data or reject the data. Please see the next section for details about how you can query the data in the staging tables. Until you click Yes, import the data the data will remain in the staging tables. De-Duplication De-duplication occurs at two levels. The first level is related to the client record. For client records, ClientTrack will attempt to match records using identifying information. For other records, referred to as transaction record, in the import ClientTrack will use the unique identifiers from the source file to de-duplicate. Client De-duplication The ClientTrack HMIS 4.1 CSV uses strict comparison for client de-duplication. Strict comparison de-duplication is intended to find the most appropriate client previously entered in to the system. At least three elements used to determine who to match must be present on [6]

7 either side of the import and must be an exact match. These elements include First Name, Last Name, Social Security Number, and birth date. The first Client found matching the criteria will be updated with the data from the source file. Transaction De-duplication Transaction records refer to any non-client record in the import. Transaction records include enrollments, assessments, services, and all other types of data imported. Upon initial import, ClientTrack will create transaction records. These records are then mapped (associated) with the unique identifier from the CSV and the endpoint from which it came for future updates when processing later import requests. For example, let s say your import includes an enrollment for a client. On the first import, ClientTrack will check to see if that enrollment s unique identifier, ProjectEntryID, exists. On first import that unique identifier will not exist and ClientTrack will create the enrollment. ClientTrack will also create a record in the table IntegrationRemoteKey that stores the ProjectEntryID for that endpoint. The next time that specific enrollment record is imported the import will see that the record already exists. ClientTrack will override the existing record in ClientTrack with the imported data. It s important to remember that when importing a transaction record ClientTrack will always create a new record unless that specific transaction record from the source endpoint has already been imported. If it the record has already been imported then ClientTrack will override the existing ClientTrack record. Viewing Data in Staging Tables As mentioned above, ClientTrack s two-stage import allows administrators to review data before final import. This data can be reviewed using query designer. The staging tables used include: Integration.HUDCSV4_1_Client Integration.HUDCSV4_1_Disabilities Integration.HUDCSV4_1_EmploymentEducation Integration.HUDCSV4_1_EnrollmentCoC Integration.HUDCSV4_1_Exit Intergration.HUDCSV4_1_HealthAndDV Integration.HUDCSV4_1_IncomeBenefits Integration.HUDCSV4_1_Project Integration.HUDCSV4_1_Services Data in these tables is structured to match the CSV structure in the 4.1 standard. The tables also include data related to the import itself including the taskid of the import task, the line number, and a unique id for the staging import record. The line number may be helpful in determining [7]

8 where the record exists in your source data. For example, a line number of 64 would refer to row 64 in your source file. Please note that the first row of a valid CSV file is the head and it s not loaded into the integration staging tables. In the following screenshot example, query designer is used to view data in the table integration.hudcsv4_1_enrollment The staging tables for completed tasks are cleared out regularly to manage database size. Editing Mapping In the import section of this document you ll find instructions for mapping imported data in the import workflow. In this section we describe how mapping can be updated for an endpoint. Mapping is associate with each endpoint. Editing mapping can be accomplished from the integration endpoint search form. This form is found in the Global Administration menu group, under System Integration. From each endpoint, the options for editing the mapping are accessed from the search button. In the following screenshot, SSVF Provided Public Benefit mapping is depicted: [8]

9 Import and Security Two security measures are present and used in an import, client access and transaction access. These measures are setup through the endpoint. All records are created with the owning organization being the organization assigned to From Endpoint and the User running the import is assigned as the creator of the new records. Client Security Restriction is used when creating new client records that are not found to match to existing clients. When an Information release is selected the clients created will be assigned that info release with a start date of date of import and no expiration date. The Client Security Restriction will be used to set the appropriate values for the release of information selected. Transaction security the option selected is given to all new records imported. Import and Deleted Items marked as deleted in the CSV file will create or update an existing record with that status for the associated record. The CSV Import will not delete a client record if the transactional records are indicated as deleted (and have a deleted date) they will be updated but the client record itself will not ever be deleted by import unless the client record is originally imported as deleted a client record with the deleted status is then created. Potential Issues The following potential issues are all related to the source data meeting the standard. In each case, you will see an error in the import indicate which record caused the error. Please note that records with errors will not be imported to the staging tables. Duplicate or Missing IDs Source files with duplicate or missing IDs will result in errors during import. Files generating these errors are invalid and cannot be imported. Resolving these issues requires acquiring a valid file from the source. Incomplete Data In some cases you may have missing data in your source data that trigger validation on ClientTrack s forms and workflows. Barriers are one example of this. ClientTrack s barriers form allows users to track physical disability, developmental disability, chronic health condition, HIV/AIDS, mental health, and substance abuse on the same form. If the source data is missing a barrier, ClientTrack s barriers assessment will warn the user they didn t record all the required data. In such a case, users should either acquire the appropriate response or record Data Not Collected. [9]

10 Missing Required Data In addition to unique IDs, there are multiple other fields required by the 4.1 standard. If these data are missing then ClientTrack will not be able to import the data and errors will be generated. Errors of this kind have the format of fieldname is Missing. For example, if the information date is missing from IncomeBenefits.csv ClientTrack will return the error InformationDate is Missing for that file. The resolution for issues of this kind is to acquire a valid CSV file. Additional Support If you need further help or have any questions about this update then please contact us. ClientTrack support professionals are available by phone at You may also submit a support question, issue, or suggestion using the help link found at the top right of your ClientTrack solution. Please do not CSV files to ClientTrack. The CSV files contain personal identifying information and is not a secure transport method. ClientTrack s support ticket system is secure and you can upload files to a support ticket. Feature Notes Version Control Revision Number Publication Date Description /9/2015 Initial release [10]