Orbis Cascade Alliance Archives & Manuscripts Collections Service. AT Cleanup & Migration Preparation Guidelines. Introduction

Transcription

1 Orbis Cascade Alliance Archives & Manuscripts Collections Service AT Cleanup & Migration Preparation Guidelines Produced by the Archival Collection Management Working Group of the Content Creation & Dissemination Team Eva Guggemos, Pacific University (chair) Mark Carlson, University of Washington Sarah Shipley, Seattle Municipal Archives (co-author of this document) Amber d Ambrosio, Willamette University Nathan Georgitis, University of Oregon (co-author of this document) Jodi Allison-Bunnell, Alliance (ex officio) Version 1.1, February 2017 Introduction Migration of collections data to ArchivesSpace from AT or other sources provides an opportunity and, in many cases, the necessity to review and edit data to ensure complete and accurate migration. This document provides an overview of goals, roles and responsibilities, and methods of preparing data for migration. See other guidelines for migrating data to AS from other sources, e.g. MARC, Excel, and Access. The goals of the migration process are: To ensure complete and accurate data migration To ensure data functions well in AS and within institutional workflows To take the opportunity to resolve any outstanding issues with data To facilitate publication of finding aids that comply with Describing Archives: A Content Standard (2nd Edition) and the Alliance s EAD Best Practices (Version 3.7)

2 Roles and Responsibilities A&M Collections Service Participants will be responsible for preparing collections data for migration, with some assistance provided by A&M Collections Service. The A&M Collections Service has created a Data Cleanup Checklist to help participants prepare collections data for migration. Specific roles and responsibilities will vary according to the following scenarios: A. A&M Collections Service Participants with AT hosted by Alliance, moving to AS hosted by LibraryHost B. A&M Collections Service Participants with AT hosted locally, moving to AS hosted by LibraryHost C. A&M Collections Service Participants with AT hosted locally, moving to AS hosted locally These roles and responsibilities are described in Migration Process section below. In general, the A&M Collections Service will: Provide documentation and training Coordinate communications with LibraryHost, for those migrating to AS hosted by LH Consult with A&M Collections Service participants on migration issues In general, A&M Collections Service participants will: Complete training and read documentation Prepare AT data for migration to AS Issue Level 5 AT login to LibraryHost, for those with locally hosted AT migrating to AS hosted by LH Resolve metadata errors needing correction in order for migration to proceed LibraryHost will: (for those migrating to AS hosted by LibraryHost) Access AT and run AT to AS Migration Tool Resolve migration errors as possible during migration Provide migration logs to A&M Collections Service participants so that they can correct metadata errors

3 Migration Process Listed below is the general sequence of events by scenario. Institution-specific schedules will be created and agreed on by LibraryHost, the CCD Program Manager, and the institutions. The migration process will vary according to the following scenarios: D. A&M Collections Service Participants with AT hosted by Alliance, moving to AS hosted by LibraryHost E. A&M Collections Service Participants with AT hosted locally, moving to AS hosted by LibraryHost F. A&M Collections Service Participants with AT hosted locally, moving to AS hosted locally A. Institutions with AT hosted by Alliance, moving to AS hosted by LibraryHost Step Description 1 Institution prepares AT data for migration according to documentation and training provided and suspends data entry in AT according to schedule communicated to CCD Program Manager Responsibility Institution 2 Institution contacts CCD Program Manager to request migration Institution 3 CCD Program Manager contacts Alliance IT Manager who makes SQL backup (NB: backups run nightly regardless of requests; it may be most efficient to plan around those rather than requesting one-off backups) Alliance 4 IT Manager creates SSH tunnel, AT Level 5 login for AT and sends to LibraryHost Alliance 6 LibraryHost accesses and runs AT to AS migration tool LibraryHost 7 LibraryHost reviews migration log and AS data and resolves issues as possible LibraryHost 8 LibraryHost contacts CCD PM and institution if more AT cleanup required LibraryHost 9 Institution completes AT cleanup Institution 10 Institution contacts CCD Program Manager to request migration re-try Institution 11 CCD Program Manager contacts IT Manager to make SQL backup Alliance 12 CCD Program Manager contacts LibraryHost to request migration Alliance 13 Repeat Steps 6-12, as necessary All 14 Upon successful migration IT Manager makes SQL backup, closes AT instance, retains SQL backup for one year Alliance B. Institutions with AT hosted locally, moving to AS hosted by LibraryHost Step Description 1 Institution prepares AT data for migration according to documentation provided and suspends data entry in AT according to schedule communicated to CCD Program Manager Responsibility Institution 2 Institution makes SQL backup of AT Institution

4 3 Institution creates SSH tunnel, AT Level 5 credentials for AT and sends to LibraryHost (LH strongly prefers not to work with SQL dump) Institution 5 Institution notifies CCD PM to request migration Institution 6 LibraryHost accesses AT and runs AT to AS migration tool LibraryHost 7 LibraryHost reviews migration log and AS data LibraryHost 8 LibraryHost notifies CCD PM and institution if more AT cleanup required LibraryHost 9 Institution completes additional AT cleanup if necessary Institution 10 Institution contacts CCD Program Manager to request migration Institution 11 Repeat Steps 6-10, as necessary All 12 Upon successful migration institution makes SQL backup, closes AT instance, and retains SQL backup for one year Institution C. Institutions with AT hosted locally, moving to AS hosted locally Step Description Responsibility 1 Institution prepares AT data for migration according to documentation provided and suspends data entry in AT 2 Institution prepares AT server / client for migration (see Guidelines for Migrating Data) Institution makes SQL backup of AT Institution ensures AT client is upgraded to version 2.0 Institution ensures AT client is connected to the backup copy of the server Institution installs AS Migration Plugin to AT client Institution Institution 3 Institution installs and configures AS (see Configuration Settings) Institution 4 Institution runs AT to AS migration tool, reviews logs and AS data Institution 5 Institution completes additional AT cleanup if necessary Institution 6 Institution clears AS data and makes SQL backup of AT Institution 7 Institution runs AT to AS migration tool, reviews logs and AS data Institution 8 Repeat Steps 4-7 as necessary Institution 9 Upon successful migration institution makes SQL backup, closes AT instance, and retains SQL backup for one year Institution

5 Preparing Data for Migration A&M Collections Service participants may take different approaches to preparing collections data for migration depending on the quality of collections data, the technical expertise of staff, and the availability of human resources. For example: Those with consistent data and well formed records may opt to identify and resolve problems during migration, rather than thoroughly review collections data in advance of migration Those with inconsistent data and poorly formed records may opt to identify and resolve issues in advance of migration in order to allow time to address issues and to simplify migration Those with technical expertise may opt to resolve issues with collections data by executing scripts against the AT MySQL database Those with less technical expertise may opt to resolve issues with collections data by editing collections data by hand in AT A&M Collections Service Participants should determine their approach based on their knowledge of their collections data and available resources. The A&M Collections Service has prepared an AT Data Cleanup Checklist to guide participants in the process. See the ArchivesSpace document Guidelines for Migrating Data from Archivists Toolkit to ArchivesSpace for basic guidance. General Notes on Methods 1. Identifying issues in AT Many issues may be identified in AT Review EAD BPG and AT collections data to anticipate issues Use the AT Data Cleanup Checklist to review data, identify issues, note extent, determine cleanup Prioritize issues for resolution based on impact on migration and nature and extent of cleanup 2. Editing data in AT by hand Many issues may be resolved by hand in AT Modify AT search editor to search fields under review See AT Manual Modifying Search Settings Customize AT search results to display fields under review See AT Manual Customizing list screens 3. Editing data in EAD by batch Some issues may be resolved in batches by exporting to EAD and using an XML editor Oxygen has limited ability to batch edit files using Find>Find/Replace in Files See additional resources for editing and validating EAD: EAD Toolkit (general resource): Yale Method for Batch Validating EAD (geared toward import to AS)

6 4. Running scripts on AT MySQL database Some issues may be resolved efficiently by running scripts on AT MySQL database Requires access to the MySQL backend of your AT database Contact your IT support to run some example scripts to identify issues if you do not have access to MySQL backend or are unfamiliar with SQL 5. Test Migration, Error Reports, Data QC Some issues may be identified during test migration Other issues may be identified only by reviewing data in AS after test migration Identify test batch of finding aids that reflect various practices, record complexity Review test batch before and after test migration Review report of migration warnings and errors For guidance on interpreting migration warnings and errors, see AT2AS Migration Logs 6. Editing data in AS by hand or by batch find and replace Issues that don t prevent migration may be resolved in AS after migration AS has a flexible search interface AS batch find and replace feature may allow users to more easily edit data in bulk See documentation: AS Batch Find and Replace 7. Running scripts on AS MySQL database Data may be editing in AS after migration using scripts ArchivesSpace Solr index must be rebuilt after each editing session Data in some fields, such as publish field in resource and accession tables, may be edited easily Editing other fields requires a deep understanding of database structure and table relationships Using the Data Cleanup Checklist The A&M Collections Service has prepared an AT Data Cleanup Checklist to guide participants in the process of identifying and resolving issues with data. The checklist includes descriptions of common data migration issues, notes on how to identify issues, places to track the extent and status of issues, and notes on how to resolve issues. Problem Problem Details Priority How to Identify Extent and Status How to Fix - in AT How to Fix - SQL Strategy Brief description of issue, noting relevant AT field or data Detailed description of issue Priority for resolution based on impact on data migration Notes on how to identify in AT Indicate extent of unresolved issues; status of issue resolution Notes on how to fix in AT, including whether to fix before or after migration SQL script examples See the Detailed Notes on Methods below for additional details.

7 Detailed Notes on Methods Review the documentation on under Archivist Toolkit. Download the AT2AS mappings. Review Local Practices Review AT collections data and descriptive practices to identify non-standard, local practices and use the AT mappings to see how your data and practices will migrate to AS. Do you have data in User-Defined Fields in Accessions Records, Resource Records, etc.? See if AS has an appropriate defined field for the data Decide if you want to continue the non-standard, local practice If not, standardize in AT before migration How was your data originally populated into Archivist Toolkit? Depending on how data was added to AT, some data may have bypassed AT validation rules. In this case, for fields that have specific formats, you may wish to review your data to make sure it fits those formats. Do you use Created By and Modified By metadata in your workflows or reports? AT audit metadata does not migrate to AS Date Field Review The following example is for the date field, but the same techniques could be used for other fields such as Resource Identifiers, Accession Numbers, Finding Aid Unique Identifiers or any other field where your institution uses a standard format. To identify resource and resource components with invalid date ranges: In AT: Under Setup-Configure Application, choose Resources and select the begin and end fields to include in the search editor and add a number for the return screen order. Do the same for the Resource Components table. Including resource components, use the greater than and less than dropdowns to search the begin and end dates according to the range of dates you have in your collection. Examples: End less than 1800 or Begin less than 1800 End greater than 2016 or Begin greater than 2016 In SQL: For dates, running the following scripts may identify some incorrect formats: Examples:

8 SELECT * FROM [your database name].resourcescomponents where datebegin > 2016 or dateend >2016 or datebegin < 1800 or dateend < 1800 SELECT * FROM [your database name].resourcescomponents where datebegin > dateend; SELECT * FROM [your database name].resourcescomponents where length(dateexpression) > 9 or length(dateexpression) < 4 and dateexpression is not null; User Defined Fields You may want to review your user-defined fields to ensure they have been used consistently. In AT: Under Setup-Configure Application, choose your table (Accessions, Resources) and select the user defined fields to include in the search editor and add a number for the return screen order (see the AT documentation if more specifics are needed: Individually search each user-defined field with Is Not Empty selected in the AT search editor, and review your results. You may also search for particular phrases if helpful. In SQL: Example:SELECT * FROM [your database].accessions where userdefinedtext11 is not null; SELECT * FROM [your database].accessions where userdefinedtext1 like %parks% ; Text Data Cleanup To find elements with odd characters or carriage returns, use the following example. The note Content field or any lengthy text field is a likely candidate. Some of the characters may be legitimately used in some cases. In AT: SQL: Search the field for the specific character. SELECT * FROM [your database].archdescriptionrepeatingdata where notecontent like '[@#$]' or notecontent like CONCAT('%', CHAR(13)+CHAR(10),'%'); To remove trailing whitespaces from fields: UPDATE [your database].`archdescriptionrepeatingdata` SET `notecontent` = RTRIM(`noteContent`); You can also use sql to batch edit fields you wish to change. For example, if you have a certain phrase you would like to take out of a field such as if you had included the component type in the identifier. *Be careful when making this type of change that you have thought through the consequences of losing information or creating duplicates. Always backup your data before making bulk edits. UPDATE [your database].`resources` SET `resourceidentifier1` = REPLACE(`resourceIdentifier1`,'Series', '')

9 WHERE `resourceidentifier1` like '%Series%'; De-Duplication ArchivesSpace will treat duplicate data in fields in different ways during migration. For duplicate data in some fields, such as EAD FA Unique Identifier, AS will change and note in the log. For duplicate data in others fields, like Subjects and Names, AS will merge duplicates and note in log. Controlled Vocabularies: Subjects must have Type attributes with values: Function; Genre / Form; Geographic Name; Topical Term; or Uniform Title. Names may be non-unique in ArchivesSpace if they are from different sources. Non-standard source vocabulary values for Subjects and Names must be added to AS Controlled Value list Subject Source or Name Source prior to migration. To check for duplicates in AT, list all and then sort and review. Alternately, copy all to Excel and use logic functions to identify duplicate values. To resolve duplicates in AT, highlight the duplicate headings and use Merge function to combine them. To check in the sql backend, run the following scripts: SELECT subjectterm, count(subjectterm) FROM [your database name].subjects group by subjectterm order by count(subjectterm) DESC ; SELECT sortname, count(sortname) FROM archivist_toolkit.names group by sortname order by count(sortname) DESC; Review any terms that have a count greater than one. Containers In ArchivesSpace 1.5, a new data model was implemented for handling containers. A review of these changes may be found here: In cases where you do not use barcodes and have parallel runs of box numbers within a resource or accession, changes will need to be made to ensure that distinct boxes remain distinct are not merged to a common container record. For example, if no changes were made, Box 1 in a Series 1 of Resource 1 and Box 1 in a Series 2 of Resource 1 would be considered the same box. In order to keep this problem from arising, ArchivesSpace has provided a way to address this using the ArchivesSpace Barcoder Plugin ( after migrating to a AS 1.4 version, but before upgrading to AS 1.5. LibraryHost will run the ArchivesSpace Barcoder Plugin during the migration process for those migrating to AS hosted by LH.

10 AT2AS Migration Logs The AT to AS Migration Tool generates a report of record conversion errors and warnings. Library Host will provide the report to A&M Collection Service Participants migrating to AS hosted by LH. Note that server error messages likely indicate problems with the migration process rather than the data and should be referred to the appropriate contact for handling the technical aspects of the migration. The following are examples of migration errors and their meaning. Example No. Error Type Error or Warning Explanation 1 Invalid Date End date: 1871 before begin date: 1885, ignoring end date Record:: Resource Component: A 105/ref10 Identifies an invalid date range, e.g The end date is ignored, meaning it does not migrate. The log notes the resource identifier (A 105), so the record can be reviewed. 2 Dates - Normal Values digits 3 Duplicate EAD FA Unique Identifier 4 Duplicate Subject Term 5 Duplicate Subject Term (with differences in the subject term record) End date: before begin date: , ignoring end date Record:: Resource Component: Ax 642/ref691 Duplicate EAD Id: ORU_Ax715.xml Changed to: ORU_Ax715.xml ##wf7 Endpoint: AT Identifier: Subject->Maps Re-using existing ASpace record: /subjects/292 Endpoint: AT Identifier:Subject->Parks Status code: 400 Status text: Bad Request {"error":{"terms":["subject must be unique"]}} Identifies an invalid date range, e.g. one in which end date comprises fewer digits than begin date. The end date is ignored, meaning it does not migrate. The migration tool doesn t allow duplicate EAD ids. The migration moves forward, but the tool adds a suffix to make the value unique. Indicates a duplicate subject term. The tool merges these terms and will redirect any links to the already existing term. There is no need to act on this error. This will come up for duplicate subject terms that are not exact duplicate records such as having a different source value. AS does not allow for this and it will need to be resolved by the institution. 6 Invalid Rules for a Name Endpoint: AT Identifier: Name_Person->Williams, Vera B. Status code: 400 Status text: Bad Request {"error":{"names/0/rules":["invalid value 'english'. Must be one of: aacr, dacs, local, rda, resource description and access"]},"warning":null,"invalid_object":"#<jsonmodel(:a gent_person) {\"names\"=>[{\"primary_name\"=>\"williams\", \"source\"=>\"naf\", \"name_order\"=>\"inverted\", \"rest_of_name\"=>\"vera B.\", \"rules\"=>\"english\", \"sort_name\"=>\"williams, Vera B.\", \"use_dates\"=>[], \"authorized\"=>false, \"is_display_name\"=>false, This indicates that there is an invalid value for the Rules field for an agent. The person is named, so the value can be fixed.

11 \"sort_name_auto_generate\"=>true}], \"agent_type\"=>\"agent_person\", \"publish\"=>false, \"agent_contacts\"=>[{\"telephones\"=>[], \"name\"=>\"williams\"}], \"jsonmodel_type\"=>\"agent_person\", \"linked_agent_roles\"=>[], \"external_documents\"=>[], \"rights_statements\"=>[], \"notes\"=>[], \"dates_of_existence\"=>[], \"related_agents\"=>[]}>"} 7 Missing required field value Endpoint: migration=archivisttoolkit AT Identifier: Status code: 200 Status text: OK {"errors": ["Server error: #<:ValidationException: {:errors=>{\"instances/0/instance_type\"=>[\"property is required but was missing\"]}}>"], "saved": []} This shows that the resource with the identifier has an instance that is missing an instance type. A similar message is produced for any field that is missing a required value. 8 Field value is too long. Endpoint: AT Identifier:Accession->68/2010 Status code: 400 Status text: Bad Request {"error":{"deaccessions/1/description":["must be 255 characters or fewer"]},"warning":{},"invalid_object":"#\"the collection is available for research.\", \"restrictions_apply\"=>false, \"external_ids\"=>[{\"external_id\"=>\" 6949\", \"source\"=>\"archivists Toolkit Database::ACCESSION\"}], [SEE Full error message here: 2.amazonaws.com/aspacemigrationtools/AT+Migration+Gu idelines--rev pdf] This shows that the value in the deaccession description field in Accession 68/2010 is too long, containing more than 255 characters, which is the max length for the field in AS. 9 Location data Endpoint: AT Identifier: Location->Knight6 Status code: 400 Status text: Bad Request {"error":{"coded_errors":["location_fields_error"]},"warnin g":null,"invalid_object":"#<jsonmodel(:location) {\"building\"=>\"knight\", \"floor\"=>\"6\", \"external_ids\"=>[{\"external_id\"=>\"12380\", \"source\"=>\"archivists Toolkit Database::LOCATION\"}], \"coordinate_1_indicator\"=>\"unspecified\", \"jsonmodel_type\"=>\"location\"}>"} Links & Resources Other institution s migration experiences:

12 Problem Problem Details AS Priority How to Identify Extent of Issue How to Fix SQL Search Strategy AT search Strategy Version Dates - Required in Resource records Dates - Normal Values digits All resource records must have both a date statement (e.g ) and start and end dates AS 1.4 High Search AT date fields Add start and end dates where they are missing <unitdate> must have Begin Date AS 1.4 High Search AT date fields and End values (or) rely on that are BOTH the same number migration error log of digits, e.g. 4, 6, or 8 digits with - Export to EAD, correct date values End Date later than Begin Date in - Alter collection <unitid> to allow import to AT without order for both to migrate overwriting original - Compare edited with original, restore links from accessions to resource, - Compare edited with original, restore extent and container summary - Compare edited with original, reassign location data - Delete original, and restore collection <unitid> See AT Manual p Log into AT with appropriate permissions. Go to Setup > Configure > Resources (and/or Accessions). Double click on DateBegin field. Edit the Return Screen Order field to 3 (this will place DateBegin in the 3rd column of your browse list). Save and repeat for DateEnd field; Edit Screen Order field to 4 (this will place DateEnd in the 4th column of your browse list). After saving, browse your Resources or Accessions; double-click on the column header for DateBegin or DateEnd to sort by date.the smallest/largest in these two date fields will now be sorted. Dates - Normal Values - 9+ digits <unitdate> values more than nine digits in Begin Date or End Date will not migrate to AS Dates - Normal Values digits <unitdate> values less than four digits in Begin Date or End Date, e.g. yyy, is poorly formed data that will migrate to AS AS 1.4 High Search AT date fields (or) run mysql script AS 1.4 Low Search AT date fields (or) run mysql script - Search date fields and correct - Configure AT search editor to display dates - Configure AT resources to display dates - Search for dates less than Correct by hand SELECT * FROM [your See previous database name].resourcescompo nents where length(dateexpression) > 9 or length(dateexpression) < 4 and dateexpression is not null; SELECT * FROM [your See previous database name].resourcescompo nents where length(dateexpression) < 4 and dateexpression is not null; Dates - Begin Date later than End Date Dates - Improbable Begin Date or End Date Container Numbers - Multiple Box 1s Identifiers with Series Designation <unitdate> values in which Begin Date is greater than End Date multiple top <container>s with value 1 within a resource record (parallel runs of box numbers) <unitid>s with values including component type, e.g. <unitid>series I</unitid> AS 1.4 Search AT date fields (or) run mysql script - Configure AT to display begin and end date for Resouces, Components - Search resources, components by date - Review results AS 1.4 Search AT date fields - Configure AT to display begin and end date for Resouces, Component - Search recources, components for begin and end dates beyond range of collections - Ex. End less than 1800; Begin less than Ex. End greater than 2016; Begin greater than Review results AS 1.5 High Manual review of AT resource records AS 1.4 High Search AT identifier field Who: LH When: Duiring Migration - Barcode plugin between 1.4 and 1.5 migration -Edit out "series" and similar designations SELECT * FROM [your See previous database name].resourcescompo nents where datebegin > dateend; See previous or "How to fix" column.

13 Subjects - Duplicate Headings Subjects - Non-Standard or Variant Source Vocabularies Subjects - archiveswest browsing terms not updated from nwda Subjects - Invalid Types <controlaccess><subject>s with AS 1.4 duplicate values will migrate to AS but may be merged depending on Source and Type values If the source of the browsing terms was not updated from "nwda" to "archiveswest" when the EAD Best Practices was updated in May 2015, the terms do not function in Archives West Subjects with types other than the following will not migrate: Function (657) Genre / Form (655) Geographic Name (651) Occupation (656) Topical Term (650) Uniform Title (630) Review in AT (or) Review in Excel - To identify duplicates for small vocabularies... - Open Subjects in AT and sort subject terms in alpha order - To identify duplicates for large vocabularies... - copy and paste all subject terms, types, sources to Excel and alpha sort by Subject Terrm - select Col A, Find&Replace "---" with "-;" - select Col A, Find&Replace "--" with ";" - select Col A, Data>Text to Columns>Delimited by ";">Destination=$F$1:$G$1 - in Line1, Col D insert Formula>Logical>If with Logical_test: A1=A2; value_if_true: DUPE; then copy down for all lines - in Line2, Col E insert Formula>Logical>If with Logical_test: A2=A1; value_if_true: DUPE; then copy down for all lines - Copy ColumD, ColumnE and Paste Special>Values - Select All and Sort by ColumnD, smallest to largest - Select lines with FALSE in ColumnD and sort by ColumnE, smallest to largest - Delete lines with FALSE in ColumnD AND False in ColumnD - Select All and Sort by ColumnA, smallest to largest - To resolve duplicates... AS 1.4 High Review in AT (or) LH - In Subjects, sort by Source and browse for variant source terms - If necessary, clean up data in Lookup Lists>Subject Term Source - Note non-traditional source values (for LH to add values to AS list) - Add values to AS Controlled Value list for Subject Source (or) - Copy all subject terms, types, sources to Excel - Select All>Data>Remove Duplicates by Col C (sources) - If necessary, clean up data in AT Lookup Lists>Subject Term Source - Note variant source values (for LH), or: - Add values to AS Controlled Value list for Subject Source AS 1.4 High Search AT subject term sources (or) LH - Use Setup>Lookup Lists>Subject term source and edit lookup list value and code from nwda to archiveswest. AS 1.4 High Review in AT - Use Setup>Lookup Lists>Subject Term Type to identify invalid types and merge types if necessary update archivist_toolkit.lookupl istitems set code ="archiveswest", listitem = "Archives West" where code ="nwda"; If needed, configure the Subjects browse list to display sources. See AT Manual p Log into AT with appropriate permissions. Go to Setup > Configure > Subjects > Source. Directions for displaying/sorting same as under Date fields above. Sort your Subjects list by source; then edit as necessary.

14 Names - Duplicate Headings Names - Non-Standard Source Vocabularies Names - Invalid Types <controllaccess><persname > and <corpname> with duplicate values will migrate to AS but may be merged depending on Source and Type values AS 1.4 High Review in AT (or) Review in Excel Names with name types other than "Person" or "Corporate body" or "Family" will not migrate - To identify duplicates for small vocabularies... - Open Names in AT and sort names in alpha order - To identify duplicates for large vocabularies... - copy and paste all sort names, name types, sources to Excel and alpha sort by name - select Col A, Find&Replace "---" with "-;" - select Col A, Find&Replace "--" with ";" - select Col A, Data>Text to Columns>Delimited by ";">Destination=$F$1:$G$1 - in Line1, Col D insert Formula>Logical>If with Logical_test: A1=A2; value_if_true: DUPE; then copy down for all lines - in Line2, Col E insert Formula>Logical>If with Logical_test: A2=A1; value_if_true: DUPE; then copy down for all lines - Copy ColumD, ColumnE and Paste Special>Values - Select All and Sort by ColumnD, smallest to largest - Select lines with FALSE in ColumnD and sort by ColumnE, smallest to largest - Delete lines with FALSE in ColumnD AND False in ColumnD - Select All and Sort by ColumnA, smallest to largest - To resolve duplicates... AS 1.4 High Review in AT (or) LH - In Names, sort by Source and browse for variant source terms - If necessary, clean up data in Lookup Lists>Name Source - Note variant source values (for LH), or: - Add values to AS Controlled Value list for Subject Source (or) - Copy all name terms, types, sources to Excel - Select All>Data>Remove Duplicates by Col C (sources) - If necessary, clean up data in AT Lookup Lists>Name Term Source - Note variant source values (for LH), or: - Add values to AS Controlled Value list for Name Source High Review in AT - Use Setup>Lookup Lists>Name Type to identify invalid types and merge types if necessary See previous, but configure Names list instead. User Defined Fields Text Data Cleanup - Trailing Whitespace Inconsistent use of user defined fields AS 1.4 High - Review local practices to identify local, non-standard use of fields - Configure AT to search and display user defined fields - Search each user-defined field with Is Not Empty" - Search each user-defined field for keywords if helpful - Review results AS 1.4 Low Run script on AT MySQL SELECT * FROM [your database].accessions where userdefinedstring1 is not null; SELECT * FROM [your database].accessions where userdefinedstring1 like %parks% ; UPDATE [your database].`archdescripti onrepeatingdata` SET `notecontent` = RTRIM(`noteContent`); See AT Manual p Log into AT with appropriate permissions. Go to Setup > Configure > [Accession/Resources] > [User Defined Fields]. Open each User Defined Field that has been in use at your organization and make sure that the checkbox for "Include in Search Editor" is selected. Close the Configure dialogue pane. Click on the Accessions/Resources module as appropriate; then the Search button. Search within each applicable User Defined Field for "Is Not Empty" and (if helpful) an additional keyword.

15 Text Data Cleanup - Carriage Returns AS 1.4 Low Run script on AT MySQL SELECT * FROM [your database].archdescriptio nrepeatingdata where notecontent like '[@#$]' or notecontent like CONCAT('%', CHAR(13)+CHAR(10),'%'); Text Data Cleanup - Odd Characters Accessions - Date Type AS 1.4 Low Search AT fields for specific characters Accessions with inclusive date AS 1.4 ranges may result in unpopulated Date Type fields. When: After Migration Search AS and edit by hand