Orbis Cascade Alliance Content Creation & Dissemination Program Digital Collections Service. OpenRefine for Metadata Cleanup.

Transcription

1 Orbis Cascade Alliance Content Creation & Dissemination Program Digital Collections Service OpenRefine for Metadata Cleanup Produced by the Digital Collections Working Group of the Content Creation & Dissemination Team Janet Hauck, Whitworth University (chair) Anneliese Dehner, Metadata Applications Librarian, Alliance (author of this document) Laura Zeigen, Oregon Health & Science University Julia Simic, University of Oregon Theodore Gerontakos, University of Washington Jodi Allison-Bunnell, Alliance (ex officio) Version 2.0, June 2017 Table of Contents Overview Install & run OpenRefine Create an OpenRefine project Get acquainted with OpenRefine OpenRefine tools OpenRefine recipes Export cleaned up data OpenRefine resources Overview OpenRefine is often described as a power tool for cleaning up data. This guide is intended to support Alliance libraries in using OpenRefine to clean up digital object metadata for central aggregation. With OpenRefine Alliance libraries can perform batch remediation on records exported from their local system. After batch editing in OpenRefine, the data can be exported as an Excel, CSV, or TSV file and re-ingested into the local system. This video (~7 min.) provides a great introductory overview of OpenRefine functionality:

2 Install & run OpenRefine Requirements & limitations Web browser : OpenRefine runs in a web browser, but does not require an internet connection. OpenRefine does not work with Internet Explorer. Record limits : OpenRefine slows down when cleaning up a file of ~50,000 records. System requirements : OpenRefine is a java application, and you need to have a Java Runtime Environment (JRE) installed on your computer to run OpenRefine. If you don t already have one installed, you can download and install it from by going to the site and clicking Free Java Download. Install OpenRefine 1. Download OpenRefine beta 1 from 2. Install OpenRefine: Windows or Linux : When you download OpenRefine from the address above, you are downloading a zip file. To install OpenRefine, unzip the downloaded file wherever you want to install the program*. Mac : When you download OpenRefine from the address above, you are downloading a dmg (disk image) file. Open this file, and you will be prompted to drag the OpenRefine application into the Applications folder*. * OpenRefine will run wherever you put application, but the location has to be a local drive as problems have been reported trying to run OpenRefine from a Network drive. Run OpenRefine 1. Navigate to OpenRefine and open the program: Windows : Navigate to the folder where you installed OpenRefine and double-click openrefine.exe or refine.bat. Mac : Navigate to the Applications folder and click the OpenRefine icon. If you get the message that Refine cannot be opened because it is from an unidentified developer, open it as follows: Hold the control key while you click the OpenRefine icon. From the resulting menu, select Open and a popup will open. Select Open to launch OpenRefine.

3 Linux : In a terminal window, navigate to the folder where you installed OpenRefine. Type./refine 2. The OpenRefine interface is accessed via a web browser. When you run OpenRefine, a window should open in your default web browser pointing at the address If this doesn t happen automatically, open a web browser and type in this address. If you plan to use OpenRefine often, it may be helpful to bookmark this address. If you encounter problems installing or running OpenRefine, a good source of support is the OpenRefine mailing list and forum. If you encounter problems installing OpenRefine on Windows, you may want to check the thread on installing OpenRefine on Windows 7. Create an OpenRefine project Data formats OpenRefine can parse a variety of data file formats. The formats currently supported include: TSV (tab-separated values) CSV (comma-separated values) XLS, XLSX (Excel) JSON (javascript object notation) XML, RDF as XML Google Sheets RDF N3 triples You can also point OpenRefine at a URL to a data file or a Google Sheet, if it is published publicly. Create a project Follow the steps below to create an OpenRefine project: 1. First open OpenRefine, then open a web browser and go to You will see a page similar to the screenshot below: 2. From the left tab select Create Project.

4 3. Click This Computer. 4. Click Choose Files, locate your data file, and choose. 5. Click Next >> and you will see a Configure Parsing Options page similar to the screenshot below. On this page you will see a preview of how OpenRefine will interpret your data. And you will see options that can be selected to ensure the data imports into OpenRefine correctly. These options will vary depending on the type of data you are importing. 6. Look under Parse data as to verify that the correct file format is selected. 7. Click in the box after Character encoding and select UTF Check Parse next 1 line(s) as column headers. 9. If you want numbers and dates to be formatted as numeric and date data, check Parse cell text into numbers, dates, It is generally easier to clean up data with this unchecked. 10. Quotation marks are used to enclose cells containing column separators. If you are importing a CSV file: Check this. If you are importing a TSV file: Do NOT check this. If you are importing an Excel file: This isn t an option. 11. Check Store blank cells as nulls. 12. Look at the preview to make sure your data looks right (eg. Is a CSV or TSV file s data is separating into columns as expected). 13. Enter a Project name in the top right box.

5 14. When the preview looks okay, click the Create Project >> button at the top right, and your project will open. 15. When your new project opens, double-check that the number of rows in your OpenRefine project match the number of records in the collection (in your system). If these do not match, you may have line breaks in your TSV file that are resulting in incorrect parsing of your records by OpenRefine. You can remove these line breaks in Google Sheets: 1. Open the TSV file in Google Sheets. 2. Use Find and Replace to find \r\n and replace all instances with. 3. Return to step 1 of Create a Project, above. Get acquainted with OpenRefine The layout of OpenRefine After creating a project, the project will open in the basic OpenRefine interface, which resembles the screenshot below. OpenRefine displays data in a tabular format like a spreadsheet. Each row usually represents one record in the data, while each column represents a metadata field or a type of information. Rows & navigation

6 In the upper left, the total number of rows is shown. This number should match the number of records in your collection. By default OpenRefine shows only 10 rows of data/page. You can change this by clicking another number in the navigation bar, after Show:. Navigate to the next page of rows by clicking next at right. Columns Several OpenRefine tools are accessed from the drop-down menu icon at the top of each column. From this menu, you can perform the setup actions listed below. Additional tools are discussed in the next section OpenRefine tools. To reorder and remove columns, click the drop-down menu icon at the top of the first column, labeled All. Hover over Edit columns in the drop-down menu. Click Re-order / remove columns (see screenshot below). To rename a column, click the drop-down menu icon at the top of the column. Hover over Edit column in the drop-down menu, and click Rename this column (see screenshot below). To sort data, click the drop-down menu icon at the top of the column you want to sort by, and select Sort from the drop-down menu.

7 OpenRefine tools The tools described here can be combined to make quick work of data cleanup. Undo In OpenRefine you can undo, and redo, any number of steps you have taken in an OpenRefine project. This means you can experiment with editing your data or try out transformations and undo if you need to. The Undo and Redo options are accessible in the left panel, to the right of Facet/Filter. Follow the instructions below to undo steps: 1. Click the Undo/Redo tab, and you will see a list of all the steps you ve taken so far. 2. Click on the last numbered step you want to preserve. This will automatically undo all the changes made since that step.

8 Split multi-valued cells When a field contains multiple values, it s not possible to batch edit the values with OpenRefine facets and transforms. Before faceting or transforming a column, its multi-valued cells need to be divided. Split multi-valued cells by following the steps below. And remember to rejoin the values when you re finished editing the column. 1. Click the drop-down menu icon at the top of the column. 2. Hover over Edit cells in the drop-down menu. 3. Select Split multi-valued cells... from the flyout menu and a box will open that looks like the box below: 4. In the box, enter the separator currently separating the cell values in the column. 5. Click OK and you will see that each cell in the column contains only one value. Join multi-valued cells After splitting multi-valued cells and transforming their values, it s best practice to rejoin them right away. To join cell values and restore the original structure of your data, follow the steps below:

9 1. Click the drop-down menu icon at the top of the column. 2. Hover over Edit cells in the drop-down menu. 3. Select Join multi-valued cells... from the flyout menu and a box will open that looks like the box below: 4. In the box, enter the separator that should separate the values. Generally this separator will be a semicolon and a space ; 5. Click OK and your multiple values will be back together in one cell. Facet Facets are one of the most useful features of OpenRefine. They can quickly give you an overview of the data in a project. And they can help you clean up your data to be more consistent. To create a Text Facet for a column, follow the steps below:

10 1. Click on a column s drop-down menu icon. 2. Hover over Facet in the drop-down menu. 3. Click on Text facet in the flyout menu, and you will see a new facet box in the Facet/Filter panel at left. The values in your new facet can be sorted alphabetically ( name ) or by count. The values are sorted by name in the screenshot below. Creating a facet is the first step in several OpenRefine batch editing tasks. Here are a few things you can do after faceting on a column. Filter with facets Filter your data by clicking on one of the values in the Facet/Filter panel. Once your data has been filtered, you can edit data within this subset. It is important to note that when you have filtered the data displayed in OpenRefine, any cleanup you do will only apply to the rows that match the filter (the data currently being displayed). Create more facets Create additional facets to further refine your subset. Clean up inconsistencies with facets Clean up inconsistent values within the Facet/Filter panel. For example, there are several variations of Titan in the set shown in the screenshot above. To clean up inconsistencies like this, follow these steps: 1. Hover over one of the values in the Facet/Filter panel. 2. Click edit in the value row, and a box will open like the one below.

11 3. Type a new value in the box and click the Apply button to change the value for several records at once. In this example, all 33 instances of the value Titan,. will be updated to the value Titan when Apply is clicked. Clean up inconsistencies with clusters Inconsistent values can also be cleaned up by clicking on the Cluster button, within the Facet/Filter panel. OpenRefine s Cluster function finds similar values within the faceted column and allows you to merge these variations together into a single value. To use the OpenRefine s cluster feature, follow the steps below: 1. Click cluster and a Cluster & Edit column box will open like the one below:

12 In each row of the Cluster & Edit table, you will see a cluster of values. For each cluster, OpenRefine tells you the Cluster Size (the number of variations in the cluster), the Row Count (the number of rows affected by this cluster), Values in the Cluster, a Merge? Column with checkboxes, and a New Cell Value column (for the preferred value). By selecting different options from the Method and Keying Function drop-down menus, you can apply different clustering algorithms. Sometimes one Keying Function or Method will catch variations that the others miss, so it can be worthwhile to try different options. 2. Identify a cluster row to merge. 3. Click the Merge? checkbox in the cluster s row. 4. Type your preferred value in the row s New Cell Value box. 5. Click the Merge Selected & Close button. Facet by blank There are two ways to find empty fields in a column. 1. Look at the list of facet values in the Facet/Filter panel. If there are any empty cells in the column, a blank facet will appear in the list. Click blank to view these rows in your data.

13 2. Or you can use the shortcut shown in the screenshot below (Click the drop-down icon at the top of a column > hover over Facet > hover over Customized facets > Click Facet by blank ) Text filter In addition to facets, you can filter data in OpenRefine by applying a Text Filter, which finds a text value in a column and then displays all records/rows containing the value. To apply a text filter, follow the steps below: 1. Click the drop-down menu icon at the top of the column you want to filter on. 2. Select Text filter from the drop-down menu, and the filter options will display in the Facet/Filter panel at left.

14 3. In the box that appears in the Facet/Filter panel, type the text you want to use as a filter. Your data will filter as you type, only displaying rows which contain your typed text in the relevant column. In the example above, 4219 records are displaying because they all have the value london in the field Place of Publication. It is important to note that when you have filtered the data displayed in OpenRefine, any cleanup you do will only apply to the rows that match the filter (the data currently being displayed). To remove a text filter, click the x at top left of the filter box. When you remove the filter, you will see all of your data again. Fill down Sometimes you want to fill an empty column with a single value. One way to do this is described in the steps below: 1. Hover over the first cell in the column, and you will see the word edit. 2. Click on the cell and a box will open for cell editing. 3. Type a value in the box and click Apply. 4. Click the drop-down menu icon at the top of the column. 5. Hover over Edit cells in the drop-down menu. 6. Select Fill down from the flyout menu, and OpenRefine will fill the column s cells until it reaches a cell with an existing value. If you start with an empty column, OpenRefine will fill all the cells in the column with the value you typed in the first cell.

15 Transform OpenRefine supports transformations, which are ways of manipulating data in columns. Transformations are normally written in a special language called GREL (Google Refine Expression Language) which can include regular expressions. To some extent GREL expressions are similar to Excel functions, although they mostly focus on text manipulations instead of numeric functions. Transformations are useful for standardizing the format of data in a column without changing the values (eg. date format, inverted names). And they are also handy for find and replace. To start writing transformations, follow the steps below: 1. Click the drop-down menu icon at the top of a column. 2. Hover over Edit cells. 3. Select Transform and you will see a dialog box like the one below: 4. Enter a GREL (Google Regine Expression Language) expression in the box (there are several in the below OpenRefine recipes section), and in the Preview table you will see your original data in the left column and your transformed data in the right column. Select what you would like OpenRefine to do when your GREL expression fails to transform your data ( Keep original is usually a good choice here; it means OpenRefine will do nothing in these cases).

16 5. When you are happy with the transformation, click OK. OpenRefine recipes OpenRefine recipes, for common metadata issues, are provided in the How to fix column of the tables below. The bold terms in the recipes reference tools described in the above section, OpenRefine tools. Dates Noncompliant Compliant How to fix ca. YYYY YYYY~ Transform the values in the Date column by if(value.contains('ca'),value.replac e('ca.','').replace('ca. ','')+'~',value) [YYYY?] YYYY? Transform the values in the Date column by if(value.contains('['),value.replace ('[','').replace(']',''),value) Month DD, YYYY YYYY-MM-DD Transform the values in the Date column by

17 value.todate('mmmm dd, yyyy').tostring('yyyy-mm-dd') MM/DD/YYYY YYYY-MM-DD Transform the values in the Date column by value.todate('mm/dd/yyyy').tostring( 'yyyy-mm-dd') YYYY; YYYY YYYY-YYYY Transform the values in the Date column by if(value.contains(';'),value.split(' ;')[0].trim()+'-'+value.split(';')[- 1].trim(),value) YYY0s YYYu Transform the values in the Date column by if(value.contains('-'),value,value.r eplace('0s','u')) YY00s YYuu Transform the values in the Date column by if(value.contains('-'),value,value.r eplace('00s','uu')) YYth century YYuu Transform the values in the Date column by if(value.contains('century'),(value. replace('th century','').tonumber()+1).tostring( )+'uu',value) YYYY Month DD YYYY-MM-DD Transform the values in the Date column by

18 value.todate('yyyy MMMM dd').tostring('yyyy-mm-dd') Month YY YYYY-MM Transform the values in the Date column by value.todate('mmmm yy').tostring('yyyy-mm') Month DDth YYYY YYYY-MM-DD Transform the values in the Date column by value.replace('th','').replace('rd', '').replace('nd','').todate('mmmm dd yyyy').tostring('yyyy-mm-dd') Languages Noncompliant Compliant How to fix Language x, Language y Language x; Language y Transform the values in the Language column by value.replace(', ','; ') Names Noncompliant Compliant How to fix First name Last name Last name, First name If all Creator or Contributor values are personal names: 1. If cells in the Creator or Contributor column contain multiple semicolon delimited values, split multi-valued cells, entering ; in the prompt. 2. Transform the values in the Creator or

19 Contributor column by copying the script below and pasting in the Custom text transform if(split(value,' ').length()>2,split(value,' ')[-1]+', '+split(value,' ')[0]+' '+split(value,' ')[split(value,' ').length()-2], value.split(' ').reverse().join(', ')) 3. In the Creator or Contributor column, join multi-valued cells, entering ; in the prompt. If the Creator or Contributor values are a mix of personal and corporate names: 1. Facet or apply a text filter to create a subset of personal names. 2. Follow the steps above to invert the personal names. Name x or Name y Name x; Name y Transform the values in the Creator or Contributor column by copying the find & replace script below and pasting in the Custom text transform value.replace(' or ','; ') Rights Noncompliant Compliant How to fix [absent] For permission to publish, please contact Pine State University Libraries at (555) org/publicdomain/zero/ 1.0/ Rights transferred to Pine State University Libraries. For permissions, please If the same rights statement applies to all records: 1. Enter the rights statement in the first cell of the Rights column. 2. Fill down. 1. Facet the Rights column. 2. Edit the noncompliant value in the Facet/Filter panel, replacing the value

20 contact admin/rightsmanageme nt with a compliant value. Type Noncompliant Compliant How to fix [DCMI Type values are absent] Text If the same Type value applies to all records: 1. Enter the Type value in the first cell of the Type column. 2. Fill down. If the Type value varies, but it can be derived from a value in another column: 1. Facet and/or apply a text filter on another column, creating a subset of records of the same type. 2. Enter the Type value in the first cell of the Type column. 3. Fill down. Still image StillImage 1. If cells in the Type column contain multiple semicolon delimited values, split multi-valued cells, entering ; in the prompt. 2. Facet the Type column. 3. Edit the Still image value in the Facet/Filter panel, replacing the value with StillImage. 4. In the Type column, join multi-valued cells, entering ; in the prompt. OR Transform the values in the Type column by

21 value.replace('still image','stillimage') Export cleaned up data When you are finished working with your data in OpenRefine, you can export it. The export options can be accessed by clicking the Export button at top right in the OpenRefine interface. Export formats include HTML, Excel, CSV (comma-separated values), and TSV (tab-separated values) Additional OpenRefine resources OpenRefine Documentation for Users OpenRefine Forum Google Refine Expression Language