PROCESSING AND CATALOGUING DATA AND DOCUMENTATION - QUALITATIVE

Transcription

1 PROCESSING AND CATALOGUING DATA AND DOCUMENTATION - QUALITATIVE INGEST SERVICES UNIVERSITY OF ESSEX... HOW TO SET UP A DATA SERVICE, 8-9 NOVEMBER 2012

2 PRE - PROCESSING Liaising with depositor: consent anonymisation data format documentation Preparation of initial processing plan Ingest Services agrees plan and begins processing

3 FILE FORMATS Qualitative data (text) extensible Mark-up Language (XML) text according to an appropriate Document Type Definition (DTD) or schema (.xml) Rich Text Format (.rtf) plain text data, ASCII (.txt) Digital audio data Free Lossless Audio Codec (FLAC) (.flac) Digital image data TIFF version 6 uncompressed (.tif)

4 PROCESSING PLAN SN 6124: Being a Doctor

5 QUALITATIVE DATA PROCESSING Being a Doctor: a Sociological Analysis, semi-structured interviews All interview transcripts checked: read thoroughly, rectify anonymisation already carried out some names, locations and hospitals were anonymised 2 locations used a and b, which were too close to the real names of the towns renamed to x and z! transcripts formatted to standardised template data listing prepared Other: checking and conversion of audio / video / multimedia materials

6 TYPICAL TRANSCRIPT TEMPLATE

7 FORMATTED INTERVIEW TRANSCRIPT

8 DATA LISTING a key finding aid at file level We compile a data listing depositor may send in completed excel template standard fields plus bespoke fields may need to extract information from data compiled as excel and then as pdf. Supplied in both formats

9 DOCUMENTATION DATA LISTING

10 TYPICAL QUALITATIVE ANONYMISATION CHECKS SN 6228: Discourse of the School Dinners Debate conversation of instant messages had to anonymise online screen names names removed from audio (first time done). Only mentioned early on BUT we had to listen to 30 hours of audio

11 CONTEXTUAL INFORMATION Wider contextual information about project and data background, project history, aims, objectives, hypotheses publications based on dataset Data collection methodology and processes sampling data collection process fieldwork, topic guide temporal/geographic coverage secondary data sources used Useful documents are: End of award reports, topic guides, consent form, project web pages, publications

12 DOCUMENTATION PROCESSING: BEING A DOCTOR Creation of a User Guide (research report, interview schedule, information for participants and consent form) Data List (Excel and PDF) Citation file Read and Note files (only Read file external)

13 USER GUIDE convert Word / RTF / Excel to Adobe PDF paper documentation scanned and converted to PDF (rare) files may be combined according to type bookmarks and headers added create metadata HTML and XML formats create additional documentation (interview with depositor, press reviews, glossaries)

14 QUALITATIVE STUDY USER GUIDE

15 COMPLEX DATA COLLECTIONS working closely with depositors, large scale legacy data set and disclosure issues SN 5801: Concepts of Healthy Eating Food Research: Phases I and II, interview transcripts; 73 diaries; 6 observation field notes conducted without thought of archiving. Involvement and consultation with PI throughout the archiving process (seeking approval, clarifications) PI's reflections on archiving are available here: pdf

16 DIGITISATION OF DATA Non-digital data selectively digitised: paper-based materials: signed consent forms, handwritten field notes, old hardcopy materials photographs analogue audio or audio-visual recordings For audio-visual data we digitise as: images scan and save as TIFF; collate groups as PDF audio: convert to WAV, disseminate as mp3 video: convert to JPEG

17 DIGITISING TEXTUAL DATA Create image file anonymise a copy with black marker prior to scan scan and save as TIFF image file used for poor typeface, handwritten, text with tables and graphs Create searchable PDF collate TIFFs and convert to PDF (Adobe Acrobat) bookmark PDF file for navigation, with contents page and headings Create rich text: optical character recognition (OCR) convert TIFF to RTF format; optional mark-up in XML requires detailed checking and proofreading Transcription keep digital data as close to the original as possible create consistent structure; templates XML where possible

18 REEL-TO-REEL DIGITISATION Our best loved study: The Edwardians oral history collection conducted by the British Library with tight spec from us agreed metadata 1203 reels converted by external supplier no filtering or post-processing applied dead air checks and long gaps edited out processing /metadatabase created batches returned every 3 weeks, audit and random listening. MD5 checksum confirmed 2517.wav files produced HUGE files!!! 2 terabytes of files Dissemination solution? Consent agreed, so just technical

19 AUTOMATING QUALITATIVE PROCESSING Forthcoming Requirements XML versions of content can be indexed to facilitate faster searching on an online system semi-automated system to store and convert deposited transcripts conversion of deposited transcripts to standardised rich text (rtf) and TEI XML format for online dissemination such as via older Qualidata Online system

20 BACKEND QUALITATIVE PROCESSING SYSTEM