Workshop on Web Archiving

Transcription

1 Workshop on Web Archiving MODULE 1: WEB ARCHIVING Niels Brügger Asger Harlung

2 Program, KU Workshop part 1: Web archiving Coffee Workshop part 2: Existing collections Short break Workshop part 3: Doing your own archiving (incl. hands-on) Wrap-up; feedback Open discussion: Thoughts and questions 2

3 Module 1: Web Archiving Introducing ourselves and NetLab Why archive the web Two examples; dr.dk s history and Probing a Nations Web Three kinds of digital content WWW as technology What is web archiving? Methods of web archiving Challenges for the web crawler Crawling advantages/disadvantages Characteristics of the archived web 3

4 Introducing Ourselves and NetLab Niels Brügger Professor (MSO, with special responsibilities) in Internet Studies and Digital Humanities, Head of NetLab, and of the Centre for Internet Studies, specialising in internet research since Asger Harlung MA in ICT and learning, has previously worked with research in digital rhetoric, and supporting creativity development in learning processes and the rest of NetLab is: Janne Nielsen Assistant Professor, PhD, Media Studies, AU; author of the book: Using Web Archives in Research an Introduction (2016) Ulrich Have IT Architect, MA in Information Studies. At NetLab, he is developer of tools for research using archived web material and does data analytics. 4

5 Introducing Ourselves and NetLab A research infrastructure for internet research. Part of the Danish research infrastructure Digital Humanities Lab (DIGHUMLAB). Established in Research driven development of research infrastructures. 5

6 Digital Humanities Lab Language Tools (KU) Media Tools (AU) Interaction & Design (AAU & SDU) NetLab Audio and visual materials Online Collecting data for specific projects Archived IT architect Netarkivet (the Danish national web archive) NetLab Forum , micro stipends 6

7 Ditigal Impact on the Humanities We are living in an ever more digitised world We know that why are they stating the obvious? 7

8 Ditigal Impact on the Humanities (based on Alvin Toffler: TheThird Wave (1980)) Neolithic revolution, ca. 12,500 years ago: Agriculture is the first technological revolution, and replaces hunter-gatherer culture. Hundreds or thousands of years between technogical advances. Industrial revolution (I and II), ca s. Decades between technological advances, but faster advancements since late in the 19th century ( second industrial revolution ) Post-industrial age, information age, digital age 1950 s and onwards. Years, months, sometimes weeks between technological leaps. State of the art today may be obsolete before the end of the year (a state of perpetual future shock ). 8

9 Ditigal Impact on the Humanities 2000: 75% of the world s data was stored in analog form (paper, film, photographic prints, vinyl, magnetic casette tapes, etc.), 2007: 7% analog, 93% digital 2012: Only 2% of all stored data was stored in analog form. Mayer-Schönberger & K. Cukier (2013): Big Data: A revolution that will transform how we live, work, and think. Houghton Mifflin Harcourt Publishing Company, New York, 2013, pp

10 Why Archive the Web? To preserve the cultural heritage To preserve a stable research object To be able to document and illustrate a study 10

11 dr.dk s History 11

12 Dr.dk s history Started in 2007 Analyse the historical development of dr.dk Partly based on archived web content 12

13 Probing a Nation s Web Domain from Small Data to Big Data Web sphere Web element Web Website Web sphere 13

14 Probing a Nation s Web Domain from Small Data to Big Data The historical development of an entire national web:.dk The project is a collaboration with Netarkivet

15 Probing a Nation s Web Domain from Small Data to Big Data Brutto list of 'probes : Size e.g. bytes Space e.g. geolocalisation Structure e.g. network of hyperlinks Liveliness e.g. domain names and updating Content e.g. degrees of openness, files, software types, language, website textual elements, semantics 15

16 Digitised Formerly analogous media, transferred to a digital form. Born Digital Has not previously existed in any other form than digital. Reborn Digital Born digital content which has been gathered and preserved, and to some extent has been changed in the process. 16

17 WWW as Technology Web element 17

18 WWW as Technology Web 18

19 WWW as Technology Web site Web element Web Website 19

20 WWW one among other internet protocols: http Hyper Text Transfer Protocol URL Uniform Resource Identifier (Locator) html Hyper Text Markup Language Constructing a URL on WWW: protocol://subdomain.domain.topdomain/path// 20

21 Network of computers http Computer (webserver) URL, dr.dk Web s = html-files html html html html html html html html html html html html http Computer (user) The browser (Safari, Firefox...) translates html into writing, pictures etc. Computer (webserver) as database, CMS (Content Management System), URL dr.dk Computer (webserver) as database, URL, e.g. dmi.dk Images Words Weather Heading Comp. X Comp. Y Web s = patched together in an empty shell (stylesheet) of material from databases 21

22 Small Exercise: Source Code 22

23 What is Web Archiving? International Internet Preservation Consortium s definition: the process of gathering up data that has been published on the World Wide Web, storing it, ensuring the data is preserved in an archive, and making the collected data available for future research. ( Any form of deliberate and purposive preserving of web material. (Brügger, 2011:25) Brügger, Niels (2011): Web Archiving Between Past, Present, and Future. IN: Mia Consalvo and Charles Ess (eds.) The Handbook of Internet Studies. Wiley-Blackwell. 23

24 What is Web Archiving? Macro archiving Cultural heritage institutions Preserve as much as possible Big and varied data IT expertise, advanced technology, computer power Micro archiving Individual researcher/research group Stablize a concrete research object, here-and-now No experience, no advanced technology or computer power 24

25 Methods of Web Archiving Web crawling (hyperlink crawling) Screen image Screen filming Harvesting via API (Delivery from producers) 25

26 Web Crawling domain.com 26

27 Web Crawling crawler 0 domain.com

28 URL URL URL URL URL crawler domain.co m domain.dk domain.dk domain.dk crawler domain.dk 28

29 Challenges for the crawler JavaScripts Content based on Flash Interactive s Streamed content Websites with access limitations (password, captcha) Cookies, adds, plugins etc. Robots.txt ki/captcha Deep web (e.g. databaser, ftp-server, password-protected content, hidden content, s not linked to, dynamic content based on requests). 29

30 Pages not being crawled Not crawled robots.txt domain Not crawled password protected domain Not crawled script Not crawled too deep 30

31 Elements not crawled _ Netarkivet 31

32 Elements not crawled _ Netarkivet 32

33 Elements not crawled _ Internet Archive 33

34 Crawling, Advantages The entire in full length Hyperlinks, link source as well as target Look and feel of live web (with limitations) Automatic (partly, evaluation and trouble shooting) Machine readable, enables search, sorting, analysis Access to metadata (crawl logs) Robust format (html) Big data-analysis (content analysis, network analysis, etc.) 34

35 Crawling, Disadvantages Some objects not archived, e.g. videos and streamed content, and applications based on Flash, JavaScript etc. Temporal inconsistencies Difficult to delimit in terms of spatial extent Risk of web crawler being caught in bot traps (some monitoring is necessary) 35

36 Characteristics of the Archived Web What is archived is not a 1:1 copy of the material one attempted to archive It is versions/reconstructions: Created in the process of archiving On the basis of a number of choices made by the archiver (harvesting strategy, settings, etc.) The choices made have consequences for what is archived The archived objects are re-assembled in the archive replay 36

37 Characteristics of the Archived Web The archived version is deficient because of: Technical challenges Web s specific characteristics: dynamic, unpredictable Potential asynchronicity between updating and archiving archiving takes time certain elements cannot be archived It is an added challenge that we do not know what is missing: Not much documentation No baseline to compare with 37

38 Characteristics of the Archived Web As scholars using archived web as an object of study, it is important that we are aware of the pitfalls and sources of error inherent in the material. 38

39 During the Olympics in Sydney in 2000, I wanted to save the website of the Danish newspaper JyllandsPosten. I began at the first level, the front, on which I could read that the Danish badminton player Camilla Martin would play in the finals a half hour later. My computer took about an hour to save this first level, after which time I wanted to download the second level, Olympics But on the front of this section, I could already read the result of the badminton finals (she lost). The website was as a whole not the same as when I had started; it had changed in the time it took to archive it, and I could now read the result on the front, where the match was previously only announced. N. Brügger: Archiving Websites, 2005, pp

40 Characteristics of the Archived Web Exactly what is preserved? Content, user experience or something else entirely? an archive also must be sure that the document is translated in an authentic manner. In this case, authenticity means that the document must both include the context and evoke the experience of the original. (Lyman, 2002 s. 41) 40

41 Characteristics of the Archived Web It is versions/reconstructions: The archived objects are re-assembled in the archive replay 41

42 Link list Named entities Online web archiving html + files IN CONTRAST TO DIGITIZED COLLECTIONS: TO A LARGE EXTENT ARCHIVED WEB IS ALREADY MARKED UP HTML, FILE NAMES... 42