Questions & answers. Thierry BOUCHE Université Joseph Fourier (Grenoble) DML-EU May 22th 2003

Transcription

1 Questions & answers Thierry BOUCHE Université Joseph Fourier (Grenoble) DML-EU May 22th 2003 A digitisation program supported by C.N.R.S. and ministère de la Recherche, managed by the Cellule MathDoc

2 1. What? 2. Why? 3. Which principles? 4. For whom? 5. How? 6. How many? 7. Demonstration 8. Outline of the production workflow DML-EU Göttingen May 22th

3 1. WHAT Digitise for archiving and diffusion the backrun of French mathematical journals NUMDAM PHASE I Title Since Owner Volumes Pages Articles Ann. inst. Fourier 1949 Assoc. AIF Journées É.D.P CNRS Publ. math. I.H.É.S IHÉS Bull. Soc. math. France 1872 SMF Mém. Soc. math. France 1964 SMF Ann. École normale sup ÉNS Contract with the commercial publisher. DML-EU Göttingen May 22th

4 2. WHY Mathematical litterature never becomes obsolete ; It s usefull to other sciences in an asynchronous fashion ; It s valid only when considered as a whole, building a network of (international) references. Economics are unimportant ; Digital format makes it possible to keep the archive alive, with easy access and hyperlinks. Heterogeneous data can also be bound together. DML-EU Göttingen May 22th

5 3. PRINCIPLES Make accessible and visible the French mathematical heritage. Sustain Fench-speaking mathematicians and publications ; DML-EU Göttingen May 22th

6 3. PRINCIPLES Make accessible and visible the French mathematical heritage. Sustain Fench-speaking mathematicians and publications ; Strengthen the journals, authors and searchers ; DML-EU Göttingen May 22th

7 3. PRINCIPLES Make accessible and visible the French mathematical heritage. Sustain Fench-speaking mathematicians and publications ; Strengthen the journals, authors and searchers ; Find a common way to these 3 (sometimes conflicting) interests. DML-EU Göttingen May 22th

8 3. PRINCIPLES Make accessible and visible the French mathematical heritage. Sustain Fench-speaking mathematicians and publications ; Strengthen the journals, authors and searchers ; Find a common way to these 3 (sometimes conflicting) interests. Not a new edition. DML-EU Göttingen May 22th

9 3. PRINCIPLES: INTEGRITY Publisher s rôle The journals remain owners of the articles they have published ; The economic balance of the journals is preserved through the moving wall ; Cross linking with publisher s site. DML-EU Göttingen May 22th

10 3. PRINCIPLES: INTEGRITY Publisher s rôle The journals remain owners of the articles they have published ; The economic balance of the journals is preserved through the moving wall ; Cross linking with publisher s site. Data integrity The end object is an article, digitised at a sufficiently high resolution in order to insure higher faith to the original than a mere xerox copy ; Metadata, automatic enhancements, as well as the HTML interface all subject to change during the project s lifetime are kept apart from the original (scanned) data. DML-EU Göttingen May 22th

11 3. PRINCIPLES: INTEROPERABILITY Indexing and links produced by MathDoc must be available to large independant databases (international: DML, interdisciplinary: DIEPER, etc.) Platform independance (XML) ; Keyboard independance (Unicode) ; Math dialect independance (MathML/T E X/??). DML-EU Göttingen May 22th

12 4. FOR WHOM Journals Long term archiving the math content indepedently of journals economic fortune ; Give far wider visibility to this content than to its custom readership (adding references to it in a national database, then included in large international or interdisciplinary databases). Authors Enhance visibility of and ease of access to their work. Searchers Give easy access to original works to mathematicians, historians and scientists through a variety of search methods. DML-EU Göttingen May 22th

13 5. HOW Rights Diffusion agreement with the journals ; Author s agreement in cooperation with the journal ; First page identifying the journal, providing the bibliographic data of the article, and reminding the copyright agreement. DML-EU Göttingen May 22th

14 5. HOW Rights Diffusion agreement with the journals ; Author s agreement in cooperation with the journal ; First page identifying the journal, providing the bibliographic data of the article, and reminding the copyright agreement. Methods Exhaustivity of collections ; Download units: articles (PDF ou DjVu) ; Respect of the paper s format ; Rich and generic metadata easily reusable (XML/Unicode) Tagged bibliographical references. DML-EU Göttingen May 22th

15 5. HOW: SPECIFICATIONS Technics Monochrome digitisation at 600 dpi ; Assembly of each article in one multipage file (with anything not belonging to the article removed, if needed); OCRed text (automatic non corrected 99,9 %) for searching only, hidden for the end-user ; Keyed in (99,99 %) and tagged bibliographies (author, title, year); Metadata as a single XML/Unicode per physical volume;. Exhaustive (automatic) as well as sampled (manual) quality control. The files served are reengineered (with PdfT E X) in house to have bibliographical data, ownership and fair use agreement printed on the cover. DML-EU Göttingen May 22th

16 5. HOW: LINKS Internal networking From an erratum to the corrected article (and backwards) ; From a bibliographical reference to the paper itself ( NUMDAM link) ; From the served file to its metadata (stable URL). DML-EU Göttingen May 22th

17 5. HOW: LINKS Internal networking From an erratum to the corrected article (and backwards) ; From a bibliographical reference to the paper itself ( NUMDAM link) ; From the served file to its metadata (stable URL). External networking From a digitised article to its review (Math. Reviews, ZentralBlatt Math, Jahrbuch über die Forstschritte der Mathematik) ; From an article cited in the bibliography to the to its review ; Backward links expected... DML-EU Göttingen May 22th

18 6. HOW MANY: DATA Title Since Owner Volumes Pages Articles Authors 1 Wall Ann. inst. Fourier 1949 Assoc. AIF Journées É.D.P CNRS Publ. math. I.H.É.S IHÉS Bull. Soc. math. France 1872 SMF Mém. Soc. math. France 1964 SMF Ann. École normale sup ÉNS Total phase I Before and after identification. DML-EU Göttingen May 22th

19 6. HOW MANY: LINKS Title Articles Link ZM MR Errata Biblios Bib. ref. Link ZM MR Link NUMDAM Ann. inst. Fourier Journées É.D.P Publ. math. I.H.É.S Bull. Soc. math. France Temporary total DML-EU Göttingen May 22th

20 Summer 2000 Funding by CNRS. CHRONOLOGY Sept March 2001 Writing up schedule of conditions, call for proposition. Jan.-Dec Cataloguing and checking database. Dec Choice of operator validated by CNRS. Jan.-May 2002 Setting up production chain (operator) and quality control (Cellule MathDoc). Aug The first pages delivered! Jan Annales de l institut Fourier & J.E.D.P. released, first version of up and running. Mar Writing up schedule of conditions, call for proposition for NUMDAM II. June 2003 End of the industrial phase of NUMDAM I, public access to the full featured whole collection scheduled for the autumn. Expected start of the phase II. Next Robust & versatile archiving (files, metadata, updates and corrections, dynamic pages... ). Better integration with live contents (integration of recent paper s metadata, links to the actual full-text for subscribers). Enhance our served DjVu. DML-EU Göttingen May 22th

21 HALL OF FAME Pierre Bérard/Yves Laurent & Laurent Guillopé Directors of MathDoc initiators, political choices, negociations Thierry Bouche Project manager technical choices, executive direction and coordination of all the actors (also responsible for the logo, page layout, and PdfT E X wizardness) Elizabeth Cherhal Engineer metadata conversions, DTD, XML Hélène Bégnis Engineer, project coordinator temporary staff management, website design and programming, intranet & quality control management. Catherine Barbe-Zoppis Engineer database creation and maintenance, automatic quality control, website administration Claude Goutorbe Engineer Edbm wizard (content indexing and search interface), reference matching and linking Plus 2 part time temporary staff : cataloguing, manual quality control, 1 external consultant for call to tender, 1 legal advisor. DML-EU Göttingen May 22th

22 II. NUMDAM WEBSITE DML-EU Göttingen May 22th

23 III. PRODUCTION WORKFLOW 1. General organization 2. Preliminary catalogue 3. Quality controls automatic manual 4. Database structure 5. Indexer and links 6. Served files (The forthcoming slides were produced by MathDoc staff) DML-EU Göttingen May 22th

24 GENERAL ORGANIZATION Production flow 1/2 Journals Preliminary catalogue Vendor Digitization provider Controls Monopage tiff files Multipage tiff files XML files (raw) PDF and DjVu files Archiving DML-EU Göttingen May 22th

25 GENERAL ORGANIZATION Production flow 2/2 XML files Internal controls & corrections Raw PDF and DjVu files SQL Database Identification Author database Author rights management contracts creation EDBM XML Link matching TeX User files DML-EU Göttingen May 22th

26 PRELIMINARY CATALOGUE The preliminary catalogue was done inside cellule MathDoc by gathering available data from existing sources (journal, databases, etc.) making bibtex files checking them manually against the documents converting this to XML with a perl script. DML-EU Göttingen May 22th

27 QUALITY CONTROLS Quality control procedure Automatic check Perl LOG (detected errors) Refection Synthesis Data files : XML, TIFF, PDF and DjVu Sample selection Perl Sample Visual monitoring Check-list Php Validation (XML, TIFF, PDF, DjVu files) State of errors Database MySQL Import in database DML-EU Göttingen May 22th

28 QUALITY CONTROLS The automatic control is exhaustive. It checks exhaustivity/integrity of the delivered file system validity of each file internal coherence between data and metadata (number of articles, pages, etc.) The manual control is performed on a sample, it checks whatever requires a human eye: Scanned image quality (TIFF monopage) Segmentation of articles (TIFF multipage) OCR The generated metadata and its tagging There is also a manual check of all anomalies reported by the automatic control. DML-EU Göttingen May 22th

29 MANUAL QUALITY CONTROL Set in the process of check Visual monitoring : checklist File in the process of check Errors types Aims: Control global quality of files Keep records of all controlled files and any errors DML-EU Göttingen May 22th

30 MANUAL QUALITY CONTROL Some statistical data A set of pages is automatically controlled in 8 hours. & A sample of 315 one page tiff is controlled in 1 day A sample of 125 articles is controlled in 4 days by a person working full-time. DML-EU Göttingen May 22th

31 AUTOMATIC QUALITY CONTROL LOG CHECKING Example of a LOG file Analyse of the volume 1994, issue 57 of the «Mémoires de la SMF» Indication of a new article with NUMDAM identifier and start/end page Problem with the djvu file of the article : wrong number of pages DML-EU Göttingen May 22th

32 AUTOMATIC QUALITY CONTROL Quality Control: Automatic Procedure Aims: - check digitized files coming from provider - compare all possible sources for same information Previous examination of volumes number of pages, of articles Set of XML files describing each volume structure Set of digitized files - monopage files: tiff - multipage files: tiff, pdf, djvu, xml Perl program Log Files: ctrl.log ctrl_vol.log ctrl_xml.log ctrl_tif1.log ctrl_art.log ctrl_djvu.log ctrl_pdf.log ctrl_ocr.log ctrl_tif2.log DML-EU Göttingen May 22th

33 AUTOMATIC QUALITY CONTROL I- Check directory structure, file names and existence - Observe numdam id, file name and file location:. numdam volume id <-> directory name. numdam article id <-> directory and file name - Verify the existence of each expected file according to metadata. Monopage files: for each article - all pages between first and last scheduled page - same number of pages as indicated in Xml Metadata. Multipage files: - pdf, djvu, tiff and xml (ocr) files DML-EU Göttingen May 22th

34 AUTOMATIC QUALITY CONTROL Check example: identifiers Directories and digitized files: Metadata in volume file «AIF_1949 1_.xml»: <notice> <idvol>aif_1949 1_</idvol> <article type="normal"> <idart>aif_ _0</idart> <pagedeb systnum="arabe" pagination="normal" typepag="normal">103</pagedeb> <article type="normal"> <idart>aif_ _1</idart> <pagedeb systnum="arabe" pagination="normal" typepag="normal">113</pagedeb> <article type="normal"> <idart>aif_ _0</idart> <pagedeb systnum="arabe" pagination="normal" typepag="normal">121</pagedeb> DML-EU Göttingen May 22th

35 AUTOMATIC QUALITY CONTROL No missing file according to metadata Metadata in «AIF_1949 1_.xml» <notice> <idvol>aif_1949 1_</idvol> <idart>aif_ _0</idart> cphys>page0113.tif page0114.tif page0115.tif page0116.tif page0117.tif page0118.tif page0119.tif page0120.tif page0121.tif page0122.tif page0123.tif </cphys> <idart>aif_ _1</idart> <cphys>page0123.tif page0124.tif page0125.tif page0126.tif page0127.tif page0128.tif page0129.tif page0130.tif </cphys> <special type="photo"> <cphys>spec0010.tif</cphys> </special> Series of digitized files DML-EU Göttingen May 22th

36 AUTOMATIC QUALITY CONTROL II- Check all file formats For each digitized file, validate - file type ( xml, tiff, pdf, djvu ) - validity of the file Example: «badly formed» xml file file BSMF_ _.xml: <idvol>bsmf_ _</idvol> <revue> <issn> </issn> <acronumdam>bsmf</acronumdam> </infos> <numerisation> <idphys>bsmf_ _</idphys> <datescan>27/09/2002</datescan> <infos>tif CCITT Group4 #-bà 600dpi </numerisation> </notice> > xmllint BSMF_ _.xml BSMF_ _.xml:20: error: Opening and ending tag mismatch: infos and numerisation </numerisation> ^ DML-EU Göttingen May 22th

37 AUTOMATIC QUALITY CONTROL File format checking examination of all format files: > file BSMF_ _.xml BSMF_ _.xml: XML document text > file BSMF_ _1.djvu BSMF_ _1.djvu: data > file BSMF_ _1.pdf BSMF_ _1.pdf: PDF document, version 1.4 > file BSMF_ _1.tif BSMF_ _1.tif: TIFF image data, little-endian > file BSMF_ _1.xml BSMF_ _1.xml: XML document text Error example showed by the log file «ctrl_ocr.log» «-> PB: /ums4/numdam/bsmf1/bsmf_1881 9_/BSMF_ _2/BSMF_ _2.xml is not an XML file!» >> file /ums4/numdam/bsmf1/bsmf_1881 9_/BSMF_ _2/BSMF_ _2.xml /BSMF_ _2.xml: Apple Old Partition data block size: first typ, number of blocks: , DML-EU Göttingen May 22th

38 AUTOMATIC QUALITY CONTROL File validity checking. programs for each file format: tiff, pdf, djvu, xml. check they are «well formed» Error example showed by the log file «ctrl_djvu.log» djvups ASENS_1959_3_76_1_1_0.djvu > test.ps [ ] Entrée de décodage endommagée *** [ ] Impossible d'obtenir l'image de la page 0 *** (DjVuToPS.cpp:2158) *** 'void DjVuToPS::print (ByteStream &, const GP<DjVuDocument> &, GUTF8String)' DML-EU Göttingen May 22th

39 AUTOMATIC QUALITY CONTROL III- Check existence of all digitization metadata XML metadata expected from digitalization process Provider has to fill in metadata about volume, article, special Example: volume «AIF_1949 1_.xml» <notice> <numerisation> <idphys> AIF_1949 1_</idphys> <datescan>29/05/2002</datescan> <infos>tif CCITT Group4 à 600dpi</infos> </numerisation> <pages>196</pages> <article type="normal"> <idart>aif_ _0</idart> <cphys>page0015.tif page0016.tif page0017.tif page0018.tif page0019.tif page0020.tif page0021.tif</cphys> </article> <special type="photo"> <pagedeb systnum="arabe" pagination="normal" typepag="special">0</pagedeb> <pagefin systnum="arabe" pagination="normal" typepag="special">0</pagefin> <cphys>spec0010.tif</cphys> </special> DML-EU Göttingen May 22th

40 AUTOMATIC QUALITY CONTROL IV- Check metadata structure and encoding We compare XML files at each stage: original xml <> xml completed by provider <> end xml -The Import / Export procedure of metadata from MySql database shows wrong character encoding. If they are badly coding they will be different or missing from the end xml file. - It also shows:. mistakes or corrections introduced by provider on xml volume description. errors in xml file structure files are different after reconstruction DML-EU Göttingen May 22th

41 AUTOMATIC QUALITY CONTROL Error example with unicode character Volume «BSMF_ _.xml» <article type="normal"> <idart>bsmf_ _0</idart> <titre xml:lang="fr">champs spinoriels et propagateurs en relativité générale</titre> <langue>fr</langue> <biblio> <bibitem>[1] <bauteur>bogoljubov (N. N.)</bauteur> et <bauteur>širkov D. V.) </bauteur>. <btitre>introduction à la théorie quantique des champs</btitre>. Paris, Dunod, <bannee>1960</bannee> (Travaux et Recherches mathématiques, 5).</bibitem> <bibitem>[2] <bauteur>bruhat (Yvonne)</bauteur>. <btitre>propagateurs et solutions d'équations homogènes hyperboliques</btitre>, C. R. Acad. Sc. Paris, t. 251, <bannee>1960 </bannee>, p </bibitem> <bibitem>[4] <bauteur>duffin (R. J.)</bauteur>. ߞ <btitre>on the characteristic matrices of covariant systems</btitre>, Phys. Rev., t. 54, <bannee>1938</bannee>, p </bibitem> Unicode character «ߞ» badly encoded -> correction made to bibliography data: DML-EU Göttingen May 22th

42 AUTOMATIC QUALITY CONTROL Error example about page number Volume «BSMF_ _.xml» <article type="normal"> <idart>bsmf_ _0</idart> <pagedeb systnum="arabe" pagination="normal" typepag="normal">153</pagedeb> <pagefin systnum="arabe" pagination="normal" typepag="normal">160</pagefin> LOG file: «--> diff : value of [[pfin]] line 301 v1: [160] v2: [159]» Wrong end page number coming from our initial examination -> provider has changed this value on its xml file it appears when we compare both files DML-EU Göttingen May 22th

43 AUTOMATIC QUALITY CONTROL V-Compare metadata with information from digitized files Information is automaticly extracted from provider files and compared with data coming from the xml volume description. Tag from tiff file (mono and multipage) -file width, resolution, date, copyright -number of pages - check unicity for monopage files, - compare with article metadata for multipage files. Tag from pdf and djvu file : number of pages. Metadata list from ocr xml file DML-EU Göttingen May 22th

44 AUTOMATIC QUALITY CONTROL Examples: number of pages pdf > pdfinfo PMIHES_ _0.pdf Pages: 13 Page size: 595 x 748 pts Optimized: yes PDF version: 1.4 volume «PMIHES_1959 3_.xml» <article type="normal"> <idart>pmihes_ _0</idart> <pagedeb>5</pagedeb> <pagefin >17</pagefin> <langue>en</langue> </article> langage (*n) tiff > tiffinfo PMIHES_ _0.tif TIFF Directory at offset 0x13618 Subfile Type: (0 = 0x0) Image Width: 4960 Image Length: 6234 Resolution: 600, 600 pixels/inch Bits/Sample: 1 Compression Scheme: CCITT Group 4 Photometric Interpretation: min-is-white Software: "Pixel Translations Inc., PIXTIFF Version 5" TIFF Directory at offset 0x2a1a0 Subfile Type: (0 = 0x0) Image Width: 5007 Image Length: 6272 Resolution: 600, 600 pixels/inch TIFF Directory at offset 0x400ea Subfile Type: (0 = 0x0) Image Width: 5002 Image Length: 6268 TIFF Directory at offset 0x52ff8 Ocr file PMIHES_ _0.xml <article idart="pmihes_ _0"> <body> <page numero="1"> THE DEFINITION OF EQUIVALENCE OF COMBINATORIAL <page numero="2"> <page numero="13"> DML-EU Göttingen May 22th

45 DATABASES View on Global Procedure Original XML metadata files Completed XML metadata files 1 Initialisation 2 DataBase Add digitization data 3 - Data check, Corrections - Author identification 4 End XML Files DML-EU Göttingen May 22th

46 DATABASES from Xml metadata to MySql database DB Table Update For each new volume XML file <-> 1 volume (XML dtd) <=> -> Volume record -> n 'Article ' normal, preface, postface, erratum, corrigentum, addendum -> m 'Special' title page, summary, index, advertising, photography DML-EU Göttingen May 22th

47 DATABASES MySql database overview main 'article' table author table reference author table main 'special' table main 'volume' table DML-EU Göttingen May 22th

48 DATABASES Volume tables overview file «PMIHES_1959 2_.xml» <notice> <idvol>pmihes_1959 2_</idvol> <revue> <issn> </issn> <acronumdam>pmihes</acronumdam> </revue> <tome> <numero>2</numero> <annee>1959</annee> </tome> <editeur>institut des Hautes Études Scientifiques</editeur> <pages>68</pages> <numerisation> <idphys>pmihes_1959 2_</idphys> <datescan>13/08/2002</datescan> <infos>tif CCITT Group4 à 600dpi</infos> </numerisation> </notice> 'Volume' Volume Table Volume information : numdam id acronyme number, year, fascicule, serie editors Journal information: issn title publication information (country, publisher) Corresponding records in DB 'Journal' DML-EU Göttingen May 22th

49 DATABASES Article tables overview file «AIF_1950 2_.xml» <article type="normal"> <idart>aif_ _0</idart> <pagedeb systnum="arabe" >161</pagedeb> <pagefin systnum="arabe" >164</pagefin> <ordre>0</ordre> <auteur> <nom>zariski</nom> <prenom>o.</prenom> </auteur> <titre xml:lang="fr">sur la normalité analytique des variétés normales.</titre> <langue>fr</langue> <biblio> <bibitem>[1] <bauteur>c. CHEVALLEY</bauteur>, <btitre>"on the theory of local rings"</btitre>, Ann. of Math., vol. 44 (<bannee>1943</bannee>), pp </bibitem> </biblio> <cphys>page0171.tif page0172.tif page0173.tif page0174.tif</cphys> </article> Article Table type (article, preface, erratum,) numdam id and others id:. Math. Reviews,. ZentralBlatt Math,. JahrBuch für Mathematik title(s), author(s), contributor(s) langage(s) first and last page, order on page, order in xml file abstract bibliography monopage file list relations Corresponding records in DB DML-EU Göttingen May 22th

50 DATABASES Special tables overview file «PMIHES_ _.xml» <special type="photo"> <pagedeb systnum="arabe" pagination="normal" typepag="special">0</pagedeb> <pagefin systnum="arabe" pagination="normal" typepag="special">0</pagefin> <cphys>spec0008.tif</cphys> </special> Special Table type (title page, summary, index, advertising, photography) numdam id langage(s) first and last page order on page order in xml file monopage file list Corresponding record in DB DML-EU Göttingen May 22th

51 TOWARD THE WEB! INDEXER/LINKS The content of the database is exported back to XML, it is inputed into cellule MathDoc s search engine edbm formerly developped for the web interface to the ZentralblattMath database. Full text and tagged bibliography entries are indexed. Links are added as much as possible. DML-EU Göttingen May 22th

52 TOWARD THE WEB! SERVED FILES A simple pdft E X program is executed for each file served. It generates a first page containing the identification of the journal archive, bibliographical data of the article, the Numdam stable URL, a reminder of fair use, and a notice that the Numdam program produced the file. The server is configured for page serving of the files (PDF files are linearized, DjVu files served as indirect) DML-EU Göttingen May 22th

53 Thank you! DML-EU Göttingen May 22th