Gedcom Map Program "PrepMaps" Preparing tsv-files for interactive Online Maps from ged-files

Transcription

1 Preparing tsv-files for interactive Online Maps from ged-files Table of Content 1 Main Screen Functionality Prerequisites Description Program sequence Definition file (mmsd-file) Program start by command line Search criteria Options Buttons Tab "Options / Header line tsv-file" Method for compression Extract from a log file Appendix Copyrights, License agreements and Others Setup of the tsv-file File name Description of the columns Examples of tsv-files Processable MAP + _GOV Structures of ged-files Standard Structure "_LOC Records" "0 _PLAC" "0 _PLAC_DEFN" Programs without MAP, LATI, LONG and _GOV Diedrich Hesmer Seite 1 von 12

2 1 Main Screen 1.1 Functionality This program helps to show genealogists graphically on a world map Where do our ancestors come from? Where are their descendants? It creates up to 8 tsv files from a ged file, which must be UTF-8 encoded, to create interactive maps with OpenStreetMap background or Google Maps to be integrated into the own website. There, markers are placed in each location found and the frequency of the respective surnames can be displayed. A linking of the persons with the OFB data of "ofb.genealogy.net" 1 and the GOV data of "gov.genealogy.net" 2 is possible. The ged files must therefore include the coordinates specified by the tags MAP with LATI and LONG. If these data are missing, they can be supplemented with the program "ModPlac" shown below. The ancestors and/or descendants of an individual or a group of persons born in a specified place within a defined period are selected and transmitted in compressed form. All INDI and FAM records are evaluated for this purpose. The surnames with a key date are listed for each location found. For the ancestors, the most distant date is displayed. The most recent date is shown for the descendants. The main difference between the maps that can be created with the tsv files and a local family book is the assignment of where the persons came from and where they moved to. This results in a completely different 1 OFB: Online Heritage Books, a project of the "Society for Computergenealogy, e.v." (CompGen) 2 GOV: Historic Gazetteer, a project of the "Society for Computergenealogy, e.v." (CompGen) Diedrich Hesmer Seite 2 von 12

3 point of view. You can sees which surnames are found from or up to a date at this place. This in extension with the surrounding places forms an unusual perception of the own data. Example maps with OpenStreetMap: German English Example maps with Google Maps: German English For details, see the report "Serving Online Maps with PrepMaps" from Otto Riehl. An exemplary application may be found here for Krasna, a colony with German inhabitants in Bessarabia. Both are accessible by the Homepage. During the process, a log file is created that can be opened with any text editor. The log file has the name of the ged file, extended by "_pm" and the file extension ".log" and is stored in the directory of the ged file. 1.2 Prerequisites For the output, only persons will be taken into account, who are containing in the ged-file: information of date of birth, christening, death or burial and additionally location data (PLAC) with their coordinates (MAP.LATI/LONG) Persons with the entry "DEAT Y" and a PLAC text are also included, even without a date. See paragraph "Processible MAP + _GOV structures of ged files" on page 12 for details of the programs concerned. 1.3 Description The following describes the program sequence, the mmsd definition files, the search criteria, the options, the effect of the buttons, the logic and the log file Program sequence After starting the program, the above main screen appears. Fields with missing or incorrect entries are highlighted in red. This means that an analysis is not yet possible. Only INDI and FAM data records are analyzed. At the first call all fields are empty or set to their system value. Load the ged file, select the search criteria for the selection of persons and the options. After defining all definitions, these should be saved in a mmsd control file (see next section) for later reuse. "Start" starts the import of the ged file. All INDI and FAM data are read. Missing or incorrect PLAC texts, GOV texts, LATI or LONG texts are listed in a log file and are not used internally. o LATI/LONG texts with "," are replaced by ".". o Long LATI/LONG texts are truncated after 6 digits after the ".". o Missing surnames are replaced by "_". o Optionally defined PLAC entries, e.g. "unknown", are omitted. After importing, the search criteria are compared with the data in the ged file. If a year of birth is missing, the year of christening is used as a substitute. If there is a match, the person is considered a finding. The persons found by this search are the "base group". From the persons of the base group their ancestors or descendants are searched now, independently of the place and year of their birth, christening, death and burial. From the total quantity of all found persons, in the following called "findings", a compression takes place according to the procedure described below on page 7. The tsv files has 9 fixed columns, 1 row for each location found. The "LATI;LONG" combination, which is unique, is formed as a key field from the coordinates, even with different spellings of the respective location. Up to 8 tsv files are created according to the selected options (Add, Create for, Output texts). By "Close" an ini file with the names of the mmsd control files is created, which are made available again with the new start and the program ends. Diedrich Hesmer Seite 3 von 12

4 When restarting the program, it searches for the ini file. If found, the fields are filled according to the settings of the mmsd file from the last "Close". These can be changed before the next "Start" Definition file (mmsd-file) These files are used to save all selected options. They are created and managed with this program and should be saved under a name that is as meaningful as possible. It is recommended to save all mmsd files together in a common directory (e.g. "C:\Data\Map\Files\"). In the "DropDown" bar of the mmsd definition files in the lower part of the window, click on the arrow to open a list with the last mmsd files used up to the number defined under "Settings" in the start center. The file selected here is loaded. You can also start the PrepMaps program by double-clicking on an mmsd file in the explorer Program start by command line The program is designed that it can be started with command line parameters. Therefor it can also be processed with the GSP Utility "BatchRun", where it belongs to the "Form 1" parameters (see program description "BatchRun.pdf", available on the homepage). The following call is required: "mmsd-file with full path"[,/q]. The file specification must be enclosed in ". The ",/q" command is optional and causes processing in the background so that a window is only opened if errors or questions occur. Otherwise, the program ends automatically. Without this command the program starts normally with the specified mmsd file. Example command line: "C:\0\group-gov-test2.mmsd",/q Search criteria The search criteria determine the search for the persons concerned. To do this, select : The Starting point for the analysis: o Individual The field "INDI Id" must contain the complete ID number of the person in the form "@Inn@". o Group of persons This allows the selection of a group of people from a location within a period. To do this, the exact text as it appears in the ged file must be specified for each of the following PLAC or the _GOV indicator. These are best transferred via "Copy&Paste" from the ged file. In addition, the following must always be specified Years in the form "YYYY-YYYY" (from-to). The year is determined from the DATE texts. If two 4-digit numbers are found there, as in FROM YYYY TO YYYY, BET YYYY AND YYYYY, the first value is used for ancestors and the last value for descendants. Beside the Gedcom formats also the German formats " " or "before Easter 1820", "1870/1875", " " are accepted. Persons with missing entries are not included in the base group. The Add to this... group of persons: o Ancestors for all persons of the base group and/or o Descendants for all persons of the base group The groups are calculated with HUSB, WIFE and CHIL from the FAM data records. Therefor FAMC and FAMS do not have to be evaluated. For this calculation, all persons are used, independent whether place and date data are available. However, only persons with place and date specifications are taken into account in the later compression Options Options for processing and output of the data are Diedrich Hesmer Seite 4 von 12

5 Create tsv for... the output of the map o OpenStreetMap and/or o Google Maps. Output texts controls the language (German and/or English) of the output texts in the tsv file. Affected are the variable texts of the header lines given below and the fixed coded base texts for the links. These are: o " Link zum Geschichtlichen Ortsverzeichnis (GOV)" " Geburtsorte auswärtiger Personen OFB" " Sterbeorte weggezogener Personen OFB" " weitere Nachnamen, ", "ab " und "bis " resp. o " Link to the Historic Gazetteer GOV" " Places of Birth of Foreigners Local heritage book OFB" " Places of Death of Emigrants Local heritage book OFB" " further surnames, ", "from " und "to " Max Persons in List allows you to narrow down the surnames displayed for each location in the list. If "0" is specified, up to 1,000 persons are listed. The system value is "10". Link to "ofb.genealogy.net" allows linking to the OFB data base stored on the specified CompGen server: o Names linked to the names listed in the list o Places linked to the respective places. o With which OFB The corresponding name of the "Online OFB" of ofb.genealogy.net is to be entered here. The original spelling (upper/lower) must be used. o Test calls the specified OFB at genealogy.net by the browser. If it is not found, an error message appears in the browser and the spelling has to be corrected Buttons Following buttons are available: [ Symbols with an open folder ] Loads a ged-file. [ Edit ] Pushing by the left mouse button opens the ged-file by the text editor for viewing. Pushing the right mouse button starts "GedShow" with the ged-file. [ Edit tsv ] opens a dialog for selecting a tsv file by a text editor. [ x ] Deletes all entries of the search criteria. [ Paper Symbol ] Clears all definitions and sets the defaults. [ Symbols with an open folder ] Loads a mmsd file. [ Symbol with floppy disk ] Saves a mmsd file. [? ] Opens the description. [ Info ] Shows the folder where the ini-file is stored (contains all settings at the last "Close"). [ Start ] Starts the processing. [ Edit Log ] Opens the log-file by the text editor. [ Close ] Writes the ini-file with all mms files of the drop down list and closes the program. (Won't be done by the upper right x-button) Diedrich Hesmer Seite 5 von 12

6 Gedcom Map Program "PrepMaps" Tab "Options / Header line tsv-file" Options for the log-file MAP errors in log-file lists missing MAP and therefor no LATI and LONG data exists. LATI/LONG errors in log-file lists LATI and LONG errors. Suppress "Base group" suppresses the output of "Starting point of search " Further Options BAPM as christening add to CHR uses the BAPM data for the christening if no CHR is found. Due to settings, some genealogy programs store the christening data under the BAPM tag instead of CHR. If both are present, the data of CHR is used. Suppress PLAC texts... suppresses all PLAC texts specified in the text field, e.g. "unkown;unbekannt", when reading the ged file. The texts can be specified in lower case, semicolons must separate several Definition oft he Header lines The screen above shows the fields of OpenStreetMap and Google Maps for the headers of the tsv file, here "- en" for English. Nine lines are fixed. These can be modified with the following exceptions: "Latitude" and "Longitude" are mandatory see Note below. Display language allows the selection of the displayed language. Modify texts allows the change of the respective two fields manually by overwriting. When activating, a warning pops up. If this is affirmed, the fields are released, some buttons and the own option are locked. This lock is released by "Check + Release". o Load standard overwrites all four texts by the system data. o Check + Release validates the texts of the four fields and releases the locked buttons and options again. This ensures that only valid data for the headers is stored in the mmsd file. Note: With OpenStreetMap, the lines are only used for internal processing. If you change them, the templates must also be changed accordingly. Recommendation: A change should only be made if it is absolutely necessary. With Google Maps, the text of the columns are transferred directly into the display of the popup. The texts can be adapted according to your needs. Diedrich Hesmer Seite 6 von 12

7 For both applies: "Latitude" and "Longitude" may not be changed! 1.4 Method for compression Compression takes place in several steps: 1. From the "findings" determined above, only persons are used who contain a text for PLAC, MAP.LATI and MAP.LONG for at least one of the events BIRT, CHR, DEAT and BURI in the ged file and who contain a determinable year in DATE for at least 1 of the 4 events. Entries in the form "DEAT Y" without date, but with PLAC are considered as a determinable year. 2. From the remaining persons, "LATI;LONG" combinations from all events are used as key fields and for each combination the listed persons are assigned with their surname, the earliest (ancestors) / latest (descendants) year and the event. 3. Within each combination, the surnames are sorted in descending order according to their frequency and within the frequency an ascending (ancestors) / descending (descendants) order according to the year is carried out. 4. The defined number (default = 10) of the most frequent surnames at each location is then listed according to the column description of the tsv file in the attachment. If "0" is entered, all names, max. 1000, are listed. Diedrich Hesmer Seite 7 von 12

8 1.5 Extract from a log file Listing during reading: Time stamp and file name. Persons not found due to incorrect references in the ged file. Non-interpretable date specifications with details of the person concerned and their ID number, so that a correction is possible. Missing or incorrect LATI/LONG data. Incorrect GOV information. Additional information is listed: The base quantity of persons (without ancestors/descendants). Number of nodes of the earth quarters. Created on: :04:09 - PrepMaps-Version: Base: "d:\ofb\1\paf\riehl\map\krasna gedcom_u_mpl.ged" from: :08:33 Modified file: "d:\ofb\1\paf\riehl\map\krasna gedcom_u_mpl_pm*.tsv" Records - INDI: , FAM: > Tag/attribute faulty => Select replacement value *** Date error *** -> 1 BIRT -> 2 DATE 16 DEC => Select: 0000 ## [Emilia /Ternes/] 3 Entries in list *** LATI/Long error *** -> 3 MAP => 4 LATI > LATI/LONG error -> not useable 1 Entries in list *** MAP missing, no LATI/Long *** -> 2 PLAC => 2 PLAC,Baden,DE-BW -> 2 PLAC => 2 PLAC Hausstädt,Bremervörde,DE-NI Entries in list *** Starting point of search Andreas, Schuck - BC: Christina, Lauber - BC: Entries in list Ancestors: 20 Persons taken into account Number of nodes: NE: 8 NW: 3 SE: 0 SW: 0 Total: 11 Descendants: 29 Persons taken into account Number of nodes: NE: 18 NW: 7 SE: 0 SW: 0 Total: :04:13 Creation completed Diedrich Hesmer Seite 8 von 12

9 2 Appendix 2.1 Copyrights, License agreements and Others This and other relevant information can be found in the appendix of the "Map-Start" description. They apply here without restriction. 2.2 Setup of the tsv-file The file contains a header line with the column headers and for each location found a data line. As tsv-file the tab (Tabulator) is used as separator. Comma, semicolon, space or others could not be used, because the data itself contains them File name The file name consists of the file name of the ged file and, depending on the selected options, the following letter additions, so that "my_file.ged" becomes "my_file_pmyz-xx.tsv", where the following applies: "pm" = PrepMaps "y" = "o" for "OpenStreetMap" or "g" for "Google Maps" "z" = "v" for ancestors (Vorfahren) or "n" for descendants (Nachkommen) "xx" = "de" for Deutsch or "en" for English The files are saved in the folder of the ged file Description of the columns Column 1/A Contains the earth quarter data for column 3/C + 4/D and is determined from the entries MAP.LATI/LONG of the ged file. Allowed are: Nord and East, South and East, Nord and West, South and West Example: Layer formation in quarters for group formation in Google Maps. North and East Latitude + N and Longitude + E North and West Latitude + N and Longitude - W South and East Latitude - S and Longitude + E South and West Latitude - S and Longitude - W There are no groups in OSM. Here this information is evaluated differently. Column 2/B Contains the exact text from PLAC of the ged file, 1 line per location, with different spelling at the same LATI;LONG coordinate, the first found. Example: Emmental,Bender,Bessarabia, Pervomaisc,Căușeni District,MDA Column 3/C Contains the specification for the latitude in the form xx.xxxxxx without leading letters and will be determined from the entries MAP.LATI of the ged file. Example: Card reference system WGS84 (decimal degree with dot and - sign) Column 4/D Contains the specification for the longitude in the form xx.xxxxxx without leading letters and will be determined from the entries MAP.LONG of the ged file. Example: Column 5/E Contains, if _GOV exists, the link of the GOV identifier to the GOV system by concatenating " and GOV. Example: Column 6/F Contains a summary of surnames that had an event in this place. The output is sorted by the frequency of occurrence of the respective surname. Example: "Andres, from 1715, 35x Diedrich Hesmer Seite 9 von 12

10 Bachmeyer, from 1812, 23x [[ Deichert]], from 1884, 9x Einbeck, from 1745, 1x Gross, from 1745, 9x Müller, from 1745, 19x Neuner, from 1745, 9x Ottersbach, from 1756, 2x Paulus, from 1795, 4x Perscheid, from 1745, 9x 9 other names, 78x " Column 7/G Contains the number of persons found with their ancestors or descendants in this location, not the number of entries. Column 8/H Contains link to birth/christening, if applicable. If option "Link places with OFB" is active, and BIRT or CHR occur in the place > 1x, then create link. Example: aisc%2cc%c4%83u%c8%99eni+district%2cmda&e=geburt&sort_by=zuname Column 9 Contains link to death/burial, if applicable. If option "Link places with OFB" is active, and DEAT or BURI occur in the place > 1x, then create link. Example: aisc%2cc%c4%83u%c8%99eni+district%2cmda&e=tod&sort_by=zuname Diedrich Hesmer Seite 10 von 12

11 2.2.3 Examples of tsv-files Structure of a tsv file created by PrepMaps for OSM Structure of a tsv file created by PrepMaps for Google Maps Diedrich Hesmer Seite 11 von 12

12 2.3 Processable MAP + _GOV Structures of ged-files MAP with LATI and LONG (Coordinates) and _GOV (Identifier) are exported differently by genealogy programs, if processed at all. To my knowledge, there are four types of export. Each of these types can be processed with PrepMaps, but not in all combinations within a ged file. MAP, LATI and LONG are standard tags that are processed by a larger number of programs. _GOV is a user-defined tag that can only be processed by a smaller number of programs. PrepMaps first searches for the default structure. If this is not found, the other structures are searched. The structures listed are exemplary and contain only the tags concerned and their references, if any Standard Structure 1 BIRT 2 PLAC Brosowo, Kulm, Bromberg, Danzig-Westpreussen, Deutsches Reich 3 MAP 4 LATI N LONG E _GOV BROOWOJO93FH (only sporadically) Used among others by Program Ahnenblatt, Ages!, Brother's Keeper, FinFamily, FTM, Gen_Plus, GEN_DO!, Gramps, WebTrees "_LOC Records" 1 BIRT 2 PLAC Brosowo, Kulm, Bromberg, Danzig-Westpreussen, Deutsches Reich 3 (Reference to a _LOC record) Und dann im _LOC Datensatz _LOC 1 NAME Brosowo 1 MAP 2 LATI N LONG E _GOV BROOWOJO93FH Used among others by Program Ahnenblatt, Ages!, Familienbande, GEN_DO!, GES2xxx; PC-Ahnen (only _GOV) "0 _PLAC" 1 BIRT 2 PLAC Baden bei Wien, Niederösterreich, Österreich Und dann unter "0 _PLAC" 0 _PLAC Baden bei Wien, Niederösterreich, Österreich 1 MAP 2 LATI LONG _GOV BADDENJN88CA (only TNG) Used by Program Mac Stammbaum, Mobile Family Tree, Reunion, TNG "0 _PLAC_DEFN" 1 DEAT 2 PLAC Kirchheim unter Teck Und dann unter "0 _PLAC_DEFN" 0 _PLAC_DEFN 1 PLAC Kirchheim unter Teck 2 MAP 3 LATI N LONG E Used by Program Legacy no _GOV Programs without MAP, LATI, LONG and _GOV "PrepMaps" requires at least the data of the tags LATI and LONG in the ged file. If these are missing, as for example with PAF, then these and optionally also _GOV can be added in the standard form with the GedMap program "ModPlac" for every PLAC in INDI and FAM records. In addition to the program, a corresponding reference file must be created manually with a spreadsheet as a tsv file (tabulator as separation character). Details can be found in the description "ModPlac_e.pdf". Diedrich Hesmer Seite 12 von 12