eha qgis-versioning Documentation Release documentation

Transcription

1 eha qgis-versioning Documentation Release documentation February 24, 2016

2

3 Contents 1 Introduction Summary Requirements Hard requirements Soft requirements Plugin functionality In a nutshell Typical workflow Plugin group types in pictures Database artifacts Selecting features before checkout Procedure When problems occur Conflict management Errors and exceptions Inner workings Central database Working copy database Project contributors SC Apavil Ramnicu Valcea Oslandia ehealth Africa Other i

4 ii

5 This is the temporary location of the qgis-versioning plugin documentation. Click this link for the original page. Date Last generated on at 13:43 Contents 1

6 2 Contents

7 CHAPTER 1 Introduction 1.1 Summary The qgis-versioning plugin provides versioning of geographical features stored in PostGIS databases. A typical use case met by the plugin involves one or more users checking out a local working copy to edit features in (potentially) offline mode using SpatiaLite and committing changes back to the PostGIS server. More information can be found in the plugin s original page page. Note: Most of the screenshots contained in this document were made on an Ubuntu workstation. A few ones were made on a Windows workstation. 3

8 4 Chapter 1. Introduction

9 CHAPTER 2 Requirements This section is a list of both hard and soft requirements (aka suggestions). 2.1 Hard requirements Requirements considered hard are those you cannot really live without. The plugin may work but you may experience problems to make it work. The two basic hard requirements are about QGIS and PostgreSQL/PostGIS. Another pertains to naming conventions. In a nutshell, those hard requirements are : QGIS 2.8+ PG 9.x (ideally 9.2+) names asked by he plugin must begin with either a _ (underscore) or a letter followed by any lowercase non accented letter, digit or underscore without spaces and up to 63 characters long QGIS Recent versions of the plugin were tested with QGIS 2.8. It is worth mentioning that plugin versions < 0.2 were developed for older versions of QGIS, which may incidentally ship with older versions of spatialite/sqlite. As of version 0.2 (Aug 2015), spatialite version 4.x is supported by the plugin. This in turn makes any QGIS versions that come with older spatialite versions unusable with the plugin (for spatialite checkouts at least). Another key dependency of the plugin is ogr2ogr. Although the plugin does not depend on the most recent features of ogr2ogr, it is wise to stick to the version bundled in QGIS 2.8+ (2.8+ because newer versions of ogr2ogr shipped in newer versions of QGIS should be backwards compatible) PostgreSQL/PostGIS The plugin should work on any minor revision of the PostgreSQL 9.x series. It was tested successfully on PostgreSQL 9.2 and 9.4. There are no hard requirements on PostGIS as such on the part of the plugin. Packing the most recent version supported in your PostgreSQL installation should be sufficient. 5

10 2.1.3 Naming conventions Operation of the plugin is best ensured by sticking to the PostgreSQL naming rules. As suggested here : PostgreSQL uses a single data type to define all object names: the name type. A value of type name is a string of 63 or fewer characters. A name must start with a letter or an underscore; the rest of the string can contain letters, digits, and underscores. Warning: Do NOT use empty spaces in any identifier the plugin asks you to supply. Do NOT use accented characters (e.g. German umlaut or French accent aigu ) For optimal operation, avoid using spaces in the full path name of files intended to be used by the plugin, for example spatialite files. Even though PostgreSQL object names can contain capital letters, the plugin does not currently support object names other than in lowercase letters (plus digits and underscores as mentioned above). Even though the plugin ensures some level of protection in that respect, it is best to stick to those conventions when naming a new PG checkout (see later for an explanation), a branch or any other name the plugin asks you to provide. The same applies to spatialite filenames (SL checkout). 2.2 Soft requirements Soft requirements are more like best practice suggestions. As the saying goes : Your Mileage May Vary Separate schema As will be explained in more detail later in this document, the qgis-versioning plugin operates historization by adding columns to each table in a particular database together with a revisions table that holds all revision information. For a number of reasons, it is wise to isolate your geographic data in a schema other than the public schema. As mentioned here :... store no data in the public schema. The specific context of the previous quote pertains to backup and restore procedures but the advice also applies for the qgis-versioning plugin. 6 Chapter 2. Requirements

11 CHAPTER 3 Plugin functionality 3.1 In a nutshell Depending on the type of layer group (unversioned/versioned PostGIS database, PostGIS/Spatialite working copy) the qgis-versioning plugin provides a different set of functionalities, as summarized in the following table. Icon Unversioned Versioned Working copy (PG/SL) Definition X X X Branching Checkout Spatialite (SL) Checkout PostGIS (PG) X Commit changes X X X Help X Start versioning (historize) X Check if working copy up to date X View revisions The following table shows the combination of plugin text information and icons as a function of group type selected in the QGIS legend. 7

12 Group type Menu Comments No group No group item selected in legend Unversioned Only option : historize Versioned Textinfo : DB + schema + branch= + rev= Working Copy (SL) Working copy (PG) Mixed layers Same Name Textinfo : filename + working + rev= Textinfo : DB + schema + working rev= Layers in group do not share the same database or schema Groups must have different names Empty group Selected group is empty 3.2 Typical workflow The following sections present how the qgis-versioning plugin is used in QGIS. A later section will detail what the plugin does in the PostgreSQL database Unversioned database The sequence of operations begins with data in a PostGIS (PG) database that is loaded in a QGIS layer group. Note: The qgis-versioning plugin operates on QGIS layer groups, not on individual layers. At that stage the data is unversioned and the only option on the layer group, except for help, is to historize the database in the current schema. 8 Chapter 3. Plugin functionality

13 Clicking on the historize button ( ) will generate a warning from the plugin that four new columns will be added to all tables in the selected database schema. As the warning mentions, all datasets in the database will be versioned even though a subset of datasets (tables) was initially selected by the user to be loaded as layers in QGIS. Upon accepting, the plugin creates a versions table in the formerly unversioned database schema, creates a new schema by appending _trunk_rev_head to the current schema name, indicating the original schema now has one line of versioning called trunk, and loads the selected layers in a new QGIS group. That is the begining of the versioning journey. Note: The symobology of the original layers, if any, is not carried over to the versioned database layer group Versioned database The plugin menu for a versioned layer group shows 5 icons. The space-separated text items on the left identify four components : name of database name of schema name of branch (initial branch has default name trunk) revision number (or head if latest) Three operations can be performed on a versioned layer group : 1. checking out a working copy (PG or SL) 2. viewing specific revisions 3. branching Checking out a working copy Checking out a working copy creates one of two new layer groups, either a PG checkout ( ) or a spatialite checkout ( ) : 3.2. Typical workflow 9

14 a spatialite layer group, called working copy (or named with the full path name of the spatialite file created if a group called working copy already exists) is created in the legend a PG layer group, the name of which is made up of the user provided schema name In both cases, properties of individual layers in the groups will clearly show provenance (spatialite filename on the file system for spatialite checkouts or schema. view_name for PG checkouts). Working copies The qgis-versioning plugin allows users to work on two types of working copies : PostGIS (PG) and Spatialite (SL). Once a database is versioned, users can checkout a working copy in either PostGIS or Spatialite formats. In the former case, a new schema is created on the PostGIS server with the features for the selected layers in SQL views. In the latter, a local Spatialite file is created. SL working copies allow users to work offline, since the database is held in a file on the local machine. PG working copies require a live access to the central database. A peculiarity of spatialite working copies is that features can be selected prior to checking out as explained in Selecting features before checkout. Working copies can also be updated with changes committed by other users in the central PG database. The following image shows the three icons found in the menu bar for a working copy for : spatialite PostGIS On the left, either the name of the schema in the central database (for PG checkout) or the name of the spatialite file appears together with the current working revision number. Note: The working rev number equals the latest revision number in the database (head) plus one. The reason is that when work will be committed back to the central database it will be committed to revision = working rev if no other commits were made by other users in the meantime or whichever revision number > working rev. The two basic operations that can be performed on a working copy are to either update ( ) or to commit changes ( ). Updating implies synchronizing the current working copy with the central database. If the current working copy is not synchronized (late) with the central database, data is uploaded to the working copy. Data is either integrated directly in the working copy or a conflict resolution workflow is launched in the event local edits conflict with database revisions newer than that at which the working copy was initially checked out. As the name implies, committing involves uploading the changes to the central database where they will be integrated and the revisions table updated with a new record. 10 Chapter 3. Plugin functionality

15 Viewing revisions The view icon ( ) shows the contents of the revisions table stored in the schema that was originally versioned. The user can select one or more revisions by clicking the checkbox before the revision number. Selecting one or more revision numbers will result in one group per revision being created in the QGIS legend tree. Each of those groups show all layers at the specific revision number. The default name of those groups is branch_name + revision + the revision number, for example trunk revision 1. At the top of the dialog, a checkbox called Compare selected revisions gets enabled when two revisions are checked, allowing the user to compare between the two revisions. Instead of getting two layer groups each containing all features, clicking on that checkbox creates a single group called Compare revisions X vs Y (with X < Y) where features that differ between the two compared revisions are given a rule-based symbology as a function of whether they were created, updated or deleted Typical workflow 11

16 A special case named intermediate identifies features that are transient items. An example would be a point feature that would have been moved between the two revisions but which has a parent feature in revision X and a child feature in revision Y. Note: The Compare selected revisions checkbox is automatically unchecked and disabled if the number of selected revisions is not equal to two (2). Branching Branching involves the creation of a new schema in the database. The new schema becomes another line of versioning of the original schema. Note: Even though branches are to hold independent versioning histories, they still result in a commit in the revisions table. 3.3 Plugin group types in pictures The following screenshots illustrate the various layer groups produced by the qgis-versioning plugin. Note that default group names (e.g. working copy for spatialite checkout) were modified to be more expressive. Unversioned database 12 Chapter 3. Plugin functionality

17 Versioned database Note: The reason why there are fewer points for the versioned database has to do with feature duplication/multiplication in the original table. See Committing data for an illustration between the original table contents and the view generated by the historization process. Spatialite checkout 3.3. Plugin group types in pictures 13

18 PostGIS checkout Branch 14 Chapter 3. Plugin functionality

19 View revision (full mode) Note: The leftmost point at the top shows that at that revision (here 1) the feature was there. It is not in the head view provided by the versioned database view but it shows in the unversioned database because features are added continusouly in the tables once historization has begun. See Committing data for details. View revision (diff or comparison mode) 3.3. Plugin group types in pictures 15

20 3.4 Database artifacts This section shows database artifacts created by the plugin and discusses users permissions in the context of the plugin Basic workflow : historization First, we begin with an existing database and schema (in this case epanet_test_db and epanet, respectively) with two geodata tables : junctions (points) and pipes (lines). 16 Chapter 3. Plugin functionality

21 Historizing the database will add two things : 1. An additional table called revisions to the original schema 2. An additional schema called epanet_trunk_rev_head with N views (N being equal to the number of tables in the original schema) 3.4. Database artifacts 17

22 The additional schema s name is made up of the original schema plus the name of the branch, which defaults to trunk for the first commit brought about by the historization process, plus the revision number, in this case rev_head because that schema hold the latest (head) revision. Once the database is versioned, branches other than trunk can be added. Clicking on a text box to supply the name of a new branch. Here is the result for the name mybranch. will present the user with 18 Chapter 3. Plugin functionality

23 Note the added schema, called epanet_mybranch_rev_head, which also contains two views of the original dataset. After adding the mybranch branch, the revisions table now shows two entries : one for the initial commit (historization) and another one for the new branch Database artifacts 19

24 3.4.2 Committing data After the database is versioned, users can start checking out working copies and edit features. When they are happy with their edits, they may commit their changes to create a new revision in the branch they have checked out from (usually trunk ). This is the pop up windows that appears at commit time : The committer now has the option to choose another PG username if integrating data from another contributor. See user permissions for more detailed explanations of typical user roles in PostgreSQl. After clicking OK and giving some time for the upload to get through, the user should see this : Let us look at an example. 20 Chapter 3. Plugin functionality

25 Say a user edits the pipes dataset and commits the results a total of 4 times including the initial commit. The revisions table will show 4 records. If we look at the pipes view in the trunk_rev_head schema, we see three features : Fig. 3.1: Pipes view Looking at the corresponding pipes table in the original schema, we find the whole story : Fig. 3.2: Pipes table Notice how the view only shows those features (pid = 7,8,9) beginning at the latest (HEAD) revision, number 4 in our case. The table shows the details of the versioning history of the three features at HEAD revision : pid 1 started at rev 1 and ended at 2; it was edited to become pid 4 pid 4 shows its parent is pid 1; it existed only at rev 3 to become pid 7 pid 7 shows its parent is pid 4; it shows neither trunk_rev_end nor trunk_child, so it is at HEAD (latest revision) pid 2 existed at rev 2, then it was edited to become pid 5 pid 5 shows its parent is pid 2; it existed only at revision 3 and became pid 8 pid 8 shows its parent is pid 5; like pid 7, it is at HEAD revision pid 3 existed at rev 2, where it was edited to become pid 6 pid 6 shows its parent is pid 3; it existed only at revision 3 and became pid 9 pid 9 shows its parent is pid 6; like pid 7 and 8, it is at HEAD revision More details can be found in the Inner workings section Database artifacts 21

26 3.4.3 User permissions considerations Users of the plugin in QGIS must be able to create objects in the PostgreSQL database, like tables (e.g. the revisions table), schemas and views (e.g. to create new branches or PG checkouts). The plugin requires database users to be owners of all tables in the source schema. This is best achieved by configuring a group role with write access to the source schema as owner of all tables and put individual users in that group. In the more recent versions of the plugin, another type of user with read-only operations may be used in the commit workflow. In a typical organization, some users will be allowed to commit changes to the database while others may only either view the data or submit modifications of their own (e.g. in the form of a spatialite file) to a committer. In the latter case, the committer has the opportunity to identify someone else as data editor at commit time. This ensures some level of traceability as to who edited features and who committed (or approved or further modifed before final commit). For this to be possible, all potential data editors must have a corresponding username in the database and ideally be associated with a read-only group role. Although the plugin does not enforce a non null Author field in the revisions table (e.g. in tests), the current default behaviour of the plugin adds the following information in the Author field : OS:OS_username.committer_pg_username.[read-only]_pg_username OS specifies the committer s operating system (e.g. Linux, MacOS , Windows7 ). OS_username is the username of the committer on his (her) workstation. committer_pg_username defaults to the committers user name in PostgreSQL (not editable at commit) and [read-only]_pg_username is the bit of information in the the Author field that may be selected by the committer. It defaults to the current committers pg username. We use square brackets to indicate that the committer may also select committer-level users as data editors. 22 Chapter 3. Plugin functionality

27 CHAPTER 4 Selecting features before checkout Before checking out a local spatialite working copy, one can select features from the versioned tables to work on locally. Any layer in the group can have features selected. In the case of a layer with no features selected, the whole dataset will be checked out by the qgis-versioning plugin. This allows users to select only those features they are interested in editing rather than the whole dataset. Note: Feature selection prior to checkout only applies to spatialite checkouts. It has yet to be implemented for PG checkouts. 4.1 Procedure For each layer in the group, select features you want checked out. This can be done in a number of ways in QGIS. When ready to checkout, click on the layer group and click on the spatialite checkout button ( ). At that point any layer with selected features will pop this warning to let the user know a subset of features will be checked out : 23

28 Complete the rest of the default spatialite checkout workflow and check that only a subset of features was retrieved for the layers you selected features for. In our example, only the selected polygons (yellow above) and all points (since no feature selection was performed on the point layer) were checked out : 24 Chapter 4. Selecting features before checkout

29 CHAPTER 5 When problems occur Multiple modifications to the same dataset amongst a number of users inevitably bring about conflicts. This section shows how editing conflicts are managed by the plugin. Specific errors or exceptions are also mentioned. 5.1 Conflict management For users to be able to commit modifications to the database, their working copy must be up to date with the database. Take the example of two users checking out SL working copies from HEAD revision X. Each on their workstations, they are editing features in their working copy. One of the users will inevitably commit his(her) changes to the database first. This will create another entry in the revisions table, that is increment the HEAD revision to X + 1. When the second user tries to commit, a message will warn that the working copy needs to be updated : Note: It is wise to always update a working copy before trying to commit. Updating directly by clicking will check if the working copy is up to date. If it is, a message will let you know : 25

30 Else, the working copy is updated to the latest revision, for example If edits made by the first committer do not conflict with the second user s working copy, updating will proceed normally. Changes made by the other user will be merged into the second user s working copy. All is well and the second user can now commit his(her) changes. In the event there are conflicting edits, the second user will be presented with this message after updating : A new layer is created by the plugin for every dataset that contain errors and it is displayed in the working copy layer group. The name of that layer is made up of the original layer name to which the _conflicts string is appended : 26 Chapter 5. When problems occur

31 The figure above shows two conflicting features in one layer. Highlighted is the edit the second user is trying to commit. The attributes table shows the id of the conflicting feature. The conflict is resolved by deleting the unwanted entry in the conflict layer, either mine or theirs and saving the edits. On deletion of mine, the working copy edition is discarded; on deletion of theirs the working copy edition is appended to the feature history (i.e. the working copy feature becomes a child of the last state of the feature in the historized database). Once the conflict table is empty, committing can proceed. Note: On deletion of one conflict entry, both entries are removed (by a trigger) but the attribute table (and map canvas) are not refreshed. As a workaround, the user can close and re-open the attribute table to see the actual state of the conflict table. Since deleting mine implies discarding one s own change, then committing will result in no change being committed. A useful tip if you deleted the wrong one (e.g. theirs and you meant to delete mine ) and did not save yet : CTRL-Z (undo) to the rescue. In the more general case, multiple editions can be made to the same feature. Therefore child relations must be followed to the last one in order to present the user with the latest state of a given conflicting feature. Warning: Known bug : Updating a working copy may indicate it is up to date when in fact it is not. This may happen if for example the checkout (working copy creation) was done on trunk and then another branch was created afterwards. The working copy gets stuck at the latest revision of the branch it was checked out from. The only way to get around this is to checkout a fresh working copy from the desired branch. Edits made in the other working copy are still there and need to be integrated manually in the most recent working copy. Tip : clicking on the view icon will tell you what the latest revision number is. 5.2 Errors and exceptions Although errors are generally managed within the plugin, specific circumstances may trigger errors or Pyhton exceptions. This section shows some of those errors and how they can be avoided or recovered from Errors and exceptions 27

32 28 Chapter 5. When problems occur

33 CHAPTER 6 Inner workings This section is intended to explain more technical aspects of the versioning implemented by the qgis-versioning plugin. 6.1 Central database Once a database is versioned, three operations can be performed on table rows: INSERT, DELETE and UPDATE. To be able to track history, every row is kept in the tables. Deleted rows are marked as such and updated rows are a combined insertion-deletion where the deleted and added rows are linked to one another as parent and child. A total of five columns are needed for versioning the first branch: PRIMARY KEY a unique identifier across the table branch_rev_begin revision when this record was inserted branch_rev_end last revision for which this record exists (i.e. revision when it was deleted minus one) branch_parent in case the row has been inserted as a result of an update, this field stores the id of the row that has been updated branch_child in case the row has been marked as deleted as a result of an update, this field stores the id of the row that has been inserted in its place. For each additional branch, four additional columns are needed (the ones with the prefix branch_ ). Note: A null value for branch_rev_begin means that a row (feature) does not belong to that branch. SQL views are used to see a snapshot of the database for a given revision number. Noting rev the revision we want to see, the condition for a row to be present in the view is: (branch_rev_end IS NULL OR branch_rev_end >= rev) AND branch_rev_begin <= rev In the special case of the latest revision, or head revision, the condition reads: branch_rev_end IS NULL AND branch_rev_begin IS NOT NULL Note: Since elements are not deleted (but merely marked as such) in an historized table, care must be taken with the definition of constraints, in particular the conceptual unicity of a field values. 29

34 Views for revisions must be read-only and historized tables should never be edited directly. This is a basic principle for version control : editions must be made to working copies an then committed to the database. Please note that by default PostGIS 9.3 creates updatable views. 6.2 Working copy database For each versioned table in the working copy, a view is created with the suffix _view (e.g. mytable_view). Those views typically filter out the historization columns and show the head revision. A set of triggers is defined to allow operating on those views (DELETE, UPDATE and INSERT). The DELETE trigger simply marks the end revision of a given record. The INSERT trigger creates a new record and fills the branch_rev_begin field. The UPDATE trigger creates a new record and fills the branch_rev_begin and branch_parent fields. It then marks the parent record as deleted, and fills the branch_rev_end and branch_child fields accordingly. 30 Chapter 6. Inner workings

35 CHAPTER 7 Project contributors The following organizations have played a key role in the development of the qgis-versioning plugin. 7.1 SC Apavil Ramnicu Valcea The GIS Office of SC Apavil Ramnicu Valcea, Romania initially funded the development of the plugin through a European development project. 7.2 Oslandia Oslandia is the initial developer of the plugin and the official maintainer on GitHub. 7.3 ehealth Africa ehealth Africa joined the development effort in Fall 2015 and has contributed several enhancements since. 7.4 Other List of individual code contributors. 31