Introduction to GDAL/OGR

Transcription

1 Introduction to GDAL/OGR Johannes van der Kwast UNESCO-IHE Institute for Water Education Version 3.1, September 2015 OpenCourseWare 1

2 Contents 1 Introduction Learning objectives Prerequisites Installation Using the command line Opening a command prompt Viewing the contents of a folder Changing directories Creating files Copy, move, rename and delete files Command history GDAL - The Geospatial Data Abstraction Library Retrieving information from GIS data Raster data Vector data Reprojecting files Introduction Raster data Vector data Change raster properties Resize an image Change raster format Spatial queries of vectors Convert Comma Separated Files Batch conversion

3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first part of these exercises introduce you to the DOS command prompt. If you are already familiar with DOS commands, you can skip chapter 3 and start immediately with chapter 4. These exercises are provided as OpenCourseWare and come without support. For support, please register to our related short courses that can be found at Learning objectives After these exercises you will be able to: Use the DOS prompt Understand files and directories and their relative and absolute paths Retrieve information (metadata) from raster and vector data Convert raster and vector data formats Reproject raster and vector data formats Change raster properties Perform spatial queries on vector layers Convert comma separated files Convert spatial data to Google KML. 2 Prerequisites 2.1 Installation For these exercises GDAL needs to be installed, preferably using the OSGEO4W distribution package. The GDAL website: The OSGEO4W website: 3

4 3 Using the command line During the excersises we are going to use the Command Prompt. If you already know how to use the the Command Prompt (aka DOS prompt), you can skip this section. 3.1 Opening a command prompt To open a command prompt you can go do: Start All Programs OSGeo4W OSGeo4W Shell Now you will see the Command Prompt on your screen: 4

5 C:\> is called the command prompt. It shows your current working directory (or folder). In this case it means that you are at drive C (which is called the root). You cannot delete or edit the command prompt. You'll have to type the commands after the > of the command prompt. How does your command prompt look like? Give drive name, directory and (sub)directories. It may differ from the example in the screenshot. 3.2 Viewing the contents of a folder In this section, you will view the contents of a directory by using the dir command. The dir command stands for "directory." Type the following command after the prompt: dir After typing a command press the <Enter> key to execute. You will see something similar to this: This is called a directory list. A directory list is a list of all the files and subdirectories that a directory contains. If you type dir /? you will see all options of the dir command. For example, if you type dir /w you will see the directory listing in columns. 5

6 3.3 Changing directories Look at the list on your screen. All the names that have <DIR> beside them are directories (or folders). The others are files. The size of the files is also displayed. You can see a list of the files in another directory by changing to that directory, and then using the dir command again. Type the following commands: md John md Peter These commands create subdirectories called John and Peter. Check if the directory has been created by listing the directories. Type: dir Also try: dir /AD that only lists the directories Now to go to the newly created directory "John" by typing cd John cd means "change directory". You will see that the command prompt changes to C:\...\John> 6

7 DOS is not case sensitive (it interprets upper and lowercase in a similar way), so you could also have typed cd john. List the contents of the directory John by typing dir Now you'll see that the directory is not empty. There are to directories, namely. and.. Actually, the directory is empty, but it also shows the relative paths:. means "current directory".. means "mother directory (or one directory up)" Type: cd.. Check the command prompt: we're back in the directory where we started the exercise. Now type : cd. What is the result of this command? Explain why. We can go back to directory John using the relative path: cd.\john Note that this is the same as cd John. 7

8 Or the absolute path (in my case): cd C:\Users\jkw\John If we want to go to the root (drive C) we can do it using the absolute path: cd \ or the relative path: cd..\..\.. Go back to the directory called John. You can choose if you want to use the relative or absolute path. Type cd..\peter Look at the prompt and explain what this command did. Both ways of changing directories are equivalent. Why bother than? Well, if you move your files that are organized in directories and subdirecties to another drive (e.g. drive D), than your absolute paths will not work, while your relative paths will still refer to the same locations. This is specifically important when writing scripts, which we'll do later. Now we're going to delete the directory called Peter. First move one directory up, type: cd.. than type: rd Peter rd means "remove directory". List the directory to check if Peter has been removed. If you want to change to another drive, e.g. drive D, you use this command: d: Now go back to c by typing: c: You'll be back in the last directory accessed on the C drive. Move back to the directory where you created John before continuing with the next exercise. 3.4 Creating files Of course you can create files using your Windows software, but since you're learning the command prompt you'll learn how to create a simple ASCII text file. 8

9 Type: copy con listdirectories.bat Type: dir /AD Type <CTRL>-<Z> (keep the control button pressed and type z) Your screen looks like this: Check if the file is in the directory list. What is the size of the file? Now display the contents of the file on the screen. Type: type listdirectories.bat Now type: listdirectories Congratulations! You have created your first batch file. Batch files are scripts that can be used to execute commands in batch. In this case the file executed dir /AD Because batch files always have the file extension.bat, the computer knows that this is a batch file and will execute the commands in the file. We'll come back to this later. Let's create another file. Type: dir > list.txt This command will not show the result of the dir command on the screen, but saves it to an ASCII file called list.txt. So we can use > after a command to save its results to a file instead of printing it to the screen. Check the contents of the file: type list.txt 9

10 If the list is larger than the Command Prompt screen, you have to scroll. With large text files it is easier to use: type list.txt more The result of list.txt is given to the more command which displays the results in pages as big as your window. Press <ENTER> to see the next line. Press <SPACE BAR> to see the next page. Press <CTRL-C> to stop. You can use this last key combination to stop any command if it isn't doing what you like, it crashed or it takes too long. Type: more Nothing happens, because the more command expects input from another command. So you can wait forever. In this case you can stop the execution of the more command by using <CTRL-C>. Now try this: type listdirectories.bat >> list.txt Display the result: type list.txt more What happened? In summary: > saves the result of a command to a new file. If the file on the right hand of > already exists, it will be overwritten. >> appends the result of a command to an existing ASCII file. uses the result of the command on the left hand side in the command on the right hand side of the. This is called a pipe. Use <CTRL-C> to stop the execution of a command. We can also use these operators to create so called lock files. These are used in scripting. The script will check if a certain file exists. If it exists, the program will wait, if it is removed, the program continues. Therefore, these files can be empty. You can create a lockfile with: type NUL > lockfile.txt 10

11 3.5 Copy, move, rename and delete files Now we can copy the file list.txt to a new file by typing: copy list.txt newlist.txt We can also copy list.txt to subdirectory John: copy list.txt john\newlist.txt Now we move newlist.txt to the directory John: move newlist.txt John Go to the subdirectory John and check the directory listing. Because it is a bit confusing to have a copy of list.txt in subdirectory John, we're going to rename it: rename list.txt listjohn.txt Check the directory listing again. Save the directory listing to a file called dirjohn.lst Now I'll introduce so called wildcards. Type dir Type dir *.txt dir *john.* dir *.??t dir *.?xt Explain the function of * and? We can also append textfiles: copy dirjohn.lst+listjohn.txt listappend.txt Because we made copies, we want to remove duplicate file. Type: del newlist.txt You can now remove all files by typing: (Make sure you are in the right folder!!) del *.* (or simply: del.) 11

12 Now remove the directory John. 3.6 Command history Sometimes you often use the same command. There are several tricks to type them more efficiently. Type <F3> This repeats the last command used. Clear the command prompt and press the right arrow button several times. You can see that the characters of the previous command are repeated. If you press the up and down buttons, you can browse through the previously used command. When you use <TAB> while typing a path or a filename, it will try to automatically complete it. With the doskey command we can do even more. Type doskey /h This prints all the commands you typed during this session to the screen. If you close the command prompt, the command history will be lost. Save the command history to a text file using one of the commands previously learned. In this way you can edit the command history file in e.g. notepad. If you remove all the wrong commands and you save the file with the.bat extension, you can execute all of the commands in batch. Try this for a few of the commands you have learned so far. You can close a command prompt either by clicking on the cross in the corner, typing exit and pressing enter, or choosing Close when right clicking the Command Prompt icon on the task bar. 12

13 4 GDAL - The Geospatial Data Abstraction Library GDAL is a translator library for raster geospatial data formats that is released under an X/MIT style Open Source license by the Open Source Geospatial Foundation. As a library, it presents a single abstract data model to the calling application for all supported formats. It also comes with a variety of useful commandline utilities for data translation and processing. See for more info: Download the GDAL using the OSGeo4W installer: Download the dataset for this exercise and save it to your harddrive e.g. D:\gdalExercises The folder contains freely available GIS data: roads.shp: srtm_37_02.tif: gem_2011_gn1.shp: road map from OpenStreetMap ( tile of a Digital Elevation Model (DEM) from the Shuttle Radar Topography Mission (SRTM) ( borders of Dutch communities, freely available from CBS (Statistics Netherlands) and Kadaster ( Before continuing, make sure that you are in the right folder. Open the OSGeo4W Shell (Start All Programs OSGeo4W OSGeo4W Shell) Change directory to D:\gdalExercises 13

14 5 Retrieving information from GIS data 5.1 Raster data One of the easiest and most useful commands in GDAL is gdalinfo. When given an image as an argument, it retrieves and prints all relevant information that is known about the file. This is especially useful if the image contains additional tag data, as is the case with TIF files. When working with satellite imagery, this is an extremely useful way of keeping track of the images location in long./lat. coordinates as well as the image projection. Execute the following command: gdalinfo srtm_37_02.tif What is the size of the image? What is the coordinate system? What is the EPSG code? 5.2 Vector data Sometimes similar information is needed from a vector image, for this there is ogrinfo. Execute the following commands: ogrinfo -al roads.shp more ogrinfo -al gem_2011_gn1.shp more What are the coordinate systems of these shapefiles? Lookup the EPSG codes of both shapefiles at You'll need this later. There exists a useful online tool to convert.prj format (as included by ESRI files) to EPSG code: 14

15 6 Reprojecting files 6.1 Introduction In this exercise we want to make a map of the community of Delft with the main roads and the relief. Because the datasets are in different formats we have to reproject them to a common coordinate system. Here we reproject all datasets to the Dutch Amersfoort/RD New projection. 6.2 Raster data GDAL has the capability to change a raster coordinate system using the following syntax: gdalwarp -t_srs EPSG:... <input> <output> The -t_srs argument specifies the target coordinate system. If the source coordinate system is unknown it must be specified with the -s_srs argument. EPSG:... specifies the EPSG code of the projection. We are now going to reproject a Digital Elevation Model (DEM) acquired by the Shuttle Radar Topography Mission (SRTM). You can use this website to download DEM's for your area of interest: In order to reproject the DEM from WGS-84 lat/lon to Amersfoort/RD New we use this command: gdalwarp -t_srs EPSG:XXXXX srtm_37_02.tif dem_rd.tif Replace XXXXX with the proper EPSG code for Amersfoort/RD New (see one of your previous answers using ogrinfo). Execute the command and visualize the result in QGIS. 6.3 Vector data For vector data again ogr is used to convert the OpenStreetMaps road data to the Amersfoort/RD New projection. Execute: ogr2ogr -t_srs EPSG:XXXXX roadsreprojected.shp roads.shp Replace XXXXX with the proper EPSG code. Note that with ogr the output filename is typed before the input filename! With gdal and most other tools this is different. 15

16 7 Change raster properties 7.1 Resize an image The gdal_translate command gives us several options to modify raster images. gdal_translate can be used simply to change the size of an image using the -outsize parameter, which takes two integer values as the xsize and ysize respectively, or two percentage values to scale the image. The syntax is: gdal_translate -outsize newxsize newysize inputfile outputfile Let's apply it to our DEM: gdal_translate -outsize 15% 15% dem_rd.tif resized.tif Visualize the original image and the resized image in QGIS. Explain what happened. 7.2 Change raster format gdal_translate's primary function is to change between image formats. Resizing and changing the image format can also be combined into one step by using both the -outsize and -of parameters. The basic syntax is: gdal_translate -of FORMAT inputfile outputfile All supported formats can be found here: Now we are going to convert the DEM from geotiff to PCRaster format. PCRaster is open source software for spatial dynamic modelling and has its own GIS format ( Execute: gdal_translate -of PCRaster -ot Float64 dem_rd.tif dem.map Visualize the result in QGIS The -ot argument is needed to specifiy that it is continuous data, so PCRaster will interpret the map as a scalar (continuous) data layer. Use -ot Int32 to convert to integer (thematic) maps. Use -ot Byte to convert to a boolean PCRaster map. 16

17 8 Spatial queries of vectors For our map of Delft we want to do the following GIS analysis: Select the community of Delft from the community map and save it into a new shapefile; Intersect the community boundaries of Delft with the road map of the Netherlands. We can use a spatial query to select a feature from a vector map. What is the attribute in the community map containing the names of the communities? You can use either ogrinfo or QGIS to answer this question. Execute the following command: ogr2ogr -f "ESRI Shapefile" -where GM_NAAM='Delft' -a_srs EPSG:28992 delft.shp gem_2011_gn1.shp This will save the feature with GM_NAAM Delft to a new file called delft.shp. The argument - a_srs EPSG:28992 is used to assign the Amersfoort/RD New projection to the output file. The argument -f defines the output format. Now open in QGIS the reprojected DEM (dem_rd.tif), the reprojected road map (roadsreprojected.shp) and the community of Delft (delft.shp). 17

18 9 Convert Comma Separated Files Sometimes you want to reproject coordinates in an ASCII file, e.g. which has been saved in a spreadsheet program. Here we will convert the coordinates in a comma separated ASCII file (locations.csv) to a new ASCII file (locations_reprojected.csv). View the contents of locations.csv using notepad First, we have to create a virtual data source by creating an XML control file. On your windows computer open notepad and type the XML code in the screenshot below. Use indentations of three spaces. Save the file as "locations.vrt" in the gdalexercises folder. Some explanation about the XML file: <OGRVRTLayer name="locations"> should correspond with the <SrcDataSource>locations.csv</SrcDataSource> <GeometryField encoding="pointfromcolumns" x="lon" y="lat"/> indicates the columns with the coordinates that you want to convert. Execute the following command: ogr2ogr -t_srs EPSG: f "CSV" locations_reprojected locations.vrt -lco GEOMETRY=AS_XY In this example locations.csv with lat/lon WGS-84 coordinates is converted to locations_projected.csv with Amersfoort/RD New projection. Note that the file is saved in the folder with the same name as the output file. Use notepad to check locations_reprojected.csv. What is saved in each column? In the same way we can convert the comma separated file to a shapefile. Execute: ogr2ogr -f "ESRI Shapefile" locations.shp locations.vrt Visualize the shapefile in QGIS by plotting the locations over the DEM, road map and Delft community border. Make a nice map. We can also convert the locations to Google Earth KML format. Type: ogr2ogr -f KML locations.kml locations.vrt locations 18

19 Note that you don't need to specify an output projection (-t_srs), because for KML this is always WGS 84 (EPSG:4326). Visualize the results in Google Earth in Windows. If you don't have Google Earth yet, you can download it from: Double click the file and it opens in Google Earth. What are the objects in our CSV file? 19

20 10 Batch conversion Desktop GIS programmes are very useful for GIS operations, but are hard to use if we have to repeat the same task for many GIS layers. Then scripting can be a solution. Here we have an example dataset from a land-use model of Dublin. The data are in IDRISI raster format, with one layer for each year between 1990 and Our task is to convert all layers to.tif format. Unzip landuse.zip to a folder called "landuse". Open the command prompt and "cd" to this folder. Make a batch file with the following commands: Try to understand the code. This is a for loop that loops over all *.rst files in the folder. %%f is the variable that contains the filename of each file. With echo we can print something to the screen. Here we print %%~nf, which is the part of the filename before the dot that separates it from the extension. Then we use the gdal_translate command with output format GeoTiff. At the end of the line we add the.tif extension to the filename. Now you can execute the batch file and check the results. 20