dbase (DBF) Reader/Writer

Transcription

1 FME Readers and Writers 2013 SP1 dbase (DBF) Reader/Writer The dbase Format (DBF) Reader/Writer allows FME to read and write data in the DBF format. All DBF files are formatted according to the dbase III specification. The DBF Reader/Writer reuses the Relational Table Reader/Writer s CSV to expand its capabilities from being usable only in the FME Universal (Quick) Translator to also being usable in FME Workbench and Universal Viewer. te: Any single DBF file can have a maximum file size of 2 GB, a limit imposed by the dbase III specification. Files larger than 2 GB may be readable, but not officially supported. Files larger than 2 GB are not writable, and will produce an error message. Overview A DBF file defines a single table within a database. The feature attribution of a FME feature are the columns and values of the DBF database table. There is no geometry or dimension to the features created from the DBF files. They are all undefined. Therefore, none of the features created are viewable because there is no graphical component to the features. DBF files store only feature attribution. The DBF format has one physical file. The extension is added to the basename of the DBF file. DBF Quick Facts About Quick Facts Tables Format Type Identifier Reader/Writer Licensing Level Dependencies Dataset Type Feature Type Typical File Extensions Automated Translation Support User-Defined Attributes Coordinate System Support Generic Color Support Spatial Index Schema Required DBF Both Base ne Directory or File File base name.dbf Yes Yes Never Yes

2 dbase (DBF) Reader/Writer Transaction Support Geometry Type Encoding Support dbf_type Yes Geometry Support Geometry Supported? Geometry Supported? aggregate no point no circles no polygon no circular arc no raster no donut polygon no solid no elliptical arc no surface no ellipses no text no line no z values n/a none yes Reader Overview The DBF reader module produces FME feature for each table entry held in the DBF files residing in the given directory. The DBF reader first scans the directory for all DBF files which are defined in the mapping file. It processes only the specified files if IDs lines are available. Otherwise, all files in the directory are read. The DBF reader extracts data from the file one row at a time, producing FME features before passing them on to the rest of the FME for further processing. When the file is exhausted, the DBF reader moves on to the next file in the directory. Optionally a single DBF file can be given as the dataset. In this case, only the single file is read. The reader supports dbase III, dbase IV and FoxPro files. dbase (DBF) Reader Parameters File Contents Character Encoding This optional parameter controls which character encoding is used to interpret text attributes from the DBF file. If the value is not set, then the character encoding will be automatically detected from the source DBF file. If the value is set, it will take precedence over the automatically detected character encoding. This parameter is useful when the character encoding information stored in the DBF file is missing or incorrect. Schema Attributes Additional Attributes to Expose This parameter exposes Format Attributes in Workbench when you create a

3 FME Readers and Writers 2013 SP1 workspace: In a dynamic scenario, it means these attributes can be passed to the output dataset at runtime. In a non-dynamic scenario where you have multiple feature types, it is convenient to expose additional attributes from one parameter. For example, if you have ten feature types and want to expose the same attribute in each one, it is easier to define it once than it is to set each feature type individually in the workspace. Writer Overview The DBF Writer writes all attributes of a feature to a DBF file. Features of the different feature types are written to different DBF files. te: te that this format does not contain separate Workbench writer-level parameters, so the Writer Parameters button is disabled. Feature Representation In addition to the generic FME feature attributes that FME Workbench adds to all features (see About Feature Attributes), this format adds the format-specific attributes described in this section. The DBF feature attributes consists of the column name that were in the DBF table. All DBF features contain a dbf_type attribute, which is always set to dbf_none as there is no geometry to DBF features. This shows that the feature was generated from a DBF file. Attribute Name dbf_type Contents The DBF geometric type of this entity. Range: dbf_none Default: dbf_none Format Mapping File Directives te: FME translations were originally based entirely on Mapping Files. Mapping files still exist under the surface but the interface has been almost entirely replaced by Workbench's graphical interface. Information on mapping files is included in this manual for technical reference purposes. Mapping Files are ASCII text files that contain a series of rules that specify the FME readers, writers, and transformations (in Workbench, these are represented by transformers). A mapping file (.fme) is a series of commands for FME to perform. Mapping files use functions and factories to transform the data. They also contain the definition and parameters for the readers and writers. A mapping file can be run through the FME

4 dbase (DBF) Reader/Writer Quick Translator. Before FME Workbench was designed and developed (about 2001), this was the only way to configure a translation process. You can create a mapping file either by manually programming it or by using FME Workbench. In Workbench, there is still an Export as.fme tool on the toolbar. The Workbench file format itself (.fmw file) is partially a mapping file with an XML header. When FME runs a workspace it is converted into a mapping file. Since mapping files are written in a plain ASCII format, so you can use any text editor to edit them. To see what a mapping file looks like: select one or more transformers in Workbench, copy them, and then paste them in a text editor. The mapping file equivalent of those transformers will be pasted. Directives and Reader/Writer Keywords Directives are processed by the reader or writer. Directives are prefixed by the current <ReaderKeyword> or <WriterKeyword> in a mapping file. By default, the keywords for formats are the format shortname (viewable in the Formats Gallery, or in the Format Quick Facts tables. Reader Directives The suffixes listed are prefixed by the current <ReaderKeyword> in a mapping file. By default, the <ReaderKeyword> for the DBF reader is DBF. DATASET Required/Optional: Required This is the name of a directory containing one or more DBF files, or a single DBF file. The extension for DBF files is.dbf. Example: DBF_DATASET /usr/data/dbf/input Workbench Parameter: Source dbase (DBF) File(s) DEF Required/Optional Required Each DBF file may optionally be defined before it can be read. The definition specifies the base name of the file, and the names and the types of all attributes. Example: <ReaderKeyword>_DEF <basename> \ [<attrname> <attrtype>]+ The following table shows the attribute types supported

5 FME Readers and Writers 2013 SP1 Field Type char(<width>) date number(<width>, <decimals>) logical memo Description Character fields store fixed length strings. The width parameter controls the maximum number of characters that can be stored by the field. padding is required for strings shorter than this width. Date fields store date as character strings with the format YYYYMMDD. Number fields store single and double precision floating point values. The width parameter is the total number of characters allocated to the field, including the decimal point. The decimals parameter controls the precision of the data and is the number of digits to the right of the decimal. Logical fields store TRUE/FALSE data. Data read or written from and to such fields must always have a value of either true or false. The reader can read dbase III, IV and FoxPro memo fields.when writing, only dbase III format memo fields are supported. The example below is a DEF line for the trees DBF file that has the attributes name and id_number: IDs DBF_DEF trees \ name char(30) \ id_number number(11,0) Required/Optional Optional This optional specification limits the available and defined DBF files read. If no IDs are specified, then all defined and available DBF files are read. The syntax of the IDs keyword is: <ReaderKeyword>_IDs <basename> \ <basename1> \ <basenamen> The basenames must match those used on the DEF lines. The example below selects only the pipeline DBF file for input during a translation:

6 dbase (DBF) Reader/Writer DBF_IDs pipeline ENCODING Required/Optional Optional This optional specification controls which character encoding is used to interpret text attributes from the DBF file. If the value is not set, then the character encoding will be automatically detected from the source DBF file. If the value is set, it will take precedence over the automatically detected character encoding. This directive is useful when the character encoding information stored in the DBF file is missing or incorrect. Example: <ReaderKeyword>_ENCODING <character encoding> Workbench Parameter Character Encoding Parameter <character encoding> Description The character encoding to use when interpreting text attributes. Must be set to any of the following values: ANSI - this means use the current OS language BIG5 EUC ISO OEM SJIS UTF-8 CP437 CP708 CP720 CP737 CP775 CP850 CP852 CP855 CP857 CP860 CP

7 FME Readers and Writers 2013 SP1 Parameter Description CP862 CP863 CP864 CP865 CP866 CP869 CP932 CP936 CP950 CP1250 CP1251 CP1252 CP1253 CP1254 CP1255 CP1256 CP1257 CP1258 ISO ISO ISO ISO ISO ISO ISO ISO ISO ISO ISO TRIM_PRECEDING_SPACES Required/Optional: Optional This option specifies whether the reader should trim preceding spaces of attribute values. If the option is set to YES, then preceding spaces in attribute values will be discarded. If the option is set to NO, then preceding spaces will be left intact. The default value is YES. Workbench Parameter: Trim Preceding Spaces

8 DeLorme GPL Reader EXPOSED_ATTRS This directive allows the selection of format attributes to be explicitly added to the reader feature type. This is similar to exposing format attributes on a reader feature type once it has been generated; however, it is even more powerful because it enables schema-driven applications other than Workbench to access and leverage these attributes as if they were explicitly on the schema as user attributes. The result of picking a list of attributes is a comma-separated list of attribute names and types that will be added to the schema features. Currently all reader feature types will receive the same set of additional schema attributes for a given instance of the reader. Required/Optional Optional Mapping File Syntax t applicable. While it is possible for FME Objects applications to invoke this directive, the required format is not documented. This directive is intended for use in our GUI applications (for example, Workbench) only. Workbench Parameter Additional Attributes to Expose Writer Directives The suffixes shown are prefixed by the current <WriterKeyword> in a mapping file. By default, the <WriterKeyword> for the DBF writer is DBF. DATASET, DEF The DBF writer processes the DATASET and DEF keywords as described in the Reader Keywords section above. However, it does not make use of the IDs keywords. Unlike the reader, the writer requires a DEF line for each file being written. ENCODING The ENCODING directive is used to specify which character encoding should be used when writing text attributes into DBF files. If the value of this directive is not set, the current OS language is used. The syntax of the ENCODING writer directive is the same as the ENCODING reader directive, as described in the Reader Directives section. DeLorme GPL Reader FME can read files in the DeLorme GPL format

9 FME Readers and Writers 2013 SP1 DeLorme GPL is a GPS log file generated by DeLorme products. The GPS log file stores a single, continuous GPS track. Track points include information on speed and course. te: To use FME to translate to or from this format, you must install GPSBabel. GPSBabel is a free tool for converting between various GPS formats. Download GPSBabel from DeLorme GPL Quick Facts DeLorme GPL Quick Facts Format Type Identifier Reader/Writer Licensing Level Dependencies Dataset Type Feature Type Typical File Extensions Automated Translation Support User-Defined Attributes Coordinate System Support Generic Color Support Spatial Index Schema Required Transaction Support Geometry Type Encoding Support DELORME_GPL Reader Professional GPSBabel File tracks, track points.gpl LL84 Never Yes delorme_gpl_type Geometry Support Geometry Supported? Geometry Supported? aggregate yes point yes circles no polygon no circular arc no raster no donut polygon no solid no elliptical arc no surface no ellipses no text no line yes z values no none no

10 DeLorme GPL Reader Reader Overview FME considers a single DeLorme GPL file to be a dataset. All features in each dataset use the LL84 coordinate system. DeLorme GPL Reader Parameters DeLorme GPL Reader Parameters Schema Attributes Additional Attributes to Expose This parameter exposes Format Attributes in Workbench when you create a workspace: In a dynamic scenario, it means these attributes can be passed to the output dataset at runtime. In a non-dynamic scenario where you have multiple feature types, it is convenient to expose additional attributes from one parameter. For example, if you have ten feature types and want to expose the same attribute in each one, it is easier to define it once than it is to set each feature type individually in the workspace. Format Mapping File Directives te: FME translations were originally based entirely on Mapping Files. Mapping files still exist under the surface but the interface has been almost entirely replaced by Workbench's graphical interface. Information on mapping files is included in this manual for technical reference purposes. Mapping Files are ASCII text files that contain a series of rules that specify the FME readers, writers, and transformations (in Workbench, these are represented by transformers). A mapping file (.fme) is a series of commands for FME to perform. Mapping files use functions and factories to transform the data. They also contain the definition and parameters for the readers and writers. A mapping file can be run through the FME Quick Translator. Before FME Workbench was designed and developed (about 2001), this was the only way to configure a translation process. You can create a mapping file either by manually programming it or by using FME Workbench. In Workbench, there is still an Export as.fme tool on the toolbar. The Workbench file format itself (.fmw file) is partially a mapping file with an XML header. When FME runs a workspace it is converted into a mapping file. Since mapping files are written in a plain ASCII format, so you can use any text editor to edit them. To see what a mapping file looks like: select one or more transformers in Workbench, copy them, and then paste them in a text editor. The mapping file equivalent of those transformers will be pasted

11 FME Readers and Writers 2013 SP1 Directives and Reader/Writer Keywords Directives are processed by the reader or writer. Directives are prefixed by the current <ReaderKeyword> or <WriterKeyword> in a mapping file. By default, the keywords for formats are the format shortname (viewable in the Formats Gallery, or in the Format Quick Facts tables. Reader Directives The directives listed below are processed by the DeLorme GPL reader. The suffixes shown are prefixed by the current <ReaderKeyword> in a mapping file. By default, the <ReaderKeyword> for the DeLorme GPL reader is DELORME_GPL. DATASET The value for this directive is the DeLorme GPL file to be read. Required/Optional Required Mapping File Syntax DELORME_GPL_DATASET /usr/data/data.gpl

12