Performing XML Data Validation in the Global Force Management Data Initiative

Transcription

1 Performing XML Data Validation in the Global Force Management Data Initiative by Frederick S. Brundick ARL-TR-4742 March 2009 Approved for public release; distribution unlimited.

2 NOTICES Disclaimers The findings in this report are not to be construed as an official Department of the Army position unless so designated by other authorized documents. Citation of manufacturer s or trade names does not constitute an official endorsement or approval of the use thereof. Destroy this report when it is no longer needed. Do not return it to the originator.

3 Army Research Laboratory Aberdeen Proving Ground, MD ARL-TR-4742 March 2009 Performing XML Data Validation in the Global Force Management Data Initiative Frederick S. Brundick Computational and Information Sciences Directorate, ARL Approved for public release; distribution unlimited.

4 REPORT DOCUMENTATION PAGE Form Approved OMB No Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports ( ), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY) March REPORT TYPE Final 4. TITLE AND SUBTITLE Performing XML Data Validation in the Global Force Management Data Initiative 3. DATES COVERED (From - To) October 2006 to December a. CONTRACT NUMBER 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) Frederick S. Brundick 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) U.S. Army Research Laboratory ATTN: AMSRD-ARL-CI-IC Aberdeen Proving Ground, MD PERFORMING ORGANIZATION REPORT NUMBER ARL-TR SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR S ACRONYM(S) 11. SPONSOR/MONITOR S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited. 13. SUPPLEMENTARY NOTES 14. ABSTRACT This report describes the validation performed on data produced by the Global Force Management (GFM) Data Initiative (DI) project. Extensible Markup Language (XML) was chosen for the data exchange protocol because of its popularity and widespread support. The GFM data model, written in an object-oriented form, not only normalizes data, but also defines objects with parent/child hierarchical relationships. Since XML is a hierarchical language, the GFM XML schema is able to perform a more thorough analysis of hierarchical data than the same data presented in relational form. Due to the limitations of XSD tests, I wrote XML Stylesheet Language: Transformations (XSLT) scripts to perform additional structural and business rule validations on GFM XML data. This report contains descriptions and sample code from all of the GFM XSD modules. After an introduction to data validation with XSLT, the tests performed by both scripts are shown and explained. I present instructions on how to validate GFM XML data, along with sample results, and discuss the strengths and shortcomings of the validation process. 15. SUBJECT TERMS XML, XSL, data schema, data validation 16. SECURITY CLASSIFICATION OF: a. REPORT Unclassified b. ABSTRACT Unclassified c. THIS PAGE Unclassified 17. LIMITATION OF ABSTRACT UU 18. NUMBER OF PAGES 82 19a. NAME OF RESPONSIBLE PERSON Frederick S. Brundick 19b. TELEPHONE NUMBER (Include area code) Standard Form 298 (Rev. 8/98) Prescribed by ANSI Std. Z39.18 ii

5 Contents List of Figures List of Tables Acknowledgments Executive Summary vii viii ix xi 1. Introduction 1 2. XML and Schemas Background The GFMXSD GFM Simple Data Types GFM Relational Table Types GFM Hierarchical Table Types GFM Table Elements Main GFM XSD File Validating Data With Transformations Background XSLT Validation Overview Structural Validation Link Validation Category Code Validation Validation of Override Pairs Validation of Date/Time Groups Detection of Mandatory Elements Type Associated With Proper Item iii

6 3.3.7 Consistent References Single Root Node Putting the Parts Together Validation of Business Rules Introduction Link Type Validation Person Type Category Code Validation Organisation Validation Billet Validation Link Endpoint Types Required Associations Aligning Person Types General Information Performing Validation Testing Schema Validation XSLT Validation Analysis XSD and XSLT XML Limitations Alternatives References 47 Appendix A. Security Markings 49 Appendix B. Validation Constraints 51 Appendix C. Valid Example 55 List of Symbols, Abbreviations, and Acronyms 63 Glossary 64 Distribution List 67 iv

7 List of Figures Figure 1. Hierarchical data schema Figure 2. Canonical (wrapper) data format Figure 3. Files comprising the GFM XSD Figure 4. Simple data type Figure 5. Enumerated data type Figure 6. Complex type Figure 7. Optional element Figure 8. Object Item type Figure 9. Object Item Group Figure 10. Object Item Hierarchy type Figure 11. Root (main) type Figure 12. Simple table definition Figure 13. Hierarchical table definition with tests Figure 14. Equivalent SQL referential test Figure 15. Uniqueness test Figure 16. Object Item keys Figure 17. Object Item Alias keyrefs Figure 18. Relational data hierarchy keyrefs Figure 19. Default empty template for mode mode name Figure 20. Template that checks for invalid OBJ ITEM ASSOC links Figure 21. Named template missing-endpoint that produces output Figure 22. Template that checks for invalid OBJ TYPE ESTAB OBJT DET links, part Figure 23. Template that checks for invalid OBJ TYPE ESTAB OBJT DET links, part Figure 24. Template that compares CAT CODE with child element Figure 25. Template that finds incorrectly overridden fields Figure 26. Template that finds invalid DTG ranges Figure 27. Template that finds invalid effective datetime values v

8 Figure 28. Template that finds Object Types without Establishments Figure 29. Template that finds Object Items with incorrect Object Types Figure 30. Redundant reference Figure 31. Template that finds Object Items with incorrect Object Types Figure 32. Template that finds multiple Organisation tree roots Figure 33. Main (root) template Figure 34. Counting the number of object-oriented tables Figure 35. Template that matches OTEOD links with valid code pairs Figure 36. SQL query that finds OTEOD links with valid code pairs Figure 37. Template that matches OTEOD links with invalid code pairs Figure 38. Template that checks Person Type categories Figure 39. Template that finds ORGs that are children Figure 40. Template that reports ORGs that are not children Figure 41. Template that finds Billets that have children Figure 42. Template that checks child of Mat Type parent Figure 43. Named template that checks the category code of an Object Type Establishment. 34 Figure 44. Template that reports Crew Platform Types that do not have a Mat Type Figure 45. Template that reports Person Type roots that do not have a PTSA of ROS Figure 46. Template that reports Person Type trees that have an incorrect number of nodes. 37 Figure 47. Template that reports MilPostTypes with too few PersType children Figure 48. Template that counts Crew Platform (and Billet) elements Figure 49. Typical GFM root element Figure 50. Xerces XML validator detecting one error Figure 51. Saxon transformation engine testing XML data Figure 52. Web browser output of XSLT validation Figure 53. Web browser output of validation of relational data Figure 54. Web browser output of validation using business rules Figure 55. Sample force structure (organisation) tree Figure A-1. GFM XML data with security attributes Figure A-2. Classification markings attribute groups vi

9 Figure A-3. Object Item type with security attribute group Figure C-1. OBJ TYPE elements, part 1 of 2 (ORG TYPEs) Figure C-2. OBJ TYPE elements, part 2 of 2 (MAT TYPE and PERS TYPE) Figure C-3. OBJ TYPE ESTAB elements Figure C-4. GFM PERS TYPE SKILL ATTR element Figure C-5. OBJ TYPE ESTAB OBJT DET (link) elements Figure C-6. OBJ ITEM (ORG) elements Figure C-7. OBJ ITEM OBJ TYPE ESTAB elements Figure C-8. OBJ ITEM ASSOC element Figure C-9. Sample data with relationships vii

10 List of Tables Table 1. Object Type to Object Item associations Table 2. Allowable OTEOD combinations Table B-1. OTEOD category codes Table B-2. Assoc category codes Table C-1. Link keys for figure C viii

11 Acknowledgments I would like to thank Holly Ingham, a fellow member of the Global Force Management Data Initiative team, and William Tanenbaum, a high school student in the Science and Engineering Apprentice Program, for reading early drafts of this report. They provided valuable feedback and kept the technical discussions from being overly esoteric. ix

12 INTENTIONALLY LEFT BLANK. x

13 Executive Summary This report describes the validation performed on data produced by the Global Force Management (GFM) Data Initiative (DI) project. Extensible Markup Language (XML) was chosen for the data exchange protocol because of its popularity and widespread support. It is common practice to compare a set of XML data against an XML Schema definition (XSD) file, similar to the way that data in a relational database may be verified against the database schema. While this process will catch gross errors, such as strings that are too long, incorrect date formats, and the absence of mandatory elements, much more validation may be performed. The two basic test classes that this report describes are structural and business rule validation. One reason why XML is popular is because the values are surrounded by their element names, eliminating confusion caused by changing the order of unlabeled values. However, the GFM data model was written in an object-oriented form where the data is not only normalized (to use the database term), but also defines objects with parent/child hierarchical relationships. XML is a hierarchical language, and this fact may be used to explicitly structure the data to match the hierarchical model. The GFM XML schema is able to perform a more thorough analysis of hierarchical data than the same data presented in relational form. Tests are also defined to check the data for referential integrity; when one object refers to a second object, the latter is checked to make sure that it exists. The final structural test ensures that top-level identifiers are unique. There are limits to the tests that may be conducted by XSD. While the XML Stylesheet Language: Transformations (XSLT) was written to transform an XML file into another form, such as Hypertext Markup Language (HTML), it may be used as a programming language to validate XML data. An XSLT script has been written to perform additional structural validations on GFM XML data. Unlike XSD, XSLT includes conditional expressions, allowing validation tests such as the starting date/time group must be less than the termination date/time group. A second XSLT script enforces certain GFM business rules. While the XSD ensures that an particular value is in the set of values recognized by the data model, GFM business rules may put additional constraints on the value. This report contains descriptions and sample code from all of the GFM XSD modules. After an introduction to data validation with XSLT, the tests performed by both scripts are shown and explained. I present instructions on how to validate GFM XML data, along with sample results, and discuss the strengths and shortcomings of the validation process. While the intended audience is people with some XML experience, a glossary with acronyms is provided to define terms in the context of the report. Appendix A contains a description of the Intelligence Community Information Security Marking (IC-ISM) XSD module, appendix B lists summaries of the structural and business rules, and appendix C describes a small set of sample data. This is work in progress that is expected to grow over time. xi

14 INTENTIONALLY LEFT BLANK. xii

15 1. Introduction The Global Force Management (GFM) Data Initiative (DI) (1) uses Extensible Markup Language (XML) (2) to exchange data between servers and clients. It is important that the data be validated, not only to prevent retransmissions due to errors, but also to verify that the original data was constructed in accordance with the GFM schema, structure, and business rules. This report discusses the techniques that are used by GFM to ensure the correctness of the data. An XML data file is composed of elements (with attributes) in a hierarchical structure, while the original (non-xml) schema was written using a relational database model. Database tables contain records; each record is made up of fields, with one value in each field. The vocabulary in this report is a mixture of XML and database terms. It is simpler to call an element a table instead of using the verbose phrase the XML element that corresponds to a table in the database model. 1 The intended meaning of a word should be clear within its context. 2. XML and Schemas 2.1 Background While XML was originally designed to be used in conjunction with Hypertext Markup Language (HTML) and for document processing, it was quickly adopted as a data markup language (3). XML Schema Definition (4, 5) (commonly known as XML Schema or XSD) was adopted as a grammar to describe the structure and data types of an XML data file. It is similar to a traditional data dictionary, declaring data elements and their types, but allows the programmer to define his own data types, the structural relationships between elements, and the logical relationships. There are two tests that may be performed on an XML data file. The first is well-formedness, which ensures that the file is syntactically correct are elements nested properly, are all elements terminated, are strings properly delimited, etc.? The second is validation, which compares the file with an XSD and examines the data contents (6) are all of the elements defined, does all data conform to the type restrictions, are mandatory elements present, etc.? Section 3 presents another methodology to validate data. 2.2 The GFM XSD The GFM Information Exchange Data Model (IEDM) is an augmented subset of the Joint Command, Control and Consultation Information Exchange Data Model (JC3IEDM) developed 1 See section for other element mappings. 1

16 by the Multilateral Interoperability Programme Data Modelling Working Group (MIP DMWG) (7). The JC3IEDM is available in both relational and object-oriented XSDs. Note: All fields and tables added by GFM start with GFM. For readers who are unfamiliar with the GFMIEDM, a short explanation is in order. Every record of every table has a primary key called an Enterprise-Wide Identifier (EwID) (8). These surrogate keys are constructed so they are unique not only within a specific table or database, but throughout all systems that produce GFM data. Because the JC3IEDM is an object-oriented model, even though it is intended to be used by relational database management systems (RDBMSs), many of the EwIDs are foreign keys that refer to records in other tables. The JC3IEDM may be treated as a highly normalized database where many of the tables have a parent-child relationship. A hierarchical (object-oriented) portion of the GFMIEDM Entity-Relationship (E-R) diagram, where links represent is-a relationships, is shown in figure 1. Figure 1. Hierarchical data schema. The field above the line is the primary key for each table. In this example, obj item id is a unique key and the other EwIDs (such as org id and unit id) are foreign keys. The relational XSD uses what the Oracle Corporation refers to as the canonical XML format (9), also called the wrapper format. Each database table is an XML element, the rows in the table correspond to children of the element, and the fields are child elements of the row element. A typical example is shown in figure 2. 2 When hierarchical data is stored in canonical form, the relationships between the records is lost and validation becomes more difficult. 2 Intelligence Community Information Security Marking (IC-ISM) attributes are not shown but are described in appendix A. 2

17 <OBJ_ITEM_ASSOC_TBL> <OBJ_ITEM_ASSOC> <SUBJ_OBJ_ITEM_ID> </SUBJ_OBJ_ITEM_ID> <OBJ_OBJ_ITEM_ID> </OBJ_OBJ_ITEM_ID> <OBJ_ITEM_ASSOC_IX> </OBJ_ITEM_ASSOC_IX> <CAT_CODE>HSADMI</CAT_CODE> <SUBCAT_CODE>ALTFOR</SUBCAT_CODE> <GFM_CAT_CODE>NOS</GFM_CAT_CODE> <GFM_SUBCAT_CODE>DEFALT</GFM_SUBCAT_CODE> <GFM_OBJ_ITEM_ASSOC_S_DTG> T00:00:00Z</GFM_OBJ_ITEM_ASSOC_S_DTG> <GFM_OBJ_ITEM_ASSOC_T_DTG> T00:00:00Z</GFM_OBJ_ITEM_ASSOC_T_DTG> </OBJ_ITEM_ASSOC> <OBJ_ITEM_ASSOC> <SUBJ_OBJ_ITEM_ID> </SUBJ_OBJ_ITEM_ID> <OBJ_OBJ_ITEM_ID> </OBJ_OBJ_ITEM_ID> <OBJ_ITEM_ASSOC_IX> </OBJ_ITEM_ASSOC_IX> <CAT_CODE>HSADMI</CAT_CODE> <SUBCAT_CODE>ALTFOR</SUBCAT_CODE> <GFM_CAT_CODE>NOS</GFM_CAT_CODE> <GFM_SUBCAT_CODE>DEFALT</GFM_SUBCAT_CODE> <GFM_OBJ_ITEM_ASSOC_S_DTG> T00:00:00Z</GFM_OBJ_ITEM_ASSOC_S_DTG> <GFM_OBJ_ITEM_ASSOC_T_DTG> T00:00:00Z</GFM_OBJ_ITEM_ASSOC_T_DTG> </OBJ_ITEM_ASSOC> <OBJ_ITEM_ASSOC> <SUBJ_OBJ_ITEM_ID> </SUBJ_OBJ_ITEM_ID> <OBJ_OBJ_ITEM_ID> </OBJ_OBJ_ITEM_ID> <OBJ_ITEM_ASSOC_IX> </OBJ_ITEM_ASSOC_IX> <CAT_CODE>HSADMI</CAT_CODE> <SUBCAT_CODE>ALTFOR</SUBCAT_CODE> <GFM_CAT_CODE>NOS</GFM_CAT_CODE> <GFM_SUBCAT_CODE>DEFALT</GFM_SUBCAT_CODE> <GFM_OBJ_ITEM_ASSOC_S_DTG> T00:00:00Z</GFM_OBJ_ITEM_ASSOC_S_DTG> <GFM_OBJ_ITEM_ASSOC_T_DTG> T00:00:00Z</GFM_OBJ_ITEM_ASSOC_T_DTG> </OBJ_ITEM_ASSOC> </OBJ_ITEM_ASSOC_TBL> Figure 2. Canonical (wrapper) data format. Since XML is inherently hierarchical, it is ideally suited to handle GFM data. As figure 1 shows, a UNIT is-an ORG is-an OBJ ITEM. To phrase that another way, a Unit is composed of an OBJ ITEM, an ORG, and a UNIT. The org id and unit id fields are foreign keys that have the same value as the obj item id primary key. It is incorrect to have a record in one of these tables without having a corresponding record in each of the other tables. Of course, that is just one path through the tree. A UNIT child of an ORG could be replaced by a GFM CREW PLATFORM or a GFM BILLET, or the OBJ ITEM child could be a MAT (Materiel) or FAC (Facility). These requirements an OBJ ITEM must have an ORG, MAT, or FAC child, and an ORG must have one of three children may be explicitly recorded in the XSD. 3

18 Due to the size of the JC3IEDM, there are two separate sets of XSD files. Since the GFM model is considerably smaller than the JC3 model, the relational and object-oriented schemas were combined. In order to improve readability and facilitate configuration control, the XSD was split into multiple files. These files, which are shown in figure 3, are described from the bottom up. GFMIEDM341.xsd GFMIEDM341tables.xsd GFMIEDM341hierTableTypes.xsd GFMIEDM341relatTableTypes.xsd IC-ISM-v2.xsd GFMIEDM341simpleTypes.xsd Figure 3. Files comprising the GFM XSD GFM Simple Data Types The file GFMIEDM341simpleTypes.xsd contains all of the data types that are defined in the E-R model, translated into XML terms. The majority of the types were copied from the JC3IEDM XSD. At over 10,500 lines, this file makes a strong argument for a modular design. There are 2 main classes of data types: 1. Simple or primitive types, and 2. Enumerated types. An example of a simple type is shown in figure 4 and should be self-explanatory. <xs:simpletype name="txt_mandatory_100"> <xs:annotation> <xs:documentation>oracle datatype: VARCHAR(100)</xs:documentation> </xs:annotation> <xs:restriction base="xs:string"> <xs:maxlength value="100"/> </xs:restriction> </xs:simpletype> Figure 4. Simple data type. 4

19 The word mandatory in the type name is slightly misleading. A type may not be mandatory; only an element may be mandatory. 3 This is a reminder that an element of this type is required. This type is an extension of xs:string and has a maximum length of 100 characters. 4 The documentation element shows the equivalent SQL type. 5 An enumerated type is an extension of xs:string with a list of all possible values. The example in figure 5 was chosen because it has only two values. Most of the enumerations are fairly large, and some are huge. The documentations and type names are taken from the E-R model. All JC3IEDM enumerated type names start with DS, while GFM types begin with GF. <xs:simpletype name="ds4190_obj_type_estab_cat_code"> <xs:annotation> <xs:documentation>data type for the validation rule DS4190_obj_type_estab_cat_code</xs:documentation> </xs:annotation> <xs:restriction base="xs:string"> <xs:enumeration value="ces"> <xs:annotation> <xs:documentation>complete equipment schedule: A list of the associated ancillaries, accessories, tools, literature and spares which, when scheduled together, form a composite vehicle, equipment or store.</xs:documentation> </xs:annotation> </xs:enumeration> <xs:enumeration value="pcg"> <xs:annotation> <xs:documentation>parts catalogue: A list showing the disassembly build order of an equipment, identifying the assemblies, sub-assemblies and components which comprise the equipment (or assemblies and sub-assemblies).</xs:documentation> </xs:annotation> </xs:enumeration> </xs:restriction> </xs:simpletype> Figure 5. Enumerated data type GFM Relational Table Types The file GFMIEDM341relatTableTypes.xsd includes the file GFMIEDM341simpleTypes.xsd. At roughly 1800 lines, it is the second largest file. It defines all of the records and fields for the canonical form. It also imports the file IC-ISM-v2.xsd (10) and defines the attributes required for classification markings on data records. 6 These attributes are discussed in appendix A. 3 The definition is identical to the type txt optional All XML Schema elements are in the xs namespace and thus start with xs:. 5 The XSD best practices recommends documentation elements rather than comments because they are inline elements and therefore part of the schema. 6 The include element treats all of the declarations as part of a single application, while import references declarations in another namespace. 5

20 Figure 6 shows the definition of the CivilianPostType type. It is a complex type because it combines several elements together. By default, an all element requires that each child element appears exactly once (in the data file). The first element is named CIV POST TYPE ID and it is of type identifier20 (an EwID). Notice that the other two elements have almost the same type names except for the first two characters. This was done to emphasize that the types are related, but that the first was defined by the JC3IEDM and the second by GFM. <xs:complextype name="civilianposttype"> <xs:annotation> <xs:documentation>definition: An ORGANISATION-TYPE with a set of duties that are intended to be fulfilled by one person in private sector and non-military government organisations.</xs:documentation> </xs:annotation> <xs:all> <xs:element name="civ_post_type_id" type="identifier20"> <xs:annotation> <xs:documentation>definition: The organisation-type-id of a specific CIVILIAN-POST-TYPE (a role name for object-type-id). </xs:documentation> </xs:annotation> </xs:element> <xs:element name="cat_code" type="ds369_civ_post_type_cat_code"> <xs:annotation> <xs:documentation>definition: The specific value that represents the class of CIVILIAN-POST-TYPE.</xs:documentation> </xs:annotation> </xs:element> <xs:element name="gfm_cat_code" type="gf369_civ_post_type_gfm_cat_code"> <xs:annotation> <xs:documentation>definition: The specific value that represents the GFM class of CIVILIAN-POST-TYPE.</xs:documentation> </xs:annotation> </xs:element> </xs:all> </xs:complextype> Figure 6. Complex type. The code fragment in figure 7 shows how to mark a field as optional. The attribute is minoccurs and the value 0 means that the element may appear zero times, i.e., it is optional. Remember that the word optional in the type name is just a reminder to the human reader. <xs:element name="descr_txt" type="txt_optional_50" minoccurs="0"> <xs:annotation> <xs:documentation>definition: The character string assigned to represent the specific ORGANISATION-TYPE.</xs:documentation> </xs:annotation> </xs:element> Figure 7. Optional element. 6

21 2.2.3 GFM Hierarchical Table Types The file GFMIEDM341relatTableTypes.xsd could be included to define the schema using the canonical form; however, better validation may be achieved by taking advantage of the hierarchical nature of XML and defining hierarchical table types in the file GFMIEDM341hierTableTypes.xsd. To differentiate the force structure tree hierarchical data 7 from the data structure hierarchy, the latter is referred to as the Generalization Hierarchy. Nesting the elements also makes the data more understandable to a human reader since related data are kept together instead of being scattered throughout the GFM XML data file. Figure 8 shows part of the definition of ObjectItem. Notice that it uses sequence instead of all. The main difference, which is not relevant in GFM, is that the data elements in a sequence must appear in the specified order, while all allows any order to be used. The reason that sequence is used is to allow the ObjectItem type to be extended to derive a new type. The italicized element is explained in appendix A. <xs:complextype name="objectitem"> <xs:annotation> <xs:documentation>definition: An individually identified object that has military or civilian significance.</xs:documentation> </xs:annotation> <xs:sequence> <xs:element name="obj_item_id" type="identifier20"> <xs:annotation> <xs:documentation>definition: The unique value, or set of characters, assigned to represent a specific OBJECT-ITEM and to distinguish it from all other OBJECT-ITEMs.</xs:documentation> </xs:annotation> </xs:element>... </xs:sequence> <xs:attributegroup ref="securityattributesgroup"/> </xs:complextype> Figure 8. Object Item type. The simple types shown in figures 4 and 5 are restrictions. They are both based on the xs:string type, but the first has a maximum length of 100 characters and the second may contain only specific values. A specification is needed to state that an ObjectItem must have a FAC, MAT, or ORG child element via an extension of the ObjectItem type. If one or more simple types were being added to the ObjectItem type, a format similar to the one for restricting simple types could be used; however, a group must be used instead. The group definition is shown in figure 9. 7 Force structures are discussed in section 5.2 and appendix B. Otherwise, the word hierarchy refers to a data structure. 7

22 <xs:group name="objectitemgroup"> <xs:choice> <xs:element name="fac" type="facility"/> <xs:element name="mat" type="materiel"/> <xs:element name="org" type="organisationhierarchy"/> </xs:choice> </xs:group> Figure 9. Object Item Group. The choice element specifies that exactly one of the elements listed must appear in the data. Now the parts may be assembled to define the hierarchical version of an Object Item type. This has been done in figure 10. An ObjectItemHierarchy consists of all of the elements in an ObjectItem (figure 8) plus the ObjectItemGroup (figure 9). <xs:complextype name="objectitemhierarchy"> <xs:annotation> <xs:documentation>an ObjectItem which is part of a hierarchy. </xs:documentation> </xs:annotation> <xs:complexcontent> <xs:extension base="objectitem"> <xs:sequence> <xs:group ref="objectitemgroup"/> </xs:sequence> </xs:extension> </xs:complexcontent> </xs:complextype> Figure 10. Object Item Hierarchy type. Figure 9 states that the element ORG is of type OrganisationHierarchy. This hierarchical type is an extension of the Organisation type plus the OrganisationGroup. Except for the names, these types and this group are defined the same way as the Object Item example. The final result matches the hierarchical data schema shown in figure GFM Table Elements All of the pieces are now in place to define the table elements. The file GFMIEDM341tables.xsd includes the file GFMIEDM341hierTableTypes.xsd. Next a type for the root element is defined to allow different main XSD files to include this file. As explained in the next section, validation rules must appear in the scope of the element, so the top element may not be declared here. The enclosing elements and some table elements are shown in figure 11. The first thing to note is that every table is optional. If a GFM XML data file does not contain any MiscellaneousEquipmentType data, then it should not have a MISC EQPT TYPE TBL element. 8 8 A basic tenet of XML data is that empty elements are rarely used. 8

23 <xs:complextype name="gfmiedm34type"> <xs:annotation> <xs:documentation xml:lang="en">this element MUST be conveyed as the root element in any instance document based on this schema specification.</xs:documentation> </xs:annotation> <xs:all>... <xs:element ref="misc_eqpt_type_tbl" minoccurs="0"/><!--rel--> <xs:element ref="obj_item_tbl" minoccurs="0"/><!--rel--> <xs:element ref="obj_item_oo_tbl" minoccurs="0"/><!--oo--> <xs:element ref="obj_item_addr_tbl" minoccurs="0"/><!--both-->... </xs:all> </xs:complextype> Figure 11. Root (main) type. Similarly, a table may appear no more than once. Also, all of the elements end with TBL, a convention adopted by the JC3IEDM. Finally, compare the two highlighted elements. The first is the relational (canonical) Object Item table, and the second is the object-oriented (generalization hierarchy) version. This fact is emphasized by the <!--rel--> and <!--oo--> comments. Within a GFM XML data file, a hierarchical table element ends with OO TBL. The fourth element shown has a <!--both--> comment to denote that it is not part of any generalization hierarchy, and thus is the same regardless of the schema being used. The GFM XSD defines 46 tables; 37 are part of generalization hierarchies and 9 are purely relational. The hierarchical tables are combined into only three XML hierarchies. The rest of the file defines all of the table elements and states that each element may contain one or more child elements (records). (A sequence implicitly sets minoccurs=1.) A simple table is shown in figure 12. All relational tables use this form, changing only the element and type names. <xs:element name="obj_item_tbl"><!--rel--> <xs:annotation><xs:documentation>rel only</xs:documentation></xs:annotation> <xs:complextype> <xs:sequence> <xs:element name="obj_item" type="objectitem" maxoccurs="unbounded"/> </xs:sequence> </xs:complextype> </xs:element> Figure 12. Simple table definition. The hierarchical tables, such as the one in figure 13, start out like the relational tables. The element name ends with OO TBL instead of TBL, but the record name remains the same. The OBJ ITEM TBL contains one or more OBJ ITEMs, while the OBJ ITEM OO TBL contains one or more top-level OBJ ITEMs, which have child elements. 9

24 What complicates the definition of a hierarchical table are the key and keyref elements, which are highlighted in figure 13. Each of these elements has a name, a selector with an XPath 9 expression, and a field with a second XPath expression (11). In addition, a keyref contains a reference to a key. <xs:element name="obj_item_oo_tbl"><!--oo--> <xs:annotation><xs:documentation>oo only</xs:documentation></xs:annotation> <xs:complextype> <xs:sequence> <xs:element name="obj_item" type="objectitemhierarchy" maxoccurs="unbounded"/> <xs:annotation><xs:documentation>foreign keys in OBJ_ITEM tree must equal primary key</xs:documentation></xs:annotation> <xs:key name="objitemidtree"> <xs:selector xpath="obj_item_id"/> <xs:field xpath="."/> </xs:key> <xs:keyref name="testfacid" refer="objitemidtree"> <xs:selector xpath="fac"/> <xs:field xpath="fac_id"/> </xs:keyref> <xs:keyref name="testmatid" refer="objitemidtree"> <xs:selector xpath="mat"/> <xs:field xpath="mat_id"/> </xs:keyref> <xs:keyref name="testorgid" refer="objitemidtree"> <xs:selector xpath="org"/> <xs:field xpath="org_id"/> </xs:keyref> <xs:keyref name="testgfmbilletid" refer="objitemidtree"> <xs:selector xpath="org/gfm_billet"/> <xs:field xpath="gfm_billet_id"/> </xs:keyref> <xs:keyref name="testgfmcrewplatformid" refer="objitemidtree"> <xs:selector xpath="org/gfm_crew_platform"/> <xs:field xpath="gfm_crew_platform_id"/> </xs:keyref> <xs:keyref name="testunitid" refer="objitemidtree"> <xs:selector xpath="org/unit"/> <xs:field xpath="unit_id"/> </xs:keyref> </xs:element> </xs:sequence> </xs:complextype> </xs:element> Figure 13. Hierarchical table definition with tests. 9 XPath is a standard XML query language. XSD uses a subset of XPath. 10

25 The purpose of key/keyref is to test referential integrity. The selector and field are used to locate an element whose value is associated with the key s name. In this example, the selector specifies the OBJ ITEM ID. Since we have located the element that we want, the field is., which means the current element. A simple way to picture this is the assignment statement ObjItemIDTree = value-of(this OBJ ITEM s OBJ ITEM ID). The keyref s that follow specify elements whose values must be the same as a key. The highlighted keyref has a name that must be unique but otherwise is ignored. The refer attribute supplies the name of the desired key. The selector and field attributes specify that the ORG ID of the ORG child of the current OBJ ITEM is the element to compare. The equivalent comparison statement is value-of(this OBJ ITEM s ORG s ORG ID) == ObjItemIDTree. These statements may be combined into the desired result, namely, when an OBJ ITEM has a child ORG, the ORG s ORG ID must be equal to the OBJ ITEM s OBJ ITEM ID. Since this test is applied to every OBJ ITEM in the OBJ ITEM OO TBL, it roughly the same as the SQL query shown in figure 14. The value returned by the query should match the number of records in the ORG table. SELECT COUNT(*) FROM OBJ_ITEM oi, ORG o WHERE oi.obj_item_id=o.org_id; Figure 14. Equivalent SQL referential test Main GFM XSD File The GFMIEDM341.xsd file is the main GFM XSD file and is entirely devoted to validation tests. If fewer restrictions are desired, such as for GFM XML data files that are known to be incomplete, then a different main file could be written. The reason for the declaration of the GFMIEDM34Type should now be apparent, since tests must be defined within the scope (body) of an element. After including the GFMIEDM341tables.xsd file, the main file defines the document root of a GFM XML data file, namely GFMIEDM34. The first rule is verbose but easy to understand it uses the unique element to ensure that the primary key in each record (that is not a child in a generalization hierarchy) is unique. A highly edited version of the uniqueness test is shown in figure 15. The selector s XPath expression, of which only two paths are shown, must be confined to a single, long line. The actual test lists all 12 EwIDs that appear in the GFM XSD. 11

26 <xs:unique name="uniqueewid"> <xs:annotation><xs:documentation>primary EwIDs must be unique </xs:documentation></xs:annotation> <xs:selector xpath="... */OBJ_ITEM/OBJ_ITEM_ID OBJ_ITEM_ADDR_TBL/OBJ_ITEM_ADDR/OBJ_ITEM_ADDR_IX..." /> <xs:field xpath="."/> </xs:unique> Figure 15. Uniqueness test. The paths in the selector expression are separated by vertical bars, and the field s XPath is simply.. Ignoring the syntactic details, what this test indicates is that all of the fields listed in the selector must contain unique values. Each path must be long enough to unambiguously specify a single element. The foreign key element often uses the same name as the primary key element, requiring the use of long paths to differentiate between the two. The example shows that the path always starts with the TBL name and drills down to the EwID element. The first path in the example starts with a wildcard (*) because an OBJ ITEM may be a record in an OBJ ITEM TBL or an OBJ ITEM OO TBL. 10 The rest of the file consists of tests for referential integrity. There are two types of references: pointers to disparate element types ( horizontal references) and the relational version of the data hierarchy ( vertical references). The keys are defined the same way as in the GFMIEDM341tables.xsd file. Each key has the same name as its field, plus vertical keys have the prefix Rel. The details are explained with the OBJ ITEM hierarchy. Figure 16 shows the keys that are defined for fields in the OBJ ITEM. The first two selectors start with a wildcard, because horizontal tests apply to both relational and object-oriented tables. The third selector explicitly names the table element because the key is used only in vertical relational tests. <!-- ObjItem keys --> <xs:key name="objitemid"> <xs:selector xpath="*/obj_item"/> <xs:field xpath="obj_item_id"/> </xs:key> <xs:key name="orgid"> <xs:selector xpath="*/obj_item/org"/> <xs:field xpath="org_id"/> </xs:key> <xs:key name="relorgid"> <xs:selector xpath="org_tbl/org"/> <xs:field xpath="org_id"/> </xs:key> Figure 16. Object Item keys. 10 Remember that these paths are relative to the GFMIEDM34 element. 12

27 The horizontal tests in figure 17 are for the table (OBJ ITEM ALIAS) that associates an Object Item with an Alias. The desired tests are to make sure that the elements that are referenced, namely the OBJ ITEM ID and GFM OBJ ALIAS TYPE, exist in the GFM XML data file. The first keyref ensures that the OBJ ITEM ID in an OBJ ITEM ALIAS of an OBJ ITEM ALIAS TBL has the same value as an ObjItemID as defined in figure 16. The second does the same for the Alias end of the link; the key definition is not shown. <!-- ObjItemAddr --> <xs:keyref name="objitemaliasobjitemref" refer="objitemid"> <xs:selector xpath="obj_item_alias_tbl/obj_item_alias"/> <xs:field xpath="obj_item_id"/> </xs:keyref> <xs:keyref name="objitemaliasaliastyperef" refer="gfmaliastypeid"> <xs:selector xpath="obj_item_alias_tbl/obj_item_alias"/> <xs:field xpath="gfm_obji_alias_type"/> </xs:keyref> Figure 17. Object Item Alias keyrefs. The keyrefs in figure 18 match the E-R diagram in figure 1. Each EwID in a table must match the EwID in its parent table. However, there are subtle differences between these tests and the ones defined in figure 13. For one thing, the scope is entirely different. The other tests are defined inside of the OBJ ITEM OO TBL, while these keys and keyrefs are in the main GFMIEDM34 element. Another difference is the key that is referenced the object-oriented keyrefs all refer to the OBJ ITEM ID element. The OBJ ITEM OO TBL element requires that each OBJ ITEM has an ORG, MAT, or FAC child, and an ORG has one of three child elements. Since the relational elements are distinct from each other, this restriction must be imposed by the keyrefs from the bottom up. A UNIT ID must match an ORG ID, while an ORG ID must match an OBJ ITEM ID. It is still possible to create an ORG without a child, but detecting this is beyond the scope of XSD. The techniques described in the next section could perform this task, but the emphasis has been on GFM XML data in the object-oriented form. 3. Validating Data With Transformations 3.1 Background There are limits to the tests that may be performed by an XSD file. Only a subset of XPath is recognized, and conditional statements are not allowed. A separate application must be written to conduct additional validation tests, and that can be quite an undertaking. Rick Jelliffe of the Academia Sinica devised a way to use Extensible Stylesheet Language: Transformations (XSLT) (12) as a clever alternative. 13

28 <!-- ObjItem tree --> <xs:keyref name="orgref" refer="objitemid"> <xs:selector xpath="org_tbl/org"/> <xs:field xpath="org_id"/> </xs:keyref> <!-- Org subtree --> <xs:keyref name="unitref" refer="relorgid"> <xs:selector xpath="unit_tbl/unit"/> <xs:field xpath="unit_id"/> </xs:keyref> <xs:keyref name="crewplatformref" refer="relorgid"> <xs:selector xpath="gfm_crew_platform_tbl/gfm_crew_platform"/> <xs:field xpath="gfm_crew_platform_id"/> </xs:keyref> <xs:keyref name="billetref" refer="relorgid"> <xs:selector xpath="gfm_billet_tbl/gfm_billet"/> <xs:field xpath="gfm_billet_id"/> </xs:keyref> <!-- Mat subtree --> <xs:keyref name="matref" refer="objitemid"> <xs:selector xpath="mat_tbl/mat"/> <xs:field xpath="mat_id"/> </xs:keyref> <!-- Fac subtree --> <xs:keyref name="facref" refer="objitemid"> <xs:selector xpath="fac_tbl/fac"/> <xs:field xpath="fac_id"/> </xs:keyref> Figure 18. Relational data hierarchy keyrefs. XSLT was originally conceived to transform one XML document into another (13). The file GFMIEDM341flatten.xsl is an XSLT script that reads a GFM XML data file that contains hierarchical data and produces the same data in relational form. The most common use of XSLT is to indicate to a Web browser how to convert XML data into XHTML (14) (a reformulation of HTML) to make it more readable by humans. 11 GFM s script to do this is the file GFMIEDM341.xsl. Jelliffe describes how to write XSLT templates (rules) to produce error messages in XHTML based on problems that it detects in the XML data (15). The GFM package contains two XSLT scripts. These scripts assume that the data has already been validated against the GFM XSD. The first, GFMIEDM341validate.xsl, defines structural restrictions that could not be tested by the XSD. The second, GFMIEDM341businessRules.xsl, is an attempt to automate business rules that are not formally part of the model. The restrictions and allowed values are summarized in appendix B. Other scripts may be written to validate data with special requirements. Providing tools to perform data validation allows the data producers to test their systems, while data consumers may avoid tedious input checking in their applications. 11 All major Web browsers have a built-in XSLT transformation engine. 14

29 3.2 XSLT Validation Overview This report is not intended to be a tutorial on XSLT or XPath. However, XSLT is a declarative language, not a procedural language like C, Java, and most other popular languages. An overview is provided to explain Jelliffe s technique in layman s terms, followed by actual code from the GFM XSLT files. An XSLT file contains templates. Most templates contain a match attribute, which consists of an XPath expression. 12 This is very similar to the patterns in an XSD key or keyref, except the full power of XPath is available. The apply-templates element may be used to compare the XML data tree (or a specified subtree) to all of the templates or just a subset. The code in a matching template is executed, possibly producing XHTML output. Without going into too much detail, a match template may have an optional mode and/or priority. By default, all templates have no mode or priority. Each template in the desired mode is evaluated, comparing its XPath expression against the XML tree (or subtree). The one with the highest priority wins and its code is evaluated. A default template that matches every element and prints an element s contents is automatically provided to make sure that all data gets matched (and printed) and is never ignored. The other kind of template is a named template, where a fixed name takes the place of an XPath expression. This is XSLT s concession to procedural code, allowing parameters to be passed to a specific template to facilitate code re-use. These are used in the GFM scripts to format error messages. The mechanism that makes Jelliffe s technique possible is the fact that once XSLT matches a data element against a template, other templates are ignored. A template that has no contents traps an element and quietly ignores it. The template in figure catches all text elements 14 when the mode name mode is in effect and throws them away. The priority of -1 causes the template to be matched after other templates. <xsl:template match="text()" priority="-1" mode="mode_name"> <!-- strip characters --> </xsl:template> Figure 19. Default empty template for mode mode name. Most templates used in GFM validation are bad templates. They match data that has an error and generate XHTML that describes the problem that was found. All elements that are not matched are processed by a default template like the one in figure 19. The assumption is that all elements that are not bad are either good or they are irrelevant. The opposite idea is to write a template that matches good data and does nothing with it. A second template, with a similar match expression but a lower priority, produces the proper error message. The default template then catches all data elements that are left and ignores them. 12 A match template is analogous to an SQL query statement. 13 By convention, XSLT tokens are in the xsl namespace and thus start with xsl:. 14 There are seven types of nodes in the XML Tree Model. All GFM data elements are text elements. 15

30 A variation on these two approaches is a template that matches a subset of data, then uses conditional processing to determine if the data is good or bad. The first of these is the link test as described in the next section. 3.3 Structural Validation Link Validation The simplest template in the file GFMIEDM341validate.xsl, because it is the closest to procedural code, is the template that detects Object Item Associations (Assocs) that are missing either the parent or child node. The actual template is shown in figure <!-- locate nodes where an endpoint is not an OBJ_ITEM --> <!-- external OBJ_ITEMs are not supported yet --> <xsl:template priority="2" match="//gfm:obj_item_assoc" mode="link"> <xsl:choose> <xsl:when test="gfm:subj_obj_item_id=//gfm:obj_item/gfm:obj_item_id"> </xsl:when> <xsl:otherwise> <xsl:call-template name="missing-endpoint"> <xsl:with-param name="class" select=" OBJ_ITEM_ASSOC "/> <xsl:with-param name="end-name" select=" SUBJ_OBJ_ITEM_ID "/> <xsl:with-param name="end-ewid" select="gfm:subj_obj_item_id"/> <xsl:with-param name="link-ewid" select="gfm:obj_item_assoc_ix"/> </xsl:call-template> </xsl:otherwise> </xsl:choose> <xsl:choose> <xsl:when test="gfm:obj_obj_item_id=//gfm:obj_item/gfm:obj_item_id"> </xsl:when> <xsl:otherwise> <xsl:call-template name="missing-endpoint"> <xsl:with-param name="class" select=" OBJ_ITEM_ASSOC "/> <xsl:with-param name="end-name" select=" OBJ_OBJ_ITEM_ID "/> <xsl:with-param name="end-ewid" select="gfm:obj_obj_item_id"/> <xsl:with-param name="link-ewid" select="gfm:obj_item_assoc_ix"/> </xsl:call-template> </xsl:otherwise> </xsl:choose> </xsl:template> Figure 20. Template that checks for invalid OBJ ITEM ASSOC links. The priority is 2 and the mode is link. The XPath expression is very simple. The code match="//gfm:obj_item_assoc" 15 All elements defined in the GFM XSD are in the gfm namespace. 16

31 will match all OBJ ITEM ASSOC elements at any depth in the XML data tree. The choose is the outer wrapper for a multi-branch conditional statement. Each when element may be thought of as an if or else if. The otherwise element is a default ( else ) that is executed if none of the when tests is true. The test attributes are XPath expressions. While they may look intimidating, GFM uses basic patterns. The data element that matches the match expression may be treated as an argument to the template. All expressions within the template are relative to it. Therefore, the reference to gfm:subj_obj_item_id means the value of the SUBJ OBJ ITEM ID in the current OBJ ITEM ASSOC. The second fragment uses an absolute path //gfm:obj_item/gfm:obj_item_id and means the value of the OBJ ITEM ID of any OBJ ITEM in the data. The = means is equal to. The entire expression may be summed up as For each OBJ ITEM ASSOC, is there an OBJ ITEM whose OBJ ITEM ID is equal to the Assoc s SUBJ OBJ ITEM ID? If so, do nothing (the when element is empty), otherwise call the named template missing-endpoint with the listed parameters. The code is shown in figure 21. <!-- parent or child of link does not exist --> <xsl:template name="missing-endpoint"> <xsl:param name="class"/> <xsl:param name="end-name"/> <xsl:param name="end-ewid"/> <xsl:param name="link-ewid"/> <li> <xsl:value-of select="$class"/> <xsl:text> s </xsl:text> <xsl:value-of select="$end-name"/> <xsl:text> (</xsl:text> <xsl:value-of select="$end-ewid"/> <xsl:text>) does not exist</xsl:text> <br /> <xsl:value-of select="$link-ewid"/> </li> </xsl:template> Figure 21. Named template missing-endpoint that produces output. The named template starts with the parameter names (argument list); the rest is written to the output file. It uses the XHTML tags li and br. In this case, the value-of element writes the value of the named parameter. The text element writes literals that contain blanks, which are shown as symbols. Sample output is shown in section

32 Going back to the original template, the first two parameters in figure 20 are literal strings to facilitate code re-use, while the other two are values taken from the current OBJ ITEM ASSOC data element. The second choose element tests for the existence of the child OBJ ITEM of the link. This template will detect a link whose parent, child, or both is missing from the GFM XML data file. This may not be the case with other templates, which may stop after the first error. The endpoint tests performed by this template could have been done with a key/keyref pair, and in fact, those tests were originally in the GFM XSD. Links to external data stored in other systems will be added, and these tests will be embellished to check for external indicators. The links between Object Types are more sophisticated, and their validation tests must be performed with XSLT. Figure 22 shows the first part of the template. It ensures that the parent Object Type and Object Type Establishment both exist in the GFM XML data file. Again, this may be a key/keyref in the XSD if external nodes are ignored. There are three types of Object Type links (OBJ TYPE ESTAB OBJT DET or OTEOD) in the GFM model, and they are designated by the GFM OTEOD ROLE IND CD element. A normal OTEOD has a role of 0, and currently that s the only value used in GFM data. A Type 1 Role is a placeholder that shows that an Organisation must be created, but the details are unknown. This type of OTEOD should not have a child node. The third possible value for a role is 2, but it is ignored for now and is treated as if it were a Type 0 Role. These tests, which occur in the second half of the template, are shown in figure 23. The outermost choose element checks if the OTEOD is a Type 1 Role; if so, the template exits. Otherwise, the child Object Type and Object Type Establishment are checked to make sure that they exist. As it is currently written, the only difference between this template and a group of key/keyrefs is the test of the role indicator code. The text() template with a mode of link and a priority of -1 is required because of all of the data elements that are not Assocs or OTEODs. Without this default template, all other data elements would be printed. 18

33 <!-- locate nodes where an endpoint is not an OBJ_TYPE(_ESTAB) --> <!-- exception: Type 1 Roles have a parent but not a child --> <!-- external OBJ_TYPEs are not supported yet --> <xsl:template priority="2" match="//gfm:obj_type_estab_objt_det" mode="link"> <xsl:choose> <xsl:when test="gfm:estabd_obj_type_id=//gfm:obj_type/gfm:obj_type_id"> </xsl:when> <xsl:otherwise> <xsl:call-template name="missing-endpoint"> <xsl:with-param name="class" select=" OBJ_TYPE_ESTAB_OBJT_DET "/> <xsl:with-param name="end-name" select=" ESTABD_OBJ_TYPE_ID "/> <xsl:with-param name="end-ewid" select="gfm:estabd_obj_type_id"/> <xsl:with-param name="link-ewid" select="gfm:obj_type_estab_objt_det_ix"/> </xsl:call-template> </xsl:otherwise> </xsl:choose> <xsl:choose> <xsl:when test="gfm:obj_type_estab_ix= //gfm:obj_type_estab/gfm:obj_type_estab_ix"> </xsl:when> <xsl:otherwise> <xsl:call-template name="missing-endpoint"> <xsl:with-param name="class" select=" OBJ_TYPE_ESTAB_OBJT_DET "/> <xsl:with-param name="end-name" select=" OBJ_TYPE_ESTAB_IX "/> <xsl:with-param name="end-ewid" select="gfm:obj_type_estab_ix"/> <xsl:with-param name="link-ewid" select="gfm:obj_type_estab_objt_det_ix"/> </xsl:call-template> </xsl:otherwise> </xsl:choose>... Figure 22. Template that checks for invalid OBJ TYPE ESTAB OBJT DET links, part 1. 19

34 ... <xsl:choose> <xsl:when test="gfm:gfm_oteod_role_ind_cd= 1 "> </xsl:when> <xsl:otherwise> <xsl:choose> <xsl:when test="gfm:det_obj_type_id=//gfm:obj_type/gfm:obj_type_id"> </xsl:when> <xsl:otherwise> <xsl:call-template name="missing-endpoint"> <xsl:with-param name="class" select=" OBJ_TYPE_ESTAB_OBJT_DET "/> <xsl:with-param name="end-name" select=" DET_OBJ_TYPE_ID "/> <xsl:with-param name="end-ewid" select="gfm:det_obj_type_id"/> <xsl:with-param name="link-ewid" select="gfm:obj_type_estab_objt_det_ix"/> </xsl:call-template> </xsl:otherwise> </xsl:choose> <xsl:choose> <xsl:when test="gfm:det_obj_type_estab_ix= //gfm:obj_type_estab/gfm:obj_type_estab_ix"> </xsl:when> <xsl:otherwise> <xsl:call-template name="missing-endpoint"> <xsl:with-param name="class" select=" OBJ_TYPE_ESTAB_OBJT_DET "/> <xsl:with-param name="end-name" select=" DET_OBJ_TYPE_ESTAB_IX "/> <xsl:with-param name="end-ewid" select="gfm:det_obj_type_estab_ix"/> <xsl:with-param name="link-ewid" select="gfm:obj_type_estab_objt_det_ix"/> </xsl:call-template> </xsl:otherwise> </xsl:choose> </xsl:otherwise> </xsl:choose> </xsl:template> Figure 23. Template that checks for invalid OBJ TYPE ESTAB OBJT DET links, part 2. 20

35 3.3.2 Category Code Validation Every object in the generalization hierarchy has a field named CAT CODE, which states the category of its child (if any). The first template in the XSLT file checks every element in the generalization hierarchy to make sure that the CAT CODE and child element agree. It does all of the work in the match attribute instead of using a procedural conditional element. Since it is looking for bad patterns, the template finds elements whose CAT CODE is not the same as the child. One of the many templates is shown in figure 24. <!-- locate nodes where child exists but catcode is incorrect --> <xsl:template priority="2" match="//gfm:obj_type[gfm:cat_code!= OR ][gfm:org_type]" mode="tree"> <xsl:call-template name="cc-mismatch"> <xsl:with-param name="class" select=" OBJ_TYPE "/> <xsl:with-param name="catcode" select="gfm:cat_code"/> <xsl:with-param name="subclass" select=" ORG_TYPE "/> <xsl:with-param name="ewid" select="gfm:obj_type_id"/> <xsl:with-param name="name" select="gfm:name_txt"/> </xsl:call-template> </xsl:template> Figure 24. Template that compares CAT CODE with child element. The XPath expression matches an OBJ TYPE that has an ORG TYPE child but whose CAT CODE element does not have the value OR. There are a total of 34 templates that test the combinations allowed by the model. If an element contains an error, the child elements are not tested because the transformation engine has already found a match Validation of Override Pairs The GFM model adds enumerated values to existing JC3 fields. To facilitate future data exchanges between GFM- and JC3-based systems, new fields were defined containing the new enumerated values. The implementation requirement states If the GFM field is NOS (no such), then use the value in the JC3 field, else use the GFM field s value. What this means in practice is that it is invalid for both fields to have non-nos values. 16 The template in figure 25 will be executed when an ORG is found where neither the CAT CODE nor the GFM CAT CODE is NOS. There are some JC3 fields that do not include the NOS value. In these cases, the test uses the first enumerated JC3 value instead. This value is relevant only when the GFM field is NOS, but the data should be consistent. There are 11 override templates. 16 Setting both fields to NOS is perfectly valid and means the desired value is NOS. 21

36 <!-- locate nodes where each field pair is not NOS --> <xsl:template priority="2" match="//gfm:org[gfm:cat_code!= NOS ][gfm:gfm_cat_code!= NOS ]" mode="override"> <xsl:call-template name="no-override"> <xsl:with-param name="class" select=" ORG "/> <xsl:with-param name="jc3-name" select=" CAT_CODE "/> <xsl:with-param name="jc3-value" select="gfm:cat_code"/> <xsl:with-param name="gfm-name" select=" GFM_CAT_CODE "/> <xsl:with-param name="gfm-value" select="gfm:gfm_cat_code"/> <xsl:with-param name="ewid" select="gfm:org_id"/> </xsl:call-template> </xsl:template> Figure 25. Template that finds incorrectly overridden fields Validation of Date/Time Groups Many tables contain a pair of elements that define the start and termination date/time group (DTG) of the object; the start DTG must be before (less than) the termination DTG. This would normally not be a difficult test, but XSLT cannot convert the string into an actual date. 17 The trick is to delete extraneous characters, convert each string into a number, and compare the values. This may be safely done because the DTGs have been validated against the XSD, and the type has a strict pattern that must be adhered to, which includes leading zeroes where needed. The function translate is used to replace a set of characters with an empty string. This string is then converted into a number with the number function. To give an example, a typical DTG is T00:00:00Z. Removing the dashes, colons, T, and Z gives Because the date is stored in year-month-day form, the numerical values may be compared. The template for a start and termination pair is shown in figure 26. This is another example of a template that matches a set of elements, then uses a conditional (in this case, an if element) to check for an invalid condition. There are 12 templates that test DTG ranges. There are two fields whose values must be within the start and termination DTG range. The valid expression is s dtg date time < t dtg. Unfortunately, XPath 1.0 does not have a operator, so the first part of the test must be rewritten as not (s dtg > date time). The > and < and symbols are not allowed in a when element and have been replaced by > and <, respectively. The template that tests both the start and termination DTG range and the effective datetime is shown in figure There may be extensions that do this, and newer versions of XML and XSLT may have this capability, but GFM has been restricted to the original versions because they are widely available. 22

37 <!-- locate objects where an S_DTG is not less than the T_DTG --> <xsl:template priority="2" match="//gfm:obj_type" mode="dtg"> <xsl:if test="number(translate(gfm:gfm_obj_type_s_dtg, -T:Z, )) >= number(translate(gfm:gfm_obj_type_t_dtg, -T:Z, ))"> <xsl:call-template name="bad-dtg-range"> <xsl:with-param name="class" select=" OBJ_TYPE "/> <xsl:with-param name="s-dtg" select="gfm:gfm_obj_type_s_dtg"/> <xsl:with-param name="t-dtg" select="gfm:gfm_obj_type_t_dtg"/> <xsl:with-param name="ewid" select="gfm:obj_type_id"/> <xsl:with-param name="name" </xsl:call-template> </xsl:if> </xsl:template> Figure 26. Template that finds invalid DTG ranges. select="gfm:name_txt"/> The comment block at the top of the figure summarizes the actions performed. The EFFCTV DTTM element is optional, and errors will be falsely reported if it is absent from the data file. The expression test="gfm:effctv DTTM" is true if the EFFCTV DTTM element has a value. The format for EFFCTV DTTM is purely numeric, with no internal separators between the fields, so no translation is needed. The other value that must be within the start and termination DTG range is GFM ORG ORGT M DTG and its template is not shown because of its complexity. It is based on figure Detection of Mandatory Elements Many of the tests that have been discussed thus far have checked referential integrity, e.g., do both endpoints of a link exist? There is another requirement that says certain elements must contain a reference to another element. The next pair of tests ensure that the referring element exists. The must-have-ote template, shown in figure 28, finds all OBJ TYPE elements where the OBJ TYPE ID is not equal to an OBJ TYPE ESTAB s ESTABD OBJ TYPE ID. In other words, each Object Type must have an Object Type Establishment. A similar template, using the must-have-oiote mode, tests the requirement that every Object Item must be linked to an Object Type with an Object Item Object Type Establishment (OBJ ITEM OBJ TYPE ESTAB or OIOTE). This template is not shown Type Associated With Proper Item The must-have-oiote template ensures that every Object Item is associated with an Object Type. It is also important to verify that the Item and Type are compatible. For example, a GFM CREW PLATFORM must be associated with a GFM CREW PLATFORM TYPE. The Object Type to Object Item mappings are shown in table 1. Notice that a GOVT ORG TYPE may be a leaf node in the data hierarchy only if it has a category code of INTCIV, INTCMI,orNATCIV. 23

38 <!-- Pseudocode to explain this template. if (S_DTG >= T_DTG) report bad-dtg-range error else if not null(effctv_dttm) { if (not (S_DTG > EFFCTV_DTTM) and (EFFCTV_DTTM < T_DTG)) do nothing else report date-outside-range error } --> <xsl:template priority="2" match="//gfm:obj_type_estab" mode="dtg"> <xsl:choose> <xsl:when test="number(translate(gfm:gfm_obj_type_estab_s_dtg, -T:Z, )) >= number(translate(gfm:gfm_obj_type_estab_t_dtg, -T:Z, ))"> <xsl:call-template name="bad-dtg-range">... </xsl:call-template> </xsl:when> <xsl:when test="gfm:effctv_dttm"> <xsl:choose> <xsl:when test="not(number(translate(gfm:gfm_obj_type_estab_s_dtg, -T:Z, )) > number(gfm:effctv_dttm)) and number(gfm:effctv_dttm) < number(translate(gfm:gfm_obj_type_estab_t_dtg, -T:Z, ))"> </xsl:when> <xsl:otherwise> <xsl:call-template name="date-outside-range"> <xsl:with-param name="class" select=" OBJ_TYPE_ESTAB "/> <xsl:with-param name="field-name" select=" EFFCTV_DTTM "/> <xsl:with-param name="field-value" select="gfm:effctv_dttm"/> <xsl:with-param name="s-dtg" select="gfm:gfm_obj_type_estab_s_dtg"/> <xsl:with-param name="t-dtg" select="gfm:gfm_obj_type_estab_t_dtg"/> <xsl:with-param name="ewid" select="gfm:obj_type_estab_ix"/> <xsl:with-param name="name" </xsl:call-template> </xsl:otherwise> </xsl:choose> </xsl:when> </xsl:choose> </xsl:template> Figure 27. Template that finds invalid effective datetime values. select=" "/> 24

39 <!-- find OBJ_TYPE that does not have an OBJ_TYPE_ESTAB --> <xsl:template priority="2" match="//gfm:obj_type" mode="must-have-ote"> <xsl:choose> <xsl:when test="gfm:obj_type_id= //gfm:obj_type_estab/gfm:estabd_obj_type_id"> </xsl:when> <xsl:otherwise> <xsl:call-template name="missing-obj"> <xsl:with-param name="class" select=" OBJ_TYPE "/> <xsl:with-param name="miss-class" select=" OBJ_TYPE_ESTAB "/> <xsl:with-param name="ewid" select="gfm:obj_type_id"/> <xsl:with-param name="name" </xsl:call-template> </xsl:otherwise> </xsl:choose> </xsl:template> select="gfm:name_txt"/> Figure 28. Template that finds Object Types without Establishments. Table 1. Object Type to Object Item associations. Object Type Exctv Mil Org Type Unit Type Task Frmtn Type Crew Platform Type Civ Post Type Mil Post Type Prv Sctr Org Type Group Org Type Govt Org Type INTCIV INTCMI NATCIV Object Item Unit Crew Platform Billet X X X X X X X X X X X A portion of the item-matches-type template is shown in figure 29. The outer conditional determines the type of the Object Item, and each inner test verifies that Object Type is an allowable type. The code for the Crew Platform is shown because it is the simplest. 25

40 <!-- find OBJ_ITEM that does not link to proper type of OBJ_TYPE --> <xsl:template priority="2" match="//gfm:obj_item_obj_type_estab" mode="item-matches-type"> <xsl:variable name="item-ewid" select="gfm:obj_item_id"/> <xsl:choose> <xsl:when test="gfm:obj_item_id=//gfm:gfm_crew_platform/gfm:gfm_crew_platform_id"> <xsl:choose> <xsl:when test="gfm:estabd_obj_type_id= //gfm:gfm_crew_platform_type/gfm:gfm_crew_platform_type_id"> </xsl:when> <xsl:otherwise> <xsl:call-template name="item-type-mismatch"> <xsl:with-param name="item-name" select= "//gfm:obj_item[gfm:obj_item_id=$item-ewid]/gfm:name_txt"/> <xsl:with-param name="item-ewid" select="$item-ewid"/> <xsl:with-param name="item-class" select=" GFM_CREW_PLATFORM_TYPE "/> <xsl:with-param name="oiote-ewid" select="gfm:obj_item_obj_type_estab_ix"/> </xsl:call-template> </xsl:otherwise> </xsl:choose> </xsl:when>... </xsl:choose> </xsl:template> Figure 29. Template that finds Object Items with incorrect Object Types Consistent References In the JC3 model, an OBJ TYPE ESTAB is an index to an OBJ TYPE, and together they make up a compound key. Each Establishment may be thought of as a numbered variant on an Object Type. All references are required to provide both the OBJ TYPE ID and the OBJ TYPE ESTAB IX to uniquely identify the Object Type Establishment. GFM replaces the integer index with an EwID. The compound keys still function correctly, but redundancy has been introduced to the references as shown in figure 30. In this example, the OBJ ITEM OBJ TYPE ESTAB directly references the OBJ TYPE and indirectly via the OBJ TYPE ESTAB. The template with the mode objtype-matches-estab ensures that the OBJ TYPE in a reference has the same value as the OBJ TYPE in the referenced OBJ TYPE ESTAB. One of the templates for mode objtype-matches-estab is shown in figure

41 Figure 30. Redundant reference. <!-- OBJ_TYPE/ESTAB references must match the OTE s OBJ_TYPE --> <!-- check OIOTEs because they have dual references --> <xsl:template priority="2" match="//gfm:obj_item_obj_type_estab" mode="objtype-matches-estab"> <xsl:variable name="estab-ewid" select="gfm:obj_type_estab_ix"/> <xsl:choose> <xsl:when test="gfm:estabd_obj_type_id=//gfm:obj_type_estab [gfm:obj_type_estab_ix=$estab-ewid]/gfm:estabd_obj_type_id"> </xsl:when> <xsl:otherwise> <xsl:call-template name="objtype-estab-mismatch"> <xsl:with-param name="link-class" select=" OBJ_ITEM_OBJ_TYPE_ESTAB "/> <xsl:with-param name="link-ewid" select="gfm:obj_item_obj_type_estab_ix"/> <xsl:with-param name="estab-ewid" select="$estab-ewid"/> <xsl:with-param name="objtype-ewid" select= "gfm:estabd_obj_type_id"/> </xsl:call-template> </xsl:otherwise> </xsl:choose> </xsl:template> Figure 31. Template that finds Object Items with incorrect Object Types Single Root Node Each file is expected to contain an Organisation tree root node and, optionally, an OrgType tree root node. While it is possible for an XML dump of a GFM database to contain multiple roots 27