API Standard and Guidelines Part B - Technical. Version 1.0

Transcription

1 API Standard and Guidelines Part B - Technical Version 1.0 Approved October 2016

2 Crown copyright. This copyright work is licensed under the Creative Commons Attribution 3.0 New Zealand licence. In essence, you are free to copy, distribute and adapt the work, as long as you attribute the work to the Department of Internal Affairs and abide by the other licence terms. To view a copy of this licence, visit Please note that neither the Department of Internal Affairs emblem nor the New Zealand Government logo may be used in any way which infringes any provision of the Flags, Emblems, and Names Protection Act 1981 or would infringe such provision if the relevant use occurred within New Zealand. Attribution to the Department of Internal Affairs should be in written form and not by reproduction of the Department of Internal Affairs emblem or New Zealand Government logo. Published by the Department of Internal Affairs Part B Technical v1.0 Page 2 of 126

3 Table of Contents 1 API Security Introduction Definitions Standards for Securing RESTful APIs Risks Mitigation Approach Security Reference Architecture Actors and Security Functional Capabilities Building Secure APIs API Security Design Principles Authentication & Authorisation Basics OAuth 2.0 Basics Usage Patterns Authentication Authentication Techniques Anonymous Authentication Username and Password Authentication (Direct Authentication) API Keys Authentication Certificates (Mutual) Authentication Developer Authentication Multi Factor Authentication (MFA) OAuth 2.0, OpenID Connect and SAML OAuth Grant Types Basic OAuth Implementation Patterns Quick Reference Table Tokens & Credentials OAuth Scenario (Authorisation Code Grant Flow) OpenID Connect and SAML (Security Tokens) Distributed Model Profiles User Managed Access (UMA) Chained Authorisation Authorisation Authorisation Techniques Part B Technical v1.0 Page 3 of 126

4 1.6.2 API Gateway ABAC Implementation Scopes (Limited Fine Grain Access) Confidentiality and Integrity Communications Security (Confidentiality and Integrity) Token Audience and State Parameters (Integrity) Content Encryption (Confidentiality) Content Signing (Integrity) Non Repudiation (Integrity) Availability and Threat Protection Token Threat Mitigation API Development API Artefacts API Design When is an API appropriate? API Design Principles Designing an API Design Considerations Design Driven Development Granularity Interface Specification Orchestration Software Development Kits (SDK) HTTP Verbs GET POST PUT DELETE OPTIONS Other URI Construction API Offering Version Namespaces Resources and Sub-resources Part B Technical v1.0 Page 4 of 126

5 2.4.5 Word Separation Query Arguments HTTP Headers Request Headers Response Headers Custom X- HTTP Headers Returned Content Resource Scope Formats Layout JSON Property Names Consistency Singletons vs. Collections HATEOAS Consideration of state Batch Handling & Transactions API Version Control API Version Control Methods When to Version Software Configuration Management Search Caching Response Cache Object Cache Error Handling HTTP Status Codes API-Specific Error Code Human-Readable Error Message API Governance Glossary Further Reading Appendix A - IETF NFC Relating to OAuth Appendix B - RFCs in Development Appendix C - Example Interface Specification Part B Technical v1.0 Page 5 of 126

6 TABLE OF FIGURES Figure 1: API Security Considerations... 8 Figure 2: Identity Centric... 9 Figure 3: API Actors and Core Components Figure 4: Split API Support Stack Figure 5: OAuth 2.0 components Figure 6: Internal API Security Figure 7: Developer Authentication to API Access Figure 8: Consuming Application Identification Figure 9: System to System Authorisation Figure 10: Consuming Application Authorisation Figure 11: Delegated Authority Figure 12: Authentication Options Figure 13: Anonymous Authentication Model Figure 14: HTTP Basic/Digest Authentication Model Figure 15: API Key Authentication Figure 16: Certificate Authentication Figure 17: API Portal Login Page Figure 18: OAuth 2 Implementation Models Figure 19: OAuth 2.0 Models (colocation of Authorisation and Authentication services) Figure 20: Stage 2 Client Registration Figure 21: Stage 3 Client Registration Figure 22: Stage 4 Client Registration Figure 23: Stage 5 Client Registration Figure 24: Stage 6 Client Registration Figure 25: Stage 7 Client Registration Figure 26: Stage 8 Client Registration Figure 27: OpenID Connect - Distributed Authorisation and Authentication Server Figure 28: OpenID Connect and OAuth Figure 29: Distributed Authorisation Models Figure 30: UMA Figure 31: Chained Authorisation Figure 32: Authorisation Techniques Figure 33: API Gateway for Authorisation Part B Technical v1.0 Page 6 of 126

7 Figure 34: XACML Reference Architecture Figure 35: ABAC Support for APIs Figure 36: API Layering TABLE OF TABLES Table 1 - API Security Standards Table 2 - Actors and Devices Table 3 - Core Components Table 4 - OAuth 2.0 Grant types Table 5 - Quick Reference Guide Table 6 - Tokens and Credentials Table 7 - Risks and Mitigations Table 8 - Token Threats Table 9 - HTTP Status Codes Part B Technical v1.0 Page 7 of 126

8 1 API Security 1.1 Introduction Securing RESTful APIs is fundamental to the success of any API Strategy or implementation; any approach should include the following three key areas: 1. Domain of Consideration 2. Domain of Control 3. Identity-centric and Holistic View 1. Domain of Consideration Developing and securing RESTful APIs is more than just applying standards; it is a framework and state of mind that has to be understood and followed jointly by the business owners, IT architects and developers. The API Security framework must be defined at the organisation and business level and should always consider who, how and what users and applications (both internal and external to an organisation) will interact with the APIs. These considerations should be defined at the beginning of any project and driven from a desired business outcome e.g. provide real time information for the public about the closest location and address of a GP. Figure 1: API Security Considerations 2. Domain of Control The Domain of Control contains the components (defined further in this document) that need to be developed, deployed and work together to provide API security to support: Registered application developer access to the API Authenticated and authorised consuming application access to the API Part B Technical v1.0 Page 8 of 126

9 Protected communication between the API and the consuming application to ensure confidentiality and integrity The ability for applications to act on behalf of a customer 3. Identity Centric and Holistic View The security of APIs should not just be seen as a bounded solution i.e. only the components illustrated in security considerations above, but needs to be seen from a holistic perspective. It needs to incorporate existing enterprise security frameworks where the management and understanding of user identities is core. For example, securing an API that is targeted for a mobile application is not just about applying an OAuth profile, it should take into consideration how mobile devices and applications are managed and secured and how the enterprise security framework (e.g. authentication) can be leveraged. Figure 2: Identity Centric People- (or User-) centric security frameworks are key to defining the required access policies and controls for APIs. The management of Identity (this includes users, device, servers and applications) should be central to any API security framework Definitions For this standard the following definitions are used: 1. Authentication is the process of verifying the identity of a customer (or device) who presents identity credentials and authentication key(s); 2. Authentication Authority is a system entity that provides authentication services to ensure only permitted customers (or devices) gain access 3. Authorisation is the process of verifying that a customer (or device) has the right to perform an action and what they are allowed to access; 4. Availability is the ability to minimise API downtime by implementing threat protection; 5. Confidentiality is the ability to ensure information that is sent between Users, Applications and Servers is only visible to those authorised to use it; 6. Delegation is when a user authorises another user (or device) to serve as his or her representative for a particular task; Part B Technical v1.0 Page 9 of 126

10 7. Delegated Authorisation is a framework that defines how an owner of a set of resources can grant access (delegate) to a designated user or consuming application to perform actions on some of those resources on the owner s behalf, but without sharing their credentials; 8. Federation is the process that allows for the leverage and reuse of identity credentials to multiple Authentication Authorities for authentication and/or Single Sign On; 9. Integrity is the ability to ensure that information received has not been modified by a third party, also providing non-repudiation services; 10. Provisioning is the automated or manual service for aggregating and correlating identity data resulting in the creation of user (IT) accounts and the delivery of user meta data used by systems to define access policies and controls for services. 11. Threat protection is the service for protecting APIs (at the ingress and egress points of an organisation) from known threats (e.g. the OWASP top 10) by preventing misuse or loss of availability. Note: Threat protection should also be addressed at the OS hardening level and should be an integral part of the API software development; 12. User Managed Access has been developed to provide a user data delegation model which enables a resource owner to control the authorisation of data sharing and other protectedresource access made between online services on the owner s behalf or with the owner s authorisation by an autonomous requesting party; Note: A customer/user can be internal or external to a Government agency Standards for Securing RESTful APIs The table below captures (current) security standards that should be part of any API Security Strategy, ordered by type. The provisioning standards are included to complete the Identity and Access Standards table. This enables it to be used by architects to select the most appropriate option. Provisioning Provides the framework for managing the provisioning of user and their access to APIs e.g. the process for internal or external developers gaining access to API development and publishing services SPML Service Provisioning Markup Language is an XML based framework for facilitating the exchange of provisioning information (creates, updates and deletes) on user objects (e.g. an LDAP Directory). This is normally implemented to provide Just in Time (Real Time) provisioning. Although now regarded by most as a legacy standard, it is still supported by most vendors and used by niche service vendors. SCIM System for Cross-domain Identity Management. This is a RESTful API-based framework for Just In Time provisioning and, like SPML, moves away from batch- and delta-based provisioning processes. It uses a RESTful API to manage the provisioning of users. Part B Technical v1.0 Page 10 of 126

11 As SCIM is a REST API framework it should be secured using OAuth. Federation / Authentication Provides authentication and Single Sign On services to customers and secure transportation of authentication and authorisation information. SAML Security Assertion Markup Language (SAML) is an XML-based, openstandard data format for exchanging authentication and authorisation data between parties; in particular, between an identity provider and a service provider. SAML is seen as complex but is regarded as a high-level security framework. As it is based on XML, SAML can have high payload overheads and thus can result in performance issues in the mobile application space. It is still widely used but its uptake is declining. It is included in this standard to support existing New Zealand SAML instances (e.g. education, RealMe). OpenID Connect OpenID Connect is an interoperability authentication protocol based on the OAuth 2.0 framework. This is a relatively recent federation protocol that provides lightweight federation services. It, like SAML, provides SSO services and allows the secure exchange of user authentication data, but it is not as feature rich as SAML. As it is based on REST/JSON, it is perceived as the Federation service of choice for mobile services. Delegated Authorisation Provides a framework for delegating the specified access rights to a 3 rd party. OAuth 1.0a OAuth 1.0a is derived from the original OAuth 1.0 specification (RFC 5849) which provides a method for client applications to access resources on behalf of a resource owner. It is an authentication framework around the exchange of signed tokens. It has now been made obsolete by OAuth 2.0. OAuth 2.0 OAuth 2.0 is an open standard for a delegated authorisation framework. It is not backward compatible with OAuth 1.0, but is modelled on the framework with the objective of providing greater flexibility and defines specific credential (grant) flows. Part B Technical v1.0 Page 11 of 126

12 It is also based on token exchange, with the primary difference being that the tokens are secured by mandating TLS on all communication connections (RFC 6749), where the OAuth 1 tokens are digitally signed. Authorisation Standards Provides a framework for controlling access to resources. XACML The extensible Access Control Markup Language standard is XML based and defines a fine-grained attribute-based access control policy language. It provides an architectural model and policy terminology that can be used to separate out the functions of any authorisation framework. As it is based on XML it is sometime perceived as a legacy standard, but, from a RESTful API perspective, it provides a fine-grained attribute-based authorisation framework. (RFC 7061) UMA User-Managed Access is an OAuth-based access management protocol standard. It builds on OAuth 2.0 to provide additional delegated authorisation capabilities. Its key focus, for RESTful APIs, is to enable an individual (Resource Owner) to manage and define a set of access control policies that can be managed by an Authorisation Server, which controls access to a set of APIs. ALFA Abbreviated Language For Authorisation defines fine-grained authorisation rules in a JSON-like policy language. This language helps remove one of the greatest barriers for implementing XACML, which is complexity in writing of the access control policies. Two (Multi)-factor Authentication This is a method of confirming a user s claimed identity (authentication) by using two or more pieces of evidence, relating to something they know, something they have and something they are. U2F / FIDO Universal 2 nd Factor is an open authentication standard that can be incorporated into an API security framework Part B Technical v1.0 Page 12 of 126

13 TOTP / HOTP Time-based and HMAC-based One-Time Password. These can be used in any of the authentication processes that are part of the process of gaining access to APIs. The requirements for these would be based on business requirements and risk analysis of the information or service being exposed by the API. These could be used to add a second authentication factor to any existing or proposed authorisation mechanisms. JSON Security Standards This is a set of standards that provide security around the exchange of tokens, based on JSON. JWT A JSON Web Token is designed to be compact and provides trusted information that is used in the authentication process. Used in API security to pass identity information, specifically by OpenID Connect. JWE JSON Web Encryption standard provides integrity validation and can be used with or without digital signatures. JWS JSON Web Signature is a standard for signing JSON, thus providing a level of authority (where it has come from) and integrity, by proving the JWT hasn t been changed in transit. JWA JSON Web Algorithm defines the algorithms used for encrypting the JWT. JWK JSON Web Keys represent the cryptographic key used for encrypting JWT. The algorithms for these are defined in JWA. Table 1 - API Security Standards Risks APIs are another channel into an organisation s resources and information. Most organisations are accustomed to exposing a web interface, with good control over what information is released via that interface. APIs offer direct, machine to machine access to resources and information, which makes it less obvious when information is incorrectly exposed. It becomes increasingly important for internal business stakeholders to decide what information and resources should be released via this channel, and to whom. The security risks that APIs introduce will be similar to the traditional risks experienced on any web channel (web sites and web applications), except there is: Increased attack surface due to more ways in, multiple services to potentially exploit Risk of inadvertently exposing back-end data, back-end architecture and back-end applications Part B Technical v1.0 Page 13 of 126

14 Risks posed by APIs include loss of integrity, confidentiality and availability of data, for example: Loopholes retrieving API resources may offer access to more information than was intended (especially if fields requested are built straight into a DB query) Write operations offer a means of polluting data stores, feeding misinformation into a system Write operations could be used to form a Denial of Service attack by overloading the server or data store Use of wildcards in search fields can shut down APIs and back-end applications Cross site scripting attacks made possible by consuming applications not checking user inputs SQL injection into consuming applications which cause database damage at the API backend Parameter attacks such as HTTP Parameter Pollution (HPP) Man-in-the-middle attacks, modifying API requests or responses leading to data eavesdropping or misinformation insertion Subverting authentication or authorisation mechanisms to spoof messages from legitimate consumers Stealing authentication tokens to obtain information illicitly System information leakage through API error messages revealing details about an API s construction or underlying system makeup Broken Session IDs, Keys and authentication create exposure to unauthorized access through authentication factors that are not functioning because of poor security design or technology bugs Mitigation Approach API risks need to be mitigated in a number of ways. There is no single off-the-shelf security solution which can be dropped in to address all aspects of API security. APIs need to be secure by design; security needs to be built in from scratch, and be considered within the context of existing protection mechanisms. The main areas that API Security covers are: 1. Identity and Access Management (IdAM) to provide the following services: o o o Authentication Authorisation and delegated authority Federation 2. Confidentiality 3. Integrity 4. Availability and Threat Protection This ensures that: Part B Technical v1.0 Page 14 of 126

15 The consuming application is known and can only get access to API resources they are allowed to Message content has not been tampered with between consumer and provider Resources are reliably from the provider intended when the consuming application made the request The API will be available when needed, and not brought down by attacks from malicious consuming applications In order to address API security risks, a security framework is needed which encapsulates all the aspects of security listed above. 1.2 Security Reference Architecture This section describes an API Security Reference Architecture and its component parts to inform the construction of an API security framework Actors and Security Functional Capabilities Identity and Access Management defines the actors (users and devices) who interact with system components that manage and expose APIs. Figure 3 shows a typical model of API components (support stack) and actors. The actors and components are described in Table 2 and Table 3. Figure 3: API Actors and Core Components The green areas highlight internal actors while the yellow areas highlight the external actors. The components defined remain valid no matter what API architecture (internal, cloud, hybrid) is implemented. Part B Technical v1.0 Page 15 of 126

16 Actors and Devices External Users Customers - the public, other agencies and partners who use consuming applications to access resources via APIs. External application developers who build software to access resources via APIs Devices PC Browsers running web applications Mobile Devices running apps Servers running systems with server-to server communications Internal Users Internal API Developers who build APIs Internal Application Developers who build software which accesses resources via APIs Business Owners responsible for the API product(s) Security responsible for ensuring the APIs are secure Table 2 - Actors and Devices The core components of an API Security Framework (the development portal, manager and gateway) provide a grouping of functionality. These functions can be delivered with discrete applications, or bespoke code development, via COTS products or through leveraging existing devices that can be configured to provide these functions / services. Note: some of the functionality may overlap or be combined into one or more products depending on the vendor used. The following lists the functions of a mature API delivery and security framework for an agency that is working with the development community. Together, these functions provide full support for the application developer building and developing consuming applications that will use the API(s) exposed by the agency. Depending on the requirements of the agency, some of these functions might not be required e.g. if the agency API exposed is purely for public consumption and only allows consuming applications to read information, then only a solution for enforcing threat protection (i.e. Denial of Service) might be required, and this could be delivered using an existing service protection capability. Part B Technical v1.0 Page 16 of 126

17 Core Components API Portal API Manager The API Portal often provides the following functions for internal and external application developers: Discovery of APIs Analytics to monitor APIs Access to specifications and descriptions of APIs, including SLAs Social network capability to share and publish ideas Also supports the development, build and test of consuming applications. The API Manager functions cover: Centralised API administration and governance for API catalogues Management of registration and on-boarding processes for communities of API developers Lifecycle Management of APIs Applying pre-defined security profiles Security policy administration / definition Policy evaluation API Gateway API Monitoring and Analytics The API Gateway capability can provide the following: Act as the API proxy or the host acting as the primary point of access for exposed APIs Enforce threat protection, throttling and quota management Authorisation Services to control access to APIs Authentication Services to ensure only permitted users (internal/external) have access to the API Security Policy enforcement Monitoring and analytics for business and security analysts Business owners and security specialists need to be able to monitor the use of APIs: Monitor uptake of API services Define when to deprecate an old version Profile usage for business Profile usage for security baselines This helps adapt to change in usage/demand Part B Technical v1.0 Page 17 of 126

18 Core Components Credential Stores The credential stores are identity and key stores which are used to securely store: Internal and external user objects, and possibly groups API keys and secrets, certificates etc. These stores are used by the API Gateway for authorisation and authentication services Table 3 - Core Components The model can also be split, with the API support stack duplicated one set to support internal API usage and one set to support external use: Figure 4: Split API Support Stack Authentication, authorisation, confidentiality, integrity and availability can be applied across the components in the support stack, depending on component capabilities. The actual configuration and location of the API functional capabilities will vary depending on individual circumstances (e.g. some capabilities may be internal, some may be in the cloud, where API development is outsourced then internal functional stack may belong to the outsourcer etc.). Also, some components might not be required or can be developed in house Building Secure APIs Building in security starts from the ground up, so development of APIs needs to be done with awareness of the API security risks associated with the resources and information being exposed, and with appropriate mitigations in place for these API security risks. Part B Technical v1.0 Page 18 of 126

19 When developing an API is it advisable to carefully consider potential malicious use, especially: PUTs and POSTs which change internal data and could be used to attack or misinform DELETEs which could be used to remove the contents of an internal resource repository Standard secure coding practices are always recommended (see OWASP Secure Coding Principles), in line with NZISM guidance. But API development should take place with awareness of the: Design Driven Development (see section 2.2.5, Design Driven Development) OWASP Top Ten Cheat Sheet A summary of the standard attacks and mitigations REST Security Cheat Sheet REST-specific risks and how to prevent them, e.g. input validation OWASP API Security Project work in progress on API security guidance But it is worth also considering the following where your API accepts input values as parameters: OWASP Input Validation Cheat Sheet A summary of input risks and mitigations OWASP XSS Cheat Sheet how to escape inputs to prevent cross site scripting OWASP SQL Injection Cheat Sheet ensuring DB queries are built internally OWASP Parameterization Cheat Sheet examples of SQL injection and stored proc vulnerabilities It is also recommended that a security testing capability be incorporated into the development cycle which provides continuous, repeatable and automated tests to find security vulnerabilities in APIs and web applications during development and testing API Security Design Principles The following are key principles that should be applied when designing API security frameworks: 1. Design with the objective that the API will eventually be accessible from the public internet, even if there are no plans to do so at the moment 2. Use a common authentication and authorisation pattern, preferably based on existing security components: avoid creating a bespoke solution for each API 3. Least Privilege - Access and authorisation should be assigned to API consumers based on the minimal amount of access they need to carry out the functions required 4. Maximise entropy (randomness) of security credentials by using API Keys rather than username and passwords for API authorisation, as API Keys provide an attack surface that is more challenging for potential attackers 5. Balance performance with security with reference to key life times and encryption / decryption overheads 1.3 Authentication & Authorisation Basics Before looking at the technical solutions to API authentication and authorisation, this section will provide an introduction that illustrates the situations where authentication and authorisation are appropriate. Part B Technical v1.0 Page 19 of 126

20 Authorisation and authentication are intrinsically linked inside the OAuth framework which in itself is regarded as synonymous with securing APIs. OAuth uses its own terminology which is worth becoming familiar with before adopting an OAuth approach. As the OAuth framework is a commonly accepted approach to the securing of modern APIs (large companies like Google, Microsoft and Twitter use it), this section also covers an introduction to OAuth OAuth 2.0 Basics OAuth provides a more comprehensive and extensible approach to security than some of the basic authentication and authorisation mechanisms. Based on security tokens, it can be used for delegated authority such as enabling a mobile application to act on behalf of its user. The IT industry perceive the need for any production quality API security framework to be based on OAuth 2.0. In reality OAuth 2.0 is a delegated authorisation framework but it provides the foundations on which secure services can be built in order to provide the complete security solution. OAuth requires some fundamental security components in order to work, and has its own terminology for describing these components and their roles: Figure 5: OAuth 2.0 components 1. Resource Owner the person who has the right to grant a third party (e.g. a consuming application) access to a protected resource (e.g. information about themselves). Quite often the resource owner is the customer. 2. Client (or Client Application) A consuming application requesting access to a protected resource on behalf of a third party e.g. a mobile application on a user s (third party) smartphone or a web application accessed via a browser. 3. OAuth Server / Authorisation Server provides a Security Token Server / Infrastructure for managing tokens. It is responsible for issuing: o o o Authorisation (code) grant approval tokens driven by Resource Owner approval Access tokens used by the API to authorise access Refresh tokens which allow new access tokens to be requested by the client and re-issued within a specified timeframe Part B Technical v1.0 Page 20 of 126

21 4. Authentication Server This is not a component of OAuth 2.0, or defined by OAuth 2.0, but needs to be considered when defining a complete OAuth 2.0 framework. This could be a simple login capability or managed by an Identity Service Provider. 5. Resource Server This hosts the protected resources (APIs & backend applications) which only allow authenticated and authorised clients by: o o Checking the access token in each incoming API request Validating the access token against the Authorisation Server and the permitted access rights OAuth 2.0 comes with four types of grant flows, each appropriate to different situations and solution requirements: Client Credentials Resource Owner Password Credentials Authorisation Code Implicit OAuth 2.0 is appropriate where there is a requirement for third party applications to access restricted resources. This should help mitigate the risks relating to: Third party applications storing user credentials (username and password) Resource servers having to support user stores and password authentication Resource owners not being able to define granular access to resources, including duration Revoking credentials that are compromised For more details, see section 1.5, OAuth Usage Patterns Different API usage patterns require different authentication and authorisation models. Below is a short definition of each of the different usage patterns. Note: The Security components defined in the following diagrams are located, for simplicity, in the trusted zone i.e. an area managed by an agency. It is possible that these components could reside in different zones which relate to varying levels of trust (e.g. a DMZ) Internal Use only In this pattern an API is developed for internal use only by agency applications/systems. Part B Technical v1.0 Page 21 of 126

22 Figure 6: Internal API Security We see there is the need to authenticate and authorise the internal use to the internal application, and implement protection between the internal application and the API on the API Gateway Application Developer When an API is released for external use, the first interaction will be with App Developers who want to try the API out. This will normally be via the API Developer Portal. Figure 7: Developer Authentication to API Access The Application Developer needs to be authenticated to the API Developer Portal to register their new application and attain the relevant credentials which are used to secure interactions with the new application during development. The developer has to agree to the conditions of use and subsequent usage of the API can then be traced via the API keys (see Identify Consuming Application below). Part B Technical v1.0 Page 22 of 126

23 Identify Consuming Application This pattern applies when the API provider needs to know which consuming applications are using their APIs (for communication, logging and analytics purposes). Figure 8: Consuming Application Identification The consuming application is authenticated (e.g. API Key, Client Secret etc.) to use the API, but this is only used as a means of identification or registration System to System Authorisation (B2B) The system to system model is where an API is being used to enable information sharing or integration with an external system e.g. a partner agency gaining access to supporting information. Figure 9: System to System Authorisation In this model the aim is to ensure that only the correct consuming system has access to the API, and that the API is protected from malicious use. B2B models often carry sensitive information, so the consuming system needs to be authenticated to the provider for authorised access, confidentiality and integrity Authorise Consuming Application Here the pattern covers the case where different external consuming applications may be granted different levels of access to resource(s). The application s access is not dependent on which customer is currently using the application, but on which application is using the API e.g. perhaps the developer for App A pays a fee so App A gets a different quality of service from the API than App B. Part B Technical v1.0 Page 23 of 126

24 Figure 10: Consuming Application Authorisation The consuming applications must be authenticated and authorised before accessing the API. This is normally enforced at the API gateway Customer Authorisation (Delegated Authority) In the final pattern, external consuming applications may be granted different access to resource(s) depending on which customer is currently using the application e.g. learner authorises a mobile application to retrieve their own record of achievement. Figure 11: Delegated Authority The customer authenticates to the device. The device and/or the application is already authorised to use the API. The customer logs in and authorises the device and/or application to access their information. E.g. an internet banking application. 1.4 Authentication REQUIRED When securing APIs, authentication is required to identify the consumers and/or consuming applications that want to access or consume an API. Authentication enables the API provider Part B Technical v1.0 Page 24 of 126

25 to identify all consumers of an API and to confirm that the consumer requesting access is who they say they are. This doesn t automatically authorise them to access the APIs or the underlying resources. Providers should define a registration process for each category of consumer, whether system or human. The ability of knowing who is using an API cannot be overstated. This is critical when it comes time to implement aspects of the API service lifecycle, such as service deprecation, or notification of an outage. It also enables the API provider to implement different service levels for different consumers. E.g. commercial customers might have a higher request limit per day than customers not paying for the service. Making application developers register for use of the API also means they must sign up to terms and conditions that define how they might use the data they get from the API, and that they agree to ensure that their consuming applications will behave in an acceptable and non-abusive manner Authentication Techniques The diagram and table below lists the authentication techniques that can be used to secure APIs and provides guidance on when to use them. Figure 12: Authentication Options Each of these is described in more detail in the subsections below Anonymous Authentication NOT RECOMMENDED This is where the customer and the application they are using can gain access to backend API services without needing to authenticate in any way. Part B Technical v1.0 Page 25 of 126

26 Figure 13: Anonymous Authentication Model This approach can be used when the risk associated with the API is negligible e.g. an API offering publicly available information. It can be used for internal APIs if the agency s security policy allows. The downside of this model is that makes it difficult to gather effective analytics, and therefore to understand the implications of proposed changes to, and deprecation of, an API. Although it is not recommended to use this pattern it sometimes has to be included to support the scenario where a web application is making calls, on the home or login page, to a back end application server which stores the pages and the related JavaScripts. In this case it is the API Gateway that provides the security, along with the due diligence required to ensure these pages and associated JavaScripts do not contain information that should be secured. The Anonymous authentication model should be protected against typical API vulnerabilities and threats, as listed on the OWASP (Open Web Application Security Project) web site. Typically, these relate to: 1. Throttling to prevent Denial of Service attacks 2. Message analysis to block HTTP attacks parameter attacks such as cross-site scripting, SQL injection, command injection and cross site request forgery Note: If this approach is used, it might be appropriate to restrict access to the API based on other information (e.g. based on IP address). This capability can be applied using an API Gateway, or existing capabilities (e.g. firewalls, load balancers etc) may be able to provide this level of protection Username and Password Authentication (Direct Authentication) NOT RECOMMENDED In the Direct Authentication model the user is authenticated via an identity store using username and password (or hash) credentials over secure communications. Username and Password Authentication is suitable for development purposes, during training or the initial stages of development because it reduces the barrier to accessing the API. It could also be used for internal APIs, if the agency s security policy allows, and may be suitable for customers if the identity risk 1 associated with the API is low. Internal users would likely use an internal LDAP directory (e.g. Active Directory) whilst external users (e.g. customers) would require a separate identity store. The API Gateway could provide the authentication services and, for external facing APIs, also threat protection services. 1 The NZ Evidence of Identity Standard defines levels of identity risk. Part B Technical v1.0 Page 26 of 126

27 API security is provided by the Application (Web) Server acting as a trusted subsystem with TLS links to the backend API Server. The Application/Web Server invokes the backend and provides the required user ID information, which can be in the form of a session token. Figure 14: HTTP Basic/Digest Authentication Model This model can be easy to implement but has many limitations: 1. An identity store (e.g. LDAP) is required along with a full registration process for all user types (e.g. applications and application developers) 2. Cannot leverage a federated authentication model, so no single sign on (SSO), requiring re-presentation of username and password at every step 3. Password would be in clear text. If direct authentication is used, TLS would be required to secure all communications 4. Open to brute force attacks 5. Passwords have low entropy, have to be reset and managed, and are difficult to revoke at a granular level This model can be used for testing and development purposes, but is Not Recommended for production APIs; API Keys are preferred. Refer to the NZ Evidence of Identity Standard for guidance on customer authentication. If considering using this model for internal users, preference would be a Single Sign-On solution using Kerberos API Keys Authentication RECOMMENDED API keys should be used for every system-to-system interaction with a production level API. API Keys are a digital authentication mechanism, with the API key taking the form of a long string of generated characters. API keys are usually unique and can be assigned to an application, developer or user. The usual practice is for an application developer to obtain a Part B Technical v1.0 Page 27 of 126

28 key for their application from the API provider and utilise the key within their application. To obtain an API key, the developer must undergo a registration process with the API Provider. The steps involved in the registration process are dependent on the level of risk associated with the API. Figure 15: API Key Authentication At run time, the consuming application automatically passes the API Key to the API every time it requests an API resource. The API Gateway validates the API Key against the API Key Store (which can be part of the API Gateway functionality or provided by another secure device) before allowing the consuming application access to the requested API (or set of APIs) and backend resources. This model is similar to the Username and Password model, but it is the API Gateway which can be responsible for creating, managing the API key and API secret and storing a copy in the API key store for validation, rather than redirecting to an identity store for policy validation and approval. Username and password authorisation models can have high administration and response time overheads (relating to cryptographic functions). API keys are not linked to users and require no cryptographic functions. Like usernames and passwords, they come in pairs and are defined below: API Key - public unique identifier a 40+ random character string to authenticate the consuming application to the API API Secret private unique identifier- only known by the API Gateway and used to validate the API key. The API secret in this model is not passed over the network. API Keys should be used wherever system to system authentication is needed. They are suitable for simple public APIs which do not need more complex authentication models. API Keys should be used in preference to username and passwords because: 1. More secure - greater entropy than passwords random long string of characters 2. Speed API keys do not involve any hashing process, i.e. the hashing process required for passwords However, the risk is that anyone with a copy of the API key can use it as though they were the legitimate consuming application. Hence all communications should be over TLS, to protect the key in transit. The onus is on the application developer to properly protect their copy of the API key. If the API key is embedded into the consuming application, it can be Part B Technical v1.0 Page 28 of 126

29 decompiled and extracted. If stored in plain text files they can be stolen and re-used for malicious purposes Certificates (Mutual) Authentication Use this model when the API requires stronger authentication 2 than offered solely by API Keys, and the overhead of certificate management is warranted. In Certificate (Mutual) Authentication, internal and external parties are authenticated with each other. Both the consuming application and the API provider hold a digital certificate. The digital certificate can be trusted because it was issued by a mutually trusted Certificate Authority (CA). When the consuming application makes a request to the API, the server hosting the API presents its certificate to the consuming app. The app verifies the server s certificate then sends its own certificate to the server. The server verifies the client certificate and mutual trust is achieved, allowing the consuming application to use the API. Figure 16: Certificate Authentication This is Not Recommended for customer authentication as there would be a high overhead in terms of certificate management. This is Recommended for consuming application to server (gateway) or API to back end communications (if needed). Refer to the NZ Information Security Manual for guidance Developer Authentication Developer authentication will normally take place at the API Portal when gaining access to APIs. The API Portal will offer an authentication solution for developers to provide a username and password (possible two factor) login process (See Figure 17) and a user store. Further details for this are not provided in this document, but a vendor API Portal will normally provide their own authentication solution and user store, or can integrate with an existing identity and access management system. 2 Refer to the New Zealand Government Security Classification System and the New Zealand Information Security Manual for information about when stronger authentication techniques may be necessary. Part B Technical v1.0 Page 29 of 126

30 Figure 17: API Portal Login Page Once the developer has logged into the API Portal they can browse and discover the APIs available. API Portals normally require the consuming application developer to: 1. Provide contact details e.g. address 2. Register the application they are developing The API Portal should provide registration services for the client application to use: 1. API keys for basic authentication services and API monitoring 2. OAuth services and the management of Client ID sand a Client Secrets (for applications) 3. Additional production authentication and authorisation service e.g. basic, certificate etc Multi Factor Authentication (MFA) An application could use multi-factor authentication (MFA) to enhance other authentication techniques to mitigate identity risks. For example, by requesting a fingerprint from the customer in addition to their username and password (or API key). Often smart phone capabilities can be leveraged to provide this additional factor, but other options are available, like smart cards or hardware tokens. 1.5 OAuth 2.0, OpenID Connect and SAML OAuth 2.0 OAuth 2.0 is a Token-based authorisation framework and is defined and implemented using Grant Flow type patterns. These define the different types of interaction a client application can perform to gain an access token and thus access to the protected resource. There are four grant flows types supported by the OAuth 2.0: 1. Authorisation Code 2. Implicit 3. Resource Owner Password Credentials Part B Technical v1.0 Page 30 of 126

31 4. Client Credentials Grant Types The following table explains which grant flow type to use for which situation: Description Recommendations Authorisation Code Use for Internal Users or where the Customer is the Resource Owner, your agency controls the Resource Server, but the Authorisation server is not owned by the agency or is elsewhere within the organisation. The Authorisation Server provides the Authorisation Code (grant) to the client application once the Resource Owner has approved the request The client application then uses this to request the access token Recommended The most secure OAuth flow. Use for public facing APIs for Customer Authorisation pattern. Use for Internal Use Only pattern. Use the state attribute to link request and response. The Authorisation Server validates the client application using the Client ID and Client Secret. The client application has to store these credentials securely The client application authenticates to the Authorisation Server via its TLS certificate and callback URL Useful if the API requires a customer of a client application to authorise access to a protected resource provided by the API. Implicit Used when the client application resides on a device (e.g. smart phone) and cannot secure the credentials. Only the Access token is transmitted from the Authorisation Server to the client. Not Recommended The least secure grant type. If used it should ONLY be for information that is public and ONLY for GETs. Resource Owner Password Credentials Part B Technical v1.0 Page 31 of 126

32 Description Recommendations The Resource Owner s username and password are used once as the Authorisation grant to obtain an Access Token. The credentials are then discarded. Not Recommended Use for Customer Authorisation pattern if Authorisation Code grant flow can t be used. Use for Internal Use Only pattern to secure Internal APIs that use Active Directory Groups / Kerberos for authorisation and authentication if Authorisation Code grant flow is not available. Client Credentials The client application is able to obtain access to the protected resource on its own behalf. Recommended Use: Table 4 - OAuth 2.0 Grant types - For the Authorise Consuming Application pattern from device to API - Also use for Server to Server (B2B), using signed tokens - when the Customer using the client application is also the Resource Owner This table compiles the different grant types and provides recommendations for agencies implementing an API Security Framework. Recommendations are based on maximising the level of security for the APIs being exposed. OAuth 2 introduced different grants types to provide organisations with the flexibility to support a variety of client application models. These models are not device driven i.e. there is no specific device (e.g. mobile) mapping to grant type; it is the level of risk an agency is prepared to support that needs to be defined. For the application developer, the difference is in the infrastructure they need to provide e.g. the Authorisation Code model requires a managed server on which the client application runs. The Authorisation Code is the most frequently used model and as it is regarded as the most secure model it is covered in more depth later in this document. The Implicit grant is the least secure and to quote from the OAuth RFC: Implicit grants improve the responsiveness and efficiency of some clients (such as a client implemented as an in-browser application), since it reduces the number of round trips required to obtain an access token. However, this convenience should be weighed against the security implications of using implicit grants These OAuth 2.0 Grant flow types replace the two-legged and three-legged patterns used in OAuth 1.0. Part B Technical v1.0 Page 32 of 126

33 1.5.3 Basic OAuth Implementation Patterns There are three primary implementation patterns: 1. Client Credential Grant Flow 2. Authorisation Code Grant Flow 3. Distributed Resource and Authorisation Servers (UMA model see section 1.5.9) Figure 18: OAuth 2 Implementation Models As can be seen above, in most models the Resource Server resides with the Authorisation Server. But OAuth 2.0 also supports a distributed model, if needed, where the Resource Server and Authorisation Server are separate. By adding an Authentication Server component into the API Security Framework, a number of additional implementation models can be considered. The models above have separate Authorisation servers e.g. external Identity Service Providers, Google, RealMe etc. or internal Authorisation services. These Authentication Services could be housed on the same system as the Authorisation and Resource Server, and can, as would be expected, simplify the Security architecture. Figure 19: OAuth 2.0 Models (colocation of Authorisation and Authentication services) Part B Technical v1.0 Page 33 of 126

34 1.5.4 Quick Reference Table The following table provides a quick reference to identify the most appropriate authentication and authorisation model to use for the patterns defined in Usage Patterns: Vendor or Internally Developed API Gateway Capability OAuth 2.0 Grant Types API Usage Patterns API Keys Client Credentials Resource Owner Password Credentials Authorisation Code Implicit API Gateway Proprietary Identify Consuming Application Recommended Authorise Consuming Application Recommended Customer Authorisation Recommended System to System (B2B) Recommended Internal Use Only Recommended Application Developer Recommended Table 5 - Quick Reference Guide The initial consideration for any API Security Framework should be to use the Authorisation Code Grant Type, but the following provide pointers for agencies when considering their requirements: 1. It is good practice to use API Keys as the basis of all system-to-system authentication, such as consuming application to API. 2. For the Authorise Consuming Application usage pattern, the Client Credentials Grant Type is recommended but the API Keys model can be used instead. 3. For the Customer Authorisation usage pattern o the Authorisation Code Grant Type is recommended where the customer is the Resource Owner, the provider (agency) controls the Resource Server, but the Authorisation server is not owned by the provider or is elsewhere within the provider organisation Part B Technical v1.0 Page 34 of 126

35 o the Resource Owner Password Credentials Grant Type is appropriate where the customer is the Resource Owner, the provider (agency) controls the Resource Server, and the API Gateway is the OAuth 2.0 Authorisation Server 4. Use of solely API Keys for Customer Authorisation is only appropriate as a last resort 5. For the Internal Use Only usage pattern, the Authorisation Code Grant Type is recommended, where practical. Otherwise, it may be appropriate to leverage the provider agency s existing internal authentication and authorisation providers 6. For the Developer Authentication usage pattern, it is appropriate to leverage (or build) the capabilities of the developer portal (e.g. username and password) Tokens & Credentials Access tokens are becoming a standard form of access control without the need for passing credentials. Anyone with an access token (bearer token) is permitted access to the resource being controlled, which makes tokens a target for stealing or copying. Hence it is important to keep the lifetime of tokens as short as realistically possible, depending on the type of resource being exposed and business risk appetite. The OAuth framework (RFC 6749 and 6750) relies heavily on TLS for the security of the bearer token. The following RFCs (or drafts) offer additional integrity and confidentiality capability that can be applied to the bearer token (access token): 1. OAuth 2.0 Message Authentication Code (MAC) Tokens 2. JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants (RFC 7523) 3. Proof-of-Possession Key Semantics for JSON Web Tokens (JWTs) (RFC 7800) Tokens are produced by, and stored in, the Authentication Server. Each one has a specific task in the OAuth 2.0 framework. There are three types of token and three keys that should be considered in an OAuth 2.0 design: Description Recommendation Access Tokens Also called bearer tokens. No additional identity checks are carried out once this has been issued. Used by the client application to access protected resources on the provider, and it can be signed. It is a random character string that also contains scope information to allow additional access polices to be applied e.g. duration of access. It is granted by the Resource Owner via the Authorisation Grant Token Flow and enforced by Authorisation and Resource Servers. It is Required that the token be protected both in transit (TLS) and in storage (encryption) The Authorisation Request Header Field format is Required to only be used to transmit tokens It is Recommended that the life time of this token be set to 60 mins It is Recommended that scopes be used for coarse and fine grained access Form-Encoded Body Parameter is Not Recommended for transmission URI Query Parameter is Not Recommended for transmission Refresh Token Used to obtain new Access tokens when the old one It is Required that the token be protected both in transit (TLS) and in Part B Technical v1.0 Page 35 of 126

36 Description Recommendation expires or is invalid. storage (encryption) It is Recommended that the life time of this token be set to 24 hours Authorisation Code The Authentication Server sends the Authorisation Code to the Client after being granted consent by the Resource Owner. It is Recommended that the life time of this token be set to 24 hours Used to authenticate the client. API Key A 40+ random character string used in some scenarios to authenticate the client application to the API. Required when implementing an authorisation solution that uses API Keys. This is normally presented as an option in the developer API Portal. Client ID When registering an OAuth Client application with the API Portal, a Client ID is issued. Required when implementing an OAuth 2.0 model that supports Authorisation Code flow Used when interacting with the Resource Server. Client Secret Also provided when the OAuth Client application is registered. This is used with the Client ID when exchanging an authorisation code for an access token Required when implementing an OAuth 2.0 model that supports Authorisation Code flow Table 6 - Tokens and Credentials OAuth Scenario (Authorisation Code Grant Flow) This is hypothetical scenario to demonstrate a key OAuth pattern using the Authorisation Code grant flow. In this scenario IRD have developed a set of APIs that can be used by professional accounting firms to offer additional services to their customers. The assumption here is that IRD have an API Gateway, that offers: 1. API Developer Portal 2. OAuth Authorisation Server 3. Resource Server, exposing the APIs that can be called 4. That IRD are securing an API called View IRD Return with an Authorisation Code Grant Type Part B Technical v1.0 Page 36 of 126

37 Stage 1 Develop the Application (3 rd Party) The third party (in this case MyAccountantWebsite.com) develops a client application that will be exposed to their customers when they login to their website. It will allow customers of MyAccountantWebSite to authorise and setup delegated access for the MyAccountantWebSite application to view their IRD returns, but without the customer having to provide their IRD user name and password to MyAccountantWebsite. The application will use the APIs exposed by IRD, and is developed on an MyAccountantWebsite.com Application (Web) Server that can securely store security credentials Stage 2 Register the (Client) Application The developer needs to register as a user of the IRD API Portal. IRD need to verify that this person is allowed to register as a user. (This process is outside the scope of this document, and will vary depending on the sensitivity of the APIs exposed.) The authentication of the developer can be via: IRD s internal client login services An external Identity Service provided (OpenID Connect or SAML based, or Social Network Identity provider) Once the developer has been approved and has been granted login credentials (username and password) they log onto the IRD API Portal and register their client application; this is carried out over a TLS secured link. The following information is provided: The name of the application (by the Developer) Return URL (by the Developer) Client ID (by IRD) stored securely Client Secret (by IRD) stored securely The developer completes development of the client application on MyAccountantWebsite. The next steps detail how the MyAccountantWebsite customer uses the application. Note: the values of the Client ID and Secret are represented as simple strings below. In reality these are long, randomly generated strings. Figure 20: Stage 2 Client Registration Part B Technical v1.0 Page 37 of 126

38 Stage 3 Customer Sets New Service and Authorises Access The Customer logs into MyAccountantWebsite from their browser and clicks on the IRD button (View IRD) which should allow the user access to information presented by the View IRD Return API (see Figure 21: Stage 3 Client Registration below). As there is no current Access Token stored for the client application to use, the user is redirected to the IRD Authorisation Server Authorisation End Point with the following information: 1. IRD URL A (Authorisation endpoint URL) 2. Return URL (where the authorisation code will be sent) 3. Client ID = ABC (random long string that is used to identify the client application) 4. Scope = READ (defined by the application as to what the application can do) 5. State = xyz123 (random long string that is used to mitigate man in the middle attacks) Figure 21: Stage 3 Client Registration Stage 4 Authentication and Approval by the Resource Owner As the client application doesn t have an access token for this customer, or a session set up with IRD, the customer is redirected to the IRD login page. The customer then logs into IRD and will be presented with a request to accept the scopes (in this case READ) for MyAccountantWebsite access. Part B Technical v1.0 Page 38 of 126

39 Figure 22: Stage 4 Client Registration Stage 5 Provide an Authentication Code With the customer s acceptance of the scope, the Authorisation server sends an Authorisation code to the client application (with the same state parameter for the client to validate). Figure 23: Stage 5 Client Registration Stage 6 Authorisation Code is Sent to Token Endpoint To gain access to API resources, the client application sends the Authorisation Code to the Token endpoint (T on the diagram below) on the Authorisation Server, along with the Client Part B Technical v1.0 Page 39 of 126

40 ID and Client Secret it received when the client application was registered. This is used for authentication of the client application to the authorisation server. Note: The communication must be over TLS. Figure 24: Stage 6 Client Registration Stage 7 The Access Token It Provides The Authorisation Server provides an Access token back to the client application, along with a refresh token (for later use) and an expiry time for the Access token. Part B Technical v1.0 Page 40 of 126

41 Figure 25: Stage 7 Client Registration Stage 8 The Client Application uses the Access Token to Access the Resource The Client Application presents the Access Token to the Resource Server at IRD, which is verified by the Authorisation Server and the requested data is returned from the IRD back end system via the View IRD Return API to the MyAccountantWebsite client application. Figure 26: Stage 8 Client Registration This completes the OAuth scenario, as the client application has retrieved, and can use, the resource data returned from the API OpenID Connect and SAML (Security Tokens) Use of OAuth authentication security token profiles is Recommended when an API requires more secure authentication than offered solely by API keys e.g. when data is a two-way flow between the consuming application and the API. The two recommended implementation patterns (profiles) are: OpenID Connect (see section ) SAML (see section ) The preferred standard is OpenID Connect. Use SAML only if it is readily available to your developer community or has been implemented for a specific sector e.g. Education. During the authorisation token exchange process a level of authentication is required. The authentication process is basically limited to what authentication services the authorisation server can support. For example, the Resource Owner authorisation process is normally supported out of the box by the authorisation server, using username and password. The authentication service can be enhanced with authentication security tokens by implementing one of the two OAuth profiles detailed above. Part B Technical v1.0 Page 41 of 126

42 For customers and internal users this can be achieved using a brokered or federated service. Basically Federation in the context of API security provides the ability to re-use user identities in a SSO way by providing a trust relationship between the Identity Service Provider (either providing SAML or Open ID Connect support) and the Authorisation Server. This establishment of trust between the Identity Provider and Authorisation Server is normally established using certificates and mutual authentication. All communication has to be over TLS to provide Confidentiality and Integrity controls OpenID Connect OpenID Connect is a lightweight RESTful framework for identity service interaction, to provide authorisation services. It is built on top of the OAuth 2.0 framework and basically provides identity information in the form of an ID token. It introduces a new Authentication service which will run on its own or on the OAuth Authorisation server and returns an ID Token along with the access token, after the API request has been authorised. The ID Token is a JSON Web Token (JWT) that contains authenticated user information (and attributes) that the authorisation (OAuth 2 Server) / authentication server (OpenID Connect Server) provides to the client application. This token can then be used to enforce finer grained access controls by providing additional attributes that can be used by the authorisation server to apply these policies. The new JWT token has to be signed in order to address confidentiality and integrity requirements. There are also additional parameters defined that have been added to address replay attacks. In the first OpenID Connect pattern detailed in Figure 27, the Authentication and Authorisation Servers are conceptually hosted on different physical servers (the OpenID Connect server can be internal or external to the organisation). The user connects to the web application and is redirected to the OpenID Connect server for authorisation. Once authorised, the user is redirected to the Web Application which in turn exchanges the ID Token for an access token. Note: It is the token flow and exchange that is key to understand in this model. The Web Application could be an Application/ API Server and the interface to the authorisation and authentication server managed by the API Gateway A trusted link is set up between the OpenID Connect Server and the OAuth Server (Authorisation Server) which also allows additional identity information to be provided in a JWT format via an userinfo end point. This is important as the token exchange can result in the loss of user data that is required to provide fine grained access. If the OpenID Connect server was external, it would be the API Gateway that would be responsible for the interface and token exchange process. Part B Technical v1.0 Page 42 of 126

43 Figure 27: OpenID Connect - Distributed Authorisation and Authentication Server The application has to exchange the ID Token for an Access Token in this pattern. In the second pattern (below) the OpenID Connect server is run on the same server as the OAuth Authorisation server, so the ID Token and the Access Token can be issued at the same time. In both models the OpenID Connect Server can use LDAP for its back end User Store, to provide authorisation for internal users. Figure 28: OpenID Connect and OAuth Part B Technical v1.0 Page 43 of 126

44 The one final model is for an external OpenID Connect pattern, where the OpenID Connect server is an external Identity Provider (e.g. AoG service) and the API Gateway is responsible for managing the token exchange, potentially also housing the Authorisation Server. To address the risk of forged or stolen assertions, it is recommended that all communication is over TLS and tokens are at a minimum signed for authentication SAML If an OpenID Connect Service is not available, or there is only a SAML Service available, there is an OAuth authentication model that allows a SAML Identity Provider to replace the OpenID Connect service. This looks identical to Figure 27, but replaces the OpenID Connect server with a SAML provider. Like the OpenID Connect model, the SAML instance could be external or internal. The SAML profile works in a similar way to the OpenID Connect one, but there is one disadvantage with the SAML model in that there are no refresh tokens provided. The setting of time to live and session timeouts parameters for access tokens can add additional complexity and design constraints. The API gateway in this model (as with OpenID Connect) provides a threat protection service but could also act as: Authentication Server Redirect service to SAML Provider This model uses the SAML 2.0 Bearer Profile for OAuth. See RFC 7522 for details. The next sections will define how additional authorisation can be applied to this principal model to provide coarse and fine grained access controls. 3 Signing is covered in section of this document Part B Technical v1.0 Page 44 of 126

45 1.5.8 Distributed Model Profiles The above models work well when authorisation is controlled internally, but there will be situations where authorisation needs to be common across several agencies, as a shared resource. Below are some emerging OAuth 2.0 Profiles which are applicable to a distributed authorisation model. These distributed authorisation models can also leverage and support both internal and external OpenID Connect or SAML Identity Service Providers. This section addresses proposed OAuth profiles that should be considered if a distributed model is required by an agency or a group of agencies. These profiles are currently in draft and have not been fully ratified. This section only provides a high level view of the proposed models. If these models are to be implemented, it is recommended that a full assessment is performed of the current state of these proposed RFCs. OAuth 2.0 introduced a model where the Resource Server can run on a separate server. This provides for models that can support multiple instances of the Resource Server or Authorisation Server. There are two RFC models currently defined: 1. Multiple Resource Servers (UMA see section 1.5.9) 2. Multiple Authorisation Servers (Chained Authorisation see section ) Figure 29: Distributed Authorisation Models The UMA model is driven by the Kantara initiative 4 and has been productionised by a couple of vendors 5 The chained authorisation model is based on the Chain Grant Type for OAuth2 (expired) draft profile and is included here as a foundation for future development. There are currently and Part B Technical v1.0 Page 45 of 126

46 no vendors providing support for this model, therefore any intention to use this model should be carefully considered. With the introduction of the distributed Authorisation and Resource Servers model comes a number of technology scenarios which were not covered specifically by the basic OAuth 2.0 RFC. OAuth designs usually rely on these two services (Resource and Authorisation) running on the same server. The following is a list of the limitations and the RFCs that address them. 1. The OAuth standard Client Registration is a manual process. Both OpenID Connect and UMA reference a dynamic registration process. This is addressed with RFC 7591 (Dynamic Client Registration OAuth 2.0 Profile. 2. In any OAuth model, the ability to revoke token access is a key component of the design. This becomes even more important when supporting multiple Resource and Authorisation Servers. RFC 7009 (Token Revocation Profile) addresses this requirement. 3. The process for Resource Servers to query the state of an access token is not defined in the OAuth framework. This is defined in RFC 7662 (Token Introspection Profile). Each of these is explained in the following sections Dynamic Client Registration OAuth 2.0 profile (RFC 7591) Any client application has to be registered to the OAuth Authorisation server before it can gain access to Resource Server resources (e.g. APIs). This registration is normally a manual process performed by the developer. This particular profile offers a way to provide an end point on the Authorisation server so that client registration can be dynamic (e.g. a call can be made to this end point with the required registration information, and the authorisation server will register the client). This obviously speeds up the process of registration and makes it available on-demand. Another key benefit is that the (dynamic) registration process can also generate different client IDs and client Secrets for each client application installation, which can dramatically mitigate risk related to compromised client IDs or secrets Token Revocation Profile (RFC 7009) This profile defines a revocation endpoint on the authorisation server to enable clients (consuming applications) to revoke their own access or refresh tokens. This is essential should a token get into the wrong hands and be used for malicious purposes. Before allowing a client to revoke an access token and/or associated refresh tokens, the authorisation server first validates the client s credentials Token Introspection Profile (RFC 7662) This is a key requirement for a distributed model, as it allows a Resource Server to query an Authorisation Server to determine the state of a token and obtain additional meta data, including client ID, active state, scopes and [resource server] audience details. It can also provide extended attributes e.g. audience information User Managed Access (UMA) UMA was introduced to address the following requirements: 1. Support for multiple and distributed servers 2. Delegated Authorisation to a person requesting access to one or more resources Part B Technical v1.0 Page 46 of 126

47 3. The resource owner providing a pre-defined policy of what can be accessed and by whom (i.e. minimise their interaction in the current OAuth approval process) and provides a central point of control. UMA has been developed by the Kantara Initiative group. In UMA the user defines the relationship and access controls (policy) for all their resources across multiple Resource Servers. A centralised UMA Authorisation Server then keeps track of all Resource Servers associated with a given Resource Owner: Figure 30: UMA The Resource Owner introduces the UMA Authorisation Server to the Resource Servers and vice versa. They establish a token-based trust relationship which the Resource Owner authorises. The Resource Owner also defines the access control policies which grant access to potential consumers. A client application, driven by the Requesting Party who is requesting access to resources owned by the Resource Owner, attempts to access a resource on a Resource Server but will be redirected to the UMA Authorisation Server. The client application has to obtain a client key, client secret and access token from the UMA Authorisation Server before trying to get to the required resource on the Resource Server. The Resource Server will validate the access token and the permitted level of access with the Authorisation Server before allowing the client application to consume its required resources. UMA uses the following end points (a URI that is exposed on the Authorisation Server) to allow certain requests to be made and processed: 1. Token and Authorisation defined in OAuth 2.0 framework 2. Dynamic Client and Introspection defined in extended RFC (see previous section) 3. Resource Set Registration, Permission Registration and Request Party defined in the UMA specification Part B Technical v1.0 Page 47 of 126

48 These are detailed in the UMA architecture. The UMA Authorisation Server offers two APIs to support these interactions: Protection API used by Resource Servers for getting authorisation-request tokens and validating access tokens Authorisation API used by the Client Application to obtain a token for accessing a specific resource Chained Authorisation This model has been included for interest purposes only. The objective of this model is to define a process for one protected API resource to talk to another resource API, or chain a request from a client application from one resource server to another. This is done by issuing another token (JWT) based on an established trust between the two Authorisation Servers. This model theoretically could support multiple authorisation server chainings and a Master / Slave model, but at present are not covered by explicit RFCs. The two draft RFCs relating to this model are listed below, but note that both drafts have now expired: 1. A Method of Bearer Token Re-delegation and Chaining for OAuth 2 2. Chain Grant Type for OAuth2 The diagram below provides an indication of how the process would work. Note: this includes the use of JSON Web Tokens. This is not a fully defined model and would require additional investigation, review, validation and design before use. Note: Organisations do implement solutions based on Draft RFCs, but can end up with an architecture that is not ratified by the released RFC. Engagement with the RFC is always recommended if this approach is required to meet a business need. Figure 31: Chained Authorisation Part B Technical v1.0 Page 48 of 126

49 1.6 Authorisation REQUIRED Authorisation is the act of performing access control on a resource. Authorisation doesn t just cover the enforcement of access controls, but also the definition of those controls. This includes the access rules and policies, which should define the required level of access agreeable to both provider and consuming application. The foundation of access control is a provider granting or denying a consuming application and/or consumer access to a resource to a certain level of granularity. In the Authentication section the concepts of OAuth were introduced and a number of Authentication patterns were defined. This section focusses on Authorisation and provides additional patterns that work with OAuth or provides an alternative Authorisation Techniques Authentication on its own does not necessarily provide permissions to access an API or application. It merely validates that you are who you say you are. If it is used for access control, it is an all or nothing control mechanism (e.g. administration account). Once a user is authenticated (e.g. using username and password), an authorisation process will grant (or deny) them the right to perform an action or access to information. Normally this authorisation process is applied using either a coarse grained or fine grained access control process. The normal model is to provide coarse grained access at the API or API Gateway request point, and fine grained control at the backend services, but this model is changing as backend systems become more modular (e.g. microservices) and less monolithic. This will result in a need for fine grained authorisation support at the request point. With this understanding, the two types of authorisation that will be considered are defined in Figure 32. This covers: 1. Role- or Group-based authorisation 2. Policy- or Attribute-based authorisation Figure 32: Authorisation Techniques The API Gateway will provide this support either using OAuth 2.0 or its own proprietary capability Roles Based Access Controls (RBAC) In many organisations Active Directory provides the authentication services for users. Active Directory groups are then used to provide authorisation. This is classed as Discretionary Access Control (DAC): access to systems is granted by applying Access Control Lists (ACLs) directly to the user, or to the groups in which users reside. Part B Technical v1.0 Page 49 of 126

50 Note: LDAP directories can also provide this service, and in many organisations are used to provide the same service AD delivers but for external users. Active Directory (or LDAP) Groups are synonymous with roles and can be used to provide coarse-grained authorisation for APIs Attribute Based Access Controls Attribute-based access control (ABAC) defines an access control process whereby access is granted based on policies that are built using attributes e.g. a policy might state that access to a specified resource is only permitted for users who are in Sales or Marketing, who are managers, during office hours only. ABAC provides fine grain authorisation services. The really important control that ABAC provides is the ability to provide context when applying access controls, e.g. access decisions can be based on the IP address of the device, the operating system of the client and the last know transactions of a client. The recognised standard for ABAC is XACML, which is an XML based-language API Gateway API Gateways have been mentioned previously in the context of API protection. Most API Gateways on the market provide support for OAuth 2.0 and can also provide Authorisation (and Authentication) Services via a direct connection to: 1. An Identity Store containing groups 2. An Identity Access Management system 3. A Policy server ABAC Implementation Figure 33: API Gateway for Authorisation To implement Attribute Based Access Control, the current models defined use XACML. XACML (developed by OASIS), provides a reference architecture, a request / response protocol and a policy language. Part B Technical v1.0 Page 50 of 126

51 It is a highly distributed and loosely coupled architecture. It provides very useful generic definitions of the required components (services) which can be used to define any access control model. It uses the following services to define the reference architecture: 1. Policy Enforcement Point (PEP) where the request to the resource is intercepted and policy applied (based on the decision made by the Policy Decision Point) 2. Policy Decision Point (PDP) This is the policy server to which the PEP sends the request for evaluation as to whether a user should or should not have access to a resource. The PDP has access to policy and can match the credentials and request against policy to make a permit/deny decision. It can also enforce policy related obligations e.g. enhanced logging, notification and alerts, or re-routing to request additional authorisation process. 3. Policy Administration Point (PAP) The interface where the policies are developed and defined 4. Policy Information Point (PIP) Used to gather additional information about a user from Identity stores or databases to provide additional attributes that are required by the PDP to validate the policy and apply the required outcome The links and flows between these services are detailed in the diagram below: Figure 34: XACML Reference Architecture XACML is generally perceived as being difficult to write polices in, but this is being addressed in two ways: 1. OASIS are developing a Request/Response Interface based on JSON and HTTP for XACML 3.0, Version There is a JSON-based language called ALFA (Abbreviated Language For Authorization) which can be used to build XACML policies For XACML to support fine grained access for APIs requires a model such as is illustrated in the diagram below (based on the scenario in section 1.5.6). Part B Technical v1.0 Page 51 of 126

52 Figure 35: ABAC Support for APIs It follows these steps: 1. The access token is obtained during the request and exchange process 2. The access token is presented during the resource request to the PEP, which also exposes the resource 3. The access request is passed to the Authorisation Server 4. The Authorisation Server verifies the access token and passes a XACML request to the PDP. This is where additional fine-grained access can be applied. 5. The PDP authorises the PEP to allow access to the back end service Scopes (Limited Fine Grain Access) ABAC provides fine grained access controls to APIs. Based on the services (APIs) that are exposed, additional fine grained access controls can be applied using scopes. For example, a data service might provide read and write scopes. When the Authorisation code token is passed to the Authorisation Server a scope parameter is included to define what scopes the client can use. Scopes can be used to limit the authorisation granted to the client by the resource owner. These scopes are defined by the client application developer. This is an important consideration as it can impact how the API service is defined e.g. single service with multiple scopes or multiple services with single scopes. The developer has to ensure that the minimum privileges are granted to client applications to carry out the tasks (exposed by the Client Application and APIs) the user wishes the application to complete. Scopes provide a level of coarse/fine grained access and represent specific access rights e.g. the ability to read a document or write a new document (or both) is limited by the Access Token. Scopes can be used alone to define coarse/fine grained access, but these scopes need to be defined and built into the Client application / API being built. Consideration is needed to understand (for example) if: a single scope protects the service scopes are defined to protect fine-grained business functionality services should be divided into many smaller services with one scope each Once the token is issued to a client application, the access rights bounded by the scope are encapsulated in the Access Token for the length of its validation period. Part B Technical v1.0 Page 52 of 126

53 A client application may invite a user to authorise the application to act on behalf of the user. Using NZBN as an example, the client application (an accounting application for example) may invite a user to authorise the client application to update primary business data on the user s behalf. The application may ask Do you want [client application] to be able to update your business information on the NZBN Directory?. Assuming the user grants the client application this privilege, the authorisation token returned by the API will have a scope that enables update access. This scope is represented by an identifier. For example, the NZBN update scope looks like this: whereas a read-only scope would be: Confidentiality and Integrity Confidentiality and integrity covers the handling of request and response data, both in transit and at rest. The aim is to protect the payload content from unauthorised access, manipulation or faking. An API request needs to be received intact by the API, with validation as to the source of the request. Untampered API responses need to be received by the consuming application, with confirmation that they are legitimately from the API Communications Security (Confidentiality and Integrity) REQUIRED All communications to or from an API must be over TLS 1.2 or higher. Other versions of TLS and SSL should be disabled. This provides a recognised level of confidentiality that covers all communications between all components. The consuming application must validate the TLS certificate chain when making requests to protected resources, including checking the Certificate Revocation List (CRL) Token Audience and State Parameters (Integrity) Token security is particularly important to OAuth implementations; the reasons and mitigations are explained below Audience Information (Draft) Because Bearer Tokens are not aligned to any one client, they are open to reuse and misuse. To help prevent misuse of the OAuth Bearer token (Access Token) the Audience Information draft profile links the token to a specific Resource Server and scope. To do this, it defines a new audience header which is used by the client to indicate what Resource Server (or Servers) it wants to access. The client request has the required Resource Server identified in the new audience field. The Resource Server receives the request and gets the Authorisation server to validate it and return an access token to the client with that audience as part of the token s scope State State is also a parameter that can be used during the authorisation grant stage to provide a level of security to address possible man-in-the-middle attacks. The state parameter is a string of random letters and numbers that is sent to the Authorisation Server by the client when requesting an authorisation code. It is sent back to the client with the Authorisation Code and should be verified by the client application to confirm the authenticity of the response i.e. it came from the authorisation server to which the request was sent. Part B Technical v1.0 Page 53 of 126

54 Note: State is used to provide a level of integrity when using the standard format of bearer tokens. The confidence in the level of integrity can be increased in JWT tokens are used for bearer tokens. (See Integrity sections below) Content Encryption (Confidentiality) If content needs only to be visible to specific consumer end points, use encryption. However, if content only needs to be guaranteed untampered and/or from a specific source (e.g. provider) then use content signing. Content encryption enables all or part of a JSON payload to be readable only by the target consumer(s). This is useful where the content being carried by the API is sensitive, and the API request or response transits multiple stopping points. Whilst TLS protects the payload in transit, it only applies to each point to point connection between components (e.g. mobile app to API gateway). If transit components are not totally under the provider s control, it can be worthwhile performing body encryption. E.g. it may be sensible to encrypt credit card details passed between consumer and provider backend systems. It is also worth considering how much protection the information needs whilst at rest (e.g. information received from consuming applications, caches) and whether some content should be stored encrypted. Encryption is only worthwhile implementing when data sensitivity or data protection requirements drive it, as encryption is computationally intensive. It also makes it more difficult for protection mechanisms, such as API gateways, to validate and transform API content. When only the integrity of the content passed needs to be ensured, consider using Content Signing (section 1.7.4) instead. There are many existing ways of encrypting message content, built into code libraries and development tools. It is Required that any content encryption adhere to the standard algorithms laid out in NZISM (HMAC Algorithms) Content Signing (Integrity) Content signing is used to assure content integrity and proof of authorship. It can apply to the whole body of the JSON message or specific elements of that content e.g. credit card details. There are many approaches to content signing and the most appropriate approach is requirements-dependent. Standard signing algorithms exist within coding libraries, and JWT has a body which can contain verifiable (signed) JSON fields. API Gateways can also be configured to sign content objects in transit, if provided with an appropriate private key. Signing has less of a computational overhead than encryption, but can still affect performance, so it is advisable that it be used only when and where needed. For APIs, this is a developing area: there are standards currently under development to address content signing: Message Authentication Code - OAuth 2.0 Message Authentication Code (MAC) Tokens (draft) Proof of Possession - OAuth 2.0 Proof-of-Possession (PoP) Security Architecture (draft) It is Recommended that where Bearer Tokens are used, they should be signed using JSON Web Tokens (JWT) as defined in: JSON Web Token (JWT) RFC 7519 JSON Web Signature (JWS) RFC 7515 JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants RFC 7523 Part B Technical v1.0 Page 54 of 126

55 1.7.5 Non Repudiation (Integrity) Non repudiation covers the means to ensure that a consumer cannot deny making a request and, similarly, a provider cannot claim they did not send a response. To aid non-repudiation for APIs, it is important to ensure credentials are not shared between consumers and to perform comprehensive logging of API request/responses. Digital signatures are useful for not just guaranteeing authenticity and integrity, but also supporting non-repudiation. 1.8 Availability and Threat Protection Availability in this context covers threat protection to minimise API downtime, looks at how threats against exposed APIs can be mitigated using basic design principles and how to apply protection against specific risks and threats. Availability also covers scaling to meet demand and ensuring the hosting environments are stable etc. These levels of availability are addressed across the hardware and software stacks that support the delivery of APIs. There are no specific standards for availability, but availability is normally addressed under business continuity and disaster recovery standards. These standards recommend a risk assessment approach to define the availability requirements. Further information on business continuity and risks can be found at Standards New Zealand website 6. For cloud services, the New Zealand Government ICT website provides an assessment tool that includes a risk assessment tool which covers availability, business continuity and disaster recovery related questions 7. As mentioned in section 1.1.3, there are various types of risk which impact APIs. This includes threats to availability as well as confidentiality and integrity. Many threats can be mitigated through good secure coding practices, using OWASP guidelines, as indicated in section Where the resources being exposed by an API are sensitive i.e. not public data, it is advisable to perform: Threat assessment early on in the API development lifecycle Penetration test once an API is developed and published (testable) There are also automated vulnerability testing tools which can be used to give an indication of vulnerabilities in web application designs. Below is a table of risk types and some recommended approaches to help mitigation these threats: Threat Mitigation (OWASP) Exposure of inappropriate API methods to access services Protect and Limit (whitelist) the HTTP Methods (GET, PUT etc) exposed Validate Method(s) for session token / API key Part B Technical v1.0 Page 55 of 126

56 Denial Of Service attacks Throttle access to all exposed APIs. Monitor use to indicate possible DoS attacks Malicious Input, Injection attacks and Fuzzing Validate input: Secure parsing and strong typing Validate incoming content-type application/json Validate JSON content Validate XML (schema and format) Scan attachments Produce valid HTTP Return Code Validate response type Cross-Site Request Forgery Use tokens with state and nonce parameters Cross-Site Scripting Attacks Validate Input Table 7 - Risks and Mitigations Token Threat Mitigation Securing OAuth flows relies on the exchange of tokens between consuming applications and API backend servers. There is always the threat of these tokens being obtained illicitly, losing confidentiality and integrity of message content or the integrity of the sender of the token. This risk also applies to the transferring of API keys. The table below captures the main Token threats and possible mitigation strategies: Threat Mitigation Token Manufacture or modification (fake tokens and man in the middle attacks) Digital signing of tokens (e.g. JWS with JWT) or attaching a Message Authentication Code (MAC) Token disclosure man in the middle attack. The Access Token is passed in clear text with no hashing, signing or encryption. Communication Security Use TLS 1.2 with a cipher suite that includes DHE or ECDHE The client application must validate: The TLS certificate chain Part B Technical v1.0 Page 56 of 126

57 Threat Mitigation Check the certificate revocation list* *Stored locally in a file or LDAP server 8 Token Redirects. Ensure the Authentication and Resource Servers are paired and the access token can only be used in this between the specified servers Using the audience header (defined currently in a draft RFC) the client application, resource server and authorisation server can help* ensure that the token can only be used on the resource servers requested by the client and recognised by the authorisation server. Also addressed with state parameter in the header * Signing of tokens is also applicable to address token redirects Token replay where the threat actor copies an existing token (e.g. refresh token or authorisation code) and reuses it on their own request Limit lifetime of the token (e.g. 10 minutes) turning it into a short lived issue Use signed requests along with nonce and timestamps Validate TLS certificate chain when accessing Resource Server Table 8 - Token Threats API Gateway capabilities can protect against many typical API vulnerabilities and threats. Typically, these relate to: 1. Throttling to prevent Denial of Service attacks 2. Message analysis to block HTTP attacks; parameter attacks such as cross-site scripting, SQL injection, command injection and cross site request forgery 3. Controlling egress of information via the API, aligned to set access permissions/policies As well as providing (if required) access control to API functionality. 8 The process for obtaining this information is out of scope for this document Part B Technical v1.0 Page 57 of 126

58 2 API Development This section looks at APIs from a development perspective and offers the guidelines and standards required to support government agencies in designing and developing APIs. 2.1 API Artefacts There are some fundamental artefacts which comprise an API deliverable: Interface specification, accessible via some form of catalogue API code and policies Unit tests Service Level Agreement (SLA) for the API, identifying performance, availability etc. These artefacts are the responsibility of the API developer and should be mandated as a minimum set of deliverables when developing an API. All of these artefacts must be managed in a software configuration management system (SCM) so that changes can be tracked and collaborative development can be performed. 2.2 API Design Application developers will invest heavily in using your API. They will invest in learning the design and behaviour of your API, in developing and testing around your API and may even invest in developing an entire business model on your API. Good API designs will attract more developers. Conversely, bad API designs will drive away developers and drive up costs support costs, maintenance costs and operating costs When is an API appropriate? In the past, the default reaction to a requirement for capability has been to develop a web application. This is now gravitating towards APIs as the default. So when is it better to build an API than a web application? Below are some situations where an API may be more appropriate: When commercial organisations or NGOs are screen scraping data from the agency s website When the agency holds a single authoritative source of the truth Where there is a need for (near real time) information exchange with a partner agency When parts of a business process are (or may be) outsourced When the agency s service can be included as part of a larger business process When an agency needs internal systems to interact with cloud-based COTS solutions (SaaS) Where businesses require easy access to (public) information Where commercial organisations want to build capability into their own applications which will benefit the public (mobile/web apps) When an agency expects other parties to act as agents or intermediaries for their services When an agency expects to support service delivery across multiple channels Part B Technical v1.0 Page 58 of 126

59 2.2.2 API Design Principles This section assumes that API Principles defined in section 4 of Part A of this standard have already been read Future-Focused Design APIs should not be tightly coupled to legacy applications, exposing whatever capabilities the legacy system offered. Nor should they be designed to work in the way the legacy system currently works. Instead they should be consumer driven so built to expose the resources that consumers need, whether those resources are in legacy systems or new. There should not be a drive to expose an entire product via the API, or wait for the perfect backend system to be available. Just offer as much as is practically useful, then evolve it to meet needs. Design the API interface to be abstracted from the backend, so that backend systems can be changed or replaced over time without needing to change the interface. The aim is to be future-focused (whilst still pragmatic) and develop APIs to meet future needs. A good example of being future focussed is building APIs to support HATEOAS (Hypermedia as the Engine of Application State). This is where an API, in response to a consuming application's request, is programmed to return hyperlinks (URLs) which indicate the options for subsequent actions or information. See section for more details on HATEOAS Layering When designing and developing an API it is important to consider that an API is made up of distinct functional layers: Security Caching Representation Security Every API will have a security component. It is important to recognise that this is not only authentication and authorisation for access to an API, it also includes threat protection (DDoS, SQL Injection, Cross site scripting etc.) as well as availability and quality of service (QoS). When designing and developing APIs it is often cost effective to create a common framework that handles security for all APIs. See section 1, API Security for more details Caching Caching can dramatically improve the performance of an API. When designing APIs consider what, when and where to cache. Understanding how data is changed and how often it is changed is an important consideration, as well as which layer it is most appropriate to cache at. A common caching strategy should be developed for APIs that would benefit from it. See section 2.11 for more details Representation When designing and developing an API it is important to consider the representation of that API. This commonly includes an interface specification that fully describes the API. To ensure the success of an API it should be easy to consume driving a well-considered representation layer. Part B Technical v1.0 Page 59 of 126

60 Figure 36: API Layering Standards-Based Web standards have rapidly become powerful agreements, which span not just local regions but are internationally accepted and enable commonality and consistency. Using standard HTTP and URLs, the core technologies of the web, along with emergent standards such as JSON and OAuth ensures that agencies are not creating bespoke or proprietary technologies. Hence the principle is to build to the latest versions of existing open and accepted standards e.g. HTTP OpenAPI REST JSON OAuth Designing an API When designing an API, it is important to perform business process analysis to ensure that the API development is business-driven rather than technology-driven. Technology driven projects rarely meet customers needs in the long run, so it important to gain background in who could be using the API, for what. As mentioned previously, co-design is fundamental to Part B Technical v1.0 Page 60 of 126

61 driving the right API development. To help identify potential partners to involve in the codesign, consider: processes which currently depend on information the API could expose processes which require a capability an API could expose Some actors will be human (e.g. application developers) whilst some actors will be the systems which will interact with, or depend on, the API. There may be different types of actors, some public sector, some commercial entities, some public. When representatives for the potential actors are identified, start co-designing with these representatives. First and foremost, consider the requirements for the API. Application developers often couch their requirements in terms of how the API should work, rather than what the API needs to do. Don t get bogged down in the variety of proposed solutions from each developer: focus on extracting their true requirements by performing functional analysis (e.g. use cases) and data flow analysis. Then identify resources and work out the granularity needed (see sections Designing an API & Granularity). It is especially important that security and information privacy impacts are identified up front and addressed early on in the design process. Assess the information being passed and the types of access different customers and/or consuming applications should have to the API. This will help drive development of security policies alongside the design of the API. One common pitfall in API design is to map all existing data tables onto resources and develop the associated CRUD capabilities into an API. Usually this results in a poor API design and tightly couples the API design to the underlying data structure. Another common pitfall is to design APIs as an extension to, or way into, monolithic legacy systems. This should be avoided as it tightly couples the API to the legacy system. Both of these pitfalls will create issues for both provider and consumers in the long term. At this point in the design process, agility is probably more important than completeness. Share early design thoughts and interface specifications with the developer community and quickly make changes in response to their feedback. Work through some sequence diagrams with them to help pin down API interactions and inform API design thinking. The correct API design will likely not please every developer, so don t try to be all things to all developers. A rule of thumb is that you are probably on the right track if most developers are a little unhappy, but all are able to achieve their aims with the proposed design. It is important not to try to bypass evolution steps and try to build for all potential use cases right from the offset. For a more measured progression towards API delivery, start simple and focus on a single channel or interaction initially. By building onto these simple building blocks, API evolution over time naturally progress towards omnichannel applications Design Considerations The following considerations emerge when looking to develop APIs in the public sector: Understand your data, understand your consumers. Before starting work on APIs, gain a good understanding of what data you hold, the complexity of that data, its relationships to other data, the downstream dependents on that data and potential consumers of the data. It is also a good idea to understand your consumers' drivers and downstream usage or customers. Always design for REST. If the interaction appears RESTful then develop a REST API, e.g. one-to-many, but for one-to-one point solutions SOAP may be appropriate. The information supplied via an API may still be dependent on overnight batch processes for update. It is worth making the information s timeliness and update period clear to potential consumers of the API. Part B Technical v1.0 Page 61 of 126

62 APIs should not be used to handle large batch transfers because of the atomic nature of the usage i.e. if one single data write fails then the whole batch fails. APIs work better for multiple individual update transactions Design Driven Development REQUIRED When building APIs, a Design Driven Approach must be taken. This includes: Interface specification first Iterative approach Continuous integration/testing Interface Specification First The best way to design an API is in collaboration with potential consumers of that API. Creating the interface specification first makes it easier for application developers to see what the API is going to offer and how it could be used. Their early feedback will ensure that the API design is heading on the right track and will be usable and appropriate to their needs. Using a simple modelling language to define the interface specification makes it easy to view and edit API definitions. See section for more information about defining interface specifications Iterative Approach It has been acknowledged that big bang releases rarely deliver business or customer benefit. An iterative approach, with ongoing releases offering gradual improvement or evolving capability, gives a finer tuned delivery, better sense of momentum, illustrates progress and enables third parties to coordinate efforts with API developments. ICT.govt.nz Accelerate ( offers a framework which supports this kind of development and has guidance on how to deliver transformational projects. An iterative approach incorporates continuous improvement, which recognises the need to support continuing evolvement of the API beyond the first delivery. The API is a product whose capabilities should undergo continuing improvements based on consumer feedback, performance analytics and emerging needs. This aligns to Phase 6 in the Accelerate framework Continuous Integration/Testing Continuous integration provides a quick turnaround for informing API developers about breaking changes being submitted by developers who are all working on the same code base or product (e.g. API). The idea is that all developers submit code changes as often as possible (e.g. into a version control system), allowing code to go through an automated, integrated build process which ensures that nothing has broken the build. This build process could happen many times a day. Any errors are quickly identified and alerted to the team who can rectify them in a timely fashion. Tests can be written against the interface specification quite early on in the development process by developing just enough API code to enable the test to be run (stubs). The tests can then be incorporated into the automated build process, giving early warning of regression test failures. API code should not be able to progress through SDLC environments until successful test execution. Part B Technical v1.0 Page 62 of 126

63 2.2.6 Granularity There is a question as to how fine-grained an API resource should be. APIs should be designed at the lowest practical level of granularity because this makes each service simpler and allows them to be combined in ways that suit the application developer. The key principle is to design services that can be re-used and combined in different ways, and not tie the application developer to a specific way of working or sequence of API consumption just because that is how it is configured or built in the back end. If an API offers very fine-grained resources, then consuming applications end up having to make many more calls to the API in order to collect all the information they need, resulting in chattier communications. On the other hand, if the API is based on very coarse grained resources (returning everything about the resource) the response payloads could end up being enormous, may not support all API application developers needs, and the API could become cumbersome to use and maintain. There may also be a need for varying granularity within one API, depending on the purpose and use of that API. For example, if a blogging API was being created, it may make sense to offer a coarse grained resource for posting a new blog entry as a bundle of content including pictures, sound, keywords as well as the textual blog entry. But when offering the ability to like or comment on a blog entry it would make sense, for clarity and ease of use, to offer these as separate sub-resources, one for indicating a 'like' and one for submitting a comment. It is important to aim for a granularity which will prevent business logic from leaking into the API e.g. requiring API calls to be made in a certain sequence to meet with internal business processes. With the blogging example, consider the impact of a fine-grained API being offered for posting blog entries, with separate calls for adding images, sounds and metadata. If metadata is mandatory, then the API application developer needs to ensure that every blog post call is followed by a metadata call. This puts the onus on the application developer to get the sequence of calls correct and could lead to inconsistent data if they do not adhere to the rules. It also leads to close coupling between the consuming application and the API, and if the business logic changes in the future there is a downstream impact on all application developers, requiring them to modify their applications. General guidelines are: Don't aim for the finest granularity - an API should be built around each discrete and updateable resource There does not need to be a one-to-one mapping between a manual service and a corresponding API; APIs should support the process but not try to encapsulate the process A rough guide is to have one API operation for each entity lifecycle state transition Resources can be identified by reviewing a business process and identifying the key entities which underpin the process. Any entities which the organisation manages or maintains throughout its full lifecycle will be potential API resources. There is typically one API operation for each entity lifecycle state transition. There should be only one API function for one business outcome (e.g. change an address) Consider using query parameters in URLs for different granularity of resources. For example, /passengers.json could return a list of names, whilst /passengers.json?detail=full could return detailed information about each passenger in a list Interface Specification An API represents a contract between the provider and the consumer for access to provider resources. Because API consumption is a programmatic exercise, it is important to have a Part B Technical v1.0 Page 63 of 126

64 clear definition of what the API offers and how those API resources are to be accessed. This definition is the interface specification. The interface specification should be designed in advance of developing the API, as the act of working out the specification often helps think through all the issues that could impact the design of the underlying resource-handling code. It also helps consuming application developers to review the capabilities being offered to see if the capabilities meet their needs before they start developing to the specification. When outsourcing API development, the interface specification can be written in abstract as a means of defining the API the vendor should build. The specification can be handled as a separate entity, is version controllable, and it should be possible to use it as the main API documentation or reference. There are several API modelling languages available for defining the interface specification. Some are proprietary (e.g. RAML, API Blueprint) whilst others are machine-readable but not human readable (e.g. WADL). The most commonly used API modelling language today is an open standard known as the OpenAPI Specification (Swagger) maintained by the Open API Initiative. This initiative is supported by many of the main commercial entities in IT (Google, IBM, Microsoft, Atlassian, Paypal etc.). It offers a standard, programming language-agnostic (JSON/YAML) interface to REST APIs which allows both humans and computers to discover and understand the capabilities of the service. For an example of a Swagger interface specification see Appendix C - Example Interface Specification. It is important to use a modelling language to define the API interface specification as essentially it is text or code and can be maintained using source code management systems. RECOMMENDED Use OpenAPI/Swagger as the interface specification language for all APIs being developed Orchestration As a general rule using APIs as an orchestration tool is not recommended. This is due to the complexity that this can introduce to an API architecture. That said, however, simple orchestration may in some cases be appropriate. An example of this may be a mashup API that creates a new logical API resource by combining related data held in two or more back end systems. A mashup is not appropriate when you create an API that merges two distinct API products together. If a consumer requires this functionality the mashup should be performed by the consuming application. Other examples of simple orchestration could include mediation such as message transformation (JSON to XML or canonical message model transformation) Software Development Kits (SDK) RECOMMENDED It is recommended that API Providers offer an SDK to developers of consuming applications. An SDK can be considered the implementation toolset for use of an agency s APIs. By providing developers with an SDK they can build applications faster without having to understand all of the API s complexities. An SDK should provide sample code that explains the functionality of your API to potential application developers. An example of a public service API SDK can be found at the US Department of Labour GitHub repository Publication/Catalogue Once an API is in sufficient state to be offered to API consumers, the API definition should be published to an API catalogue. The primary API discoverer is the developer, so an external API must be well documented, and provide accurate and up-to-date guidance via the catalogue. Part B Technical v1.0 Page 64 of 126

65 Government are considering options for a central point of discovery for externally accessible public sector APIs. RECOMMENDED Publishing to a catalogue is recommended for internal APIs. REQUIRED Publishing to a catalogue is required for external APIs. 2.3 HTTP Verbs REQUIRED Access to REST APIs must be via the standard HTTP verbs: GET, PUT, POST, DELETE in line with the W3C standard, namely: GET POST PUT DELETE OPTIONS GET GET is used: For retrieval of information If the interaction is more like a question (i.e. it is a safe operation such as a query) The HTTP GET method is used to read (or retrieve) a representation of a resource. It can be used to retrieve information about an individual resource (e.g. customers/123456) or to retrieve a collection (list) of resources (e.g. customers). In the happy (non-error) path, GET returns a representation in JSON and an HTTP response code of 200 (OK). In an error case, it most often returns a 404 (NOT FOUND) or 400 (BAD REQUEST) if the requested resource does not exist. According to the HTTP specification, GET (along with HEAD) requests are used only to read data and not change it. Therefore, they are considered safe when used this way. That is, they can be called without risk of data modification or corruption - calling it once has the same effect as calling it 10 times. GET is idempotent, which means that making multiple identical requests ends up having the same result as a single request. Do not expose unsafe operations via GET - it should never modify any resources on the server. Examples: GET Accept: application/json,version=1.* Accept-Encoding: gzip,deflate Host: api.example.com #Success Response Content-Length: 612 Location: Date: Fri Jul 08 16:52:21 NZST 2016 CorrelationID: 65347f577ade5f abb Part B Technical v1.0 Page 65 of 126

66 Transaction-Id: 65347f577bde5949ac7bd448 "id": "names": "firstname": "John", "middlename": "Howard", "lastname": "Doe", "salutation": "Mr John Doe", "title": "Mr", "addresses": [ "type": "home", "address1": "1 Some Street", "address2": "Some Suburb", "address3": "Some City", "address4": null, "country": "New Zealand", "postcode": 1111, "type": "business", "address1": "2 Work Street", "address2": "Work Suburb", "address3": "Work City", "address4": null, "country": "New Zealand", "postcode": 2222 ] GET GET Some API GET calls will return a single resource item whilst some calls will return a collection of resource items. See section Singletons vs. Collections for more details POST POST is used: To create a piece of information or resource item To spawn an action If the interaction is like placing an order If the interaction changes the state of the resource in a way that the user would perceive (e.g. a subscription to a service) If the user be held accountable for the results of the interaction The POST verb is most-often utilized to create new resources. In particular, it is used to create sub-resources: that is, subordinate to some other (e.g. parent) resource. In other words, when creating a new resource, POST to the parent and the API takes care of associating the new resource with the parent, assigning an ID (new resource URI), etc. On successful creation, POST returns an HTTP status 201, with a Location header containing a link to the newly created resource. Part B Technical v1.0 Page 66 of 126

67 POST is neither safe nor idempotent. It is therefore recommended for non-idempotent resource requests. Making two identical POST requests will most-likely result in two resources containing the same information. The body content of a POST request will differ to that of a PUT, as the provider will create certain values such as a unique identifier for the resource and its creation date. In many cases a POST may have a small subset of the data that can exist on a resource a subsequent PUT to the resource can be performed to populate the data. Examples: #Create a new passenger resource POST "names": "firstname": "John", "middlename": "Howard", "lastname": "Doe", "salutation": "Mr John Doe", "title": "Mr", "addresses": [ "type": "home", "address1": "1 Some Street", "address2": "Some Suburb", "address3": "Some City", "address4": null, "country": "New Zealand", "postcode": 1111, "type": "business", "address1": "2 Work Street", "address2": "Work Suburb", "address3": "Work City", "address4": null, "country": "New Zealand", "postcode": 2222 ] #Successful resource creation response HTTP/ Created Location: Content-Location: Content-Type: application/json "id": 12345, "names": "firstname": "John", "middlename": "Howard", "lastname": "Doe", "salutation": "Mr John Doe", "title": "Mr", Part B Technical v1.0 Page 67 of 126

68 "addresses": [ "type": "home", "address1": "1 Some Street", "address2": "Some Suburb", "address3": "Some City", "address4": null, "country": "New Zealand", "postcode": 1111, "type": "business", "address1": "2 Work Street", "address2": "Work Suburb", "address3": "Work City", "address4": null, "country": "New Zealand", "postcode": 2222 ] PUT PUT is used: To update an existing resource item To replace an existing resource item PUT is most-often utilized to update information, PUT-ing to a known resource URI with the request body containing the newly updated representation of the original resource. However, PUT can also be used to create a resource in the case where the resource ID is chosen by the consumer instead of by the API itself: where PUT is to a URI that contains the value of a non-existent resource ID. Again, the request body contains a resource representation. This method of creation should be avoided if possible, or used sparingly. If consumer-defined resource IDs are required, use POST to create the new resource and provide the consumer-defined ID in the body representation (see POST). On a successful update, PUT returns 200 (or 204 if not returning any content in the body). If using PUT to create, return HTTP status 201 on successful creation. A body in the response is optional. It is not necessary to return a link via a Location header in the creation case since the consumer already knows the resource ID, having set it themselves. PUT is not a safe operation: it modifies (or creates) state on the server, but it is idempotent. I.e. if you create or update a resource using PUT and then make that same call again, the resource is still there and still has the same state as it did with the first call. However, were the PUT call to increment a counter within the resource, the call is no longer idempotent. Sometimes this is necessary behaviour, and it may be enough to document that the call is not idempotent. However, it's recommended to keep PUT requests idempotent. It is strongly recommended to use POST for non-idempotent requests. It is good practice to perform a GET on a resource before you perform a PUT. This means that the consuming application has the latest representation of that resource. PUT should contain the entire resource content in the message body. By performing a GET prior to the PUT the consuming application can re-send the results of the GET with any modifications. When using a resource response cache, any PUT to a resource should invalidate that cache or re-populate the cache with the updated resource. Part B Technical v1.0 Page 68 of 126

69 Examples: # Get a resource GET # The response can be cached in a response cache with a cache key of the resource e.g. cachekey 12345_1w3r4y6u # Response Body "id": "1w3r4y6u", "passenger_name": "SMITH/NICOLAS", "pnr_number": "CG4X7U", "travel_class": "business", "seat": "74J", "auxiliary_fields": [ "label": "Terminal", "value": "T1", "label": "Departure", "value": "30OCT 19:05" ], "secondary_fields": [ "label": "Boarding", "value": "18:30", "label": "Gate", "value": "D57", "label": "Seat", "value": "74J", "label": "Sec.Nr.", "value": "003" ], "flight_info": "flight_number": "KL0642", "departure_airport": "airport_code": "JFK", "city": "New York", "terminal": "T1", "gate": "D57", "arrival_airport": "airport_code": "AMS", "city": "Amsterdam", "flight_schedule": "departure_time": " T19:05", "arrival_time": " T17:30" Part B Technical v1.0 Page 69 of 126

70 # Put to a resource (update) changing the seat number PUT #The api invalidates cache object with key cachekey 12345_1w3r4y6u "id": "1w3r4y6u", "passenger_name": "SMITH/NICOLAS", "pnr_number": "CG4X7U", "travel_class": "business", "seat": "10J", "auxiliary_fields": [ "label": "Terminal", "value": "T2", "label": "Departure", "value": "30OCT 19:05" ], "secondary_fields": [ "label": "Boarding", "value": "18:30", "label": "Gate", "value": "D57", "label": "Seat", "value": "10J", "label": "Sec.Nr.", "value": "003" ], "flight_info": "flight_number": "KL0642", "departure_airport": "airport_code": "JFK", "city": "New York", "terminal": "T1", "gate": "D57", "arrival_airport": "airport_code": "AMS", "city": "Amsterdam", "flight_schedule": "departure_time": " T19:05", "arrival_time": " T17:30" Part B Technical v1.0 Page 70 of 126

71 # Response that update was successful HTTP/ No Content Content-Location: /passengers/12345/flights/1w3r4y6u DELETE DELETE is used: To delete a resource item DELETE is only used to delete a resource identified by a URI. On successful deletion, DELETE returns an HTTP status 200 (OK) along with a response body, and optionally the representation of the deleted item, or a wrapped response. Alternatively, DELETE can return HTTP status 204 (NO CONTENT) with no response body. DELETE operations are idempotent. If you DELETE a resource, it is removed. Repeatedly calling DELETE on that resource ends up as the same result: the resource is gone. However, calling DELETE on a resource a second time will often return a 404 (NOT FOUND) since it was already removed and therefore is no longer findable. This, in some opinion, makes DELETE operations no longer idempotent, but the end-state of the resource is the same. Returning a 404 is acceptable and communicates accurately the status of the call. If calling DELETE were to decrement a counter (within the resource), the DELETE call is no longer idempotent. Example: #Delete a resource DELETE OPTIONS #Response HTTP/ No Content Content-Location: /passengers/12345/flights/1w3r4y6u OPTIONS is used to retrieve information about what the consumer is allowed to do with the Resource Other PATCH is a valid HTTP verb, but is still in draft and its use is discouraged due to complexity. HEAD is also rarely used, but should retrieve metadata about a Resource, such as a hash of the data or when it was last updated. 2.4 URI Construction URI construction is important in that it is the door through which consumers enter to obtain agency resources. It should be intuitive and easy to guess what an endpoint does just by looking at the URI and HTTP verb, without needing to see a query string. As basic guidance, endpoint URLs should advertise resources, and avoid verbs API Offering RECOMMENDED Is it recommended that agencies make it clear in the URL that their offering is an API: Part B Technical v1.0 Page 71 of 126

72 Example Version APIs should have a clear indication of the version, so that application developers can ensure they are using the appropriate version for their consuming application. RECOMMENDED Header-based versioning is recommended (see section 2.9, API Version Control); however, it is recognised that some API infrastructure does not readily support header-based versioning. URL-based versioning is a viable alternative, as the version number in the URL should only change when major revisions have been made and the interface has changed substantially without backwards compatibility. For URL-based versioning the URI should include /vn with the major version (N) and v as a prefix. Agencies should not include minor version numbers when using version numbers in the API paths. Template /vversion/ Example #Get details for provider id version 1 of the API GET Accept: application/json Host: api.example.govt.nz #Get details for provider id version 2 of the API GET Accept: application/json Host: api.example.govt.nz Namespaces For APIs where an agency holds multiple responsibilities which may result in overlapping resource naming (for example MBIE could have the resource "accommodation" in two contexts: tenancy/contracts, tourism/contracts) it is recommended that namespaces be used to avoid any ambiguity. The namespace would be the first noun in the URI and should reflect the function of government being offered by this API. Namespaces may be singular or plural, depending on the situation. Template /version/namespace/ Example /v1/biosecurity/ Resources and Sub-resources RECOMMENDED Resource names should be noun-based, and collection resource names should be plural nouns, e.g. /passengers in lower case. Resource naming should be short, simple and clearly understandable. It should also be human-guessable, avoiding technical or specialist terms where possible. Part B Technical v1.0 Page 72 of 126

73 Sub-resources must appear under the resource they relate to, but should go no more than three deep i.e. /resource/id/sub-resource/id/sub-sub-resource. If you reach a third level of granularity (sub-sub-resource), it may be worth reviewing your resource construction to see if it is actually a combination of multiple first or second level resources. The URI references for resources should consistently use the same path structure to refer to resources. Sub-namespace or sub-folders should be avoided, to maintain path consistency. This allows application developers to have a predictable experience in case they are building URIs in code. /version/resource/resource-id/sub-resource/sub-resource-id Example Template /version/namespace/resource/resource-id/sub-resource/subresource-id Word Separation Hyphens have traditionally been used as word separators in URLs, as search engines (particularly Google) prefer a hyphen to split words because a hyphen is not a word character (as defined in regular expression language). This has led to hyphens, or kebabcase, being the de facto standard in the interests of readability and Search Engine Optimization (SEO). Therefore, in order to keep URLs consistently formatted, path and query string parameters should be lower case with hyphen separators for multiword names. Example Query Arguments Query arguments should be used to filter a response i.e. modify a returning result set. The general rule is: If it changes the behaviour of the result set then it should be a query argument If it changes the behaviour of the API then it should be in the path Query arguments are generally used for: Sorting or ordering the result set - e.g. sort-order=ascending Pagination - pagination is a query argument because it effectively acts as a filter and limits the result set returned. This is particularly useful with large response data sets. For more details see section When using pagination it is a good idea to inform the consumer where they can find previous and subsequent result sets using hypermedia as discussed in section of this document. Example "_links": [ Part B Technical v1.0 Page 73 of 126

74 "rel": "next", "href": " "rel": "prev", "href": " "rel": "self", "href": " ] Limiting the result set - e.g. by specifying which fields to return. This approach can be complicated and is often a decision based on functionality vs complexity. For example, it may be desirable to be able to filter a result set to a specific set of objects. Whilst this is possible, it is not a recommended approach. If this kind of flexibility is required in an API it could be a good time to consider the use of Open Data Protocol (OData). In cases where response filtering is used, providers should ensure that they use a JSON schema by default so that consumers have the ability to understand the entire resource and do not need to query the resource for the message structure. Example #Get a filtered result set for a learner GET e,dateofbirth "firstname": "Mary", "lastname": "Contrary", "dateofbirth": " ", "_links": [ "rel": "self", "href": " me,dateofbirth" ] 2.5 HTTP Headers Request Headers Request headers are supplied by the consuming application and must include the following elements: Accept Content Type Authorization Part B Technical v1.0 Page 74 of 126

75 Header definitions can be found in RFC Accept REQUIRED An Accept header is required to indicate what format the consuming application wants the information to be returned as (e.g. JSON or XML). It is preferable for the consuming application to specify the response content type that they expect, however it is acceptable to use the q parameter to specify the quality of support. Examples #Accept JSON and latest version Accept: application/json,version=* #Prefer JSON, Accept XML and latest version 1 release Accept: application/json;q=1.0,application/xml;q=0.8,version=1.* #Accept XML only and version 1.2 only Accept: application/xml;q=1.0,application/json;q=0.0,version=1.2 If the client has specified via the Accept header that it doesn t support any formats provided by the API, the server should return an HTTP 406 Not Acceptable response. The response should also include a Link header with a link to documentation or help on supported formats. The body of the representation should include an error message in human readable plain text or HTML. Example Response # Response that indicates the request failed due to the client not # supporting any response formats provided by the API HTTP/ Not Acceptable Content-Type: text/plain This API does not support any of the requested content types specified in the HTTP Accept header. Please see for details on the supported content types. The Accept header is required, however the version and q are recommended as this will depend on a provider s implementation and in some cases technology choice Content-Type REQUIRED The Content-Type header is required for all requests that include a request body i.e. POST, PUT, DELETE. Example #PUT Accept: application/json,version=1.* Content-Type: application/json Authorization: Bearer x6tlb4jaomezc5rj3cil3kxxninq "id": "33245", "names": "firstname": "John", "middlename": "Howard", "lastname": "Doe", Part B Technical v1.0 Page 75 of 126

76 "salutation": "Mr John Doe", "title": "Mr", "addresses": [ "type": "home", "address1": "1 Some Street", "address2": "Some Suburb", "address3": "Some City", "address4": null, "country": "New Zealand", "postcode": 1111, "type": "business", "address1": "2 Work Street", "address2": "Work Suburb", "address3": "Work City", "address4": null, "country": "New Zealand", "postcode": 2222 ], "qualifications":[ "id": "1a4rf54ee345", "type": "Some type", "provider": "University of life", "providerid": "0o9i8u7y6tt", "id": "8u7y6t5r4ee", "type": "Another type", "provider": "School xyz", "providerid": "1q2w3e4r5tt" ] Authorization REQUIRED Most API requests will be authorised. The Authorization header should be used for this purpose and no other. Note that the Authorization header is not the best place for an API Key. The Authorization header is comprised of a type and a token depending on the authorization type. Example #OAuth Access Token Authorization: Bearer x6tlb4jaomezc5rj3cil3kxxninq #HTTP Basic Authorization: Basic c29tzxvzzxi6c29tzsbwyxnzd29yza== Part B Technical v1.0 Page 76 of 126

77 Accept-Encoding RECOMMENDED In some cases, especially where API responses are very large, it may be worth compressing HTTP responses. (Note: where possible it is best to avoid large payloads in API responses; however, this may not always be possible.) An Accept-Encoding value of gzip instructs a provider that supports HTTP compression that the consuming application supports compressed responses. Example Accept-Encoding: gzip,deflate API Key Header REQUIRED An API key issued to a consuming application should be sent with every request made to the API. The name of the header is up to the agency, but it should not be an X- prefixed header as this use is deprecated (see 2.5.3). Sometimes API keys are passed in URIs; however, this is not considered best practice and should be avoided. Example KeyID: px680lyikasubwluzj8avbhvqk03ghtr If-Modified-Since/If-None-Match RECOMMENDED The If-Modified header is a pre-condition header used to instruct a client on the validity of the resource requested. Note that pre-condition handling is not required, but where appropriate should be considered. Pre-condition control can be a complex topic so in some cases the complexity may eclipse the value. Conditional header standards are available in Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests RFC7232. A consuming application can specify an If-Modified-Since header with a date and time. The server checks the resource to see the last modified time. If the last modification was before the date/time specified in the header, the server will respond with an HTTP 304 status code telling the consuming application that the resource they requested has not changed and therefore there is no need to return the full payload in the response. When using If-Modified headers you should also implement ETag as discussed in the response headers section ( ). Example # Request with If-Modified-Since and If-None-Match headers If-Modified-Since: Wed, 18 Jun :00:00 GMT If-None-Match: " a7c876b7e" The If-None-Match header indicates to the provider the ETag for the resource that it currently holds for that resource. Part B Technical v1.0 Page 77 of 126

78 Example Response # Response that indicates the client s current representation is still # valid HTTP/ Not Modified ETag: " a7c876b7e" Content-Length: Request tracking headers RECOMMENDED To support the tracking of API requests it is a good idea to apply a transaction ID header that will enable analysts to trace a request throughout its lifecycle. Note: there is no set naming standard for a transaction ID header. There are two common approaches to this: one is to request that the consuming application generate a unique transaction ID and the provider will capture it and respond with a corresponding correlation ID; the second is to allow the provider to generate the transaction ID as close to the edge of its domain as possible, and respond to the consuming application with that transaction ID. In this case a consuming application administrator or developer should capture and log that transaction ID for audit purposes so that it can be relayed to agency analysts if required. Example Request #Request that includes a consuming application generated transaction #ID Request-Transaction-Id: c188cb0e-1a c-0f0bb Example Response #Response to a consuming application that provided a client generated #transaction ID Request-Transaction-Id: c188cb0e-1a c-0f0bb Response-Correlation-Id: 514dbf e4d-86ee-e2001f #Response to a consuming application where there was no client #generated transaction ID #Note that the server has generated the transaction ID Response-Correlation-Id: c188cb0e-1a c-0f0bb Response Headers Response headers are supplied by the provider and must contain Content-Type Content-Type REQUIRED The Content-Type must be returned in the response, and indicate the format type the response content is in e.g. Content-Type: application/json. The Content-Type header should also include the version of the API that processed the request e.g. Content-Type: application/json,version= Location RECOMMENDED Part B Technical v1.0 Page 78 of 126

79 Where a new resource item has been created (POST), the server should respond with a 201 Created HTTP header and a Location header indicating the URI of the newly created resource item e.g. Location: A location header should also be returned by a server when it sends a 3** HTTP response, indicating a temporary or permanent redirection. A location header can be, as recommended by this standard, absolute e.g. Location: or relative e.g. Location: /v1/crownproperty/ Content-Location RECOMMENDED The Content-Location header indicates the location of the requested resource. It should be an absolute and not a relative link. Example #GET Content-Type: application/json Content-Location: "propertyid": 12345, "address1": "1 Some Street", "address2": "Some Suburb", "address3": "Some City", "address4": null, "country": "New Zealand", "postcode": 1111, "type": "commercial" Cache-Control RECOMMENDED It is important to consider caching when designing APIs, to ensure that the APIs you build can perform at scale. The Cache-Control header is only concerned with caching that occurs externally to the API layer. Common examples may be client side caching or caching within a Content Delivery Network (CDN). This header should not be concerned with caching mechanisms within the API layer such as response caches or object caches, which require a different caching strategy (see section 2.11). The Cache-Control header is comprised of the following private public: This indicates if the cache is specific to the consumer or not a setting of private will prevent the cache from being persisted in a public cache. no-cache: This instructs a cache to revalidate the resource every time. no-store: This instructs a cache not to store the response. It also ensures that no part of the request is stored too. max-age: This instructs the cache on expiry time. This can be used but it makes more sense to use the Expires header discussed below. s-max-age: Similar to the above, however it refers to a shared cache hence only relevant to CDNs and other intermediary caches. must-revalidate: This instructs the client to resend request headers and receive confirmation that the resource has not changed. Part B Technical v1.0 Page 79 of 126

80 proxy-revalidate: Similar to the above, however specific to shared caches. Example # Example caching response HTTP/ OK Date: Wed, 06 Jul :14:52 GMT Expires: Wed, 06 Jul :14:52 GMT Cache-Control: private,max-age=14400,must-revalidate Content-Type: application/json Care should be taken when using caches, especially shared or public caches. In some cases it may be more prudent to use the no-cache directive to ensure that content is not cached outside your network. This does not rule out caching within your API layer. For more information on API caching please see section HTTP Cache Control standards can be found in Hypertext Transfer Protocol (HTTP/1.1): Caching RFC Expires RECOMMENDED When using Cache-Control you should include an Expires header in the response for backward compatibility ETag (Entity Tag) RECOMMENDED The ETag header is a concurrency control header, and when in a response it is a tag that the consuming application can use when making a subsequent request to a resource. Note that pre-condition handling is not required, but where appropriate should be considered. Precondition control can be a complex topic so in some cases the complexity may eclipse the value. # Response with ETag Header # Consuming application would send this ETag in an If-None-Match #request header ETag: " a7c876b7e" Custom X- HTTP Headers X- notation headers have been deprecated as per RFC and should not be used. This is because the X- name often became a standard, making the X-notation inaccurate (e.g. x- gzip became gzip) and requiring support for both the X- and the non-x- notation. Common uses of the X- headers are examples such as the X-HTTP-Method-Override header or the X- Forwarded-For header. Where an agency feels that a custom HTTP header is required that is not available within the current HTTP standard they can use a custom header without the X- prefix Part B Technical v1.0 Page 80 of 126

81 2.6 Returned Content The returned content is the body of the message that the API returns in response to a request. This usually relates to the resource that was requested Resource Scope The amount of data to return about a resource depends on several factors: Business context - does all the resource information being returned have relevance to the business process or activity consumers will use it for? Payload size and network efficiency - how much information does the resource contain, and is it so bulky it could impact network performance? Usefulness - is the information in the returned content always useful to the consumer, or is it something they may need once but never again? Analysis of the full resource data against these factors should help determine how much resource data is returned in the response Formats REST responses should be of a format which is both human and machine readable. Even if your initial consumer is via B2B, the API should be designed, where possible, for all potential consumers. REST APIs should, by default, return content in JSON format. But it is good practice to support multiple message formats (e.g. XML, Multi-part MIME) and allow consuming applications to request the format they wish to consume (in line with the "Design for the Consumer" principle in Part A, section 4). The response format required from a GET request must be indicated by the consumer in their request using the Accept header. For example: #GET a resource and specify the response type Accept: application/xml #Response <?xml version="1.0" encoding="utf-8"?> <CrownProperyResponseBody xmlns=" <id>0219c ab6-a55b-b0de7537c426</id> <category>commercial</category> <location> <streetname>northbound Boulevard</streetName> <town>wellington</town> <postalcode>6011</postalcode> <countrycode>nz</countrycode> </location> <pricing> <rentfrequency>annual</rentfrequency> <currencycode>nzd</currencycode> <priceperunitarea> <price>1000</price> <units>squaremetre</units> </priceperunitarea> <transactiontype>rent</transactiontype> </pricing> Part B Technical v1.0 Page 81 of 126

82 <property_type>office</property_type> <areas> <internal> <minimum> <value>183</value> <units>squaremetre</units> </minimum> <maximum> <value>188</value> <units>squaremetre</units> </maximum> </internal> </areas> <detaileddescription> <text>free text describing the property in some way.</text> </detaileddescription> <tenancy> <life_cycle_status>tennanted</life_cycle_status> <nextreviewdate> </nextreviewdate> </tenancy> </CrownProperyResponseBody> The request format for a request containing a request body (POST, PUT, PATCH) must be supplied in the Content-Type request header. For example: #PUT (update) a resource Content-Type: application/json Accept: application/json #Request BODY "id": "0219c ab6-a55b-b0de7537c426", "category": "commercial", "location": "streetname": "Northbound Boulevard", "town": "Wellington", "postalcode": "6011", "countrycode": "NZ", "pricing": "rentfrequency": "annual", "currencycode": "NZD", "priceperunitarea": "price": 1000, "units": "squaremetre", "transactiontype": "rent", "property_type": "office", "areas": "internal": "minimum": "value": 183, "units": "squaremetre", "maximum": "value": 188, "units": "squaremetre" Part B Technical v1.0 Page 82 of 126

83 , "detaileddescription": [ "text": "Free text describing the property in some way." ], "tennancy": "life_cycle_status": "vacant", "nextreviewdate": " " For a good example of this in action, see the tabbed format options for presenting patient information on the FHIR 11 site. Where JSON format is used in responses, it must conform to the JSON Standard (RFC ). In line with this standard, textual content should be UTF-8 encoded by default, and any deviations from this must be limited. It in inadvisable to use APIs for returning binary data such as images in the response content; hyperlinks to images are the preferred returned response. But where images are returned, ensure that the image encoding is MIME, HTTP Multipart. HTML is allowed, but is less flexible for consumer usage (e.g. how can it be consumed by a native mobile app?), ties content to presentation and loses its benefits when building a single page application. It also reduces the ability to filter the content in transit for consumers who have fine grained control on what they can access. However, it may be necessary for some forms of API e.g. APIs with geospatial content. Avoid using bespoke formats in returned content. RECOMMENDED Returned content format should be JSON by default. REQUIRED Requests for a specific return format must be defined in the Accept header Layout Responses should be a JSON object (not an array) by default. Using an array to return results limits the ability to include metadata about results, and limits the API's ability to add additional top-level properties in the future. Don't use unpredictable properties. Parsing a JSON response where properties are unpredictable (e.g. derived from data) is difficult, and adds friction for clients. Example of good layout "response_meatadata": "item1": "value1", "item2": "value2", Part B Technical v1.0 Page 83 of 126

84 "responsearray": [ "name": "entity1", "type": "good", "name": "entity2", "type": "good" ] Example of bad layout [ "name": "entity1", "type": "bad", "name": "entity2", "type": "bad" ] JSON Property Names Within content, property names should conform to the following guidelines: Property names should be meaningful names with defined semantics Property names must be camel-case ASCII strings The first character must be a letter or underscore (_) Subsequent characters can be a letter, a digit, or underscore Reserved JavaScript keywords should be avoided Consistency Preserve backwards compatibility with existing consumers of the API by returning expected fields and employing sensible default values for missing fields. Keep consistency of terminology throughout, so that the consumer is not misled. Avoid modifying the semantics of content to new meanings, e.g. don t change a 'title' field from meaning the title of the page, to meaning the prefix for a name to the person s job title Singletons vs. Collections Some API GET calls will return a single resource item (e.g. GET whereas some calls will return a collection of resource items (e.g. GET and the handling and error treatment can be different. A single item will generally consist of all the pertinent fundamental details about the resource, and the GET will return these fundamental details. Example #GET a single passenger resource by ID Part B Technical v1.0 Page 84 of 126

85 GET Accept: application/json,version=1.* id : 12345, names : firstname : John, middlename : Howard, lastname : Doe, salutation : Mr John Doe, title : Mr, addresses : [ type : home, address1 : 1 Some Street, address2 : Some Suburb, address3 : Some City, address4 : null, country : New Zealand, postcode : 1111, type : business, address1 : 2 Work Street, address2 : Work Suburb, address3 : Work City, address4 : null, country : New Zealand, postcode : 2222 ] If passenger did not exist, the HTTP return code would be 404. If a requested sub resource did not exist e.g. addresses for passenger 54321: GET an HTTP 404 response would not be valid. The correct response would contain an empty array: #GET a single passenger address resource by ID GET Accept: application/json,version=1.* id : 54321, addresses : [ type : null, address1 : null, address2 : null, address3 : null, address4 : null, country : null, postcode : null ] A collection will take the form of an array of the given resources, including any related metadata. The collection returned should be the complete set, leaving it to the consumer to reduce the quantity of information returned by sending filters in the resource request (see Part B Technical v1.0 Page 85 of 126

86 section 2.4.6, Query Arguments). The only reason the collection should be filtered by the API provider is based on the security context; the API consumer should only gain access to what that consumer is allowed to see. It is advisable to aim for consistent naming of collection resource fields, as this enables application developers to create generic handling of the data from across various resource collections offered by the API. Collection resources should not contain binary attachments or other content that will lead to large response payloads. Note that the example below is contrived and it would be rare to expose a resource collection as large as passengers. Example #GET a collection of passenger resources GET Accept: application/json,version=1.* passengers : [ id : 12345, names : firstname : John, middlename : Howard, lastname : Doe, salutation : Mr John Doe, title : Mr, id : 98765, names : firstname : Mary, middlename : Pauline, lastname : Smith, salutation : Ms Mary Smith, title : Ms,..lots more passengers here.. : ], _links : [ rel : 12345, href : rel : 98765, href : more links here.. : ] If there were no results from a query against a filter, e.g. passengers booked on a specific flight (GET the return value would be Part B Technical v1.0 Page 86 of 126

87 a 200 response code, because the query is not invalid and there may be such data in the future HATEOAS HATEOAS, or Hypermedia as the Engine of Application State, is the principle of not just returning a response to a request, but also returning links to other useful capabilities. So if a POST has just been used to create an item, the response could return links to options to update the item, delete the item and view other items of the same type. This makes it possible for UIs which utilise an API to be automatically generated (in terms of hyperlinks) and adaptable to change (response content links). If a web page effectively captures a single state in time, then hyperlinks are transitions to other states. The HATEOAS principle suggests that REST should return not only response content (current state) but also links to transitions to other states. This could result in thinner clients and fatter APIs, and means UIs need to be built to adapt to potential new transitions, but makes it possible to update all UIs no matter what platform they sit on (mobile app, web application etc.). For example, consider the following representation of a person and sequence of requests: This example assumes that an address is a unique resource in its own right and that people are associated with addresses. (This may not always be a practical application and the example is used just to illustrate the hypermedia concept.) 1. User submits a request for a person record: # Request person details: GET Host: Accept: application/json;q=1.0 # Response with person details: HTTP/ OK Content-Location: Content-Type: application/json;charset=utf-8 Cache-control: max-age=3600,must-revalidate "personid": "d9e1a2f6-6bc9-46af-9dbb-50de7f6eacd5", "name": "Joe Bloggs", " ": "joe.bloggs@example.com", "addresses": [ "id": "5046b8f2-e8aa-4ec9-a597-34d01af0fd32", "streetnumber": "12", "streetname": "Somewhere Street", "City": "Springfield", "type": "home", "id": "18a1a85c-3b1e-41b9-909c-668f2a803b69", "streetnumber": "1", "streetname": "Some Other Street", "City": "Springfield", "type": "business" Part B Technical v1.0 Page 87 of 126

88 ], "_links": [ "rel": "self", "href": " 9dbb-50de7f6eacd5", "rel": "address1", "href": " 4ec9-a597-34d01af0fd32", "rel": "address2", "href": " 41b9-909c-668f2a803b69" ] Assume an intermediate proxy were to cache that response as directed by the cache control header. 2. The user then updates the address details changing the street number: # Request to update the address details: PUT 50de7f6eacd5 HTTP/1.1 Host: api.example.govt.nz Content-Type: application/json "streetnumber": "12a", "streetname": "Somewhere Street", "City": "Springfield" # Response that the update was successful: HTTP/ No Content Location: a597-34d01af0fd32 If the user were now to re-fetch the person details again as per step 1, the representation would contain a stale version of the address (street number 12 instead of 12a) because the address representation had been cached. This caching could occur anywhere along the network path (including public internet) between the server and client. Using hypermedia links for related resources improves the visibility of the API, and avoids these types of problems. Consider the revised person representation below: "personid": "d9e1a2f6-6bc9-46af-9dbb-50de7f6eacd5", "name": "JoeBloggs", " ": "joe.bloggs@example.com", "addresses": [ "rel": "self", "href": " 4ec9-a597-34d01af0fd32", Part B Technical v1.0 Page 88 of 126

89 "rel": "self", "href": " 41b9-909c-668f2a803b69" ], "_links": [ "rel": "self", "href": " 9dbb-50de7f6eacd5" ] Using this approach, the embedded object is replaced with a link to the real object. Although this approach requires a larger number of network requests, it avoids the stale data problems and reduces payload size, which is an important consideration for APIs being used by mobile devices or over high latency networks. 2.7 Consideration of state In almost all cases a REST API should be entirely stateless. As part of processing it is possible for an API to gather context and pass this to a downstream system, however an API should not maintain that context for future requests. This should not be confused with caching, as discussed in section 2.11 of this document. 2.8 Batch Handling & Transactions APIs are not designed for large payloads i.e. batch handling for retrieving or uploading batches of data. APIs are geared towards stateless, usually synchronous, web-like requests for individual discrete data transactions. However, batch handling can be achieved through bundling multiple calls to the same API. This helps achieve atomicity of transactions and aids recoverability in event of errors. 13 When handling transactions, it is important to consider the troubleshooting and recovery aspects of error handling. This includes visibility of transaction progress and the ability to perform root cause analysis. To achieve this, it is advisable that logging is performed on all transactions coming through an API, with accurate timestamping, so that monitoring tools can visualise transaction progress. It also requires transaction identifiers (See section 2.5.1, Request Headers) to be built into transactional API calls to ensure the transaction is traceable end-to-end. In some cases, it may be deemed appropriate to provide an asynchronous batch type capability using an API. This would usually be implemented in a scenario where legacy system impact is a concern. An example of this could be a bulk creation of person records in a database based on a batch event in a consuming legacy application. In such a scenario it would be preferable for the consuming application to treat each person as a unique event and to POST to a person API for each new unique resource. Whilst this may seem like an unnecessary overhead, by individualising the transactions each success or error scenario can be handled in its own right and the consumer can be informed of their transaction status. If this is not possible, due to some system restriction, it is possible to asynchronously POST multiple person details to an API. This type of interaction should not be attempted 13 See for best practice Part B Technical v1.0 Page 89 of 126

90 synchronously as large batches will tie up HTTP threads and in some cases require client/server timeout control. Example Request The following is an example of a singleton create person request: "names": "firstname": "John", "middlename": "Howard", "lastname": "Doe", "salutation": "Mr John Doe", "title": "Mr" This is an example of a bulk person create API call: #POST a collection of person resources POST Accept: application/json,version=1.* "people": [ "names": "firstname": "John", "middlename": "Howard", "lastname": "Doe", "salutation": "Mr John Doe", "title": "Mr", "names": "firstname": "Mary", "middlename": "Pauline", "lastname": "Smith", "salutation": "Ms Mary Smith", "title": "Ms", "...lots more people here...": "..." ] Example Response Note that the API is unable to respond with headers such as Location as this request should be treated as asynchronous. 200 OK Part B Technical v1.0 Page 90 of 126

91 2.9 API Version Control API Version Control Methods There are two main API versioning methodologies commonly used. There are positives and negatives to both approaches and a large amount of debate over which is the most RESTful. Rather than stipulate a methodology to use in this standard, below is a description of each methodology in order of preference. Both, however, are acceptable. What is more important is that APIs are versioned and that there is an understanding of when and why to version an API Accept Header Versioning This is usually considered the most RESTful way to version APIs because the resource path remains pure and it is possible to provide more version flexibility for clients. It is, however, technically more difficult to implement, and in many cases commercial API management/gateway products do not support or work well with this approach. Header based versioning should be performed using the Accept header where a consuming application requests an API version as defined in an Accept header. Wildcards (*) are used by the consuming application to indicate acceptance of the latest major or minor version of an API. Example #Get details for provider id latest minor version of the API GET Accept: application/json, version=1.* Host: api.example.govt.nz #Get details for provider id version 1.1 of the API GET Accept: application/json, version=1.1 Host: api.example.govt.nz #Get details for provider id latest version of the API GET Accept: application/json, version=* Host: api.example.govt.nz The response should include the version of the API that responded to the client request in the Content-Type header. Example Content-Type: application/json,version= URI (Path) Versioning URL-based versioning is a viable alternative, as the version number in the URL should only change when major revisions have been made and the interface has changed substantially without backwards compatibility. For URL-based versioning the URI should include /vn with the major version (N) and v as a prefix. Agencies should not include minor version numbers when using version numbers in the API paths. This is probably the easiest method to implement, but a number of REST purists feel that it compromises the URI and does not offer enough flexibility to consumers. If there is a requirement to support a large number of historical versions of APIs it can mean that there is complexity introduced with the number of URIs to maintain. Part B Technical v1.0 Page 91 of 126

92 Example #Query version of version 1 of the search API GET #Query version 2 of the search API GET The response should still indicate the version of the API that was called. This can be done as above, using the Content-Type header or in the Location header, as the version in the path indicates the API that was called When to Version Simply put, an API should be versioned when a change is considered breaking. One of the benefits of an API is that, if it is well designed, there should be fewer breaking changes. In government however, there are likely to be situations where legislative changes enforce a new version of an API and deprecation of all previous versions Breaking Changes A change is a breaking change if any consuming application requires changes to work with the new version. i.e. the new version will not successfully process messages provided by existing consumers. A breaking change should be considered as a major version change e.g. 1.3 to 2.0. Examples of breaking changes include: The removal of any property from the response representation The change of datatype for an existing property, or a change from optional to required The removal of any resource, or HTTP Verb support A change to the way errors are handled Any change to existing resource URIs Non-Breaking Changes A change is non-breaking change if any message that would have been processed by the previous version will be successfully processed by the new version (i.e. backwards compatible). This will enable an existing consumer of the previous version to work with the new version without requiring modification. A non-breaking change should be considered as a minor version change e.g. 1.1 to 1.2. Such changes include: The addition of new properties to the JSON representation The addition of new resources The addition of support for new HTTP Verbs (new operations) on existing resources Support for new custom headers, e.g. for request tracing Software Configuration Management It is important to remember that version control is more than just versioning the resource. An API will inherently have associated code and artefacts. Consider what comprises an API and include these as a logical artefact stored and managed in a software configuration management (SCM) system. It is a good idea to try and capture all components in a format Part B Technical v1.0 Page 92 of 126

93 that can be controlled by SCM. A good example of this is UML diagrams; these cannot be version controlled unless captured in a text like format. See or for examples textual UML modelling. An API artefact should, where possible, comprise the following: Any API code API specification (OpenAPI) Unit test scripts such as JUnit or Mocha. Continuous integration support files such as grunt.js or Ant scripts Search Search should be implemented using GET requests with filters provided as query string parameters (see 2.4.6, Query Arguments). Search can be used by API consumers to filter or sort information they want from an API. For instance, GET /groups/search?status=active would enable the consumer to filter the groups resource on status. Alternatively, GET /groups/search?status=active&sort=status,name enables the consumer to return active groups ordered by status and name. It can also be useful to allow the consumer to choose which fields they want back from the API, so they only receive the content they need in the API response. Example #Get a list of groups that are either active or inactive, sorted by #name and status GET name,status "groups": [ "id": "0219c ab6-a55b-b0de7537c426", "name": "API lovers", "status": "active", "id": "1219c ab6-a55b-b0de7537c427", "name": "Developers", "status": "active", "id": "7219c ab6-a55b-b0de7537c42c", "name": "Risk analysts", "status": "active", "id": "3219c ab6-a55b-b0de7537c42h", "name": "Writers", "status": "active", "id": "g219c ab6-a55b-b0de7537c425", "name": "Authors", "status": "inactive" ], Part B Technical v1.0 Page 93 of 126

94 "_links": [ "rel": "self", "href": " "rel": "API Lovers", "href": " a55b-b0de7537c426", "rel": "Developers", "href": " a55b-b0de7537c427", "rel": "Risk analysts", "href": " a55b-b0de7537c42c", "rel": "Writers", "href": " a55b-b0de7537c42h", "rel": "Authors", "href": " a55b-b0de7537c425" ] In rare circumstances it is possible that an extremely complex search API may run into query string or URL length limitations, although this is probably an indication that the design of the search operation is too complicated. The exact length limit is server and client dependent. For example, the Apache server has a default limit of 8KB, Microsoft IIS has a default of 16KB, and many browsers are limited to around 2KB maximum URL length. Where the total length can potentially exceed 2KB, the only option is to implement the search operation using the POST verb with an appropriately designed request object to contain the search parameters. Template POST /version/namespace/search-resource Example Request POST /v1/people/person-search?page=5 HTTP/1.1 Host: Content-Type: application/json "startdatebefore":" ", "position": "Manager", "businessunit": "Operations",... many other search parameters... Part B Technical v1.0 Page 94 of 126

95 Example Response HTTP/ OK Content-Type: application/json "pagesize": 10, "page": 5, "totalitems": 77, "totalpages": 8, "items": [... lots of people objects here... ] "_links": [ "rel": "first", "href": " "rel": "prev", "href": " "rel": "next", "href": " "rel": "last", "href": " ] Other considerations include special characters or filters that add additional complexity due to URL encoding requirements. For example, the search function might need to support < or > operators, or special characters for non-english language support. Additionally, structured query language (SQL) words should be avoided as common threat detection filters may block requests containing words such as DROP, ALTER or DELETE in case they are SQL injection attacks. See OWASP XSS Cheat Sheet and OWASP SQL Injection Cheat Sheet for details. Paging behaviour for search results should be consistent with the interaction as described in section 2.4.6, Query Arguments. Pagination can be implemented as a page number and page size, offset and limit, or continuation token, depending on the scale and changeability of the search results. Note: It is worth pointing out that since the HTTP protocol treats POST operations as unsafe, the result is not cacheable. This applies even if the response has a Cache-Control header included, so consider the performance impacts on the API if this a high volume operation Caching Caching enables faster responses from APIs and reduces server load. It is good for information which is frequently requested but does not change very often. The aim is to retrieve information once but re-use many times. Part B Technical v1.0 Page 95 of 126

96 There are a number of caching methodologies available, as discussed in the cache-control headers section ( ). This section focuses on two types of caching that should be owned and controlled by the provider, primarily for APIs Response Cache A response cache is a cache that contains the response to a GET request on a unique resource. The response should be cached as close to the consumer as possible while remaining inside the control boundaries of the agency. Response caches are usually fast, inmemory caches. Many off the shelf API gateway/management solutions provide a built-in response cache. When using a response cache, agencies should ensure that they understand and monitor their cache to ensure that stale objects are kept to a minimum and that they have sufficient system memory to service caching loads. This means ensuring that the cache is refreshed once a user has updated the information or a cache timeout has occurred. Consider the following example regarding response cache: Part B Technical v1.0 Page 96 of 126

97 Part B Technical v1.0 Page 97 of 126

98 Object Cache An object cache is used to cache objects that are fundamental to the function of an API but may not change on a regular basis. For example, an API may include a tax code validation step. Tax code references are held in a back end database. To prevent redundant requests to the database, the API layer could cache tax codes in its object cache and refer to these until the cache becomes invalid. The diagram below details an example sequence Error Handling Error handling is important because API consumers see the API as a black box, and when an error occurs they need to know how to handle it. Hence error responses need to be informative, yet avoid information leakage about the internals of the back end system. Errors Part B Technical v1.0 Page 98 of 126

99 should be handled in both a human- and machine-consumable way. When an error occurs, the response body should contain: The HTTP status code An API-specific error code, which API support staff will be able to look up to identify what has gone wrong A human readable error message (possibly including selective technical details if the API is a development/test release for developer consumption only) HTTP Status Codes RESTful APIs should always use the standard HTTP error responses when an error occurs, but different codes may be pertinent depending on the HTTP verb being used and the quantity of data being retrieved (e.g., single item vs. list/collection of data). The following tables give an example of good practice use of HTTP error codes: GET Response Status Codes Code Meaning Description 200 OK The request was successful and the response body contains the representation requested 302 FOUND A common redirect response; you can GET the representation at the URI in the Location response header 304 NOT MODIFIED Your client's cached version of the representation is still up to date 401 UNAUTHORIZED The supplied credentials, if any, are not sufficient to access the resource 404 NOT FOUND The requested representation was not found 429 TOO MANY REQUESTS Your application is sending too many simultaneous requests 500 SERVER ERROR An internal server error prevented return of the representation response 503 SERVICE UNAVAILABLE We are temporarily unable to return the representation. Please wait and try again later. POST or PUT Response Status Codes Part B Technical v1.0 Page 99 of 126

100 Code Meaning Description 200 OK The request was successful, and the resource was updated. The response body contains the updated representation 201 CREATED The request was successful, a new resource was created and the response body contains the representation 400 BAD REQUEST The data given in the POST or PUT failed validation. Inspect the response body for details 401 UNAUTHORIZED The supplied credentials, if any, are not sufficient to create or update the resource 404 NOT FOUND 405 METHOD NOT ALLOWED You can't POST or PUT to the resource 429 TOO MANY REQUESTS Your application is sending too many simultaneous requests 500 SERVER ERROR We couldn't create or update the resource. Please try again later DELETE Response Status Codes Code Meaning Description 204 OK The request was successful; the resource was deleted 401 UNAUTHORIZED The supplied credentials, if any, are not sufficient to delete the resource 404 NOT FOUND 405 METHOD NOT ALLOWED You can't DELETE the resource 429 TOO MANY REQUESTS Your application is sending too many simultaneous requests Part B Technical v1.0 Page 100 of 126

101 500 SERVER ERROR We couldn't delete the resource. Please try again later Table 9 - HTTP Status Codes 14 Where a consumer attempts to call a non-existent API end point, respond with a 501 Not Implemented status code API-Specific Error Code Error responses need to supply enough information to allow the API consumer to react appropriately to the error, yet not give away too much. But when a critical error occurs it is important for the API provider to be able to trace the root cause and fix it as soon as possible. One way of achieving this is to return an API-specific error code in the response to consumer. When the consumer reports the error to the API support team, the consumer can relate the associated API-specific code. The support team can then look this code up and ascertain exactly what has gone wrong and who needs to address it Human-Readable Error Message The human readable error message should be as informative as is useful to an end customer, without offering too many technical details (e.g. An account with this ID already exists ). It is important to avoid revealing system information in the human readable response, such as composition of the backend system (e.g. component names), as this informs malicious consumers as to what vulnerabilities and back doors to look for. It is also important not to confirm or deny sensitive information, such as username in a username/password combination, as this informs potential attackers which criteria they have got correct. 15 In non-production instances of the API it may be appropriate to offer more verbose error messages with some technical detail (e.g. Body should be a JSON object ). However, it is still recommended not to reveal too much about the internal systems which underpin the API, as non-production systems still reflect a lot of the composition of production. Example "errors": [ "code": 12345, "description": "There has been an error - retrying the request will not succeed. Please contact support", "code": 98765, "description": "A more detailed description of the error if appropriate" ], 14 See for a map of HTTP Status Codes 15 See Information_Leakage_and_Improper_Error_Handling Part B Technical v1.0 Page 101 of 126

102 "_links": [ "href": " "rel": "support" ] Part B Technical v1.0 Page 102 of 126

103 3 API Governance TBD 4 Glossary Term AD API ASCII CA CDN DHE DMZ DoS ECDHE HATEOAS HPP HTTP IETF JSON JWA JWE JWK JWS JWT LDAP MAC OWASP Definition Active Directory Application Programming Interface American Standard Code for Information Interchange Certificate Authority Content Delivery Network Diffie-Hellman Ephemeral Demilitarized Zone Denial of Service Elliptic Curve DHE Hypermedia As The Engine Of Application State HTTP Parameter Pollution Hyper Text Transfer Protocol Internet Engineering Task Force JavaScript Object Notation JSON Web Algorithms JSON Web Encryption JSON Web Key JSON Web Signature JSON Web Token Lightweight Directory Access Protocol Message Authentication Code Open Web Application Security Project Part B Technical v1.0 Page 103 of 126

104 PBD RAML REST RFC RO RS SAML SCIM SEO SLA SOAP SPML SQL SSO STS TBC TBD TLS URL URI WSDL XSD WADL XACML XML YAML Primary Business Data Rest API Modelling Language Representative State Transfer Request for Comments (IETF) Resource Owner Resource Server Security Assertion Markup Language System for Cross-domain Identity Management Search Engine Optimization Service Level Agreement Simple Object Access Protocol Service Provisioning Markup Language Structured Query Language Single Sign On Security Token Service To Be Completed To Be Done Transport Layer Security (superseded SSL) Uniform Resource Locator Uniform Resource Identifier Web Service Definition Language XML Schema Definition Web API Description Language extensible Access Control Markup Language extensible Markup Language YAML Ain't Markup Language Part B Technical v1.0 Page 104 of 126

105 5 Further Reading OWASP REST Security OWASP API Security Project OWASP Top Ten Cheat Sheet OWASP Secure Coding Principles NZ Protective Security Using HTTP Methods for RESTful Services Reserved JavaScript Keywords REST API Resource Modelling Government ICT Strategy 2015 OpenAPI Specification HTTP 1.1 Standards RFCs Part B Technical v1.0 Page 105 of 126

106 Appendix A - IETF NFC Relating to OAuth 2.0 RFC number and Title High Level Description RFC 6749 The core OAuth 2.0 RFC defining the authorisation framework. The OAuth 2.0 Authorization Framework RFC 6750 The OAuth 2.0 Authorisation Framework: Bearer Token Usage How to use bearer tokens in HTTP requests to access OAuth 2.0 protected resources. Any party in possession of a bearer token (a "bearer") can use it to get access to the associated resources (without demonstrating possession of a cryptographic key). To prevent misuse, bearer tokens need to be protected from disclosure in storage and in transport. RF7521 Assertion Framework for OAuth 2.0 Client Authentication and Authorisation Grants Common framework for OAuth 2.0 to interact with other identity systems using an assertion and to provide alternative client authentication mechanisms. RFC 7522 Security Assertion Markup Language (SAML) 2.0 Profile for OAuth 2.0 Client Authentication and Authorization Grants The use of a Security Assertion Markup Language (SAML) 2.0 Bearer Assertion as a means for requesting an OAuth 2.0 access token as well as for client authentication. RFC 7523 JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants Use of a JSON Web Token (JWT) Bearer Token as a means for requesting an OAuth 2.0 access token as well as for client authentication. RFC 6819 OAuth 2.0 Threat Model and Security Considerations Security considerations for OAuth beyond those in the OAuth 2.0 specification, based on a comprehensive threat model for the OAuth 2.0 protocol. RFC 7591 OAuth 2.0 Dynamic Client Registration Protocol Mechanisms for dynamically registering OAuth 2.0 clients with authorisation servers. RFC 7592 Methods for the management of OAuth 2.0 dynamic client Part B Technical v1.0 Page 106 of 126

107 OAuth 2.0 Dynamic Client Registration Management Protocol registrations for use cases in which the properties of a registered client may need to be changed during the lifetime of the client. RFC 7662 OAuth 2.0 Token Introspection Method for a protected resource to query an OAuth 2.0 authorisation server to determine the active state of an OAuth 2.0 token and to determine meta-information about this token. Provides authorisation context of the token from the authorisation server to the protected resource. RFC 7519 JSON Web Token (JWT) URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is used as the payload of a JSON Web Signature (JWS) structure or as the plaintext of a JSON Web Encryption (JWE) structure, enabling the claims to be digitally signed or integrity protected with a Message Authentication Code (MAC) and/or encryption. RFC 7800 Proof-of-Possession Key Semantics for JSON Web Tokens (JWTs) How to declare in a JSON Web Token (JWT) that the presenter of the JWT possesses a particular proof-of-possession key and how the recipient can cryptographically confirm proof of possession of the key by the presenter. RFC 7009 OAuth 2.0 Token Revocation Proposes an additional endpoint for OAuth authorisation Servers. This allows clients to notify the authorisation server that a previously obtained refresh or access token is no longer needed, which enables the authorization server to clean up security credentials. RFC 7636 Proof Key for Code Exchange by OAuth Public Clients OAuth 2.0 public clients utilising the Authorisation Code Grant are susceptible to the authorisation code interception attack. This specification describes the attack as well as a technique to mitigate against the threat through the use of Proof Key for Code Exchange. Appendix B - RFCs in Development The following are RFCs which will be pertinent to this standard but are currently under development: Part B Technical v1.0 Page 107 of 126

108 Description High Level view Chain Grant Type for OAuth2 Authentication Method Reference Values A method by which an OAuth protected service, can use a received OAuth token from its client, and in turn act as a client and access another OAuth protected service in a 'chained' profile. Establishing a registry for Authentication methods: Retina Scan Facial Recognition Fingerprints Geolocation information Proof of possession of a hardware key Knowledge based authentication Multi-channel authentication Multi-factor authentication One-time password Personal Identification Number Password based Risk Based SMS confirmation messages Proof of possession of a software key Confirm by telephone User presence test Voice biometrics Windows integration Closing Open Redirectors in OAuth OAuth 2.0 Authorization Server Discovery Metadata OAuth 2.0 JWT Authorization Request OAuth 2.0 Mix-Up Mitigation OAuth 2.0 for Native Apps A Method for Signing HTTP Requests for OAuth OAuth 2.0 Token Exchange Phishing attack on failed redirection from the Authorisation Server. Discovery of endpoints and authorisation server capability. Send request parameters in the form of JWTs rather than encoded in the request URI. To address Malicious Endpoint attacks. This RFC recommends external user-agents like in-app browser tabs as the only secure and usable choice for OAuth, rather than embedded user-agents. A method for offering data origin authentication and integrity protection for HTTP requests. Defines a protocol for a lightweight HTTP and JSON based Security Token Service (STS) covering requesting of tokens from an Authorisation Server. Part B Technical v1.0 Page 108 of 126

109 OAuth 2.0 Message Authentication Code (MAC) Tokens OAuth 2.0 Authorization Server Discovery Metadata OAuth 2.0 Proof-of- Possession (PoP) Security Architecture A proposal to use MAC Tokens in HTTP requests to access OAuth 2.0 protected resources. Discovery of endpoints and authorisation server capability. The proposal of the development of the OAuth 2.0 proof-of-possession security mechanism. Appendix C - Example Interface Specification The following illustrates a sample API Interface Specification. The specification is written in Swagger and visualised here (for readability) via a Confluence plugin however most API Developer Portal products will support importing OpenAPI specifications. Part B Technical v1.0 Page 109 of 126

110 Part B Technical v1.0 Page 110 of 126

111 Part B Technical v1.0 Page 111 of 126

112 Part B Technical v1.0 Page 112 of 126