HTTPS File Transfer. Specification

Transcription

1 HTTPS File Transfer Specification Version Apr-2017

2 Date Version Description 30-Aug Original Version 30-Jun Added FAQ 29-Jun ilink administration added 1-Sep Updated for new layout 4-Apr Revised TLS version to 1.2 minimum Page 2

3 Table of Contents 1 Introduction Quick Start Integration Options ilink connectivity setup ilink URLs Setup Connectivity Form Using the Command Line Scripts Downloading curl Configuration Uploading Files Example File Names Handling Errors Downloading Files Example Handling Errors Developing Custom HTTPS Connectivity Recommendations Handling Errors Installing the Client Certificate Protecting Your Certificate Upgrading Your Certificate Appendix: File Transfer Protocol Uploading Files HTTP Request HTTP Response Handling Errors Listing Files for Download HTTP Request Page 3

4 7.2.2 HTTP Response Handling Errors Downloading a File HTTP Request HTTP Response Handling Errors Deleting a File HTTP Request HTTP Response Handling Errors Appendix: Westpac HTTPS addresses Test Environment Production Environment Appendix D FAQ Page 4

5 1 Introduction This document defines Westpac s WIBS HTTPS file transfer protocol. The HTTPS file transfer protocol allows partners to transfer files securely and reliably over the internet. It provides a standard mechanism for transport using existing infrastructure (such as web servers, proxy servers and firewalls). Communication uses standard TCP/IP ports to prevent the need for network firewall changes. SSL is used to provide encryption of data between partners, and authorisation assures the identity of each partner. The intended audience of this document is: Server administrators who wish to use the provided command line scripts, and Software developers who wish to implement this messaging protocol in their software. Knowledge of HTTP and SSL is assumed. Page 5

6 2 Quick Start Customer task Westpac task 1. Qvalent implementation consultant creates an ilink test account for the customer s technical contact. 2. Customer contact completes ilink connectivity form in test ilink. 3. Customer develops HTTPS connectivity solution 4. Qvalent implementation consultant configures file routing. 5. Customer undertakes testing in the test environment. 6. Once customer is satisfied that testing is complete a sign off is required to progress into production. 7. Qvalent implementation consultant creates an ilink production account for the customer s technical contact. 8. Customer contact completes ilink connectivity form in production ilink. 9. Qvalent implementation consultant configures file routing. 10. Customer tests the HTTPS connection in the live environment. 11. Once this testing is successful customers can perform low value live testing of the other Westpac products that are being implemented. Page 6

7 3 Integration Options The HTTPS file transfer service is provided to allow your system to easily upload files to Westpac and download files from Westpac. You have two integration options: 1. Use the provided command-line scripts. These scripts use the curl commandline program and standard in and out streams to transfer files. Minimal configuration work is required. 2. Write code to generate HTTPS requests sent to Westpac and interpret the responses you receive. A moderate amount of development work is required, and you will need previous expertise with HTTPS. These options are further detailed in the following sections. Page 7

8 4 ilink connectivity setup In the early stages of your Westpac project you will be asked to provide the contact details of the IT person who will be responsible for setting up your HTTPS connection. Once these details are received you will be provided with an ilink login to enter your IP addresses. The ilink connectivity process has the following steps 1. The Qvalent implementation consultant will provide the user s technical contact with a login to the ilink test instance. 2. Fill in the setup connectivity form and submit 3. Notify your Qvalent and Westpac implementation managers that you have completed the connectivity form 4. The Qvalent implementation consultant will configure the file routing and notify all stakeholders when this is done 5. User to send and receive test files to confirm the HTTPS connection 6. Once the Qvalent implementation consultant has received confirmation that all relevant testing has been completed steps 1 5 will need to be repeated in the production environment. 4.1 ilink URLs Test Production Setup Connectivity Form To setup your connectivity, click the Connectivity menu option at the top of the screen, then press the Setup Connectivity button. The Setup Connectivity will be displayed where you can enter the following details: PGP Key If you have elected to PGP encrypt files sent to Westpac, upload your PGP key here. Otherwise, this field will not be shown on the screen. IP addresses The WIBS solution has a white list of IP addresses accepted for each user. Users need to provide the IP address or addresses that their incoming requests will be coming from, this is the external IP address taking into account any proxy servers or other externally facing network infrastructure. This can be found by logging on to ilink on your server and taking the browser address shown in the IP addresses section of the connectivity form. After you press Save, you will see the WIBS server details on the Setup connectivity page. Westpac s key this is the public key that you will need to use to decrypt the files you receive from WIBS. This will only be displayed if encryption is enabled. Your key - You can use these fields during testing to confirm which keys you have loaded into ilink. This will only be displayed if encryption is enabled. Your credentials - This section contains the details of the client certificate that you will use to authenticate with the WIBS server. IP Addresses lists the IP addresses allowed to access this connectivity Page 8

9 5 Using the Command Line Scripts Command line scripts have been provided for Unix and Windows that demonstrate the use of curl to upload and download files. The scripts have been designed to exit with an error code if the file could not be successfully uploaded or download. Error details will be written to the standard error stream. 5.1 Downloading curl You will need to download the curl client for your platform from Be sure to download the correct distribution for your server s operating system. You must download a version that includes SSL support. If you are using Windows, we recommend that you download the binary distribution listed under the heading Win32 - Generic labelled Win /XP, binary and SSL and maintained by Günter Knauf. This document does not provide a direct link because you should always download the latest version. 5.2 Configuration You need to edit the configuration script before you can use the upload or download scripts. The configuration script is named config.cmd on Windows and config.sh on Unix. Open the file in a text editor and make the following changes: 1. Set the CERT_FILE variable to the file path of your.pem certificate file, e.g. c:\filetransfer\certificate.pem 2. Set the CERT_PASSWORD variable to the password for your.pem certificate file. 3. If you require a proxy to connect to the internet: a. Uncomment the first PROXY_OPTIONS line and change proxy_address to the host name of your proxy server, and proxy_port to the port number your proxy server listens on. b. If your proxy server requires authentication, uncomment the second PROXY_OPTIONS line and change username to your proxy username and password to your proxy password. 4. If you are uploading and downloading files in the test environment, leave the BASE_URL variable set to If you are uploading and downloading files in the production environment, set the BASE_URL variable to 5. Set the CURL_EXE variable to the file path of the curl executable file, e.g. C:\curl \curl.exe 5.3 Uploading Files The upload script is named uploadfile.cmd on Windows and uploadfile.sh on Unix. It takes one argument, which is the file name to send to Westpac. The file contents are read from standard in. Page 9

10 5.3.1 Example The following example shows how to send a file from the file system: Windows: Unix: uploadfile.cmd mts _01.txt < c:\mts txt uploadfile.sh mts _01.txt < c:\mts txt Note that this allows the file name sent to Westpac to be different from the file name on disk. Important Security Note: If the file contains credit card numbers, PCI DSS regulations require that you do not store the file unencrypted on disk. Instead, you must decrypt the file in memory and stream that data to the input stream of the upload script File Names The file name sent to Westpac must be unique. If you send a different file with the same name as a previous file, you will receive a HTTP 400 Bad Request error response. To ensure uniqueness of file names, include the file type, date and a daily sequence number in the file name Handling Errors If you receive a network error while transmitting your file, you can simply run the same upload command again (re-using the same file name). This will never result in a duplicate file because Westpac will silently ignore duplicate files with the same name. If you receive any other error, it is likely a configuration issue you need to investigate based on the error message provided. 5.4 Downloading Files The download files script is named downloadfiles.cmd on Windows and downloadfiles.sh on Unix. It downloads all your available files and removes them from the download list (preventing duplicate downloads). By default, the file contents are echoed to standard out. You will need to modify this script to save the files to an appropriate location and trigger any required file processing. Important Security Note: If the file contains credit card numbers, PCI DSS regulations require that you do not store the file unencrypted on disk. Instead, you must read the standard out stream from the script and encrypt the file in memory before saving it to disk Example To download all available files, run the following command: Windows: Unix: downloadfiles.cmd downloadfiles.sh Handling Errors To download the available files, the script performs the following operations: 1. Get a list of all the available files. 2. For each file in that list: a. Download the file. Page 10

11 b. If the file has not already been processed, run your custom file processing. c. Delete the file (removing it from the available file list). Operations 1, 2a and 2c can fail due to network errors. In that event, simply run the download script again and the file will be downloaded again. The file is not deleted until it is downloaded and processed, which allows you to retry the download operation in the event of a failure. If you receive an error not related to network connectivity or your processing, it is likely a configuration issue you need to investigate based on the error message provided. Page 11

12 6 Developing Custom HTTPS Connectivity This section gives an overview of developing your own connectivity code to use the HTTPS file transfer service. It details the requirements of your system and gives instruction on how to integrate the file transfer protocol into your system. This section does not contain details of the file transfer protocol itself. That information is in section 7. Note that this section only gives a general overview of what needs to be done. Specific details are not provided since they vary depending on the actual technology you use. For this reason, Westpac can provide only limited assistance to customers using custom integration. We recommend that you only undertake a custom integration if you have previous experience with HTTPS and client certificates. To use the Westpac HTTPS file transfer service, you will need a HTTP client that supports the following standards: HTTP TLS 1.2 or higher Recommendations Log detailed information about each file you upload and download. This information will help you diagnose any potential problems you may encounter later. You should not log your certificate details or any full credit card numbers contained in the file. Protect your certificate file. If necessary, restrict the permissions on this file so that only your application can access the certificate. 6.2 Handling Errors If you encounter a network error or HTTP 500 response code, your system should automatically retry the request. You should wait at least 5 minutes before retrying the request, and you should only retry for a maximum of 4 hours before you investigate the cause of the issue manually. 6.3 Installing the Client Certificate You must use an SSL client certificate to communicate with the Westpac server. This certificate will be provided to you by Westpac. You may not use a certificate from another source (e.g. Verisign). Your certificate is generally provided in PKCS12 (or PFX) format. This file format requires a password for the certificate. You will be provided with this certificate password. Do not confuse this password with the password you use to login to any Westpac websites. Your certificate password will always start with a C. A few HTTP libraries require the client certificate file to be in PEM format. If this is the case, please ensure that you get the certificate in the correct format. Once you have obtained your certificate in the correct format, you must reference it from your HTTP library. The details of this will vary between HTTP libraries, so consult the documentation for your HTTP library. Page 12

13 Be sure to document the procedure to install the certificate, since the certificate must be replaced every 2 years. 6.4 Protecting Your Certificate Your certificate file is used to authorise your requests to the Westpac server. Do not give your certificate file to anyone. Never your certificate in clear text. Always use a secure method to copy your certificate file to your server. 6.5 Upgrading Your Certificate Your certificate is valid for a maximum of 2 years. When your certificate is near expiry, you will be contacted by Westpac with details on how to get a new certificate file. The steps to install your new certificate will vary depending on your technology, so it is important to document this procedure for later use. Page 13

14 7 Appendix: File Transfer Protocol This section documents the HTTPS file transfer protocol for uploading and downloading files. Authentication is based on your client certificate and the IP address that your request comes from. 7.1 Uploading Files HTTP Request To upload a file, send a HTTPS POST to the following URL: You must choose a unique file name for each file you upload. To ensure uniqueness of file names, include the file type, date and a daily sequence number in the file name. An example upload HTTP request is shown below. Note that represents an ASCII carriage-return line-feed sequence. POST /services/messaging/rest/uploadedfiles/de _01.txt HTTP/1.1 Content-Type: application/octet-stream Host: ws.support.qvalent.com:443 Content-Length: 19 This is a test file Note that no special HTTP headers are required. You simply include the file name in the URL and the file contents in the HTTP request body HTTP Response The table below lists the possible HTTP status codes that can be returned. HTTP Status Code Meaning 202 Accepted The file has been accepted for processing. The file can still be rejected at a later time if it is malformed, but the file transfer has been successful. 400 Bad Request You have reused an old file name for a new file. You must use a unique file name for each file you upload. 403 Forbidden Either: You have not specified your client certificate correctly. Check your configuration. Or your request has come from an IP address that has not been registered in Westpac s system. 404 Not Found The file name you have specified is not acceptable. The file name may contain letters, numbers underscores, dashes and full stops. Retry your upload request with a different file name. 500 Internal Server Error 503 Service Unavailable The Westpac server has encountered an internal error. Retry the same upload request again later. You have tried to upload too many files at the same time. You should upload only one file at a time. Page 14

15 An example upload HTTP response is shown below. Note that represents an ASCII carriage-return line-feed sequence. HTTP/ Accepted Content-Length: 0 Date: Mon, 30 Aug :24:20 GMT Handling Errors If you receive a network error or HTTP 500 status when transmitting your file, you can simply run the same upload command again (re-using the same file name). This will never result in a duplicate file because Westpac will silently ignore duplicate files with the same name. If you receive any other error, it is likely a configuration issue you need to investigate based on the error message provided. 7.2 Listing Files for Download HTTP Request To get the list of files for download, send a HTTP GET to the following URL: You can choose the format of the file list using the HTTP Accept header as shown in the following table: HTTP Accept Header text/plain text/xml or application/xml Response Format The list of files is in plain text with each file name on a different line. If no files are available for download, the HTTP status code will be 204. If files are available for download, the HTTP status code will be 200. The list of files is in XML format as shown in the example in the next section. An example HTTP request is shown below where the file list is requested in plain text format: GET /services/messaging/rest/filesfordownload/ HTTP/1.1 Accept: text/plain Host: ws.support.qvalent.com:443 An example HTTP request is shown below where the file list is requested in XML format: GET /services/messaging/rest/filesfordownload/ HTTP/1.1 Accept: application/xml Host: ws.support.qvalent.com: HTTP Response The table below lists the possible HTTP status codes that can be returned. HTTP Status Code Meaning 200 OK The list of files for download is returned in the response body. 204 No Content No files are available for download (plain text only). Page 15

16 HTTP Status Code Meaning 403 Forbidden Either: You have not specified your client certificate correctly. Check your configuration. Or your request has come from an IP address that has not been registered in our system. 404 Not Found The URL in the request is incorrect. It should be /services/messaging/rest/filesfordownload/ 500 Internal Server Error The Westpac server has encountered an internal error. Retry the same request again later. An example plain text HTTP response is shown below. Note that represents an ASCII carriage-return line-feed sequence. HTTP/ OK Expires: 0 Pragma: no-cache Cache-Control: no-store, no-cache, must-revalidate, proxy-revalidate Content-Type: text/plain Date: Mon, 30 Aug :24:20 GMT Transfer-Encoding: chunked 16 File1.txt File2.txt 0 An example XML HTTP response is shown below. Note that represents an ASCII carriage-return line-feed sequence. HTTP/ OK Expires: 0 Pragma: no-cache Cache-Control: no-store, no-cache, must-revalidate, proxy-revalidate Content-Type: text/xml Date: Mon, 30 Aug :24:20 GMT Transfer-Encoding: chunked e8 <?xml version="1.0" encoding="utf-8"?> <downloadfilelist xmlns=" <file> <filename>file1.txt</filename> <filename>file2.txt</filename> </file> </downloadfilelist> Handling Errors Requesting the list of files for download does not change the server state in any way, so you can safely request the list as many times as you need to. If you receive a network error or HTTP 500 status when requesting the list, you can simply repeat the same request again. Page 16

17 If you receive any other error, it is likely a configuration issue you need to investigate based on the error message provided. 7.3 Downloading a File HTTP Request To download a file, select the file name from the list of files available for download and send a HTTP GET to the following URL: An example download HTTP request is shown below: GET /services/messaging/rest/filesfordownload/file1.txt HTTP/1.1 Host: ws.support.qvalent.com: HTTP Response The table below lists the possible HTTP status codes that can be returned. HTTP Status Code Meaning 200 OK The file contents are returned in the response body. 403 Forbidden Either: You have not specified your client certificate correctly. Check your configuration. Or your request has come from an IP address that has not been registered in our system. 404 Not Found The file name you specified no longer exists in the list of files available for download. 500 Internal Server Error The Westpac server has encountered an internal error. Retry the same download request again later. An example plain text HTTP response is shown below. Note that represents an ASCII carriage-return line-feed sequence. HTTP/ OK Expires: 0 Pragma: no-cache Cache-Control: no-store, no-cache, must-revalidate, proxy-revalidate Content-Type: application/octet-stream Date: Mon, 30 Aug :24:20 GMT Transfer-Encoding: chunked 13 This is a test file Handling Errors Downloading a file does not change the server state in any way, so you can safely download the file as many times as you need to. If you receive a network error or HTTP 500 status when downloading the file, you can simply repeat the same download request again. Page 17

18 If you receive any other error, it is likely a configuration issue you need to investigate based on the error message provided. 7.4 Deleting a File HTTP Request To delete a file from the list of files available for download, send a HTTP DELETE to the following URL: An example download HTTP request is shown below: DELETE /services/messaging/rest/filesfordownload/file1.txt HTTP/1.1 Host: ws.support.qvalent.com: HTTP Response The table below lists the possible HTTP status codes that can be returned. HTTP Status Code Meaning 204 No Content The file was deleted from the list successfully. 403 Forbidden Either: You have not specified your client certificate correctly. Check your configuration. Or your request has come from an IP address that has not been registered in our system. 404 Not Found The file name you specified is not valid. Ensure that the file name you are trying to delete was copied exactly from the list of files for download. 500 Internal Server Error The Westpac server has encountered an internal error. Retry the same delete request again later. An example delete HTTP response is shown below. Note that represents an ASCII carriage-return line-feed sequence. HTTP/ No Content Date: Mon, 30 Aug :24:20 GMT Transfer-Encoding: chunked Handling Errors If you receive a network error or HTTP 500 status, you can simply repeat the same delete request again. If the file has already been deleted, the server will ignore the delete request and return a HTTP 204 status. If you receive any other error, it is likely a configuration issue you need to investigate based on the error message provided. Page 18

19 8 Appendix: Westpac HTTPS addresses 8.1 Test Environment Upload File Files for Download List Download/Delete File Production Environment Upload File Files for Download List Download/Delete File Page 19

20 Appendix D FAQ Q) I execute the downloadfile.cmd and receive the following error: Curl: (22) The requested URL returned error: 502 A) The file you are trying to fetch may have a space in its name. Contact Westpac and ask them to check that there is no space in the filenames being fetched. Page 20