SFTP Connection How to connect via SFTP & upload Files Version 1.2 October 2017
Table of Content 1. Introduction... 2 2. Technical Pre-Conditions... 2 2.1. Hardware requirements... 2 2.2. Software requirements... 2 3. How to generate and save SSH key pair... 2 4. How to connect to SFTP server... 7 4.1. Using TurboFTP client... 7 4.1.1 Collecting session details... 7 4.1.2 Connecting to the SFTP server... 7 4.1.3 How and where to upload files... 10 4.2. Using WinSCP client... 12 4.2.1 Collecting session details... 12 4.2.2 Connecting to the SFTP server... 12 4.2.3 How and where to upload files... 15 5. File submission guidelines... 16 5.1. General requirements... 16 5.2. Preparing accepted files... 16 6. Support Contacts & Service availability... 18 1
1. Introduction The Regulatory Reporting Hub (RRH) will support the customers with flexible data formats and delivery mechanisms. The different MiFID II services may be employed to combine and transform inbound customer data in a very flexible manner. The Regulatory Reporting Hub will support the formats CSV and XML for submission of data for transaction reporting. The platform will provide a fully automated data feed. For the Transaction Reporting service, we will offer the possibility to securely exchange files with the customer via SFTP protocol in addition to the file upload via GUI. In order to securely exchange files with SFTP RRH server, the users first need to have a valid certificate to login into our portal for administrative and monitoring purposes, and a separate SSH key pair (public/private key). For the time being, the users are asked to generate this key pair themselves and send only the public key to the RRH team regulatory.services@deutscheboerse.com in order to validate and import it into the server. Section 3 describes how such key pair in details. This document Describes how to generate keys in order to connect to a SFTP server Explains step-by-step how to utilize user-friendly SFTP clients to get access to the server Describes how to generate correct inbound files as well as their corresponding archives Covers necessary contact data for technical and functional support with service offering times 2. Technical Pre-Conditions Below are the hardware and software requirements to connect to the Regulatory Reporting Hub server using SFTP protocol. 2.1. Hardware requirements There are no particular hardware requirements to access to the Regulatory Reporting Hub server. The server can be accessed with any computer running one of the below mentioned SFTP clients. 2.2. Software requirements To transfer files to or from a server via SFTP, following are two commonly used clients: WinSCP (version 5.1.4) TurboFTP (version 6.30) In this guide we introduce both clients and show how to use those in order to connect to the server successfully. Any other SFTP client program like FileZilla can be used as well. 3. How to generate and save SSH key pair Please note that we recommend separate SSH key pairs for Simulation and Production due to security reasons. In Windows, PUTTY Key Generator (PuTTYGen) can be used to generate your SSH key pair. Note that the key pairs generated should be different. First, if needed download the PuTTYGen from the PuTTY download page PuttyGen Site and install it on your computer. Second, obtain and prepare to use a text editor such as Notepad++ that does not insert unwanted characters and metadata into a text file. After that follow the steps below: Step 1: Open the PuTTYGen application and select RSA for Type of key to generate and choose the key length for Number of bits in a generated key. The key length must be at least 2048. 2
Figure 1 - Select the type and length of keys Step 2: Click on the Generate button to get the prompt requesting to move the mouse for generating some randomness in keys. Then your keys will be created. Figure 2 - Generate randomness in keys Step 3: Put a suitable comment in the Key comment field so that you will remember what the keys are used for. Also type a passphrase in the Key passphrase field to use when accessing the private key and confirm it in Confirm passphrase field. You can use a key without a passphrase, but this is NOT recommended. This passphrase is designed to encrypt the private key on disk, so you will not be able to use the key without first entering the passphrase. 3
Figure 3 - Enter comment and passphrase Step 4: Click on the Save private key button to keep the private key securely in a local repository. The key should have the extension.ppk and named according the following rule (See Figure 4): Figure 4: Naming convention for private key Example: 529900G3SW56SHYNPR95.ppk 4
Figure 5: Save the public/private key via Save Button Step 5: To save the public key there are two options available which are BOTH supported by the RRH server. They are as follows: Option 1: Click on the Save public key button and select the same location to store it. The public key file however should have the extension.pub so that it will be readable by a regular text editor like Notepad++. The name of the public key must follow the rule (See Figure 6): Figure 6: Naming convention for public keys Example: 529900G3SW56SHYNPR95.pub Option 2: Select and copy the complete text in the boxed labelled Public key for pasting into OpenSSH authorized keys file. To do that, first open Notepad++ and confirm that the End of Line (EOL) format is set to UNIX/OSX Format as Figure 5 shows. This will assure that there are no extraneous characters in the public key file. 5
Figure 7 - Save the public key via Copy & Paste As in Option 1, the public key must have the extension.pub and named following the rule stated above. After generating the key pair, the user has to keep/store the private key in a safe place. The public key however should be (published on key server and) sent to the RRH team together with its corresponding key fingerprint in order to validate it. This Key fingerprint is shown on the PuTTYGen window (Figure 8). In our example the hash value is equal to fe:a0:68:63:84:7c:47:e3:17:f4:21:e4:fa:9f:ec:39. 6
Figure 8: Hash value of the public key 4. How to connect to SFTP server To connect via SFTP there exist specialized and appropriate clients. In this document only two clients are introduced: TurboFTP and WinSCP. 4.1. Using TurboFTP client TurboFTP is an easy-to-use FTP client program with an Explorer-like interface that allows browsing remote directories, downloading or uploading files with drag and dropping operation. Here are the steps needed to follow. 4.1.1 Collecting session details Before connecting to the server, users first need to know the following information provided by the RRH team. Side Address: this is the IP address of the SFTP server (environment). Port: this is the port number of the connection. User ID: this should equal the LEI of your organisation. Service Connectivity Environment IP Port SFTP Internet Simulation 194.36.239.249 Production 194.36.239.247 24 Note that SFTP usually uses port 22 by default, so that you need to adapt it to 24. To use TurboFTP the following further information are also required: Site Name: a user will be asked to provide a name that helps him/her remember the server he/she is connecting to. This site profile will be saved to the FTP Address Book with an entry title of the given name. This is a user s choice (e.g. yoursite-name). Initial Local Directory: it is optional and should be the folder in which the user s test cases are available. 4.1.2 Connecting to the SFTP server In order to connect via TurboFTP, users need to do the following: Step 1: Start TurboFTP and a Login Dialog will appear. 7
Figure 9 - TurboFTP login dialog Step 2: Open menu-item "Connect" and click on "Address Book". The user should see the following screen: Figure 10 - Open Address Book Step 3: Right-Click on "New Site" and put your session details given in Subsection 4.1.1 under General tab as: 8
Figure 11 - Enter session details within General tab Step 4: Switch to the Security tab and check the blue-framed boxes as Figure 12 shows below. After that enter the following input parameters: Password Encryption is based on SHA1 hash algorithm Secure Connection Type should be set to SFTP over SSH2 Port number equals 24 Public Key is the path to the folder in which the public key is stored Private key is the path to the folder in which the private key is stored Password is the passphrase used to protect the private key Figure 12 - Enter session details within Security tab 9
Step 5: Next, click on the Connect button to login. If the login was successful, the users should see output like the following showing a Login successful message, the connection details (the user ID and the IP address of the server) as well as the remote directory which contains tree folders IN, OUT and ARCHIVE (Figure 13). These folders are described as follows: The IN folder: this is the location where the inbound files need to be uploaded by users for processing. The OUT folder: this is the location from where the users can download the response files generated by the RRH system. These response files will be available for download for 20 work days and afterwards archived for 5 years. That means, users needs to download their files from this folder within 25 days. The ARCHIVE folder: this is a container which is composed of the following two subfolders: IN: it contains the inbound files already submitted by users. OUT: this is the location where the response files are stored. Currently, this folder is empty and it does not contain any archived response files. Figure 13 - Successful login via TurboFTP The user is now connected and free to upload and download files via TurboFTP. 4.1.3 How and where to upload files After connecting to the server users can upload files using Drag & Drop function. Before starting with uploading any file, the user should take into account the file requirements (RQ1, RQ2,, RQ9) listed in Section 5. If users first adjust their files according these requirements, then they can upload them via TurboSFTP as follows: First select the local files or directories to be transmitted from the local directory Then drag selected file and drop it onto the remote target folder named IN. 10
Figure 14 - Drag & drop the files into the input folder It takes 5-10 minutes for the files to be uploaded, processed and viewed on the dashboard on RRH portal: https://simu.regulatoryreportinghub.com/transactions/ Once the files have been processed successfully by the RRH system, the corresponding response files (initial response file, subsequent response files, NCA feedback files and trade status files) will be automatically generated in the OUT folder. Currently these response files are provided in two different zipped formats CSV_GZ and XML _ZIP as specified in the specification (CSV+XML upload file spec_val v1.0_clean Tab File Name Conv. ) available on the RRH portal. Figure 15 - Download the response files from the OUT folder 11
4.2. Using WinSCP client WinSCP (Windows Secure Copy) is a free, open source file transfer tool for Windows. As TurboFTP this client allows secure file transfers between the client's local computer and the remote server. 4.2.1 Collecting session details As with TurboFTP the following connection information should be available for users in order to connect with WinSCP: Host name Host name: a user will be requested to provide the IP address of the SFTP server (environment). Port number: this is the port number of the connection. User name: this must be equal the LEI of your organisation. Private key file: a user has to specify the path to his/her private key. Service Connectivity Environment IP Port SFTP Internet Simulation 194.36.239.249 Production 194.36.239.247 24 4.2.2 Connecting to the SFTP server To get access to the server, users need to do the following: Step 1: Start WinSCP and a Login Dialog will appear. Figure 16 - WinSCP login dialog Step 2: Set first the File Protocol as SFTP and then enter the values of login credentials described in Subsection 4.2.1. 12
Figure 17 - Login credentials in WinSCP Step 3: Choose Directories under Environment and click on Browser button to select the path to the local directory, in which the files to be submitted are located. Figure 18 - Set the path to the local repository Step 4: Press Login to connect 13
Figure 19 - Pressing "Login" to connect Step 5: After clicking on Login a dialog screen will be appeared showing the personal data and requesting to enter the corresponding Key passphrase set for the private key (If the private key has been protected). Figure 20 - Enter a passphrase Step 6: If the connection was successful, users will see the content of default remote directory as shown here 14
Figure 21 - Successful login via WinSCP 4.2.3 How and where to upload files After connecting to the server users can upload files (for details about file requirements refer to Section 5) using Drag & Drop function. This works as follows: First select the local file to be transmitted from the local directory Then drag selected file and drop it onto the remote target folder named IN. That means, all input files have to be submitted to this folder to start/trigger file processing process. Figure 22 - Drag & drop the files into IN folder within WinSCP It takes 5-10 minutes for the files to be uploaded, processed and viewed on the dashboard on RRH portal https://simu.regulatoryreportinghub.com/transactions/ Once the files have been processed successfully by the RRH system, the corresponding response files (initial response file, subsequent response files, NCA feedback files and trade status files) will be automatically provided in the OUT folder. Currently these response files are generated in two different zipped formats CSV_GZ and XML_ZIP as specified in the specification (CSV+XML upload file spec_val v1.0_clean Tab File Name Conv. ) available on the RRH portal. 15
Figure 23 - Download response files from OUT subdirectory 5. File submission guidelines Before beginning to upload any files to the RRH server the submitters/users are kindly asked to consider some requirements regarding the file naming and formats. The purpose of these requirements is to provide guidance to users in the preparation of inbound files and ensure that uploaded files meet the technical specifications. Any file not meeting these specifications will be rejected and not processed - requiring corrective action and delay in processing. 5.1. General requirements The following preconditions need to be satisfied before any upload: RQ1: Inbound files need to be generated in csv or xml format s (i.e..xml and.csv ). The capital letters are also accepted (i.e..xml and.csv ). RQ2: Inbound files should be complied with the file naming convention defined in the file specification (CSV+XML upload file spec_val v1.0_clean Tab File Name Conv. ). Any file which is not renamed according this convention will be automatically rejected and not processed by the RRH system. RQ3: Files to be submitted are only accepted in a zipped format; this does not apply to uploads via web application (GUI). RQ4: Within each archive file only one file per zip archive is allowed RQ5: The file name of archive and included inbound file should be identical (upper and lower-case extensions should be taken in to account). RQ6: The maximum file size limit for uploads is up to 400 MB (compressed). Larger file need be broken down into individual files. Each file should possess a unique name. RQ7: The limit the number of files uploaded should not exceed 50 files per day. RQ8: The maximum transactions per file is 3 million on average. RQ9:The minimum data upload frequency is one minute. 5.2. Preparing accepted files Before uploading files to RRH server, it is important to know and to use the right file format/extension for the content. In the following are the steps how to prepare valid CSV or XML files for uploads Step 1: Make sure that the original files having extension.xml or.csv (or in capital letters)) are created in the correct structure specified in the current release (CSV+XML upload file spec_val v1.0_clean Tab File Name Conv. ). In case of XML, the files 16
need first to be validated against current XSD, which is available on RRH portal. Any invalid XML file will be rejected without any further processing. In this step, it is recommended to use an editor like Notepad++ and Sublime Text to ensure the correctness of the file structure. In case of CSV, one of the most common ways in generating and editing CSV file format is using Excel. In doing so, there are a few common issues when importing CSV file into Excel. These issues include for example improper numeric interpretation and blank columns and rows. Numeric entries as dates are often incorrectly interpreted by Excel, and the file contains blank rows and columns that need to be removed. Step 2: Original files (i.e..xml and *.csv ) should be named according to current file naming conventions defined in the file specification (CSV+XML upload file spec_val v1.0_clean Tab File Name Conv. ): Environment _SenderID_Timestamp _RegulationFileType.FileFormat Where Environment: indicate the environments and can take only two values SIM for simulation or PRO for production. In the current release only SIM is allowed. SenderID: is the sender ID which can be a LEI (20 characters) or an allowed ID in case of full delegation. Timestamp : is the sending date time in format YYYYMMDDThhmmsssss. RegulationFileType: consists of 9 characters identifying the regulation type contained in the file. More about allowed regulation combinations are given in the sheet File Name Conv. within the file specification cited above. FileFormat: denotes the File Format. It should be CSV or XML or csv or xml. In case of CSV the files created need to be in UNIX format. Notepad++ allows to convert files from Windows to Unix by clicking on Edit, selecting the EOL Conversion, and from the options that come up select UNIX Format. Examples: SIM_529900G3SW56SHYNPR95_20170717T135626001_MIXXXXXXT.CSV SIM_529900G3SW56SHYNPR95_20170717T135626001_MIXXXXXXT.csv SIM_529900G3SW56SHYNPR95_20170704T135626001_MIXXXXXXT.XML SIM _529900G3SW56SHYNPR95_20170713T135626001_EMXXXXXXPX.XML SIM _529900G3SW56SHYNPR95_20170713T135626001_EMXXXXXXPX.xml Step 3: If the structure and the name of an original file are correct (Step 1 and 2), it needs to be compressed using one of the compression utilities like 7-Zip and PeaZip to obtain an archive of the extension.zip. That means, the name of the resulting archive should look like: SIM_529900G3SW56SHYNPR95_20170717T135626001_MIXXXXXXT.zip Step 4: The extension of the resulting archive should be changed from zip to XXX_GZ or XXX_ZIP or XXX_gz or XXX_zip where XXX to be replaced by XML, CSV, *.xml and *.csv ). (depending on the format of the original file). Examples: SIM_529900G3SW56SHYNPR95_20170717T135626001_MIXXXXXXT.CSV_GZ SIM_529900G3SW56SHYNPR95_20170704T135626001_MIXXXXXXT.XML_ZIP SIM 529900G3SW56SHYNPR95_20170713T135626001_EMXXXXXXP.XML_GZ SIM_529900G3SW56SHYNPR95_20170717T135626001_MIXXXXXXT.csv_gz SIM_529900G3SW56SHYNPR95_20170704T135626001_MIXXXXXXT.xml_zip 17
6. Support Contacts & Service availability For technical queries related to the application and platform features please contact our customer technical support team. English and German speaking support is available from 08:00 to 18:00 CET from Monday to Friday excl. German bank holidays. We will use all reasonable efforts to respond within 24 hours after being contacted. Hotline Germany: +49-(0) 69-2 11-1 77 55 Hotline UK: +44-(0)20-7862-7755 Email: cts@deutsche-boerse.com 18