CLI USER MANUAL. Made by Neit Consulting, s.r.o. Date of last update: Version: 2.0

Transcription

1 CLI USER MANUAL Made by Neit Cnsulting, s.r.. Date f last update: Versin: 2.0

2 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic Summary A. Preface... 3 A.1. Pre-installatin requirements... 3 A.2. Installatin... 3 B. CLI usage... 5 C. Return cdes Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 2

3 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic A. Preface Manual serves as a guideline fr wrking with the client CLI that enables prcess data fr archiving. The CLI client is cntrlled frm the cmmand line t allw its batch and autmated prcessing and must be installed n each cmputer frm which the data will be archived. We designate such a cmputer as a Client Statin. The CLI client is written in Java 8 and is fully cmpatible with massively used systems. The CLI client can back up any files and flders, cnnect t the database, and exprt the data t a CSV file that is then prcessed as a regular file using a user-defined query. Currently supprted databases are Oracle, MSSQL, MySQL, PstgreSQL. Yu can use recursive prcessing f all subflders when prcessing flders. In data prcessing, CLI client data can als be cmpressed. Fr the CLI client t prcess the data, it needs a cnfiguratin file called the "FTD prperty file", with the FTD abbreviatin "File type definitin", the Czech translatin is "Definice Typu Subru". This file cntains all settings fr the crrect prcessing f archived data. The CLI client can dwnlad and validate these files frm the Neit Metadata server. In the case f a malfunctin, the CLI client can send an errr mail t a predetermined , such as an administratr. Each client statin must be registered with Neit Metadata Server, which is implemented using IP Addresses. Cmmunicatin between the CLI client and the Neit Metadata server is secured by SSL encryptin with RSA certificates. A.1. Pre-installatin requirements Oracle Java 1.8 a higher, JRE versin als wrks Optimal allcated memry RAM: 256MB (default) A.2. Installatin - Uplad the CLI client flder t the client statin. - There are certificates fr encryptin in the CLI flder. There are default, which can be replaced by the custm yu create. A brief prcedure fr creating custm certificates fr CLI: Private KeyStre (cakeystre.jks) fr keypair backup, nt nly fr CLI, but als fr Master a Slave Servers. Own Keystre (cskeystre.jks) Own TrustStre (cstruststre.jks) must cntain certificate signed by certificatin authrity. Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 3

4 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic Generate self-signed RSA keypair (client_priv_publ_self-signed.p12) t private KeyStre. Afterwards exprt keypair fr anther imprt Imprt keypair t KeyStre (cskeystre.jks) Exprt X.509 public certificate (client_publ_self-signed.cer) frm keypair -> then imprt t TrustStre (cstruststre.jks) While creating custm certificates, it is necessary t change cnfiguratin in file tmcatusers.xml. - Cntent f cnfiguratin file (cnfig.prperties) is used fr client cnfiguratin. Fr example, this cnfiguratin file includes default settings fr maximum uplad times and attempts, which are recmmended nt t change if yur needs are nt different. Fr prper CLI functinality, the fllwing items must be set: mtdurl Neit Metadata server address dwnladtimeut defines timeut and waiting time fr server respnse, default value = 7200 upladcnfirmatintimeut defines waiting time t cnfirm successful uplad t Clud, default value = 7200 dwnladwaittime waiting time fr server a message abut dwnlad success, default value = 20 upladwaittime waiting time fr a message frm server abut uplad success, default value = 20 maxdwnladattempts maximal number f attempts t dwnlad, default value = 3 maxupladattempts maximal number f attempts t uplad, default value = 3 keypass passwrd fr key decryptin ladkeystrefrmresurces, ladtruststrefrmresurces If true, this is a cnfirmatin that the CLI shuld retrieve certificates frm the internal resurce, and the items listed belw will remain blank. If false is set, items must be set. These are external paths t certificates and passwrds t them. keystrepath, truststrepath path t keystre, truststre, if ladfrmresurces is false keystrepass, truststrepass passwrd t keystre, truststre - The cnfiguratin file "sql.prperties" is used t cnfigure t wrk with databases and need t fill it nly if yu want t prcess data directly frm the database. In this case, yu need t fill in the type and address f the database, the prt n which it is running, the database name, the user, the passwrd, and the end f the SQL query that will be used t exprt the data frm the Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 4

5 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic database. Yu can use system variables t parameterize the SQL query. E.g.: - Set sys_ftd_id=6 - db.sqlquery = SELECT * FROM filecs INNER JOIN filetypedef ON filecs.ftd_id = %sys_ftd_id%cli usage As has been said, the IP address f the client statin must be registered and enabled n the Neit Metadata server. The registratin request can be dne by simply executing the cmmand "java -jar cs.jar -g lg". This peratin fails, but yu tell the server that they are trying t cnnect t an address they d nt knw and the server adds them between the IP addresses pending apprval. Once yur IP address is apprved by yur system administratr, yu can start wrking nrmally. In practice, it may lk like this: After adding yur address t the server, it is recmmended that yu run help, which will make it easy fr yu t wrk with the client at first. Fr each parameter, examples are given fr its use. Cmmand java -jar cs.jar -h -g / --getdf: Dwnlad the apprpriate definitin file type by its name (write withut extensin.cfg!). -a / --alldf: Dwnlad all definitin file types frm the server. -s / --save: Defines the final path where the files r file are t be stred. -c / --defcfg: The path t the definitin file when uplading t the server. -u / --upf: File r flder fr uplad t server. -r / --recurs: Parameter fr uplad, recursive search f flder tree structure. -z / --zip: The uplad parameter adds the files t the flder and skips it. -q / --sqlcfg: The path t the SQL cnfiguratin file. -n / --skipv: It skips checking t uplad the file t the strage. -t / --transc: Overrides the cmpressin settings during transfer. -h / --help: View help. Examples: Dwnlad all file type definitins frm server: java -jar cs.jar -a java -jar cs.jar -a -s D:\ Dwnlad the apprpriate definitin file type by its name: Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 5

6 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic java -jar cs.jar -g lg java -jar cs.jar -g lg -s D:\ Verify that the definitin file is crrect: java -jar cs.jar -c lg.cfg java -jar cs.jar -c /etc/cs/lg.cfg Uplad files r flders java -jar cs.jar -c lg.cfg -u access.lg java -jar cs.jar -c lg.cfg -u access\lgs\ java -jar cs.jar -c D:\lg.cfg -u D:\access.lg Uplad files r flders in.zip java -jar cs.jar -c lg.cfg -z -u access\ java -jar cs.jar -c D:\lg.cfg -z -u D:\access\ Uplad files frm flder using recursive brwse java -jar cs.jar -c lg.cfg -r -u access\ java -jar cs.jar -c D:\lg.cfg -r -u D:\access\ Uplad files frm flder using recursive brwse in.zip archive java -jar cs.jar -c lg.cfg -z -r -u access\ java -jar cs.jar -c D:\lg.cfg -z -r -u D:\access\ Exprt SQL data t CSV file java -jar cs.jar -q sql.prperties -s data.csv java -jar cs.jar -q D:\sql.prperties -s D:\data.csv Exprt SQL data t CSV file and it s subsequent uplad n server By using parameter -u and path t specified file, file is nt remved frm disk after uplad. java -jar cs.jar -q sql.prperties -c lg.cfg java -jar cs.jar -q D:\sql.prperties -c D:\lg.cfg -u D:\data.csv Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 6

7 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic File type definitin File type definitin (FTD) is stred n Neit metadata server and describes each ne grup f data defined by FTD Administratr. The FTD cntains a name, a list f extensins that can be archived by the FTD, the indexing f archived data by their extensins, the authrized IP address f the client statins fr the given FTD, the strage lcatin t be archived, the list f TAGs allwed fr the given FTD, and s n. Fr CLI client purpses, it is nly necessary t remember the name f the FTD by which we btain the "FTD prperty file" frm the Neit Metadata server. Which, as already mentined, is essential fr data prcessing. On the fllwing lines, yu will find an example f ne FTD prperty file named CSVFile.cfg. There are examples f tw tag definitins in this FTD prperty file. Tag values can be changed. The standard value fr a tag can als include a system variable, in the frm: %variable_name%. The value f the system variable s defined is added when the CLI client is started. It is therefre pssible t run the CLI client repeatedly in sequence, always changing the system variables as needed befre starting. FTD administratr can make sme tag value prescribe a regular expressin that must cmply with the value yu entered. Fr example, if FTD administratr defines a regular expressin that requires 4 digits, yu cannt enter anything else in that tag. #DO NOT CHANGE! - D nt change, it cntains the name f ftd and the extensin frmat f the requested file t be upladed t the server type=csvfile extensins=csv anyextensinallwed=false File type definitin name List f allwed extensins fr a given FTD A switch that allws yu t als uplad file with nt specified extensin in FTD. #FILE TYPE DATA SETTINGS - Define tags where individual tags representing filename filters can be edited, and the file can match data types, such as Fr text, date, r IP address. Lines beginning with "#" are infrmative nly. Example 1: #Required=true #Regex=[a-z].* The switch determines whether TAG is mandatry r nt. (True / false) Regular expressin that must fill in the value yu fill in. FTD Administratr shuld prvide yu with mre detailed infrmatin n the permitted values. Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 7

8 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic #Descriptin=tag descriptin It is used t describe the tag. There may be infrmatin frm the FTD Administratr fr CLI users. Fr example, the allwed regular expressin values. TagDefinitinIdentifier=standard value Tag identifier, ie its name + its default value. If after = is nthing, the tag des nt have a standard value. Example 2: #Required=false #Regex=.* #Descriptin=hidden system tag NOTDESTROY= Tag is ptinal. Any number f characters crrespnds t it. Tag descriptin is hidden system tag. Tag name is NOTDESTROY and has n standard value. File type definitin dwnlad Dwnlading a FTD prperty file is a basic requirement fr further file prcessing. By default, it is dwnladed t the flder frm which the CLI client is running. Yu can use the "-s..." parameter t define a different path fr the CLI Client t be stred. The FTD prperty file can be dwnladed in tw ways. It is dwnlading nly ne specific prperty file by name r all available prperty files fr the client statin frm which the CLI client is running and save it t the "Templates" flder. File uplad T uplad the file successfully, use -u parameter, with a cmbinatin f the crrect FTD. The -u parameter can be further cmbined with the -z parameters fr archiving and -r fr the recursive behavir f the tree structure file. There are several cmbinatins f these parameters that are described here: Single file uplad uplading single file, e.g. in csv frmat Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 8

9 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic C:\cludstrage\ut\cs_jar>java -jar cs.jar -c ftd_templates\csvfile.cfg -u C:\csv_test\utput_10K_1.csv Flder uplad The client scans the flder and uplads all the files it finds. Hwever, it des nt search fr flders. C:\cludstrage\ut\cs_jar>java -jar cs.jar -c ftd_templates\csvfile.cfg -u C:\csv_test Uplad flder recursively The client searches the flder and all f its subflders. The fund files are then upladed t the server. C:\cludstrage\ut\cs_jar>java -jar cs.jar -c ftd_templates\csvfile.cfg -r -u C:\csv_test Uplad single file in archive Uplading the archive t the server. The client cmpresses the selected file and then sends it t the server. After uplading successfully, the temprary archive is deleted n the disk. C:\cludstrage\ut\cs_jar>java -jar cs.jar -c ftd_templates\typezip.cfg -z -u C:\csv_test\utput_10K_1.csv Uplad flder in archive The client searches fr that flder and cmpresses all the files int the archive, which it then uplads t the server. After a successful uplad, the temprary archive will be deleted frm the disk. In this flder, the client des nt underg subflders. C:\cludstrage\ut\cs_jar>java -jar cs.jar -c ftd_templates\typezip.cfg -z -u C:\csv_test Uplad flder in archive recursively The client searches the flder and all its subflders. All files will be cmpressed int ne archive, which will then be upladed t the server. After uplading successfully, the temprary archive is deleted frm the disk. C:\cludstrage\ut\cs_jar>java -jar cs.jar -c ftd_templates\typezip.cfg -z -r -u C:\csv_test File type definitin validity verificatin Verifying FTD befre using it will guarantee its crrectness. Cntrl is achieved using parameter -c help nameftd.cfg. Exprt SQL data t CSV file The client als allws yu t exprt SQL data t a CSV file. Yu can als exprt directly t the server. T d this, yu must have attached database cnfiguratin infrmatin. Only -q and -s are used t select just exprt and save. After -q the path t the cnfiguratin file fr the database is written and the name f the file t exprt the Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 9

10 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic csv file is written t -s. If yu d nt specify a path r a.csv name fr the -s parameter, the name f the saved file is generated by the database name, day and time, and then saved t the current flder. C:\cludstrage\ut\cs_jar>java -jar cs.jar -q sql.prperties -s data.csv If the file is being exprted frm the database and then upladed t the server, the -s parameter fr the lcatin is replaced by -c with the cnfig fr FTD. The file will be deleted frm the disk after it has been upladed t the server. Basically, the file is upladed t the server in.csv frmat. By adding the -z parameter, the file will be cmpressed and sent t the server in this frm. C:\cludstrage\ut\cs_jar>java -jar cs.jar -q sql.prperties -c ftd_templates\csvfile.cfg C:\cludstrage\ut\cs_jar>java -jar cs.jar -q sql.prperties -c ftd_templates\csvfile.cfg -z Using parameter -u and path t specified file, the file is nt remved frm disk after uplad. C:\cludstrage\ut\cs_jar>java -jar cs.jar -q sql.prperties -c ftd_templates\csvfile.cfg -u data.csv Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 10

11 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic B. Return cdes Pssible return cdes when wrking with a CLI client. The CLI client returns nly the numeric cdes represented by the lnger descriptin. Fr mre infrmatin, check the CLI client lg. The fllwing return cdes are generally described in the table belw. 0 Success The requested peratin was successful. 1 Unspecified errr frm Exceptin Unexpected Applicatin Errr. Cnsider submitting feedback. 2 Cnfiguratin missing r empty errr The cnfiguratin file is empty r missing. Check CLI Client Cnfiguratin. 3 Params parse errr Failed t lad specified parameters. Check the entered parameters. 4 Unspecified params entered Unrecgnizable parameters have been entered. Check the entered parameters. 5 Invalid cmbinatin f params entered Incmpatible cmbinatins f parameters were entered. Crrect the specified parameters. 6 RestClient initializatin errr Client culd nt lad SSL cnfiguratin. Check CLI Client Cnfiguratin. 7 Cnfiguratin Master Server URL invalid r empty Unable t cnnect t Master Server URL. Check the client's CLI cnfiguratin t see if the URL is incrrect r missing. 8 KeyStre r TrustStre path nt set There are n paths fr individual SSL certificates. Set paths t SSL certificates in client CLI cnfiguratin. 9 Cnfiguratin Exceptin Errr in cnfiguratin file. Check CLI Client Cnfiguratin. 10 Definitin cnfig file required params missing r invalid Mandatry parameters are nt set in the cnfiguratin file. Check CLI Client Cnfiguratin. 11 MD5 validatin errr File uplad failed. Repeat the actin. 12 Received invalid data frm server Errr receiving data. Cnsider submitting feedback. 13 File extensin mismatch with File data extensin The file cannt be sent because its extensin is nt allwed n the server. Enable the extensin n the server r use anther FTD. Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 11

12 Neit Cnsulting s.r.. Washingtnva 1624/5, Praha 1, Czech Republic 14 Maximum attempts reached fr jb The maximum number f attempts t transfer data has been exceeded. Check the errr messages in the lg. 15 Errr server respnse The actin did nt happen as expected. Check the errr messages in the lg, and cnsider submitting feedback. 16 Maximum waiting time reached fr jb Check the errr messages in the lg. Neit Cnsulting s.r.., Washingtnva 1624/5, Praha 1 IČ: , DIČ: CZ , registrvaná u OR vedenéh Městským sudem v Praze, ddíl C, vlžka Strana 12