2014 TIMi S.A.S. TIMi: Faster predictions, better decisions.

Transcription

1

2 Training objectives What will you be able to do at the end of the training? 2 Build Anatella scripts to transform data. Extract and load data to and from Anatella. Become proficient at using Anatella. Presentation Exercises

3 3 Training Agenda What is Anatella The Anatella Environment Basic Operations of Anatella Anatella boxes you cannot live without Practical exercise

4 What is Anatella? Anatella is an ETL: it extracts, transforms, and loads data 4 Anatella is a data Transformation tool known as an ETL tool, an acronym for Extract, Transform and Load Data file Extract Transformations Load Transfomed results

5 is user-friendly. Anatella is User-Friendly: Most data-transformations are meta-data-free: you don t need to care about the meta-type of a column. In this regard, Anatella is like MS-Excel: In MS-Excel, you don t need to specify the data-type of your columns/cells, neither do you in Anatella. Anatella is only slightly more complex than MS-Excel. Most data-transformations are code-free: You only need to connect "boxes": Filter (where) Group by order by Select education, count (education) as count from table where sex="female" group by education order by count

6 The advantage of Antella Fast to execute & debug, easier to understand 6 Fast to execute It has been optimized to the maximum Fast to debug Data can be viewed after each step (not only at the final output) Easy to understand graphic interface accessible to non-programmers Step-by-step logic; no coding required. SQL Anatella 6

7 7 Training Agenda What is Anatella The Anatella Environment Basic Operations of Anatella Anatella boxes you cannot live without Practical exercise

8 The Anatella environment Anatella has a user friendly interface with no coding required. 8 Menu: Menus list for quick actions. Data Table: Results of action displayed here Action Properties: Where action box properties are modified Log: Log file of actions and transformations kept here

9 The Anatella environment Data is Extracted, Transformed and Loaded in an easy intuitive way 9 Extraction from scratch, from text files, from gel, from DB Transformations on the data: sorts, aggregations, calculations, graph analysis, Load into flat files, gel, DB

10 10 Training Agenda What is Anatella The Anatella Environment Basic Operations of Anatella Anatella boxes you cannot live without Practical exercise

11 Building and running Anatella scripts Anatella transforms graphs composed of linked boxes using arrows 11 Arrows indicate that the data at the output pin of one box is used as input for the following box Boxes indicate operations on the underlying data The flag is used to show the termination of the graph

12 Building and running Anatella scripts Example of how to build a transformation graph Select the connect mode to build arrows. In this mode, click on the output pin of the outgoing box then on the input pin of the incoming box to create an arrow 2. Drag and drop boxes From the right panel to the middle frame to add them to the graph 3. Double-click on a box to edit its properties in the lower left frame 4. Right-clicking on the flag and selecting the green arrow will run the complete graph

13 Building and running Anatella scripts Testing scripts for intermmediate results 13 Click on run to switch to run mode In run mode, when clicking on an output pin, the graph runs from the last saved result till this output pin The status bar shows the overall progress of the calculation This icon (the rotating cube) shows that the box is currently running

14 Building and running Anatella scripts Running a transformation graph 14 Box Description Run to finish line: Click on this flag to run the graph from the last cached point to the flag. This is a very useful method of testing scripts efficiently and quickly however it is not advised to production situations. Delete all caches and run to finish line: Click on this flag to run the graph from the beginning of the graph until the flag. This method deletes all saved caches on the graphs. It is best practice to use this method in production.

15 15 Training Agenda What is Anatella The Anatella Environment Basic Operations of Anatella Most essential Anatella boxes Practical exercise

16 Operations of Standard Boxes It is important to understand the most commonly used boxes. 16 Extraction and Loading Transformation Automation

17 Extraction boxes 17 Boxes Description Extraction types: The boxes above are used for extracting data from flat file or from Gel files. Gel files are highly optimized data file formats that is unique to the Anatella software. Example of box parameters: Read.csv Column delimiter File name

18 Automation boxes 18 Box Description Global runner : Often scripts will be required to run automatically. The global runner box is used as an end box for a script. In Anatella s top menu there is a global runner icon, if that is clicked, all boxes linked to a global runner box will run.

19 Automation boxes 19 Box Description Parallel run: The parallel run box is used to run a series of Anatella scripts using one script. Often scripts are built in isolation to do a certain transformation, the parallel run box allows users to create a list of scripts and runs all the scripts in the specified order.

20 Transformation boxes 20 Box Description Append box: This box is used to append on table to another. It is equivalent to a UNION statement in SQL Office Agent Amount DBN Adam R 400 JHB Paul R 450 CPT Lilly R620 GMR Jenny R300 Office Agent Amount PTA Dela R 320 ELN Chris R 470 CPT Adam R 800 JHB John R 120 Union Office Agent Amount DBN Adam R 400 JHB Paul R 450 CPT Lilly R620 GMR Jenny R300 PTA Dela R 320 ELN Chris R 470 CPT Adam R 800 JHB John R 120

21 Transformation boxes 21 Box Description Single Join box: Used to joining two tables on a specific key value. The key value must be sorted beforehand. Office Agent Amount DBN Adam R 400 JHB Paul R 450 CPT Lilly R620 GMR Jenny R300 Office Area JHB 63 DBN 21 PTA 83 CPT 112 Left Join Office Agent Amount Area DBN Adam R JHB Paul R CPT Lilly R GMR Jenny R300

22 Transformation boxes 22 Box Description Single Join box: Used to joining two tables on a specific key value. The key value must be sorted beforehand. Office Agent Amount DBN Adam R 400 JHB Paul R 450 CPT Lilly R620 GMR Jenny R300 Office Area JHB 63 DBN 21 PTA 83 CPT 112 Inner Join Office Agent Amount Area DBN Adam R JHB Paul R CPT Lilly R

23 Transformation boxes 23 Box Description Single Join box: Used to joining two tables on a specific key value. The key value must be sorted beforehand. Office Agent Amount DBN Adam R 400 JHB Paul R 450 CPT Lilly R620 GMR Jenny R300 Office Area JHB 63 DBN 21 PTA 83 CPT 112 Full Outer Join Office Agent Amount Area DBN Adam R JHB Paul R CPT Lilly R GMR Jenny R300 PTA 83

24 Transformation boxes 24 Box Description Multi Join box: Similar to the single join key, however tables can be joined on multiple join keys specified by the user. The keys do not have to sorted as the complete slave tables will be stored in memory. Office Agent Amount DBN Adam R 400 JHB Paul R 450 CPT Lilly R620 GMR Jenny R300 Office Area JHB 63 DBN 21 PTA 83 CPT 112 Multiple Left Join Office Agent Amount Area Target DBN Adam R R 750 JHB Paul R CPT Lilly R R 750 GMR Jenny R300 Agent Target Adam R 750 Lilly R 750 Michel R

25 Transformation boxes 25 Box Description Sort box: This box is used to sort data. It is very commonly used in Anatella as a mandatory task to do before other tasks can be complete. For example, data has to be sorted on the join key before joining. Office Agent Amount DBN Adam R 400 JHB Paul R 450 CPT Lilly R 620 GMR Jenny R 300 Office Agent Amount CPT Lilly R 620 JHB Paul R 450 DBN Adam R 400 GMR Jenny R 300

26 Transformation boxes 26 Box Description Aggregation box: All aggregation processes are done with this box. It is equivalent to a GROUP BY statement in SQL. Office Agent Amount DBN Adam R 400 JHB Paul R 450 CPT Lilly R620 GMR Jenny R300 PTA Dela R 320 ELN Chris R 470 CPT Adam R 800 JHB John R 120 Office Amount_sum DBN R 400 JHB R 570 CPT R 1420 GMR R300 PTA R 320 ELN R 470

27 Transformation boxes 27 Box Description Data type wizard box: Used to convert data types. For example change integers to float values or to string values String manipulation box: This box is used to treat/clean strings (text). E.g.: remove brackets, convert all letters to capitals of replace words with other words. Column rename box: This box is used to rename columns in your data. It is often useful to use before loading the data to Excel/Tableau/Qlickview. Column selection box: This box is used to choose certain columns in the data. It is equivalent to a SELECT statement in SQL. Date formatter box: This is box is used to format dates to a specific format. For example, to 12/02/02

28 Transformation boxes 28 Box Description Calculator box: Used to perform calculations based on several columns and various data types. With this box, you can create and/or updates columns. Below are examples of how to use this box. Use the help tab for more information about available functions: Calculating Profit: Qty * (Price_per_unit Cost_per_unit) Concatenate name and surname name// - //surname Return a yes if x is > 10: X>10? yes : no

29 Transformation boxes 29 Box Description Calculator box: Used to perform calculations based on several columns and various data types. With this box, you can create and/or updates columns. Below are examples of how to use this box. Use the help tab for more information about available functions: Calculating Profit example: Qty * (Price_per_unit Cost_per_unit) Office Price per unit Cost per unit Quantity Profit Yokohama Tires R 40 R R Dunlop Tires R 57 R R

30 Transformation boxes 30 Box Description Calculator box: Used to perform calculations based on several columns and various data types. With this box, you can create and/or updates columns. Below are examples of how to use this box. Use the help tab for more information about available functions: Concatenate name and surname example: name// - //surname Agent Surname Concatenation Jacob Zuma Jacob-Zuma Helen Zille Helen-Zille Tony Stark Tony-Stark

31 Transformation boxes 31 Box Description Calculator box: Used to perform calculations based on several columns and various data types. With this box, you can create and/or updates columns. Below are examples of how to use this box. Use the help tab for more information about available functions: Return a yes if x is > 10 example: X>10? yes : no Agent Millions Rich Jacob 210 Yes Helen 70 Yes Tony 6 No

32 Transformation boxes 32 Box Description Filter box: This box is used to filter out rows of data that fit a certain criteria. It is equivalent to a WHERE statement in SQL Below are examples of how to use this box. Use the help tab for more information about available functions: Filter only waybills from CPT with a weight greater than 6 Loading == CPT && weight > 6 Remove null from File Reference: FileRef!= NULL Filter names with first three letters Dav left(name,3) == Dav

33 Transformation boxes 33 Box Description Filter box: This box is used to filter out rows of data that fit a certain criteria. It is equivalent to a WHERE statement in SQL Use the help tab for more information about available functions. Filter only waybills from CPT with a weight greater than 6 Loading == CPT && weight > 6 Waybill Loading Weight 1234 CPT CPT JHB JHB JHB DBN 14 Waybill Loading Weight 1235 CPT 12

34 Transformation boxes 34 Box Description Filter box: This box is used to filter out rows of data that fit a certain criteria. It is equivalent to a WHERE statement in SQL Use the help tab for more information about available functions. Remove null from File Reference example: not(isnull(fileref)) Filref Loading Weight HANLB2 CPT 5 D4355 CPT 12 C3452 JHB 14 JHB 7 JHB 5 23NULL3 DBN 14 Filref Loading Weight HANLB2 CPT 5 D4355 CPT 12 C3452 JHB 14 23NULL3 DBN 14

35 Transformation boxes 35 Box Description Filter box: This box is used to filter out rows of data that fit a certain criteria. It is equivalent to a WHERE statement in SQL Use the help tab for more information about available functions. Filter names with first three letters Dav left(name,3) == Dav

36 Transformation boxes: calculator & filterrows 36 Below a short summary of the main functions available: Operators: + - * / ^ Comparison: ==, >, <, <=, >=,!= Logical: &&, Condition: (x>a? True : False ) Format: ftoa, atof, itoa Math: abs, floor, ceil, round, sum, max, min, sqrt Char: right, left, substr, strlen, toupper, tolower, indexof Special: isnull, ndaysinmonth,nvl Constants: _pi, _e, _n, _null

37 37 Training Agenda What is Anatella The Anatella Environment Basic Operations of Anatella Most essential Anatella boxes Practical exercise

38 Thank you for your Attention For more information, please visit our website :

39 Backup up Slides The following slides are not part of the presentation. They are used occasionnaly to answer to some specific technical question.

40 Is it possible to comment code? Yes, it is. You can put comments everywhere: Directly on the graph. In the javascript. In the SQL (put "--" at the beginning of a line)

41 Can we edit directly.anatella files? Yes, for a technician. This file is a simple XML file (a text file) that is formatted so that a human can directly and easily understand and change it. For example: you can directly and easily edit the SQL statements inside the.anatella file: You can use any unicode text editor to edit.anatella files. For example, you can use the free editor EditPadLite7. Equivalent