Apache NiFi UI & Building a Flow

Size: px

Start display at page:

Download "Apache NiFi UI & Building a Flow"

Patience Sanders
6 years ago
Views:

1 Apache NiFi UI & Building a Flow

2 Purpose Apache NiFi provides a way to move data from one place to another, making routing decisions and transformations as necessary along the way

3 Main Canvas Draggable Components Actions Search Management Canvas Navigation

4 Canvas Navigation - Pan around canvas - Zoom In/Out - Fit Flow to Screen - Zoom to Actual Size

5 Draggable Components To add each object, drag the appropriate icon from the toolbar to the canvas. - Processor - Input Port - Output Port - Process Group - Remote Process Group - Funnel - Template - Label

6 Actions - Enable selected component(s) - Disable selected component(s) - Start selected component(s) - Stop selected component(s) - Copy - Paste - Group selected components - Change Color of selected component - Delete - Create template

7 Management - Summary - Controller Settings - Counters - Templates - Flow Configuration History - Data Provenance - Users - Bulletin Board

8 Basic Flow Source Connection Destination

9 FlowFile Processors Perform a single function on FlowFiles (i.e. routing, data manipulation, etc) Work independently from other processors and typically use only information found in the content and/or attributes of the FlowFile Reusable throughout a dataflow

10 Adding a Processor Drag the processor icon to the canvas to show the Add Processor dialog Select the desired processor and click Add Filter by Name Filter by Tag using the Tag Cloud View Processor description

11 (1) (2) (3) Processor (4) (1) Current state One of Invalid, Disabled, Running, Stopped (2) User Defined Name (3) Processor Type (4) Current processor statistics

12 Processor Context Menu Options Configure Change processor settings (Scheduling, properties, etc.) Start/Stop Make processor available for work Stats View historical stats (bytes read, bytes written, etc.) Data provenance View processor specific data provenance information

connections View details of outgoing connections Usage

13 Processor Context Menu Options Upstream Connections View details of incoming connections Downstream connections View details of outgoing connections Usage View processor documentation Change color Adjust background color

14 Configuring a Processor Each processor has its own configuration settings. To view usage information for a processor, right click and choose Usage. Right click a processor and choose Configure to pull up the configuration dialog The configuration dialog consists of 4 tabs Settings Scheduling Properties Comments

Processor Settings (1) (5) (2) (3) (4) (1) User editable Name (2) Penalty duration How long to prevent a FlowFile from continuing if determined to be in error (3) Yield duration How long to wait

15 Processor Settings (1) (5) (2) (3) (4) (1) User editable Name (2) Penalty duration How long to prevent a FlowFile from continuing if determined to be in error (3) Yield duration How long to wait before running if the processor is yielded by the framework (4) Bulletin level The lowest logging level that will produce UI level bulletins (5) Auto terminate relationships Allows for relationships that aren t going to be used in the flow to be quietly terminated.

16 Processor Scheduling (1) (4) (2) (3) (1) Scheduling strategy Whether or schedule the processor based on a rolling timer or CRON scheduling (2) Concurrent tasks how many threads to allow this processor to use at once (3) Run schedule how often to run the processor (4) Run duration how long to run for when scheduled

17 Processor Properties Not all processors have properties Not all processors allow user defined properties Bold properties are required and MUST be configured for the processor to be in a valid state

18 Processor States - Invalid One or more required properties are blank or improperly set, one or more relationships are not defined and are not auto terminated, etc. Hover over icon to display the cause. - Disabled The processor cannot be scheduled to run - Stopped The processor is properly configured and connected, but is not currently scheduled to run - Running The processor is currently in a running state and available for scheduling by the framework

19 Connecting Components Note: Connections are unidirectional From the source component, hover until the connection icon ( ) appears. Drag the connection icon from the source component to the destination component. If the source component is a processor, choose which relationship the connection represents in the popup and click Add.

20 Connection Context Menu Configure Change Connection settings Stats View historical stats for the connection Bring to front Ensure the connection is the topmost connection when multiple connections overlap

to the destination component Empty queue delete all

21 Connection Context Menu Go to source Bring focus to the source component Go to destination Bring focus to the destination component Empty queue delete all FlowFiles in the queue Delete delete the connection

22 Connection Details (1) (2) (3) (1) Connection Source (2) Connection Destination (3) For relationships Check all relationships that this connection will represent

Connection Settings (1) (2) (5) (3) (4) (1) User defined connection name (2) FlowFile expiration How long to allow FlowFiles to remain in the queue before automatically removing them (3) Back

23 Connection Settings (1) (2) (5) (3) (4) (1) User defined connection name (2) FlowFile expiration How long to allow FlowFiles to remain in the queue before automatically removing them (3) Back pressure object threshold the maximum number of objects that can be queued (4) Back pressure data size threshold the maximum amount of data that can be queued (5) Prioritizer configuration Allow the framework to prioritize the work queue. Drag desired prioritizers from Available prioritizers to Selected prioritizers

24 Running the Flow Once the flow is laid out and connected, it is time to start it up. To start a processor, select it and either click the Start button at the top of the UI or right click the processor and choose Start. To select multiple components, hold Shift and drag a box around the components you wish to select.

25 Further Resources RequiTest Website: Apache NiFi Website: Apache NiFi Users Mailing List: Apache NiFi Developers Mailing List:

Apache NiFi System Administration

Apache NiFi System Administration Security Configuration NiFi allows for the system to run securely over HTTPS To run over HTTPS, several properties need to be set Required Security Properties Property