Welcome to quilliup tutorials Introduction This guide will grant a beginner and an advanced developer the main guidelines for working with the different applications. quilliup contains the following applications for your business: Administration- System configuration and user management RDA- Reference Data Admin for managing you referential complementary data Quality Gates- Alert and monitor data quality Quality Gates Administration This guide covers all aspects of user functionality and administration aspects that are managed at the Quality Gates module under the management settings section. Managing Projects Main application navigation is based on projects. The project entity will be the base entity in which objects will be managed, and will also be one of the main entities for managing access permissions. Creating the first project is a precondition to create test and test related items in the Quality Gates. Path: Navigate to the side bar on the left hand side and select the following: Management Settings >> Projects Adding new projects and assign/remove relevant roles and data sources. A list of all existing projects will be displayed on the main screen, or click on Add Project to create the first project. Adding a Project Adding a new project to the system. Click "Add Project" and fill in the project details in the new project form:
Name- Name of the new project. Description- A short project description. Assigned data sources- Data sources that are relevant to the project must be assigned to the project. Only assigned data sources will be available as a testing source under the project items. Administrator can also restrict the access of specific users to specific data sources, although they were assigned to the project (see project role permissions). Assigned roles- Role is the permissions profile that users/groups are assigned to. Assigning roles to a project restricts the users that can access the project items (unassigned project are not displayed in the navigation menu bar). At least one Role must be assigned to the project so it can be accessed. After filling the above fields click on the save button. Roles and data sources can also be added or removed to existing projects by the shortcut button in the main project page. Existing Project Review, edit or delete an existing project. Navigate to the projects menu item under management settings to view existing projects details. Clicking on the project name will display a summary of the project properties. Data source- Assign or remove a data source to an existing project. Roles- Assign or remove a role to an existing project. Managing Data Sources Data source is the connection that enables quilliup to access different types of data sources to query the data. Creating a data source is a precondition to create and execute a test and test related items in the Quality Gates. Path: Navigate to the side bar on the left hand side and select the following: Management Settings >> Data Sources Under the data sources menu item you will find different data sources categories. RDBMS Data Sources RDBMS category includes relational databases connection. The native connection of quilliup to RDBMS types is JDBC driver based. Supported databases: Oracle MSSql MySql PostgreSQL IBM DB2 TeraData Vertica Netezza AS400 (DB2 based) Sybase Redshift Hadoop IBM Informix
Adding a Data Source Adding a new data source connection to the system. Click "Add Data Source" and fill in the project details in the new project form: Name- Name of the new data source. We recommend to name the data source according to the relevant system and not include the environment (e.g. dev/prod) in the name, keeping the object ready to deploy when working with environments in quilliup (i.e. DWH is a connection to the data warehouse pointing to the relevant connection in each environment). Driver- A list of native drivers (JDBC based) to different supported databases. Host- IP address or host name of the connection server. Port- Relevant to MSSql only (optional, when port in use is not 1433). Username- Username of the DB connection. It is highly recommended to provide a designated user to the application. Executing test and test related items require a read only user. However, the option of creating scripts and log results tables may require writing permissions. Password- Password of the DB connection. Database name- Relevant DB for connection. Filter by- Filter will narrow down the tables list within the connection. Application client is limited to get a list of tables under the connection up to 500. If your connection accedes that number of tables, you must use the filter and create designated connection (e.g. by schema or prefix). Assigned projects- The data source must be assigned to relevant projects. When assigned, a data source will appear in the data sources list at tests and test related objects under the specific project. A data source can be assigned or removed from a project also from the project definitions. Administrator can also restrict specific roles from accessing a data source at the project role permissions. Test Connection- Enables validating that the connection works and fetches the correct number of tables. After filling the above fields click on the save button. Existing Data Source Review, edit or delete an existing data source. Navigate to the data sources menu item under management settings to view existing data sources details. Clicking on the data source name will display a summary of the data source properties. Projects- Assign or remove a project to an existing data source. Test Connection- Check that the connection is valid. Files Data Sources Files data sources category enables users to connect to a folder path.
The mapping to a specific file is done at the test data set or any other test related entity that enables users to perform an operation on files data. For users to create a file data source, a network location has to be set first by an administrator (see Network Locations in Administration). Supported file types: *.csv *.tsv *.tbl *.txt *.xls *.xlsx The different text supported files include delimited files or fixed width. Adding a Data Source Adding a new data source folder connection to the system. Click "Add Data Source" and fill in the project details in the new project form: Name- Name of the file data source. Location- Select Network location out of the mapped locations (see Network Locations under Administration). Path- Folder path under the location specified. Filter by- Filter file extension according to the supported types. Search sub folders- When checked, list of files under the path will contain files located at sub folders. Assigned projects- The data source must be assigned to relevant projects. When assigned, a data source will appear in the data sources list at tests and test related objects under the specific project. A data source can be assigned or removed from a project also from the project definitions. Administrator can also restrict specific roles from accessing a data source at the project role permissions. Test Connection- Enables validating that the connection works and fetches the correct number of files. After filling the above fields click on the save button. Existing Data Source Review, edit or delete an existing data source. See RDBMS Data Sources >> Existing Data Sources. Qlikview Data Sources Qlikview data sources category enables users to connect to the Qlikview frond and back end data layer. Quality Gates supports 2 types of files: qvd files which contain the back end data layer and the qvw files that contain the front objects in the qlikview applications. The connection to the files will enable users to select files from a folder in a qvd connection or an object in a qvw connections. For more details see "Qlikview Data set". More details on pre-configuration required for connecting to Qlikview can be found at the installation operations tutorial. QVD Folder Data Source QVD data source is a folder based connection and is managed as the standard file connection. For more details see "Files Data Source". QVW Data Source
There are 2 main methods of connecting to Qlikview files. Remote (default) Quality Gates will connect to files that are published to the access point at the Qlikview server. A licensed qlikview client must be installed on the quilliup server. This can be validated by opening the target file in the qlikview client. Name - Name of the data source. Host - server name or ip. Authentication - Select authentication method as defined in the Qlikview: SSO & Header Authentication Single sign on When using header authentication the qlikview account should be member of Administrators, QlikView Administrators, in all clusters. Ticketing Qlikview DMS Credential User & Password - The defined user and password in the Access Point. Root Folder \ File - quilliup presents the list of qv applications according to the source folder of the applications (The full path of the root folder, can be a shared location). If the folder is located on a different server, make sure there are access rights between the servers.
The list of data entities presented in the data set can be also controlled manually. This can be useful when working in ticketing mode with a general user, but still wishing to create several restricted profiles, each of them accessing a specific list. This can be done by providing a path to the manual file. The input file should be a txt file that contains a row for each qvw file (rows are separated with a new line character). Sample File - The file used for testing the connection (quilliup will try to open it and to get access). Filter by- Filter by file name prefix. Search sub folders- When checked, list of files under the path will contain files located at sub folders. Assigned projects- The data source must be assigned to relevant projects. When assigned, a data source will appear in the data sources list at tests and test related objects under the specific project. A data source can be assigned or removed from a project also from the project definitions. Administrator can also restrict specific roles from accessing a data source at the project role permissions. Test Connection- Enables validating that the connection works and fetches the correct number of files. After filling the above fields click on the save button. Local Used to access applications on local\shared folders. in direct access, The qlikview application will be opened on the quilliup machine. A licensed qlikview client must be installed on the quilliup server. This can be validated by opening the target file in the qlikview client. QVW local data source is a folder based connection and is managed as the standard file connection. For more details see "Files Data Source". Managing Tags Tags are designed to help users to organize the different application objects under
different subjects, navigate throughout multiple perspectives, and analyze data quality by different slicers. Tag management includes tag groups which are used as dimensions, and tag values which are used as the dimension members. Each entity in the application can be assigned to multiple tag values (and tag groups). Tags list will be available at the Tag property when creating tests and tests related entities. It is highly recommended to tag all objects to allow meaningful navigation and analysis. Path: Navigate to the side bar on the left hand side and select the following: Management Settings >> Tags Adding and Modifying a Tag Group Adding a new tag group to the system. Click "Add Tag Group" and fill the group properties: Name- Tag group name. Description- Tag group short description. Group color- Tag group can be mapped to a specific color, all tag values related to the group will be visible according to the selected color. Use the "Try it" button to see how the final label will look. Tags- Type the tags that participate in the group. Use the enter key to create a new value. Values can also be removed from group. After filling the above fields click on the save button. Modifying tag group properties and values can be done by re-entering existing tag group. Managing Roles and Quality Gates Permissions Roles - Login Profile Roles are designed to create a grouped profile for users/groups that are created at the Administration user management. Each user or group that can log in to the quilliup, must be assigned to a role, which will define the log in profile and permissions. This applies to all authentication methods. Path: Navigate to the side bar on the left hand side and select the following: Management Settings >> Roles Adding and Modifying a Role Adding a new role to the system. Click "Add Role" and fill the properties:
Name- Role name. Use a name that points to the log in profile (e.g. Developers, Viewers, Quality Gates Admin, etc.). Description- A short description describing the log in profile group. Assigned Groups- Application groups that should inherit the role permissions. Assigned Users- Application users that should inherit the role permissions. Modifying role properties can be done by clicking edit or any of the shortcut options. Properties can be viewed by clicking the role name, where the option to remove role members is also available. Assigning a role to a project Authorization to projects and their items is enabled by creating a linkage between the role and the project. Once a role is assigned to the project (this is done by accessing the project properties), all those who are assigned to the specific role, will be able to access the project. Assigning a role to a project is performed by accessing the project properties. Modifying role properties can be done by clicking edit or any of the shortcut options. Properties can be viewed by clicking the project name, where the option to remove roles and data sources is also available.
A role can be restricted to perform partial functionality within a specific project. Once a role is assigned to a project, it will inherit the default permission setting. This can be modified in the project role permission settings. Managing Quality Gates Permissions quilliup allows administrators to manage flexible authorization and permissions profiles. There are several levels to quilliup permissions schema. Authorized modules - set at the Administration application under the user management. The authorized modules will appear at the home page after the log in. Allowed projects - set by the roles that are assigned to the project. Each user is eventually assigned to a role, and the role is assigned to the project. The authorized projects will appear under the "My Projects" menu item and will be accessible. Allowed data sources - set by assigning the data source to projects. Only relevant data sources will be available at the items that connect to the data source (i.e. tests, variables, etc.). Another restriction can also be defined to allow only specific users (e.g. roles) to access some of the data sources under a project. This will be detailed in the role permissions under this section. Allowed role functionality - role allowed operations are defined at the role level and at the role permissions item under each project. A role can have different permissions under each project (i.e. Developer at Migration project and a viewer under the DWH project). Managing role permissions Global permissions Role global permissions are located under each role. Path: Navigate to the side bar on the left hand side and select the following: Management Settings >> Roles Click on the role, under the role summary mode, click on "Role Global Permissions". The Role Global Permissions page contains all functionality related to the management settings of the Quality Gates. System Administrator can specify what are the allowed settings that a role can perform withing Quality Gates module.
The default setting label appears as long as you are using the default global permissions that are assigned to any new role in the application. Once the settings are changed in the Administration module, the change will not apply on existing roles, only on new created roles. Once a local change was performed for a role in the Quality Gates, the label "Local settings" will appear and only local changes may apply. Role global permissions include the following functionality: Data sources - Enable category to allow full permissions for the role. Disabling the sub category items will provide a view only permission. Available options include: Create/edit, edit only, delete and assigning a project to a data source in order to allow using it under project's items. Projects - Enable category to allow full permissions for the role. Disabling the sub category items will provide a view only permission. Available options include: Create/edit, edit only, delete and assigning a project to a data source in order to allow using it under project's items assigning a role to a project a project to make a project available for all who participate in the role. Manage Tags - Enable category to allow full permissions for the role. Disabling the sub category items will provide a view only permission. Available options include: Create/edit & manage tag values, edit & manage values only, delete tag group. Manage application security - Enable category to allow full permissions for the role. Disabling the sub category items will provide a view only permission. Available options include: Create/edit roles, edit only (including assigning projects and users/groups to the role), delete and manage permissions (global role permissions, default project permissions and project-role permissions). Deploy - Enable category to allow full permissions for the role. Disabling the sub category items will provide a view only permission. Available options include deployment of the following: All project related objects, data sources, projects, roles and tags. Import - A role who will have a permission to perform import at the specific environment, will be able to deploy any pending package, regardless to the object types it contains. Project Role Permissions (Role assigned to a project) Role permissions are located under each project and are related to all functionality under "My Projects" section. Path: Navigate to the side bar on the left hand side and select the following: Management Settings >> Projects Click on the project, under the project summary mode, click on "Role permissions" at the assigned role. The Role Permissions page under the project, contains all functionality related to the project and test related objects. System Administrator can specify what are the allowed operations that a role can perform withing the project.
The default setting label appears as long as you are using the default project permissions that are assigned to any new role assigned to the project. Once a local change was performed for a role in the project, the label "Local settings" will appear and only local changes may apply. The default settings can be set at the project level, under the project default permissions. The default will apply to any new role which is assigned to the project. The change will not apply on existing roles, Project default permissions schema is set at the Administration module, any change will not apply on existing projects, only on new created projects. Project role permissions include the following functionality: Data sources - Enable category to allow default access to any new data source which is added to the project. Disable category to allow manually adding a new data source under the project. The allowed data sources for the role under the project will available for usage when creating a data set, a variable, a script or any object that connects to a data source. Tests - Test permissions enable an administrator to create different user profiles that will work with Quality Gates. The permissions can provide independence on one hand, and restrict other users from complex or too "technical" operations. Enable category to allow full permissions for the role. Disabling the sub category items will provide a view only permission. Tests permissions include the create/edit, edit only and delete standard permissions. Execution can also be enabled, to create the option to view and execute tests without changing their definitions. The option to assign values to existing variables under tests is available as well as part as the execution and view only permission. User can also be allowed or restricted from editing the result's thresholds (this will affects the test result is considered as a modification of test definitions). Additional operations that can be restricted are the option to add a data set which was created in a test, to the shared data sets library, and the option to log the results into a database (if any of the data source is enabled for logging). Execution Flows - Execution flows permissions enable an administrator to create different user profiles that will work with Quality Gates. The execution flow permissions related to test processes and scheduling. Enable category to allow full permissions for the role. Disabling the sub category items will provide a view only permission.
Execution flows permissions include the create/edit, edit only and delete standard permissions. Execution can also be enabled, to create the option to view and execute tests without changing their definitions. The option to assign values to existing variables under execution flows objects is available as well as part as the execution and view only permission. Users can be given the option to schedule processes from the application. Library - Library permissions relate to all project shared objects. Permissions and enforcement will help the administrator to make sure that not all users can change centralized objects that can have a cross system impact. Enable category to allow full permissions for the role. Disabling the sub category items will provide a view only permission. Library permissions relate to create/edit, edit only and delete standard permissions of shared data sets, variables and scripts. Script can also be executed by an authorized user. Quality Gates Set up checklist In order to work with the platform you need at least an admin user that will have all relevant objects created and assigned to it. You can continue and add or modify any of the object any time. The application will be installed with a default entities, so you can start working. When creating your own customized entities, make sure you follow the instructions below. 1. Users/groups were set up in the administration module by your system admin. 2. Roles were created (at least one role) at the Quality Gates management settings. 3. Relevant users/groups were assigned to the role. 4. At least one project was created at the Quality Gates management settings. 5. Roles were assigned to the relevant projects. 6. Data sources were created at the Quality Gates management settings. 7. Projects were assigned to the Data sources according to relevancy. 8. Tag groups were created and populated with tag members. Now you can go to the tests menu item under your project and create your first test!