Bullhorn. Data Mirror. Introduction to Bullhorn Data Mirror Process and Setup v7.1.5

Transcription

1 Bullhorn Data Mirror Introduction to Bullhorn Data Mirror Process and Setup v

2 Contents 1. Overview of Bullhorn Data Mirror 1.1. Processing 1.2. Main Entities 1.3. Master Entities 1.4. Error Handling 1.5. Limitations 2. Configuration 2.1. Dependencies 2.2. Interfaces to Other Applications 2.3. Hosting 2.4. Environment 2.5. Event Subscriptions 2.6. Replicator Folder Replicator.jar Replicator.log Start.bat application.properties file 2.7. Task Scheduler Open Task Scheduler Create Task General Tab Trigger Tab Action Tab Conditions Tab Settings Tab Save the record 3. Continuous Monitoring 1. Overview of Bullhorn Data Mirror Data Mirror takes the data held in the Bullhorn databases and replicates it to a database hosted by the client, allowing the client direct SQL access to Bullhorn data. Data Mirror replicates the Bullhorn data into a MS-SQL Server environment. After replication, organizations may wish to migrate their data into a target standards RDBMS reporting platform and/or Database Management System. 2

3 1.1 Processing The Data Mirror is made up of two primary processes: the Master process and the Main process. These processes take new, modified, and deleted records in the core Bullhorn tables and populate your Data Mirror with these changes. There are three primary tables used to control processing within the Bullhorn Data Mirror. RequestQueue This will be populated with the IDs from the event subscription and used to create RequestStatus records for: Merge Events Events for entities that do not have a Date Last Modified Hard Deleted records RequestEntity Data Mirror uses this table to determine if replication for an entity should be driven by events or by Date Last Modified. For Date Last Modified driven entities, the table also stores the Date Last Modified value which drives replication. RequestStatus The RequestStatus table is where records are held to be processed. These records can come from the RequestQueue or from the DateLastModified process. The table includes columns for the entity type and the entity ID. These are important because the replication process loops through the records in RequestStatus and uses the entity type and ID to find the corresponding record in Bullhorn, populate the record in the client Data Mirror, and update the RequestStatus status. 1.2 Main Process and Entities The Bullhorn Data Mirror process uses the above tables to perform the two Main processes: A process that creates RequestStatus records. This is done in two ways: 3

4 Event subscriptions - Events are consumed by the Data Mirror and create RequestStatus records with a status of Queued. Deletions from the Bullhorn database and record merges are actions that generate events. Entities that do not have a Date Last Modified also throw events. These include: Candidate Education Candidate Reference Corporate User Corporate User Private Labels Corporation Department Housing Complex Sendout Note: In the event of a records merge, Bullhorn Data Mirror will process the merge as an update/delete via the event subscription. The secondary merge record will be soft deleted in the Data Mirror, and the primary record will reflect any changes as a result of the merge identified by the update event subscription. DateLastModified Process - There is a recurring process that checks the Date Last Modified for supported entities. It uses a query that sets the RequestEntity s Date Last Modified value as the start date and the current time as the end date, and pulls all the records inserted or updated within that time frame. It then adds a record to the RequestStatus table with a status of Queued for each of the returned records. Entities that support Date Last Modified are: 4

5 Appointment Appointment Attendees Candidate Candidate Business Sectors Candidate Categories Candidate Secondary Owners Candidate Skills Candidate Specialties Candidate Work History Category Private Labels ClientContact ClientContact Business Sectors ClientContact Categories ClientContact Secondary Owners ClientContact Skills ClientCorporation ClientCorporation Custom Object ClientCorporation Leads Edit History Appointment Edit History Candidate Edit History ClientContact Edit History ClientCorporation Edit History Corporate User Edit History Lead Edit History JobOrder Edit History JobSubmission Edit History Opportunity Edit History Placement Edit History Task JobOrder JobOrder Assigned Users JobOrder Business Sectors JobOrder Categories JobOrder Custom Object JobOrder Skill JobOrder Specialties JobSubmission Lead Lead Assigned To Lead Candidates Lead Categories Lead ClientContacts Lead Skills Lead Specialties Note Note Entity Opportunity Opportunity Appointments Opportunity Assigned Users Opportunity Business Sectors Opportunity Categories Opportunity Certifications Opportunity Custom Object Opportunity JobOrders Opportunity Skills Opportunity Specialties Placement Placement Change Request Placement Commission Placement Custom Object Task Tearsheet Tearsheet Candidates Tearsheet Contacts Tearsheet Jobs Tearsheet Leads Tearsheet Opportunities User Custom Object A process that inserts data using the RequestStatus records. For each Queued RequestStatus record, the Data Mirror will do the following (assuming success path): Identify the entity and entityid on the RequestStatus Call the Bullhorn database using the REST API to find the details of the entity Replicate the entity in Data Mirror For records that are updated via Date Last Modified, the record s edit history is also replicated Update the RequestStatus to Success 1.3 Master Process and Entities Master Entities are treated slightly differently than those described above. Because master entities are not modified as frequently, they are only replicated once per day via Windows Task Scheduler. This process will look for instances of the following entities and update them in Data Mirror: Business Sectors Categories Corporation Departments Corporate User Departments Field Maps 5

6 Private Label Skills Specialties 1.4 Error Handling In the event the RequestStatus process errors due to the entity/entityid not being found in Bullhorn, it is assumed this means the record was hard deleted or set to private in Bullhorn. Hard deletes mean the record was truly removed from the database, as opposed to a soft delete where the record remains but a flag is set to ensure it is no longer shown on the front end. If the RequestStatus process errors for this reason: The RequestStatus.status is updated to Retry The Exception Text is set to a value of Missing Entity. Likely due to record being hard deleted or set to private The Error Count is incremented by 1 If the error count is equal to four, then the status is changed to Error and future processing will bypass these records In the event the RequestStatus process errors for any other reason than a Missing Entity : The RequestStatus.status is updated to Retry The Exception Text is updated to the reason why the error was encountered The Error Count is incremented by 1 If the error count is equal to four, then the status is changed to Error and an is automatically generated and sent to a configured distribution list (discussed in section 2.6.4). In the event of a failure due to client-side system or environmental issues, Bullhorn uses Windows Task Schedulers to restart the (main) process every two hours. It is expected that the client will monitor the server and database for uptime. 1.5 Limitations The Bullhorn Data Mirror does not currently support replication of the following entities: Timecards People templates Job templates Company templates Placement templates Any entity marked as private The columns available via EditHistoryCorporateUser are the same as for other person records, however, we will only replicate changes for the following fields on the CorporateUser entity: Primary department, Additional (Secondary) departments, Title, Enabled/Disabled, First Name, Last Name, User type, Address, Reports to 6

7 The data Mirror is run near real-time. However, it is an asynchronous replication, which means there may be a delay. Delays could be caused by: Natural Delay - If the data is updated in Bullhorn, there may be a few seconds before the Mirror runs Errors - In the event of an error, a log of the error will be created. This could result in data in Bullhorn not being replicated until the error is resolved. Fatal Errors - While unexpected, a fatal error can occur (for example, the system hosting Data Mirror goes down). Bullhorn utilizes Windows Task Scheduler to restart this every two hours, which will ensure that the replication process automatically restarts. This could result in a delay until the next running of the scheduled task. Number of records - While configurable, Bullhorn sets limits on the number of records processed per cycle. The recommended number is 200 records and the maximum is 500 records. If the number of changes exceeds the number configured, the additional changes will be replicated during the next cycle. In other words, if there are a large number of records changing there may be a period of catch up. 2. Configuration The initial installation and configuration of Data Mirror is completed by the client with assistance from Bullhorn. A seed copy of client data is delivered via FTP, and once it has been downloaded and set up as a SQL instance on the client's server, Bullhorn will walk the client through configuring the Properties files and setting up the Task Scheduler as outlined below. 2.1 Dependencies 1. A valid Bullhorn API username, password, and key are required. 2. Server(s) with internet access is required. Technology stack outlined below. 3. Files necessary to run Data Mirror are delivered at implementation. 2.2 Interfaces to Other Applications Unless otherwise specified, this will use the default port of SQL Server Hosting 1. All data mirror data will be housed in a SQL Server Database on Client s network. 2. A Java based.jar application will be housed on a server on Client s network. 3. Client is responsible for monitoring their environment, inclusive of server, database server, and Data Mirror process to ensure the application is running. 2.4 Environment Category Title Ownership Version Web Service Bullhorn Web Service Bullhorn 2.0 7

8 Platform Java 64 bit, installed when Data Mirror is installed Bullhorn bit Operating System Microsoft Windows Server Client 2008 or 2012 Database Database User Microsoft SQL Server, Standard or Enterprise (subject to Microsoft support ) Microsoft SQL Server 1. DB user must have write and delete access 2. the DB user must have dbo full access to the environment. 3. There must be a single user with the above access rights to be used for both: installation and run the jar via task scheduler. Client. recommended: also supported: 2008, 2008R and express versions are not supported Client Database URL Hardware Microsoft SQL Server Since Data Mirror uses JDBC connection to connect to the SQL Server, the SQL URL must be a JDBC based URL.. minimum required is 8GB of RAM. recommend 500GB or 40% of the Bullhorn db you are replicating, whichever is larger. Client Client Bullhorn will work with the client to determine appropriate hardware requirements, including database storage, number of cores, and number of threads per core. 2.5 Event Subscriptions As discussed previously, Event Subscriptions are used for some entities to replicate data. These subscriptions will be created by a Bullhorn data engineer slightly before getting the backup to ensure that all data is either in the backup, or reflected in the event subscriptions. 8

9 The name of these subscriptions are used within the application.properties file within the Mirror folder. The subscription will be set up to generate events for all entities, and for updates, inserts, and deletions. An example of how this subscription would look is: ('HousingComplex,PlacementCommission,PlacementChangeRequest,Candidate,Note,S endout,joborder,tearsheet,placement,clientcontact,corporateuser,candidateeducatio n,appointment,task,candidateworkhistory,jobsubmission,candidatereference,clientc orporation,personcustomobjectinstance1,personcustomobjectinstance2,personcusto mobjectinstance3,personcustomobjectinstance4,personcustomobjectinstance5,perso ncustomobjectinstance6,personcustomobjectinstance7,personcustomobjectinstance 8,PersonCustomObjectInstance9,PersonCustomObjectInstance10,ClientCorporationCus tomobjectinstance1,clientcorporationcustomobjectinstance2,clientcorporationcustom ObjectInstance3,ClientCorporationCustomObjectInstance4,ClientCorporationCustomObj ectinstance5,clientcorporationcustomobjectinstance6,clientcorporationcustomobjecti nstance7,clientcorporationcustomobjectinstance8,clientcorporationcustomobjectinst ance9,clientcorporationcustomobjectinstance10,jobordercustomobjectinstance1,job OrderCustomObjectInstance2,JobOrderCustomObjectInstance3,JobOrderCustomObject Instance4,JobOrderCustomObjectInstance5,JobOrderCustomObjectInstance6,ClientCor porationcustomobjectinstance,jobordercustomobjectinstance8,jobordercustomobje ctinstance9,jobordercustomobjectinstance10,placementcustomobjectinstance1,place mentcustomobjectinstance2,placementcustomobjectinstance3,placementcustomobje ctinstance4,placementcustomobjectinstance5,placementcustomobjectinstance6,place mentcustomobjectinstance7,placementcustomobjectinstance8,placementcustomobje ctinstance9,placementcustomobjectinstance10') AND BhEntityEventType IN ('INSERTED,DELETED,UPDATED') Hard Deletes Hard Deletes cannot be replicated via DateLastModified, for that reason we will need a subscription for these: Placement, PlacementChangeRequest, Sendouts. 2.6 Replicator Folder Download the Data Mirror installation zip file, Data Mirror_ version-number _release.zip. First GA version is Data Mirror_7.0.10_release.zip Once that file is extracted it will create the following structures 9

10 The data_replicator_7 folder is key to starting the replication process and the properties files. There will be two additional folders within your app; the Master process and the other is the Main process. 1. The Main and Master directories will both contain the following files: Replicator.jar a. the application.properties file is the one that will hold your specific configuration. There is one variable that says either main or Master. The properties file is inline documented and easy to configure. b. Start.bat and start.sh is the exact same file in both directories. You will be creating a task Scheduler to run against this file in a step below. This is the executable jar that is run. This does not differ by client and must not be modified Replicator.log This is where the logs are written. This file will be generated the first time data Mirror is run Start.bat A batch file that runs the Replicator.jar. You will be creating a task Scheduler to run against this file in a step below application.properties file The application.properties file has a number of pieces of configuration within it. This configuration will be: Entered by Bullhorn (reflected by BH below), or Entered by Bullhorn with input from Client (reflected as BHwC ) SPECIAL-NOTE: Any changes made to the application.properties file by you should be doublechecked by Bullhorn, as this is the core to accessing the data Mirror. Field Entered by Value profilename BH main master Comments apiusername BH Username api user s username apipassword BH Password api user s password Each application.properties file will have just one of these values, which reflects which type it is replicating 10

11 apikey BH API Key api key from corporation privatelabelid BHwC Private Label ID(s) Client to provide which private labels they would like replicated. If multiple separated by comma with no spaces. customername BHwC Customer Company Name This will be part of the subject of the daily report that Data Mirror send out. You may want a unique name for each environment To BHwC xxx@abc.com Separate with a comma, no space This is where the error messages will be sent rundailyreport=true hourforreport=[0 to 23] minuteforreport=[0 to 59] timezone=est5edt standardsubscription Name BH Name of Bullhorn Data Mirror subscription from event subscriptions These are only needed if you want to alter the 1AM EST time for running report. IF you have more that one instance of DR on the same hardware you must schedule the report time to a time difference of 30 mins between reports. Only needed for the Main (not for master) server.port BH 0 (zero) This will allow the service to pick up the available port rest.clientid BH Unique client ID DataMirror specific OAuth key component rest.clientsecret BH Unique client secret DataMirror specific OAuth key component rest.tokenurl BH The rest.tokenurl parameter should not exist at all for production. If implementing in staging, then the property should be added to the file as: rest.tokenurl= rest.loginurl BH The rest.loginurl parameter should not exist at all for production. rest.sessionminutest olive BH If implementing in staging, then the property should be added to the file as: rest.loginurl= The rest.sessionminutestolive parameter should not exist at all for production. If implementing in staging, then the property should be added to the file as: rest.sessionminutestolive=1400 rest.authorizeurl BH The rest.authorizeurl parameter should not exist at all for production. If implementing in staging, then the property should be added to the file as: rest.authorizeurl= e apiurl BH This is the soap wsdl url. The apiurl parameter should not exist at all for production. 11

12 If implementing in staging, then the property should be added to the file as: apiurl= jdbc.url BHwC URL of database If running locally, will likely be: jdbc:sqlserver:// {server} \\{dbserver:port};insta ncename=[nameof thedatabase] For example: jdbc:sqlserver:// \\sql12r2:1433;insta ncename=datamirrorx If running hosted, will likely be: jdbc.url=jdbc:sqlserver://hostedname.com:{p ort};isntancename={schemaname} Example for any version before SQL-2014 jdbc.url=jdbc:sqlserver://jeremy-chart-io.cvqs cphcuaon.us-east-1.rds.amazonaws.com:14 33; instancename = ACMEDataMirror Example for SQL-2014 jdbc.url=jdbc:sqlserver://jeremy-chart-io.cvqs cphcuaon.us-east-1.rds.amazonaws.com:14 33; databasename = ACMEDataMirror jdbc.user BHwC Username jdbc.password BHwC Password SQL Server Authentication required defaultschema BHwC DATABASENAME.db o Literally, the database name followed by.dbo For example: ACMEDataMirror.dbo defaultcatalog BHwC DATABASENAME Literally, the database name. For example: ACMEDataMirror maxrecordcountper Entity BHwC 100 # of entities processed per cycle per entity type in dlmhandler and entity/error handler. Recommendation: 100 maxeventcount BHwC 2000 # of events pulled per cycle Recommendation: 2000 seedpastdays BHwC # of days between backup was taken and the Data Mirror is enabled. For safety, may want to add an additional day. This reflects how far back the Date Last Modified should be initially set in the requestentity table (f the requestentity record does not already exist). This would be for the initial seeding of records. For example, if set to 1, this would identify any requestentity records that don t exist, and set the Date Last Modified to today minus 1 day. maxthreads BHwC 12 # of threads to be used. General rule use 3x the amount of cores on the machine. maxdlmthreads BHwC 3 Number of threads processing date last modified records. maxmainthreads BHwC 3 Number of threads in the main loop. 12

13 maxerrorthreads BHwC 1 Number of threads being used by the error handler. maxtaskthreads BHwC 4 Size of thread pool for the per entity task list. 2.7 Task Scheduler Two Windows Task Schedulers need to be created, reflecting the Main and Master replication processes. The tasks will run the Start.bat file from the Master and Main folders. The differences between the two tasks are: Main: This is scheduled to run every two hours. While the Main process runs constantly, there is a failsafe to ensure that if for some reason it stops it restarts itself. This should point to the Main replicator folder. Master: This is scheduled to run every night around midnight. The Master entities do not replicate constantly. Thus, this is the process to replicate them. This should point to the Master replicator folder. The remainder of this section describes how to configure the Task Schedulers. Differences between Main and Master will be noted. If not otherwise noted, it should be assumed that Main and Master are configured in the same way. Note: The Task Scheduler should remain disabled until replication commences Open Task Scheduler Open Task Scheduler (type Task Scheduler in the Search line within Windows) Create Task Within Action menu, select Create Task General Tab Enter the following data on the General tab, entering yourself as the user. Use a different Name for Main versus Master 13

14 2.7.4 Trigger Tab Select New For Main, enter data as below: For Master: Use Daily instead of One Time Set Start time to 12:30am 14

15 Uncheck the Repeat option Action Tab Select New Enter data as below Action: Start a program Program/script: path to get to the Start.bat file Example: C:\Users\Administrator\Desktop\Main\Start.bat Start in (optional): path to the jar folder Example: C:\Users\Administrator\Desktop\Main\ Conditions Tab Enter data as below (same for main and master) 15

16 2.7.7 Settings Tab Enter data as below (same for main and master) Save the record Click OK 16

17 3. Continuous Monitoring The client should monitor the Bullhorn Data Mirror to ensure continuous and expected processing. Because of the period of time between the database backup and implementation, there may be a time that the Bullhorn Data Mirror is still catching up. Please review the Data Mirror notification (which you will configure on the properties file) to follow up with the progress of the replication. 17

18

19