HEARTLAND DEVELOPER CONFERENCE 2017 APPLICATION DATA INTEGRATION WITH SQL SERVER INTEGRATION SERVICES

Transcription

1 HEARTLAND DEVELOPER CONFERENCE 2017 APPLICATION DATA INTEGRATION WITH SQL SERVER INTEGRATION SERVICES

2 SESSION ABSTRACT: APPLICATION DATA INTEGRATION WITH SQL SERVER INTEGRATION SERVICES What do you do when your team is tasked with integrating multiple applications from multiple vendors with your organization's own in-house applications? What does integration mean to the users of the applications, to your technical team, and to the 3rd parties vendors? This session is a review of a real-world example of the approaches considered and the path took on such a project. We'll cover the constraints of the project, discuss the pro's and con's of the approach taken, and also cover what has worked, what could be better, and how it will be improved. This session will be a high-level discussion on the use of SSIS (SQL Server Integration Services) as a data integration tool, SQL Server as an integration database, DB2 as the in-house application database, ASP.net as a user front end, and finally the integration tools provided by various 3rd party vendors. Please join to learn from this projects teams experiences and to share your own

3 WHO S HERE TODAY Developers? DBA s? Other?

4 WHO I AM project lead developer application architect A developer learning about SSIS Mentor to my coworkers

5 WHY AM I HERE To provide examples and experience that may help you in your projects. Feel free to ask me questions and point out alternatives. I hope to learn from this.

6 LET S REPLACE THE OLD MAINFRAME TIME SHEET SYSTEM WITH A VENDOR APP AND

7

8

9 CONSTRAINTS

10 IN HOUSE DEVELOPER TOOLS SSIS (SQL Server Interation Services) Single purpose replication tools Visual Studio for editing SSDT (SQL Server Data Tools) FOR SSIS project type For SSRS project type For SQL Server Database project type C# ASP.net MVC SSRS home grown workflow tracking

11 The Time and activities tracking application provides and SSIS like tool, job scheduling, and API s VENDOR TOOLS LIST The preferred Personnel application provides a reporting tool that provides flat files delivered from their report scheduling tool. The Equipment tracking application provides API s for data load and unloads. But no scheduling

12 SQL Server schedule / SQL Agent jobs HOW DO WE RUN AND SCHEDULE JOBS Mainframe scheduler for running JCL type jobs: Current Employee Time and Activity Tracking Application Current Accounting application DB2 replication Equipment Management application has no scheduling so SQL Agent jobs will call API s to send data.

13 DESPITE CONSTRAINTS WHAT IS A KEY OPPORTUNITY HERE? The client gets a set of tools that work together from a common administrative application. Client also gets the new Time and Activity tracking applications for Employees and Teams Different technical groups work together on a solution that they may not have thought about on their own or using their own toolset. With our approach every developer and DBA should learn something new for their tool chest.

14 DEMOS!

15 TAKEAWAYS

16 Use it for database SOURCE CONTROL Use it for SSIS, SSRS Use it for web apps Use it for mainframe code Base deploys on source from source control

17 WORK TOGETHER

18 AUTOMATE BUILDS, DEPLOYS, AND CONFIGURATION WHY? TO SAVE TIME TO DEPLOY A STABLE SYSTEM

19 WHAT CAN BE DEPLOYED WITH AUTOMATION Database System data Test data SSRS reports ASP.net web sites SSIS packages SQL Agent Jobs anything

20 STAY DRY

21 BE PREPARED FOR TROUBLE SHOOTING SSIS SERVER ISSUES SSIS logs are hard to read even if you have access to them Exceptions in script tasks are not passed to SSIS logging?! When faced with a problem even google can t fix then delete control or package. Copy working controls to a new package. learn how to query SSISDB log tables

22 USE YOUR OWN TOOLS BEFORE USING THE APPLICATION VENDOR TOOLS

23 FUTURE FOR THIS PROJECT AND SIMILAR PROJECTS: Study ETL patterns to see what could be applied Migrate to Visual Studio 2017 Improved validation tools Package Parts Consider dedicated Workflow tools such as Microsoft WFF (Worflow Foundation?) SSAS Other BI tools

24 WHAT I HOPE WE DID. You know more about how SSIS can be used along side a much larger group of tools How you can use SSIS and C# to automate the running of data integration jobs on various systems. Hope you can learn from our decisions. Good and not so good.

25 LINKS For a great overview of the purpose of SSRS, SSIS, SSAS (section two): Anything SSIS video by Stacia Misner. Starting with: WideWorldImporters database's used for demos: worldimporters-the-new-sql-server-sample-database/

26 CONTACT INFORMATION

27 CREDITS Mainframe terminal photo by: vaxomatic

28 QUESTIONS?