HEARTLAND DEVELOPER CONFERENCE 2017 APPLICATION DATA INTEGRATION WITH SQL SERVER INTEGRATION SERVICES
SESSION ABSTRACT: APPLICATION DATA INTEGRATION WITH SQL SERVER INTEGRATION SERVICES What do you do when your team is tasked with integrating multiple applications from multiple vendors with your organization's own in-house applications? What does integration mean to the users of the applications, to your technical team, and to the 3rd parties vendors? This session is a review of a real-world example of the approaches considered and the path took on such a project. We'll cover the constraints of the project, discuss the pro's and con's of the approach taken, and also cover what has worked, what could be better, and how it will be improved. This session will be a high-level discussion on the use of SSIS (SQL Server Integration Services) as a data integration tool, SQL Server as an integration database, DB2 as the in-house application database, ASP.net as a user front end, and finally the integration tools provided by various 3rd party vendors. Please join to learn from this projects teams experiences and to share your own
WHO S HERE TODAY Developers? DBA s? Other?
WHO I AM project lead developer application architect A developer learning about SSIS Mentor to my coworkers
WHY AM I HERE To provide examples and experience that may help you in your projects. Feel free to ask me questions and point out alternatives. I hope to learn from this.
LET S REPLACE THE OLD MAINFRAME TIME SHEET SYSTEM WITH A VENDOR APP AND
CONSTRAINTS
IN HOUSE DEVELOPER TOOLS SSIS (SQL Server Interation Services) Single purpose replication tools Visual Studio for editing SSDT (SQL Server Data Tools) FOR SSIS project type For SSRS project type For SQL Server Database project type C# ASP.net MVC SSRS home grown workflow tracking
The Time and activities tracking application provides and SSIS like tool, job scheduling, and API s VENDOR TOOLS LIST The preferred Personnel application provides a reporting tool that provides flat files delivered from their report scheduling tool. The Equipment tracking application provides API s for data load and unloads. But no scheduling
SQL Server schedule / SQL Agent jobs HOW DO WE RUN AND SCHEDULE JOBS Mainframe scheduler for running JCL type jobs: Current Employee Time and Activity Tracking Application Current Accounting application DB2 replication Equipment Management application has no scheduling so SQL Agent jobs will call API s to send data.
DESPITE CONSTRAINTS WHAT IS A KEY OPPORTUNITY HERE? The client gets a set of tools that work together from a common administrative application. Client also gets the new Time and Activity tracking applications for Employees and Teams Different technical groups work together on a solution that they may not have thought about on their own or using their own toolset. With our approach every developer and DBA should learn something new for their tool chest.
DEMOS!
TAKEAWAYS
Use it for database SOURCE CONTROL Use it for SSIS, SSRS Use it for web apps Use it for mainframe code Base deploys on source from source control
WORK TOGETHER
AUTOMATE BUILDS, DEPLOYS, AND CONFIGURATION WHY? TO SAVE TIME TO DEPLOY A STABLE SYSTEM
WHAT CAN BE DEPLOYED WITH AUTOMATION Database System data Test data SSRS reports ASP.net web sites SSIS packages SQL Agent Jobs anything
STAY DRY
BE PREPARED FOR TROUBLE SHOOTING SSIS SERVER ISSUES SSIS logs are hard to read even if you have access to them Exceptions in script tasks are not passed to SSIS logging?! When faced with a problem even google can t fix then delete control or package. Copy working controls to a new package. learn how to query SSISDB log tables
USE YOUR OWN TOOLS BEFORE USING THE APPLICATION VENDOR TOOLS
FUTURE FOR THIS PROJECT AND SIMILAR PROJECTS: Study ETL patterns to see what could be applied Migrate to Visual Studio 2017 Improved validation tools Package Parts Consider dedicated Workflow tools such as Microsoft WFF (Worflow Foundation?) SSAS Other BI tools
WHAT I HOPE WE DID. You know more about how SSIS can be used along side a much larger group of tools How you can use SSIS and C# to automate the running of data integration jobs on various systems. Hope you can learn from our decisions. Good and not so good.
LINKS For a great overview of the purpose of SSRS, SSIS, SSAS (section two): https://app.pluralsight.com/library/courses/intro-dwbi-course/table-ofcontents Anything SSIS video by Stacia Misner. Starting with: https://app.pluralsight.com/library/courses/ssis-basic/table-of-contents WideWorldImporters database's used for demos: https://blogs.technet.microsoft.com/dataplatforminsider/2016/06/09/wide worldimporters-the-new-sql-server-sample-database/
CONTACT INFORMATION chuckdhanson@gmail.com https://twitter.com/chuckdhanson
CREDITS Mainframe terminal photo by: vaxomatic
QUESTIONS?