Microsoft SQL Server 2008 Integration Services

Size: px

Start display at page:

Download "Microsoft SQL Server 2008 Integration Services"

Kellie Cameron
6 years ago
Views:

1 Microsoft SQL Server 2008 Integration Services Problem Design Solution Erik Veerman Jessica M. Moss Brian Knight Jay Hackney Wiley Publishing, Inc.

3 Microsoft SQL Server 2008 Integration Services Problem Design Solution Introduction... xxv Chapter 1: SSIS Solution Architecture...1 Chapter 2: SSIS Management Framework Design...33 Chapter 3: Package Deployment and Storage Decisions...75 Chapter 4: File-Handling and Processing Methods...97 Chapter 5: Data Extraction Best Practices Chapter 6: Data-Cleansing Design Chapter 7: Dimension Table ETL Chapter 8: Fact Table ETL Chapter 9: SSAS Processing Architecture Chapter 10: Implementing Scale-Out ETL Process Chapter 11: Scripting Design Patterns Chapter 12: SSIS Package Scaling Index...423

5 Microsoft SQL Server 2008 Integration Services Problem Design Solution

7 Microsoft SQL Server 2008 Integration Services Problem Design Solution Erik Veerman Jessica M. Moss Brian Knight Jay Hackney Wiley Publishing, Inc.

8 Microsoft SQL Server 2008 Integration Services: Problem Design Solution Published by Wiley Publishing, Inc Crosspoint Boulevard Indianapolis, IN Copyright 2010 by Wiley Publishing, Inc., Indianapolis, Indiana Published by Wiley Publishing, Inc., Indianapolis, Indiana Published simultaneously in Canada ISBN: Manufactured in the United States of America No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) , fax (978) Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) , fax (201) , or online at Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Web site is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or Web site may provide or recommendations it may make. Further, readers should be aware that Internet Web sites listed in this work may have changed or disappeared between when this work was written and when it is read. For general information on our other products and services please contact our Customer Care Department within the United States at (877) , outside the United States at (317) or fax (317) Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Control Number: Trademarks: Wiley, the Wiley logo, Wrox, the Wrox logo, Programmer to Programmer, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written permission. All other trademarks are the property of their respective owners. Wiley Publishing, Inc., is not associated with any product or vendor mentioned in this book.

9 To my family: Amy, Meg, Nate, Kate, and Caleb Erik Veerman

About the Author s Erik Veerman is a Mentor for Solid Quality Mentors, focusing on training, mentoring, and architecting solutions on the SQL Server Business Intelligence (BI) platform.

Veerman has designed dozens of BI solutions across a broad business spectrum telecommunications, marketing, retail, commercial real estate, finance, supply chain, and information technology.

As an expert in on - line analytical processing (OLAP) design, extract, transform, and load (ETL) processing, and dimensional modeling, Veerman is a presenter, author, and instructor.

11 About the Author s Erik Veerman is a Mentor for Solid Quality Mentors, focusing on training, mentoring, and architecting solutions on the SQL Server Business Intelligence (BI) platform. His industry recognition includes Microsoft s Worldwide BI Solution of the Year and SQL Server Magazine s Innovator Cup winner. Veerman has designed dozens of BI solutions across a broad business spectrum telecommunications, marketing, retail, commercial real estate, finance, supply chain, and information technology. His experience with high - volume, multi - terabyte environments and SQL Server 64 -bit has enabled clients to scale their Microsoft - based BI solutions for optimal potential. As an expert in on - line analytical processing (OLAP) design, extract, transform, and load (ETL) processing, and dimensional modeling, Veerman is a presenter, author, and instructor. He led the ETL architecture and design for the first production implementation of SQL Server Integration Services (SSIS), and helped to drive the ETL standards and best practices for SSIS on Microsoft s SQL Server 2005 reference initiative, Project REAL. Veerman is also co -author of Professional SQL Server 2008 Integration Services (Indianapolis: Wiley, 2008), Expert SQL Server 2005 Integration Services (Indianapolis: Wiley, 2006), Professional SQL Server 2005 Integration Services (Indianapolis: Wiley, 2007), and lead author for the Microsoft Press SQL Server 2005 and SQL Server 2008 series, focusing on Business Intelligence Development and Implementation. As a resident of Atlanta, Georgia, Veerman participates in the local Atlanta SQL Server User s Group, a Professional Association of SQL Server (PASS) chapter. Jessica M. Moss (SQL Server MVP) is a well - known practitioner, trainer, author, and speaker for Microsoft SQL Server Business Intelligence (BI). As a Mentor with Solid Quality Mentors, she has created numerous data warehousing solutions for a variety of companies in the retail, Internet, health services, finance, and energy industries, and has delivered training courses on SQL Server Integration Services (SSIS), SQL Server Retail Services (SSRS), and SQL Server Analysis Services (SSAS). While working for a major clothing retailer, Moss participated in the SQL Server 2005 TAP program, where she developed best implementation practices for SSIS. Moss has authored technical content for multiple magazines and Web sites, and has spoken internationally at conferences such as the Professional Association for SQL Server (PASS) Community Summit, SQL Teach International Conference, and the devlink Technical Conference. As a strong proponent of developing user - to - user community relations, Moss organizes meetings for a User Group that discusses different technologies. She also actively participates in local User Groups and code camps in central Virginia. In addition, Moss volunteers her time to help educate people through the PASS organization. Brian Knight (SQL Server MVP, MCITP, MCSE, MCDBA) is the owner and founder of Pragmatic Works. He is also the co -founder of SQLServerCentral.com and JumpstartTV.com. He runs the local SQL Server Users Group in Jacksonville, Florida (JSSUG). Brian is a contributing columnist for several technical magazines and does regular Webcasts at Jumpstart TV. He is the author of ten books on the SQL Server technology. Knight has spoken at conferences such as the Professional Association of SQL Server (PASS), SQL Connections, and TechEd, as well as many Code Camps. His blog can be found at Knight lives in Jacksonville, where he loves watching the National Football League Jaguars lose on a regular basis.

About the Authors Jay Hackney is a mentor and consultant for Solid Quality Mentors, where he helps customers utilize Microsoft technology to build real-world solutions.

12 About the Authors Jay Hackney is a mentor and consultant for Solid Quality Mentors, where he helps customers utilize Microsoft technology to build real-world solutions. He has 15 years of development and consulting experience on the Microsoft data platform, and has worked with SQL Server Integration Services (SSIS) since the earliest private beta. Since then, he has used SSIS to deliver data warehousing, extract, transform, and load (ETL), and data migration solutions for many well-known companies, and has presented on SSIS topics at events such as Professional Association for SQL Server (PASS) Summit, Microsoft Tech-Ed, and SQL Server product launches. Rushabh Mehta is a Mentor for Solid Quality Mentors, and is also the Managing Director for Solid Quality India Pvt. Ltd. Mehta has actively worked with the Microsoft SQL Server and Business Intelligence (BI) platform since He has developed numerous BI solutions, including predictive -analysis systems, health-care solutions, multi - terabyte financial decision support systems, as well as designing and building an analytical platform to support the Microsoft Windows Azure platform. His experience spans a wide breadth of industrial sectors, including manufacturing, retail, finance, and government, working with clients such as Raymond James Financials, Jackson Hewitt, Publix, the U.S. Department of Veterans Affairs, the U.S. Department of Defense, and Microsoft. Mehta has also developed a number of best practices and standards for implementing BI solutions. He has also developed and delivered advanced training courses on Microsoft BI practices and technologies. An active speaker at a number of large conferences such as the Professional Association for SQL Server (PASS) and TechEd, Mehta also frequently speaks at SQL Server and BI User Groups around the world. Mehta is the Executive Vice President for PASS, the largest global user - driven community for SQL Server professionals. x

13 Executive Editor Robert Elliott Project Editor Kevin Shafer Technical Editor Paul S. Waters Production Editor Kathleen Wisor Copy Editor Paula Lowell Editorial Director Robyn B. Siesky Editorial Manager Mary Beth Wakefield Credits Production Manager Tim Tate Vice President and Executive Group Publisher Richard Swadley Vice President and Executive Publisher Barry Pruett Associate Publisher Jim Minatel Proofreader Scott Klemp, Word One Indexer Robert Swanson Cover Image istockphoto.com/wolv

15 Acknowledgments I want to first thank Jessica, Brian, and Jay for adding their valuable expertise and experience to several of the chapters of this book, and sharing their approach and code practices from the SSIS framework, to the dynamic extraction logic, to data cleansing best practices. A big collective thanks to our clients! The unique projects and challenges helped shape and refine this content. I added up the years of experience for all of us, and these pages represent about 50 years of work with data processing and ETL and that doesn t include late nights and long weekends. I also want to thank Paul Waters and Rushabh Mehta for their roles in this book as Technical Editor and contributing author, respectively. A big thanks to the Wrox team for their support, hard work, and valuable feedback. And a thanks to you, the reader, because I actually read the Amazon feedback and try to address the constructive criticism in each new work. Hopefully, the diligence has paid off. If so, help us and let the world know! Last, but not least, thanks to my family for their support. A special thanks to Amy and her encouragement and hard work in so many ways. Erik Veerman As a first - time book author, this acknowledgment holds special meaning to me. I will try to refrain from mentioning everyone I have ever met, and focus on a few special people that truly affected the outcome of this book. First of all, a huge thank you is due to Erik Veerman, the lead author on this book. He not only convinced me to join this troupe of authors, but also took the time to patiently answer all of my questions about the process. Next, thank you to Jay Hackney, Brian Knight, Rushabh Mehta, Paul Waters, and Wrox for their authoring, technical, and editing contributions. I truly appreciated your support and help. On a personal note, I would like to thank my friends and family, Mom, Dad, and Lianna, for their never - ending support and patience as I disappeared for days at a time. Finally, a few other people who I would like to recognize for their previous support are Fred Creasy, Georgia Willis, Kathi Walker, and Paul Reynolds thank you. Jessica M. Moss Thanks to everyone who made this book possible. As always, I owe a huge debt to my wife, Jenn, for putting up with my late nights, and to my children, Colton, Liam, and Camille, for being so patient with their tired dad who has always overextended. Thanks also to the makers of Guinness and other hard liquors for sustaining my ability to keep on writing technical books. Finally, thanks for all the User Group leaders out there who work so hard to help others become proficient in technology. You make a huge difference! I d like to thank Erik for this opportunity and his hard work on this book. Brian Knight Jay Hackney

Microsoft SQL Server 2008 Integration Services

Microsoft SQL Server 2008 Integration Services Problem Design Solution Erik Veerman Jessica M. Moss Brian Knight Jay Hackney Wiley Publishing, Inc. Microsoft SQL Server 2008 Integration Services Problem