BRINGING DATA LINEAGE TO YOUR FINGERTIPS

Transcription

1 DATA INTELLIGENCE ASG TECHNOLOGIES LINEAGE APPLIANCE Tailor Data Lineage to Your Enterprise Embed Data Lineage from ASG s Enterprise Data Intelligence Solution Wherever You Need It BRINGING DATA LINEAGE TO YOUR FINGERTIPS Data lineage is the key to understanding the relationships between data items across your enterprise or individual lines of business. The number and diversity of data sources, the overwhelming number of data items and the complexity of applications makes understanding these data relationships a daunting challenge for any business. ASG Technologies Lineage Appliance traces lineage through enterprise applications such as Salesforce.com and ASP, custom applications, data warehouses and data lakes. Its application understanding spans languages from COBOL to PL/SQL and JAVA. ASG Enterprise Data Intelligence Solution provides complete understanding of your enterprise data. With this understanding, enterprises can respond rapidly to compliance challenges and changing business opportunities. ASG s patented Zero Gap Lineage provides an easy-to-grasp visual snapshot of the connections between your data, leaving no gaps between disparate systems. Unique data lineage visualizations are unparalleled for the depth and scale of technologies supported. It spans mainframe, distributed and Big Data platforms. DEPLOY ZERO GAP LINEAGE QUICKLY: LINEAGE ANYWHERE The ASG Lineage Appliance provides the data lineage benefits delivered by ASG s Enterprise Data Intelligence Solution in a packaged appliance, tailored to customer deployments focused on lineage solutions. Companies who are interested solely in the Data Lineage capabilities of the Enterprise Data Intelligence solution, and who do not need capabilities such as the Business Glossary and Reference Data Management, will find everything they need in ASG s Lineage Appliance package with a lower price than for the full Data intelligence solution. The Lineage Appliance includes all of the capabilities required for sourcing, generating, and visualizing lineage, but does not include the other applications. The Lineage Appliance package also includes the Lineage Anywhere service that enables the incorporation of ASG Technology s lineage solution and visualization into any third-party business application. Lineage Anywhere is invoked as a REST web service, that enables Data Intelligence lineage graphics to be embedded in Web pages or business applications as an SVG image. The exported lineage supports a range of visualization capabilities including, Zooming by mouse wheel Scrolling by mouse drag Highlighting items by name and references/usages Optionally limiting the number of results displayed Showing attribute names or link qualifiers Clicking items for further navigation ASG Technologies Lineage Appliance is a unique solution to bring Data Lineage to the point of need, across any environment to support faster regulatory compliance and quicker more accurate decision-making.

2 DATA INTELLIGENCE ENABLING CLEARER LINEAGE FOR AUTHENTIC COMPREHENSION ASG s Data Intelligence provides: An automated tool to respond to audit requests, including queries about the calculation, movement and change of numbers. Access to data required to respond to business opportunities and build new offerings for customers. Data can be gathered to support critical business decisions in less time, allowing your enterprise to focus on making decisions rather than justifying data. Improved data-informed decision quality and reduced search time through direct access to relevant data and source elements. Reduced business risk through systematic identification of poor quality data, root cause analysis and impact analysis to determine affected reports. Prove the authenticity of their data with ASG s Enterprise Data Intelligence. This solution provides quick access to data across the enterprise and displays it efficiently for any request. ASG s Enterprise Data Intelligence provides your business with a faster, more accurate view of how your data value. The ASG Lineage Appliance package adds unique benefits to those of the Data Intelligence solution: include: Reduced cost of access to lineage-focused capabilities. ASG Technologies Lineage Appliance includes code-based ETL Increased user productivity by direct access to data lineage from preferred reporting and analysis tool used to provide audit responses and business flexibility. FULL END-USER INSIGHT INTO CRITICAL DATA Complete visualization of your data is the best tool to make informed decisions to help your business stay competitive and innovative. The ASG Lineage Appliance, will provide you quick access to accurate information to fulfill requests from any stakeholder. Our tailored solution provides you with quality insights for your data with a deployment that is quick and affordable. ASG Technologies brings peace of mind to every enterprise with information access, management and control for our customers. These solutions empower businesses to enhance workforce productivity, gain an accurate and timely understanding of the information that underpins business decisions and address compliance needs with improved visibility of cross-platform data from legacy to leading edge environments. More than 70 percent of global Fortune 500 companies trust ASG to optimize their IT investments. ASG is a global provider of technology solutions with more than 1,000 people supporting more than 4,000 midmarket and enterprise customers around the world. For more information, visit ASG Technologies or Goodlette Road North, Naples, Florida USA ASG Technologies Group, Inc. All products mentioned are trademarks or registered trademarks of their respective holders. ASG_Data_Lineage_ApplianceDatasheet_

3 I D C T E C H N O L O G Y S P O T L I G H T D a t a I n t e l ligence in the Ag e o f D i g i t a l T r a n s f o r m a t ion August 2016 Adapted from Data Lineage Management: Impact and Value by Stewart Bond, IDC #US Sponsored by ASG Technologies Big data and business analytics are uncovering insights inside of organizational data assets, driving new business benefits and competitive advantage. In parallel, increasing data integrity, security, and compliance issues are driving the need to collect more metadata, or data about the data. Analysis of metadata is another level of insight, providing data intelligence knowledge to data professionals and executives in support of decision-making processes. This paper examines emerging data intelligence solution benefits and trends in the market. It also looks at the role of ASG Technologies Group (ASG) in the strategically important market for data intelligence products. Introduction Today's business leaders are challenged to move their enterprises to the next level that of digital business transformation. To achieve this transformation requires that organizations employ digital technologies coupled with organizational, operational, and business model innovation to create new ways of operating and growing businesses. The digital transformation era is rapidly removing the boundary between the internal operations of the enterprise and its external ecosystem (e.g., customers, markets, competitors, partners, regulators) because the technologies and processes that businesses deploy are so tightly linked and interconnected with their ecosystem. Information is an integral part of digital transformation, and data is at the core of information. Leaders in digital transformation need to invest in a range of technology and people to distill insight. They also need to invest in organizational competencies focused on leveraging data in an era of complex data environments. Investing in data management as part of their digital strategy benefits business leaders by enabling more agile and well-informed critical decisions about important changes in everything from customer support to product road map changes. Data is all around us, and it is growing exponentially: IDC has predicted that by 2020, the total amount of information created worldwide will surpass 44 zettabytes. In early 2016, IDC's Hadoop and NoSQL Database Adoption Survey measured the average Hadoop cluster size among midsize and large U.S. organizations and found it to be 217 terabytes and Hadoop is only one component in modern data architectures. Just under a quarter (22%) of organizations in the U.S. have Hadoop in the data management architecture, and one-third (32%) have a form of NoSQL databases. Separately, IDC's 2016 Advanced Business Analytics survey found that in the past 12 months, 70% of organizations have expanded the number and type of data used for analytics. Information digital transformation is a difficult and multidimensional journey that organizations must undertake to adapt their current organizational processes, culture, and skills to the requirements of the new digital ecosystem. IDC's survey on advanced business analytics use in United States based organizations found that 77% of respondents had either a chief data officer or a chief analytics officer, or equivalent; 55% had both an analytics officer and a data officer; and just 21% had one person who handled both roles. This finding illustrates the need for a focus on data and analytics as separate but related disciplines. US

4 Organizations working to take advantage of information digital transformation solutions will profit by integrating data technologies inclusive of business analytics, data integration, and data intelligence solutions with organizational capabilities in order to establish a firm base for digital transformation. Data intelligence technology is part of the solution, used by individuals and processes in the organization to improve the integrity of data, and it lays a solid foundation for digital transformation. Benefits of Data Intelligence Solutions Data intelligence solutions combine the collection and analysis of metadata. These solutions help data executives and organizations uncover insights and information to gain knowledge into the data itself, thus improving operational data management, data integrity, and strategic decision making. Data intelligence solutions are emerging out of the metadata management segment of the data integration and integrity (DII) software market. Data intelligence requires left-brain and right-brain functionality to build knowledge of data, as illustrated in Figure 1. FIGURE 1 Data Intelligence Thought Model What When Where Data Intelligence Who Why How Relationship Source: IDC, 2016 The left side of the brain, responsible for facts and logic, holds the definition and technical information about the data: what it is, when it is being used, where it is, and where it came from. The right side of the brain, responsible for creativity and intuition, holds usage information about the data: who is using it, why, and how it has been transformed and how it is being consumed. Relationship bridges the two sides of the brain, providing information about the following types of connections: data to data, people to data, applications to data, and people to people. The quantitative benefits of complete data intelligence knowledge are not yet fully understood. However, with regard to the core of these solutions in the function of data lineage, IDC's survey results illustrate positive impacts on organizations, as seen in Figure 2. Across the different measures of data integrity, it is clear that organizations managing data lineage with process and technology enjoy greater positive changes in data integrity than those that haven't implemented data lineage solutions IDC

5 FIGURE 2 Lineage Management Impact on Data Integrity Q. Has your organization experienced a positive, negative, or neutral change in data integrity measurements? Q. Thinking of data integration solutions that are implemented in your environment, please select the types of use cases that are in production: data lineage Duplicates (41%) Conflicting (39%) Stale (25%) Time to find (26%) Time to prepare (31%) Compliance (46%) (% of respondents with positive measurements) Lineage (27%) No lineage (73%) n = 352 Base = respondents filtered to those who have implemented data integration software Source: IDC's Data Integration End User Survey, 2015 Data intelligence provides operational knowledge for data executives and management resources in making operational decisions on when to archive data, where to store the data, and how to protect access; in parallel, it supports knowledge of where data is coming from, going to, expanding, or collapsing in order to make strategic decisions about data initiatives, resourcing, and application of compliance policies. Data intelligence also provides strategic knowledge to help executives align data initiatives with the most critical aspects of the enterprise's business strategy, target outcomes, and competitive environment. This alignment is one of the primary activities of a chief data officer or equivalent and the basis on which to demonstrate the value of data assets. The more chief data officers are involved in data projects, the greater the opportunities for the data conversation to be elevated and for data projects to be more successful, as shown in IDC's 2015 Data Integration End User Survey. Data intelligence solutions require zero-gap lineage, or metadata captured about data as it is defined and as it traverses the IT environment, including application codes, ETL jobs, data models, SQL queries, report definitions, dashboards, social media data, and structured and unstructured data content. The continued distribution of data across hybrid and cloud environments requires that data intelligence solutions be able to interrogate the entire data population associated with business operations. IDC has seen data intelligence solutions emerging from data integration software market incumbents as well as players in the start-up and open source communities. These solutions are focused on the gaps that others have left open IDC 3

6 Trends: Data Challenges in the Digital Transformation Age The amount of data being collected, stored, and analyzed is growing exponentially and generating more digital information than ever before. Data environments are also more complex, and they are inclusive of relational, nonrelational, structured, semistructured, and unstructured data, as well as data distributed across clouds, mobile devices, social media, and big data repositories. There is growing anecdotal and quantified evidence of the diversity of the data management environment. The variety of databases, especially nonrelational databases, has mushroomed in recent years. Spending on these data management technologies has also increased. For example, analytic databases grew 61% in 2015, compared with only 1% for the relational databases that still dominate the market. Within the population of survey respondents who indicated use of NoSQL databases, 57% have replaced an existing workload, 61% are augmenting an existing workload, and 64% are using the database for new workloads. Figure 3 illustrates that the migration to cloud database technology is happening. FIGURE 3 Data Technology Deployment Q. How are each of the following data management technologies deployed? Hadoop Nonclustered relational DBMS Graph DBMS Document-oriented store (NoSQL) Key-value store (NoSQL) Wide column store (NoSQL) Cluster-based relational DBMS Streaming data processing (% of respondents) On-premises External cloud Hosted external n = 501 Source: IDC's Hadoop and NoSQL Database Adoption Survey, IDC

7 The largest percentage of on-premises deployment is associated with Hadoop and nonclustered relational databases. At the other end of the spectrum, the elasticity of cloud infrastructure can be leveraged for clustered relational databases. IDC's recent survey of data integration software users found that more organizations are integrating data in hybrid and cloud environments than are integrating data on-premises only. The cloud is re-creating silos of information and distributing that information among multiple cloud infrastructures, platforms, and applications. In our research, we Error! Bookmark not defined. discovered that organizations with data in the cloud experienced fewer positive and more negative changes in data quality measurements compared with peers with only on-premises data. Absence of knowledge about where data is, who has access to the data, how the data has been transformed, and how and where the data is being used causes data trust issues and impacts operational processes and query performance. It can also result in security breaches and fines for noncompliance. Technical challenges associated with the scale and complexity of data in the digital transformation era are only part of the problem. According to IDC's 2015 MaturityScape Benchmark: Information Digital Transformation in the United States, the majority of companies surveyed have yet to establish integrated information transformation capabilities and expertise at the enterprise level. A comparison with our business analytics survey results noted previously could be considered contradictory. However, it's likely that even if executive roles have been established, the cultural shift required for collaborative efforts between business and IT to solve data problems hasn't happened yet. Considering ASG Technologies A data intelligence solution, with zero-gap data lineage at its core, is a critical element of digital business transformation. ASG Enterprise Data Intelligence unlocks the relationships between data items across every enterprise or individual lines of business, supporting both the left side and the right side of any business' brain. Additionally, ASG helps solve problems associated with the number and diversity of data sources, the overwhelming number of data items, and the complexity of applications. Its Enterprise Data Intelligence solution provides the ability to quickly identify the source and quality of data, which allows data officers to trace the movement of data in data warehouses and big data environments on-premises, in the cloud, or in distributed environments. ASG Enterprise Data Intelligence also includes an enterprise repository, application discovery features, metadata scanning, business glossary, and reference data capabilities. Data architects using ASG's solution gain the following: Coverage of more than 220 metadata sources Macrolevel analysis of data lineage between data stores Microlevel analysis of data lineage in application code Real-time monitoring and scanning 2016 IDC 5

8 ASG's tool-agnostic solution supports the creation of custom metadata interfaces for enterprise sources, providing a complete data lineage knowledge base. The range and flexibility offered include discovery of mainframe, distributed, and other ETL code as well as parsing to eliminate end-to-end lineage gaps. ASG Enterprise Data Intelligence includes the following components: Enterprise Data Inventory Catalogs enterprise data assets Builds a business glossary of data elements for enterprisewide consistency Provides an integrated view into data information across applications and data repositories Offers reference data management to support data categorization Enterprise Data Lineage Provides zero-gap data lineage and traces critical data elements' supply chain Uses data discovery to trace data from the origin to the destination, and transformations in between Tracks every hop from source to target, allowing for expertise to be added to hops where automation cannot provide depth Interfaces to more than 220 technologies covering data warehouses, business and data modeling tools, databases, data lakes, COBOL, Java, SQL, ETL, and other languages Enterprise Data Governance Provides stewardship assignment to facilitate governance processes Offers a comprehensive interface that supports issue management for ongoing governance projects Supports standard role definitions as well customization development for enterprisespecific titles and access Provides efficiency tools for governance workflow management and execution Enterprise Impact Analysis Takes the surprise out of data changes Understands what is going to happen before making data changes by predicting immediate and long-term impact across the enterprise, beyond data stores and applications into business processes Data stewards can prove the authenticity of their data with ASG Enterprise Data Intelligence. This solution provides quick access to data across the enterprise and displays the data efficiently for any stakeholder request. Enterprise Data Intelligence can help enterprises accelerate and simplify their response process for any audit requests. It also enables management to gather accurate representations of data to support critical business decisions in less time, providing business agility with immediate results IDC

9 Opportunities and Challenges ASG's combination of data lineage and data discovery tools provides businesses with the insight to forecast outcomes with confidence. The ASG data intelligence solution uses a metadata lens to provide an accurate view of how the business is really operating. ASG's depth and history in metadata management provides a foundation for data intelligence solutions, but it also represents a challenge. The rate of change in data technology, formats, content, processing, and consumption will require adaptive and timely changes to maintain zero-gap intelligence capabilities in the increasingly complex era of digital business. ASG will see competitive pressures from market incumbents and start-ups as more software vendors try to define this market with their own solutions. The company will need to focus on its differentiators, and bring its own left- and right-brain knowledge of data intelligence to bear in order to remain competitive and help define a market it pioneered. With a road map that expands coverage to support rapidly emerging new data types, novel approaches to enriching metadata, and continued support for chief data officers and the business analytics, ASG is prepared to meet these challenges. Conclusion The concept of analyzing metadata to gain more insight into the data itself is not new. However, digital transformation in the age of mobile, social, cloud, and big data and analytics is driving a need for new levels of insight and intelligence to enable higher levels of data integrity and management. IDC believes the data intelligence market will continue to grow and change as new data, business models, and other demands continue to impact organizations. To the extent that ASG Technologies can address the challenges described in this paper, the company will differentiate itself from the competition and help propel its customers to success. A B O U T T H I S P U B L I C A T I ON This publication was produced by IDC Custom Solutions. The opinion, analysis, and research results presented herein are drawn from more detailed research and analysis independently conducted and published by IDC, unless specific vendor sponsorship is noted. IDC Custom Solutions makes IDC content available in a wide range of formats for distribution by various companies. A license to distribute IDC content does not imply endorsement of or opinion about the licensee. C O P Y R I G H T A N D R E S T R I C T I O N S Any IDC information or reference to IDC that is to be used in advertising, press releases, or promotional materials requires prior written approval from IDC. For permission requests, contact the IDC Custom Solutions information line at or gms@idc.com. Translation and/or localization of this document require an additional license from IDC. For more information on IDC, visit For more information on IDC Custom Solutions, visit Global Headquarters: 5 Speen Street Framingham, MA USA P F IDC 7