The National Advanced Research Computing and Data Management Service
Contents Executive Summary 3 Plan Highlights 3 Bridging the Gaps: the Cost of Inaction 4 Recommended Funding and Expected Outcomes 5 Recommended Immediate Steps 5 Opportunity 5 Overview of Business Case 6 Structure and Business Model 7 Scale of Investment 10 The Place of Digital Research Canada in the Research Ecosystem 16 The Need for Digital Research Canada 18 ARC is an Essential Enabler of Leading Edge Research and Innovation 18 ARC Technologies and Services Offer Clear Economies of Scale and Scope 18 Canadian ARC Investments Have Fallen Short and Have Been Fragmented 19 Canada and the World Need More Data Scientists; They Require ARC Resources to Train and Work 20 Canada Needs a National Data Infrastructure 20 A Vision for Research Excellence in Canada 21 About the Report Authors: 22 Appendix A: Market Analysis 23 Methodology 23 The Scale of Canada s Need for Advanced Research Computing Services 23 Technology Services 24 Universities and Higher Education 24 Private Sector 25 Government 26 Research Support Services 26 Universities and Higher Education 26 Private Sector and Government 27 Appendix B: What is Advanced Research Computing? 28 Appendix C: A Portfolio of Architectures to Serve All Sectors 29 The National Advanced Research Computing and Data Management Service 2
Executive Summary Canada is committed to playing a leading role in the knowledge economy and Industrial Revolution 4.0, developing a data-savvy workforce, growing firms at the forefront of technologies such as big data, the Internet of Things, cloud computing, smart materials and advanced manufacturing, precision, personalized medicine, promoting open government and making decisions based on the best available evidence. These goals require a robust national strategy for advanced research computing (ARC) and research data management (RDM). Such a strategy should leverage common infrastructure to address the academic, government and industry sectors at once, encouraging the development of a workforce with skills that transfer easily between sectors. The strategy should take advantage of economies of scale and scope made possible by a national system while incenting investment at local and regional scales. It should ensure that government investments in science and technology are supported by an enabling infrastructure that allows those projects to thrive. Digital Research Canada, with the scope and mandate proposed in this document, can implement such a strategy. Plan Highlights Digital Research Canada (DRC) will be mandated as the national service provider of computing and data infrastructure needed by researchers in academia, government and the private sector. thus enabling Canadian scientists, researchers and innovators to lead and compete globally. DRC will: Operate a capable national portfolio of advanced research computing and data platforms serving academia, government and industry; Provide a mechanism to ensure alignment between federal Science & Technology investments and the infrastructure needed to support those investments; Provide a federated national data infrastructure which promotes data sharing across regions and sectors while providing incentives for local and regional investment;, Create centres of excellence which would put Canada s best ARC experts in touch with Canada s toughest ARC problems; Provide a mechanism for the federal government, industry, provincial governments and institutions to invest in shared solutions to common problems. The National Advanced Research Computing and Data Management Service 3
Bridging the Gaps: the Cost of Inaction The proposed service will address a number of gaps and challenges in Canada s science and innovation ecosystem: The Government of Canada invests more than $2.5 billion each year through the federal granting councils, other agencies such as Genome Canada, and landmark programs such as Canada First Research Excellence Fund (CFREF), yet researchers cannot be assured of access to sufficient ARC resources to conduct their funded projects. The same uncertainty limits the productivity and impact of the government s own researchers; The consolidation and concentration of ARC technologies drives operating costs that will exceed the financial capacity of Canada s universities and that, with the current CFI funding mechanisms, creates another form of indirect cost that discourages research excellence; Both the Government of Canada and the research community are committed to principles of open research data, and yet no funding or infrastructure are in place for the storage, curation, preservation, accessibility and sharing of this data. The accelerating pace at which valuable research data is being generated, and in some cases lost, makes this gap especially pressing; The private sector under-invests in both R&D and the infrastructure that contributes to competitiveness and innovation. The US and EU have created programs to encourage adoption of these technologies in their jurisdictions. In Canada, barriers remain for such investments by the private sector. Investment in such innovation is critical for the continued growth of Canada s knowledge-based economy. In addition to these specific gaps, current vehicles for the funding, operation and direction of Canada s ARC and RDM services are fragmented. Ideally, Canada s advanced computing, storage, networking, human capital and software investments should be directed in a coordinated way, and potentially integrated and consolidated, allowing Canadian research and innovation to benefit from efficiencies and economies of scale and from the amplified results possible with coordinated strategic and operational planning. The National Advanced Research Computing and Data Management Service 4
Recommended Funding and Expected Outcomes Our preliminary calculations lead us to the following recommendations for funding. Two phases of investments over the next four years are recommended, representing a combined $275M in capital and $200M for operations, matched by $130M from institutions, industry and the provinces. Aggregate compute power and storage capacity would be increased by factors of more than 20. Canada s highly qualified experts in advanced research computing would grow from the 200 Compute Canada experts by adding another 100 Highly Qualified Personnel to serve the needs of industry and government, all of whom will assist in broader workforce development through training and internship opportunities. DRC would serve an estimated 4,000 academic research groups, 3,000 private sector firms and 3,000 government projects. The nearly $100M of total investment made in support of private sector firms can be expected to generate returns of $48 Billion in additional revenues and $4 Billion in incremental profits for those firms. 1 Recommended Immediate Steps 1. Establish and fund Digital Research Canada. Provide an initial $50 million in capital plus $30 million per year for at least 4 years of operating funds for national enterprise software, service development and administration to begin phase one of building Digital Research Canada to address the advanced research computing and data management infrastructure for industry, academia and government. 2. Committed and expected investments from the CFI should continue, augmented by the proposed Phase 1 capital. To strengthen accountability, and improve operational flexibility and efficiency, those investments should be made directly through the DRC project, augmented by direct investments from other sources. DRC would rely on Compute Canada s consultative accountability model, working with the academic community, the private sector, all levels of government and the funding agencies to ensure that services and capabilities are aligned both with the needs of the research community and with government priorities. 3. Direct federal regional development investments targeted at ARC to contribute to the common national platform. Opportunity The new service is an essential component for: the transition towards a knowledge-based economy; economic development in key areas such as advanced manufacturing and drug development; aligning and coordinating ARC/RDM resources to support major science investments in Canada; adoption by Canadian researchers in government, academia and industry of digital research methodologies such as extreme computing and big data analytics; enabling all three sectors to address increasingly complex research challenges, now and in the future; growing innovative firms and ensuring our industries remain competitive in areas such as drug development, advanced manufacturing, and IT; offering value for money for provinces/institutions and regional innovation hubs working to encourage economic development and innovation. 1 Based on ROI statistics published by International Data Corporation (IDC). The National Advanced Research Computing and Data Management Service 5
Overview of Business Case This business case proposes a shared national advanced research computing (ARC) and research data management (RDM) solution to serve the growing requirements for globally relevant research in academia, government and industry. A shared national service is the most cost effective way to enable modern research. The most effective organization structure for this undertaking requires further consideration and consultation. Clearly, a national organization will be needed, with care and thought given to embed excellence in accountability, transparency and governance. Compute Canada is an ideal structure to commence the project but the final organization would be the end product of a process that minimises the risks and by-products of what is certain to be a nationally significant transformation. Academic, government and industry researchers require very similar ARC technologies and RDM services. Technology deployment and operation can be coordinated and operated like a utility, providing uniform resources to all researchers, allowing those researchers to ignore operational details such as ownership, hosting and funding; There are economies of scope in the design, acquisition, operation and exploitation of ARC technologies that promote consolidated investments across these sectors; Certain types of technology (highly-parallel capability, or leadership, computing) offer true economies of scale, and consolidated investment across the three sectors could enable the creation of resources with up to three times the capability that might be possible without consolidation; Integrated support leverages the specialized skills and expertise of existing HQP and enables development of new HQP who are able to work in and benefit all sectors of the Canadian economy. Aggregate compute power and storage capacity would be increased by factors of over 20 (from 1.8 to 38 petaflops of compute, and from 15 to 342 petabytes of storage) over the period. This rate of increase is consistent with investments being made in ARC in other countries to support competitive innovation and research. Canada s HQP skills in advanced research computing would be strengthened through the retention of the 200 experts currently working with Compute Canada, and expansion with an additional 100 HQP serving the needs of industry and government, all of whom will assist in broader workforce development through training and internship opportunities. DRC would serve an estimated 4,000 academic research groups, 3,000 private sector firms and 3,000 government projects. The nearly $100M of total investment made in support of private sector firms can be expected to generate returns of $48 Billion in additional revenues, $4 Billion in incremental profits for those firms. 2 2 Based on ROI statistics published by International Data Corporation (IDC). The National Advanced Research Computing and Data Management Service 6
Structure and Business Model The business model includes important roles for national operations, individual research institutions, federal government laboratories and the private sector. National Operations: There are powerful benefits to a single organization having responsibility for the development, operations and delivery of a pan-canadian ARC platform and data infrastructure to deliver academic, government and industry research priorities in Canada. A single operator reduces duplication across the system, and ensures that an intrinsically shareable infrastructure is shared in a way which maximizes the benefit derived from the investment. Direct engagement with major funders of research infrastructure, including the granting councils, and CFI assures alignment of resources for national programs and investments. Federal policies promoting open data and sharing of data as a national resource can only be effectively realized in an organization which crosses provincial boundaries. Specific activities associated with national operations should include: Design, acquisition, operation and exploitation of national shared ARC technologies. This includes building a comprehensive national portfolio of compute and data architectures. The portfolio will include facilities with data security and privacy suited for hosting sensitive industry and government data. The infrastructure itself may be owned by the national organization or may be provided by a contracted service with Canadian industry depending on the cost-benefit for each element of the portfolio. A common policy framework and support system will apply regardless of hardware ownership; Creation, staffing and operation of centres of excellence providing skills and expertise to the national user community in specific domains of advanced research computing (e.g. bioinformatics, machine learning, big data analytics, molecular dynamics); Creation of a common service layer enabling ease-of-use and encouraging HQP mobility between sectors; Common resource allocation and access policies aligned with national priorities; Software licensing for national use by academia and government, as well as commercial licenses for private sector use; Engagement and coordination with key stakeholders; Coordination of national training and outreach initiatives, management and operation of skills certification programs. The National Advanced Research Computing and Data Management Service 7
Universities, Colleges and Research Institutions. Universities would continue to play a strong role in operations of the national system. These institutions host many of Canada s federally funded researchers and research institutes. They are closely connected to regional research innovation hubs and are the primary suppliers of the skilled human resources required to build a thriving digital economy. They benefit through access to ARC facilities beyond the scale which can be funded and operated by a single institution leveraging a national network of expertise and through closer ties to industry and government researchers. Specific responsibilities associated with universities include: Provision of expert support personnel to their own researchers, local government laboratories and industry. These experts, working in collaboration with the national operations team, are an invaluable resource feeding knowledge translation between sectors; Development of a cadre of HQP with skillsets desirable to all sectors. This includes participation in development of national training programs and local delivery of those programs. An example of such a role today is the national partnership between Compute Canada and Software Carpentry to deliver common training courses at campuses across the country; As the local stewards of research data, universities would participate in a federated national data system which allows research data holdings to be discovered and used by researchers in all sectors. This would be an expansion to a national scale of a successful pilot system created by Compute Canada and Canadian Association of Research Librarians (CARL). Participation can include local data hosting but, more importantly, data curation services. As it is today, the access of the academic sector to the national facility would not involve fee-for-service as long as the research program is federally sponsored. The initial funding of the research program is subject to rigorous merit review, as is the request for priority access to the national ARC resources. Alignment with federal priorities through policy and coordination between granting councils and the national ARC organization is more efficient than increasing granting council budgets and developing a complex system to pass money between all federal agencies (or all host universities) and the national ARC organization. Further, fee for service for academic research access to ARC resources is not common internationally, and would create an incentive for Canadian researchers to work abroad. University researchers pursuing unsponsored research, research funded through industrial partnerships, research funded through provincial grants or research funded by international agencies could be charged for use on a cost-recovery basis. The National Advanced Research Computing and Data Management Service 8
Private Sector (Industry). Industry benefits from the coordinated national system through access to the full portfolio of architectures, through access to workforce that transitions seamlessly from training in a university to research in industry on the same technical platform, expert local support provided by highly trained experts at universities in collaboration with national centres of expertise and access to public data assets served from the national data infrastructure. Industry provides interesting research problems to challenge both the academic research community and the expert support staff. The common platform provided by a national system is the key to mobility of people and ideas. Support for industry would be paid for through fees for service, or reimbursed through incentive programs addressing market failure scenarios. Private sector organizations would have access to national shared facilities at a level consistent with capital investment earmarked for private sector use, but access would not be restricted to specific equipment, unless required by the user (for example to meet specific security requirements). Private sector access is offered on a cost-recovery basis, with fees for use that are consistent with commercial offerings. Aside from the charging structure, private sector researchers would access shared facilities through introductory access packages for modest amounts of ARC resources and research support assistance, or through co-investment in a shared facility. An initial needs assessment/road mapping effort will be conducted to better quantify and forecast the needs of the private sector and will be repeated on a regular basis. Some provinces or regions may choose to subsidize Introductory Packages to encourage adoption, use and workforce development. Requests for such subsidies would be administered by the relevant funding organizations. Outreach and Business Development. Rather than building a new outreach and business development capability, the proposed organization will partner with numerous existing organizations across Canada that have already been created to assist new and expanding ventures accelerate their research, development and commercialization efforts, and to assist established ventures to improve their profitability and efficiency, including: NRC-IRAP MITACS Ontario Centres of Excellence, Inc. TEC-Edmonton MaRS Discovery District Startup Canada Canadian Open Data Exchange (CODX) Canadian Manufacturers and Exporters Aerospace Industries Association of Canada The National Advanced Research Computing and Data Management Service 9
Government Science Initiatives. Like the private sector, government laboratories will benefit from the national ARC system through gaining access to leading-edge national facilities, access to expert support personnel, and access to talent trained on a common platform. In addition, the national ARC system will foster closer collaborative relationships between government researchers and their counterparts in Canadian universities and industry. Specific activities associated with government laboratories include the curation and provision of government datasets of value to university and industry sectors consistent with the government s open data policies. As with the private sector, government science initiatives would have access to national shared facilities at a level consistent with capital investment earmarked for its use, but access would not be restricted to specific equipment, unless required by the user (for example to meet specific security requirements). An initial needs assessment/road mapping effort will be conducted to better quantify and forecast the needs of government, so that capital and operating investments can be properly scaled to address market needs, and this assessment will be updated on a regular basis. Scale of Investment Digital Research Canada can be established and begin generating benefits for Canada through phased investments by the Government Canada, as detailed below. Sector-by-sector estimates of technology and support investments are detailed in the appendices. These should be regarded as targets that can be achieved over time, as part of an investment program that should be re-assessed and adjusted based on demonstrated demand from the community. Nevertheless, those estimates have been used to scale a responsible program to meet Canada s need for advanced research computing capabilities. Phase 1 DRC Phase 1 is comprised of a short-term capital investment that will generate immediate benefits, combined with a sustained commitment to the core operations of Digital Research Canada. Capital investment of $50M in 2016-2017 to augment academically-focused investments by the CFI and partners, now being coordinated by Compute Canada, and to enable access to the resulting facilities by private sector firms and government labs. Compute Canada s current procurement process will allow the additional $50M to be invested and start generating benefits almost immediately; Investment of $30M per year from 2016-2020 to sustain the operation of this pan-canadian platform, including power costs, the initial creation of national centres of excellence, and research support for government users; Additional capital investment of $10M per year for data archive and preservation storage ($8M for academia, and $2M for government); Research institutions, industry and provinces would support additional operating costs of roughly $14M per year during the same period. The National Advanced Research Computing and Data Management Service 10
Detailed financial forecasts of phase 1 investments are summarized here: Phase 1: (Cdn$M) Capital Budget 2016 2017 2018 2019 2020 Higher Education Existing CFI* Plans (cash value, including provincial match) Stage 1 $40.00 $20.00 Stage 2 $20.00 $20.00 Total $40.00 $60.00 $20.00 Business $25.00 Government $25.00 RDM: Archiving and Preservation Higher Education $9.64 $10.04 $8.96 $7.16 $5.51 Government $3.44 $2.29 $1.72 $1.38 $1.38 Total Capital CFI+Provincial matching $40.00 $40.00 $20.00 DRC Investment $50.00 RDM $13.08 $12.33 $10.68 $8.54 $6.89 Total $103.08 $52.33 $30.68 $8.54 $6.89 *Proposes redirecting ARC funding from the CFI to the DRC The National Advanced Research Computing and Data Management Service 11
Phase 1: (Cdn$M) (continued) Operating Budget (Cdn$M) 2016 2017 2018 2019 2020 National Operations Personnel (Compensation, Travel & Professional Development) Management & Operations Centres of Excellence $7.43 $9.42 $10.53 $10.81 $11.10 $2.03 $2.08 $2.13 $2.19 $2.25 G&A $1.00 $1.10 $1.21 $1.33 $1.46 Outreach, workshops $0.50 $1.00 $1.50 $2.00 $2.00 Licensing $2.00 $2.20 $2.42 $2.66 $2.93 Power $7.87 $6.65 $6.55 $6.78 $6.78 Maintenance & Repairs $5.40 5.24 4.15 2.49 $1.49 Total National Operations $26.23 $27.70 $28.49 $28.27 $28.02 Research Support Academia $9.61 $10.50 $11.57 $12.83 $14.11 Private Sector $2.29 $2.36 $2.42 $2.49 $2.56 Government $2.29 $2.36 $2.42 $2.49 $2.56 Total Research Support $14.19 $15.21 $16.42 $17.82 $19.23 TOTAL $40.42 $42.91 $44.91 $46.09 $47.26 Total from Government $28.52 $30.05 $30.91 $30.76 $30.59 Institutions, Provinces, Private Sector $11.90 $12.86 $13.99 $15.32 $16.67 The National Advanced Research Computing and Data Management Service 12
Phase 2 DRC Phase 2 amplifies the capabilities of Digital Research Canada with a significant capital investment in 2017-2018, matched by incremental operating investments to better serve all three sectors of the research community. Capital investment of $120M in 2017-18, further expanding Digital Research Canada s ability to serve growing needs in all three sectors; Incremental investment of $11M per year starting in 2017 for additional power, and expansion of the national centres of excellence and research support for government users; Industry and provinces would support additional operating costs of roughly $2M per year during the same period. No additional support from academic research institutions would be required; Additional capital investment of $4.5M per year for data archive and preservation storage, $1.25M for academia, and $3.25M for government; Additional capacity [may] be needed to accommodate Canada s largest computational and data users. DRC will work closely with big science leaders, to maintain a detailed science and technology roadmap, to ensure DRC s funding will accommodate their needs. Detailed financial forecasts of phase 2 investments are summarized here: Phase 2: (Cdn$M) Capital Budget 2016 2017 2018 2019 2020 Higher Education $20.00 Business $50.00 Government $50.00 RDM: Archiving and Preservation Higher Education $1.81 $1.35 $1.08 $1.08 Government $4.51 $3.39 $2.71 $2.71 Total Capital DRC investment $120.00 RDM $6.32 $4.74 $3.79 $3.79 Total $126.32 $4.74 $3.79 $3.79 The National Advanced Research Computing and Data Management Service 13
Phase 2: (Cdn$M) (continued) Operating Budget (Cdn$M) 2016 2017 2018 2019 2020 National Operations Personnel (Compensation, Travel & Professional Development) Centres of Excellence $2.08 $2.13 $2.19 $2.25 G&A $0.00 $0.00 $0.00 $0.00 Outreach, workshops $0.00 $0.00 $0.00 $0.00 Licensing $1.30 $1.43 $1.57 $1.73 Power $2.13 $2.91 $2.88 $2.88 Maintenance & Repairs $6.00 $3.60 $2.16 $1.30 Total National Operations $11.51 $10.08 $8.81 $8.16 Research Support Academia Private Sector $1.59 $1.64 $1.68 $1.73 Government $1.59 $1.64 $1.68 $1.73 Total Research Support $3.18 $3.28 $3.37 $3.47 TOTAL $14.69 $13.35 $12.18 $11.62 Total from Government $13.10 $11.71 $10.49 $9.89 Institutions, Provinces, Private Sector $1.59 $1.64 $1.68 $1.73 The National Advanced Research Computing and Data Management Service 14
This investment plan is presented as a baseline since it provides an appropriate level of investment relative to our current understanding of demand from each sector. Aside from matching for the CFI s academic investments in 2016 to 2018, as a transition to the new model no provincial investment is assumed; however, provinces may wish to supplement the baseline investment. All such investments will be reflected in usage tracking and accounting thereby translating into access and use by the intended users. Additional capacity and capital investment will be needed to accommodate the computational and data requirements of Canada s largest science investments, including Genome Canada, national commitments to astronomy (TMT, SKA), physics (ATLAS, SNOLab) and environmental science (ONC). DRC will work closely with leaders of these big science initiatives, to create and maintain a detailed science and technology roadmap and ensure DRC s funding will accommodate their needs. The funding outlined above contemplates a higher level of federal investment, reflecting federal support for its investments in science and innovation through academia and its own science-based departments and agencies, as well as federal initiatives to spur private sector investment in R&D and the capital assets that support R&D. Additional investments will come from institutions, provinces and regional agencies, supporting local priorities in both research excellence and economic development, as well as from individual firms increasing their adoption and use of these critical technologies. The federal contribution indicated above will be reduced by private sector cost recovery, contributions from software vendors and provincial economic development contributions. These numbers are necessarily less certain as they require prediction of a rate of adoption. Private sector cost recovery is based on fees-for-service related to the use of technologies and expertise required to support capital-constrained users who have workloads too large for their company s own ARC systems, as well as matching investment by smaller firms piloting new research applications on shared facilities. The National Advanced Research Computing and Data Management Service 15
The Place of Digital Research Canada in the Research Ecosystem The current Canadian digital research ecosystem includes many stakeholders and several service organizations. Coordination among and collaboration with these entities is crucial to ensure a functioning system, and Digital Research Canada will work closely with the following partners and stakeholders: Government of Canada, including its S&T funding agencies: ensuring the right capabilities are in place to support strategic national investments, ensuring value for money and benefits to Canada; Research-performing organizations (academic institutions, private sector firms, government labs, and big science initiatives) and their researchers, scientists and innovators: implementing technology plans and roadmaps that enable each organization s effectiveness, maximizing available capacity and accessibility, and providing the support needed to maximize productivity and impact. Institutions in particular would see this investment as improving the effectiveness of Canadian research and reducing indirect cost burdens; The research data management community: building and operating a national data infrastructure capable of storing both active and archived research data and enabling policy-based services such as metadata, retention, discovery, sharing and publication. Academic institutions and government labs would be responsible for curation, setting policy details, and through peer-review making retention decisions. The expertise for research data policies, curation and management resides in the library community, and CARL is the natural organization to lead these functions. DRC would work in close partnership with the Canadian library community through CARL to ensure the smooth operation of a federated national data preservation system housed in institutions across the country; With the national advanced research network, run by CANARIE or as division of the DRC, and its networking partners (local and international): coordinating networking and ARC technology plans to ensure balanced services provided to users. Compute Canada users (particularly ATLAS) currently dominate CANARIE traffic, so coordination is essential. Integration of this function within DRC may offer opportunities for management efficiency, but is not required. In most countries ARC services are managed by a separate entity from advanced networking; With the research software funding and development ecosystem: through technology plans and road mapping, identify opportunities for shared solutions; provide resources for testing, implementation, user adoption/support and ongoing maintenance and support of useful developments; establish or join partnerships (e.g. open source initiatives) where broader collaboration and coordination benefits Canada. We encourage alignment of these funding programs to ensure the program framework leverages the appropriate expertise to foster improved performance; With the provinces of Canada: a number of provinces have been active supporters of academic ARC investments. Where provincial priorities are focussed on R&D, DRC enables continued provincial investment and allows specific benefits to the province through targeted access mechanisms that support local priorities. Where priorities include economic development, the DRC model provides new windows for investment and prioritization; With key private sector players: Vendors would applaud an investment commitment of this magnitude, although an integrated procurement organization represents a challenge for sales. Potential users would also applaud the creation of a capability that lowers barriers to adoption. Experience in other jurisdictions indicates that ease of trial encourages regular adoption and subsequent direct investment, which directly addresses current innovation gaps in Canada. The National Advanced Research Computing and Data Management Service 16
Cyberinfrastructure consist of computing systems, data repositories and services, visualization environments, and people, all linked together by software and advanced networks to improve research outcomes and enable breakthroughs not otherwise possible. Current Compute Canada Services and Infrastructure: Computing systems, data storage, data repositories, advanced services, cybersecurity* and tools, visualization, and software** Tightly coupled components of Advanced Research Computing and Research Data Management including archiving. Network run by CANARIE Research Data Policies and Management - CARL *Cybersecurity is managed by Compute Canada today with policies, programs and expert personnel. **Includes software currently funded by the CFI and CANARIE run on Compute Canada systems and is optimized and scaled by Compute Canada. Enterprise software such as Globus other services required to run a national platform currently are without a dedicated funding source and are currently being funded from Compute Canada capital funding. The National Advanced Research Computing and Data Management Service 17
The Need for Digital Research Canada ARC is an Essential Enabler of Leading Edge Research and Innovation ARC is now a critical enabling infrastructure for Canadian competitiveness in all R&D domains. The size and capacity of our resources for scientific computing and data storage are directly linked to our ability to compete internationally, to attract, retain and grow talent, and to encourage adoption of advanced methods in R&D for industry. World-class researchers migrate to the best research infrastructure, and our trading partners are investing strategically in ARC investments to enhance their positions in the global race for research and innovation leadership. ARC Technologies and Services Offer Clear Economies of Scale and Scope Advanced research computing technologies offer clear economies of scale and scope, so this is an area where bigger is clearly better. Bigger systems provide more computational power per dollar spent, provide burst capacity for projects with occasional need for large resources and allow more complex problems to be solved. Within the academic sector, the need for bigger systems has spurred consolidated investment as researchers strive to compete globally. Institutional resources have been replaced by regional resources and, more recently, by national resources, managed by Compute Canada. Integrated support for ARC users enables knowledge transfer and collaboration between researchers from different disciplines and different parts of the country. Common platforms have led to common solutions. Tools developed for particle physics are being used to solve problems in genomics, while genomics tools are being used to solve problems in the digital humanities. This kind of knowledge transfer and collaboration would extend to all users of an integrated service, including government laboratories and private sector firms, and would also encourage workforce development and mobility of HQP across all three sectors. Cybersecurity will be enhanced as a part of the coordination activity. DRC s cybersecurity stance would leverage best practices from government, industry and academic participants, including meeting the specialized needs of each. Cybersecurity will be a top-level component of DRC, at the highest level of operational priority and organizational accountability. Resiliency to intentional or unintentional problems, including security incidents, will be designed into the infrastructure. DRC infrastructure will have sufficient robustness and capacity to ensure Canada s precious research data and computing are resilient, even under adverse conditions. The National Advanced Research Computing and Data Management Service 18
Canadian ARC Investments Have Fallen Short and Have Been Fragmented Despite recent investments, funding in this essential infrastructure has not kept up with demand, with the needs of competitive science, or with investment levels seen in other competitor jurisdictions. For example, an analysis of ARC investment levels across academia, government and the private sector combined in Australia, France, Germany and the UK suggests that, for Canada to catch up to those countries recent levels of investment, a one-time investment of almost Cdn$400 million would be required, followed by continued regular investment to stay competitive. Moreover, the investments made in Canada have been made in a fragmented way across a variety of programs and sectors, each targeting only a part of the ecosystem, therefore failing to benefit from the economies of scale possible with ARC technology. Higher Education and Academia. While there has been a bottom-up push within the academic sector for consolidated investment and larger systems, there has been little coordination between federal investment in science and the enabling infrastructure needed for that science to thrive. As a result, the academic ARC system in Canada does not currently have the capacity to fully support key federal science investments. Compute Canada has been increasingly unable to meet requests for ARC resources over the last few years (with allocations fulfilling 84% of requests for computing time in 2012 but only 57% of requests for 2016) despite the fact that all applicants for these resources must rely on Compute Canada in order to conduct the research for which they have already received federal funding. Indeed, the current environment discourages large scale users in key areas such as life sciences and physics, while these areas have been a major focus of federal research investment in Canada. Using the methodology referred to above, a onetime investment of Cdn $80 million would be needed to catch up to key competitor countries in terms of ARC resources for Canada s ~65,000 academic researchers in total. Clearly, we must ensure that federal science investments are supported by a commensurate investment in ARC. Private Sector and Industry. While the academic sector has begun to take advantage of the economies of scale of advanced research computing technologies, only the largest Canadian firms have been able to afford such investments. The private sector, worldwide, is expected to make its own investments, not only in research, development and commercialization, but also in the infrastructure required to support those activities. Nevertheless it is well-known that on average Canadian firms underinvest in R&D and underinvest in the capital assets that support innovation and competitiveness. 3 For advanced research computing investments in particular, there has been a significant failure of the market to invest in ARC technologies: the methodology referred to above suggests that an aggregate investment of Cdn $175 million would be needed to catch up to key competitor countries in terms of ARC resources for Canada s ~96,000 private sector researchers. It is likely that issues of scale and readiness are added to concerns about risk to contribute to this underinvestment. This underinvestment is compounded by industry's limited opportunities for access to ARC capabilities available in other sectors, except through academic sponsorship and collaboration. Ideally, the private sector should be offered a range of mechanisms that would enable direct access to the ARC resources that these firms need to be productive and to compete and innovate. Obsidian, an Edmonton based company is benefitting from an ongoing partnership with Compute Canada team members at the University of Alberta. With an expanded mandate opportunities are in place for Compute Canada to test solutions in the private and health data management area in partnership with Obsidian. This would help to drive industry in Canada and provide real world opportunities to address challenges in personalized medicine. 3 Science, Technology and Innovation Council: State of the Nation 2014. Canada s Science, Technology and Innovation System: Canada s Innovation Challenges and Opportunities The National Advanced Research Computing and Data Management Service 19
Government Science Initiatives. Over the past year, Compute Canada has been approached by federal researchers from the Canadian Astronomy Data Centre (NRC), Statistics Canada, Canadian Nuclear Laboratories, the Canadian High Arctic Research Station, the Canadian Forest Service, the Bank of Canada, Natural Resources Canada, Agri-foods Canada, Fisheries and Oceans Canada, the Canadian Food Inspection Agency, and others. These researchers are seeking access to the same kinds of resources used by their academic counterparts, but it is not in Compute Canada s mandate to support them, except on a fee-forservice basis for each researcher. Some agencies have entered high-level discussions with Compute Canada, but each contract is currently treated on an ad-hoc basis. The high-level methodology referred to above suggests that an aggregate investment of Cdn $135 million would be needed to catch up to key competitor countries in terms of ARC resources for Canada s ~10,000 government researchers. This represents an investment shortfall of $13,500 per researcher, compared to the estimated shortfall of $1,850 per private sector researcher. The relative scale of this shortfall suggests that government researchers do not have the ARC tools they need to fulfill the missions of their departments and agencies, and that there is a significant opportunity to improve investment efficiency and productivity and impact for government science. Canada and the World Need More Data Scientists; They Require ARC Resources to Train and Work In 2012, IDC analysts estimated 90% of the world s data had come into existence within the previous 2 years (Vesset et al., 2014). The McKinsey Global Institute suggested that at current training rates, in the US alone there will be 140,000-190,000 more jobs than trained data scientists by 2018 (Manyika et al., 2011). It is reasonable to expect proportional or greater workforce shortfalls for Canadian data scientists, potentially corresponding to 15,000-20,000 unfilled data scientist jobs in Canada by 2018. The McKinsey report also estimated an employee shortfall of 1,500,000 data-savvy analysts and managers, and IDC suggests a similar number. Advanced research computing experts and data specialists are amongst the most sought after skillsets in the knowledge economy. This skillset is highly transferable and is required in most economic sectors including advanced manufacturing, life sciences, finance, film and animation, and engineering. Canada Needs a National Data Infrastructure Governments around the world have recognized that data generated through the use of public funds is a valuable asset for academia, government and industry. Canada has taken policy steps towards open government and open data, but does not have a supporting technical infrastructure which will allow this data to be exploited. Compute Canada is currently building a national data infrastructure (hardware) and has worked closely with CARL on a federated pilot system for data transfer, curation, storage, preservation and discovery. The federated model allows institutional or provincial datasets to be leveraged by researchers across the country without building a monolithic Canadian repository. By integrating a scalable national research data management system with the computational power needed to analyze the data, Digital Research Canada will provide the means to harness the power of publicly funded data. The National Advanced Research Computing and Data Management Service 20
A Vision for Research Excellence in Canada Advanced research computing underpins our national prosperity and is an essential ingredient to transform our nation from a resource-based to a knowledge-based economy. The competitive edge required to develop the workforce for sectors such as life sciences, advanced materials, aerospace, automotive and energy relies on the availability of a robust digital research infrastructure strategy in Canada. Much of the recipe to excel exists today and can be augmented to the appropriate capacity with a national vision. Building a robust digital infrastructure environment is as essential to our economic growth as the Trans- Canada Highway or our broadband connectivity. Canada is committed to building a knowledge economy, growing innovative firms, promoting open government and to evidence based decision making. This is not attainable without a robust national strategy for advanced research computing and research data management. Canada must actively develop capabilities for industry to adopt modern advanced computing and big data innovations, fully embracing the potential of scientific computing and accepting its necessity to compete internationally, to achieve research excellence and as essential infrastructure necessary to achieving open government. The National Advanced Research Computing and Data Management Service 21
About the Report Authors: Compute Canada was built from the ground up within the regions across Canada. Collectively with our regional partners we have more than 18 years of experience working with Canadian institutions, regional and national partners and industry to deliver ARC services: CC operates a pan-canadian ARC infrastructure platform that is accessible today by over 10,000 academic users across Canada, including over 3,000 faculty researchers. This platform encompasses roughly 70% of the high performance computing capacity available to the academic community. (SOSCIP also provides significant computing capability to this community); CC is leading a technology refresh of this pan-canadian platform, replacing older infrastructure with $75M of investment over the next 18-24 months, funded in part through an award from CFI, and potentially to be augmented with a further $50M of investment over the next 24-36 months, subject to funding by CFI. This technology refresh will allow CC to keep Canada at the leading edge of technology but will not meet the ongoing demand for capacity. This activity ensures value for money through consolidated procurement and operations; CC coordinates a national team of almost 200 experts, employed by and working at institutions across Canada, who offer unparalleled expertise and skills to users in a wide range of disciplines, attempting to solve a spectrum of problems of importance to Canada. This skill set is already in demand by the private sector and government researchers; The breadth of expertise of Compute Canada s team is matched by its geographic coverage, enabling CC to support the needs of research and development requirements in all economic sectors and disciplines Canada. The National Advanced Research Computing and Data Management Service 22
Appendix A: Market Analysis Methodology Canada s investment in advanced research computing technologies is compared with investment in five key competitor jurisdictions: United States and the United Kingdom: countries investing heavily in ARC; Germany and France: countries with strong traditions of industrial support; Germany s industrial sector is comparable in size to Canada s as a proportion of GDP; Australia: a country comparable in size to Canada, whose natural resources sector is comparable in size to Canada s as a proportion of GDP. ARC investment data is provided by the consulting firm International Data Corporation (IDC), which maintains country-by-country market statistics for high performance computing/data analytics technology investments. These statistics break down each country s investments by size of server (supercomputer, divisional, departmental, workgroup), industry/application area (bio-sciences, chemistry, computer-aided engineering, government, university/academic, defense, weather, etc.), and by associated investment (storage, middleware, applications, services). IDC provides historical market data for the period 2005-2014 and market projections for 2015-2019. For comparison purposes, investments are normalized in two ways: Investment (current US$) per $million of gross domestic product (GDP, current US$), provided by OECD; Investment (current US$) per researcher (FTE) overall and by sector (higher education, government and business), with data provided by UNESCO. Canadian normalized investments are compared to average normalized investments for the other five countries. Historically (2005-2014), Canada s investments are almost always less than the averages for the other countries, allowing historical underinvestment to be determined. Projected investments in the five comparison countries are used to estimate the investment required to maintain competitiveness. To be conservative, the comparison investments are calculated without including the United States. Details of this analysis are provided separately. The Scale of Canada s Need for Advanced Research Computing Services Although the technologies required to serve the needs of academia, industry and government are similar, the different structures of each sector must be considered in order to properly deliver services that would enable the level of innovation and competitiveness desired. To a large extent these services are comprised of technology services and research support services, scaled by the number of research groups to be supported in each sector. The National Advanced Research Computing and Data Management Service 23
Technology Services Technology services are largely scaled by the level of capital investment in each sector. Universities and Higher Education For academia, Compute Canada estimates the scale of investment required using both a bottom-up and top-down approach. Bottom-up. In late 2014 and early 2015, Compute Canada conducted a consultation across Canada, working with major academic disciplines such as chemistry, physics and genomics, to forecast the ARC resources each discipline would require to stay competitive internationally. 4 Compute Canada aggregated these forecasts and extended them to represent the full range of research disciplines. A technology plan was then developed, with constant annual investments, which allowed Compute Canada to expand its resources to meet the aggregate forecast demand. This plan projected a requirement for annual capital investment of Cdn$62 million per year beginning in 2016, when total resources would be expected to have caught up with demand, and future investments would only be needed to keep up with their growth. The bottom-up forecast methodology employed to derive this projection suggests that this estimate is conservative, since it incorporates demand requirements for only those groups who submitted projections and who knew their requirements well enough to make projections. By definition this forecast excludes new and emerging users and existing users who are less confident in projecting their future needs. (see Advanced Research Computing in Canada: An Overview (PDF). Top-down. Our top-down market analysis is based on methodology described above: historical ARC investment data, tracked by International Data Corporation (IDC) for five countries plus Canada, has been normalized for comparison purposes using GDP and counts of researchers by sector. This analysis projects that, in order to be competitive, Canada should make ARC capital investments of roughly Cdn$160 million per year from 2016 through 2019. Of this total, we estimate roughly 70% could be served by a national ARC capability, since there will inevitably be local investments designed to meet special needs and particular circumstances. This translates into an investment of roughly Cdn$110 million per year. These two projections are remarkably similar, given the very different methods used. Their agreement supports a projected capital investment of Cdn$60-$110 million per year, in order to meet academic needs in Canada at a competitive level. Since these needs are primarily driven by the Government of Canada s $2.5 billion annual investment in research and development conducted by the higher education sector, the proposed national strategy and investment plan assumes that these academic needs will be fully met through a national capability. Note that these market analyses also allow estimates of historical underinvestment in Canada versus comparator countries. For the academic sector, with approximately 60,000 researchers, underinvestment by the end of 2015 is estimated at Cdn$80 million. This means that a Cdn$80 million one-time investment would be needed to bring Canadian advanced research computing resources to a level comparable to the average level of resources available in Australia, France, Germany and the UK. 4 This consultation process Sustainable Planning for Advanced Research Computing (SPARC) is conducted regularly by Compute Canada. SPARC 2 is now underway and will yield updated information on the needs of the academic community by April 2016. The National Advanced Research Computing and Data Management Service 24
Private Sector For the private sector (industry), Compute Canada starts by estimating the overall scale of investment required using the top-down approach described above. This methodology suggests that overall capital investment by the Canadian private sector should approach Cdn$400 million per year from 2016 through 2019 in a robust investment scenario, compared to what IDC has forecasted for Canada averaging Cdn $280 million per year over that period. This suggests that various limits, capital constraints and/or market failure effects such as risk aversion will depress capital investment in this type of technology by roughly Cdn$120 million per year. This same data can be examined in another way, considering two types of underinvestment scenarios in the private sector: 1. Capital constraints on existing industrial users of advanced research computing. Even when a firm invests in advanced research computing, that firm must make trade offs with other demands on its capital: the firm cannot afford to invest in all the different kinds of ARC technologies that might make it more competitive, nor can it invest in systems scaled to accommodate its biggest research challenges. Instead, firms purchase systems that will meet most, but not necessarily all, of their needs; larger requirements, or requirements for technologies not purchased cannot be met with the firm s own resources. Compute Canada estimates that, of the projected robust investment level of Cdn$400 million per year, approximately 15%, or $60 million per year would correspond to such needs.(compute Canada s experience with academic research requirements indicates that roughly 30% of all compute resources demanded requires the very largest ARC systems. For the private sector, some of these demands would be met with existing investments, so we estimate 50% might be too large for those systems: 50% of 30% = 15%). 2. Risk aversion/lack of incentives for potential industrial users of advanced research computing. It is well-documented that SMEs in markets across the world are poor adopters of advanced research computing technologies and frequently miss opportunities to use those technologies to innovate and improve their competitiveness. A number of programs have been developed to improve SME adoption (e.g. iforge at the NCSA in the US, SHAPE and Fortissimo in Europe). With similar supports in Canada, the scale of usage by such new adopters could require investment equivalent to 10% of the robust investment level estimated above, or roughly $40 million per year. The investment required to address the examples of market failure described above totals approximately $100 million per year, which is consistent with the earlier estimate of a $120 million investment shortfall on an overall basis. Both of these figures are very approximate, and represent an upper boundary on the level of capital investment that might be appropriate as part of a national ARC strategy. As for the academic sector, these market analyses allow estimates of historical private sector underinvestment in Canada versus comparator countries. For the private sector, with approximately 100,000 researchers, underinvestment by the end of 2015 is estimated at Cdn$175 million. This means that the private sector would have to make a $175 million one-time investment to bring Canadian advanced research computing resources to a level comparable to the average level of private sector resources available in Australia, France, Germany and the UK. The National Advanced Research Computing and Data Management Service 25
Government For government usage, Compute Canada starts by estimating the overall scale of investment required using the top-down approach described above, supporting government researchers with levels of ARC resources comparable to those made available in other countries. This methodology suggests that overall capital investment by Canadian government labs (not including defense and production weather forecasting) should approach Cdn$140 million per year from 2016 through 2019 in a competitive investment scenario, compared to what IDC has forecasted for Canada, averaging Cdn$50 million per year over that period. This is a significant estimated shortfall. While this is a very high level estimate, and should be validated by other means, it does raise questions regarding the level of support provided to this important segment of the Canadian science and innovation ecosystem. Of this total, we estimate roughly 70% could be served by a national ARC capability, since there will inevitably be local investments designed to meet special needs and particular circumstances. This translates into an investment of roughly Cdn$95 million per year. As for the academic sector, these market analyses allow estimates of historical government underinvestment in Canada versus comparator countries. For government (not including defense and production weather forecasting) with approximately 10,000 researchers, underinvestment by the end of 2015 is estimated at Cdn$135 million. This means that government would have to make a $135 million one-time investment to bring Canadian government advanced research computing resources to a level comparable to the average level of resources available to government researchers in Australia, France, Germany and the UK. Research Support Services Research support services are largely scaled by the number of research groups to be supported in each sector. Universities and Higher Education Compute Canada currently serves the needs of the Canadian academic community with a team of roughly 200 highly skilled individuals employed by and located at 35 institutions across Canada. Most of those 200 individuals are involved in research support activities to varying degrees; on a full time equivalent basis, roughly 85 FTE team members provide research support to just over 3,000 research groups across Canada. The ratio of one (1) research support team member per 35 research groups reflects the diversity of disciplines and sophistication levels in Canadian academia. Some research groups are very productive with limited support, while others require higher levels of support. 3,000 research groups represents approximately 5% of the Canadian academic research community in 2015. Since 2011, the number of Compute Canada s research users have been growing 16.5% per year. Compute Canada projects that the number of academic research groups will grow to over 5,500 by 2019, representing almost 7.5% of the Canadian academic research community in that year. Compute Canada is currently restructuring its research support activities in order to improve service while at the same time serving growing numbers of research users. This will be accomplished in part through better training, better documentation and tutorials, and greater pan-canadian coordination of support efforts. With these initiatives, Compute Canada projects that research support staff dedicated to academic users will be able to grow the number of research groups each support professional can assist from roughly 35 groups to 50 groups by 2019, limiting growth of the overall support category from a current 85 FTEs, to roughly 110 FTEs in 2019. The National Advanced Research Computing and Data Management Service 26
A key component of this strategy will involve creation of several centres of expertise in key areas such as molecular dynamics, bioinformatics, fluid dynamics, finite element modelling, multi-physics simulation, etc. These centres of expertise will support researchers using these techniques, regardless of whether they are from academia, industry or government. Compute Canada has already started this process; by 2019, total staff involved in these centres of expertise are expected to reach 50 FTEs nationwide. Private Sector and Government For these sectors, Compute Canada has limited baseline data with which to estimate research support demands. However, if the projected technology services (capital investment) are consumed by these two sectors in proportion to their growth, each sector might be expected to have as many research groups using their proportion of investment as is projected for the academic sector that is, an expected 5,500 research groups in 2019 in each of the private and government sectors. For the private sector, this represents adoption of digital research techniques by a growing proportion of private sector researchers, ranging from 1.6% in 2016 to 5.5% by 2019. For government researchers, adoption ranges from 15% in 2016 to 53% in 2019. (These projections should be validated through further market assessments). These projected levels of adoption might argue for an additional 120 research support staff dedicated to the needs of each sector, however several factors will limit this requirement: Private sector firms and government labs should have some personnel responsible for their digital research efforts. These individuals will need training and assistance, but the objective should be selfsufficiency. The academic research support organization described above can improve the efficiency of these activities: On-campus support personnel are geographically distributed, allowing private sector and government needs to be met locally (up to a point); The centres of expertise will be available to private sector and government users, improving the effectiveness and productivity of their own support staff. Based on these factors, we project that research support in 2019 will require only half the personnel of what would otherwise be required (120 in total). In the early years, roughly 40-60 dedicated support personnel will be required, but initial support demands can be met with academic research support teams, improving the efficiency of start up, as well as increasing interactions among sectors. The National Advanced Research Computing and Data Management Service 27
Appendix B: What is Advanced Research Computing? Advanced research computing (ARC) is the engine of the digital research infrastructure (DRI) in Canada. ARC is defined as including high performance computing (supercomputing), cloud computing, mass data storage, access and archiving, as well as the associated services, software and middleware. ARC provides the needed technical expertise and research support to ensure efficient, effective and productive use of ARC resources. The major ARC elements are: Systems - high performance computing coupled with data management and archiving infrastructure. Such systems typically have a useful life cycle of 4 to 6 years. Services - customized software, middleware, data management tools, cloud computing, portals, infrastructure services, and other components by which systems are able to provide valuable outcomes. Experts - ARC systems and services require developers, system administrators, scientific computing and research support experts, trainers, and experts in communicating scientific results to stakeholders. Today, Compute Canada coordinates a national team of almost 200 experts, employed by and working at institutions across Canada, who offer unparalleled expertise and skills to users in a wide range of disciplines. This skill set is already in demand by the private sector and government researchers. Networks - Within Canada, data flows over the networks provided by CANARIE, coupled to campus and regional networks. These networks connect systems and services, and allow resource consolidation while ensuring ubiquitous access. The National Advanced Research Computing and Data Management Service 28
Appendix C: A Portfolio of Architectures to Serve All Sectors We expect user needs from all sectors to map onto a chart similar to the one below, which plots different computational jobs against the number of CPUs required to execute the job, and the total amount of time required to complete the job. Although this chart simplifies many important details, it highlights the fact that there are different classes of job which are most efficiently executed on different types of ARC technology. At the far right, peak jobs require more traditional highly-parallel capability computing. In the middle breadth jobs require grid-style systems that are more loosely coupled. On the left, capacity jobs can run on any type of system since they can be scheduled in between larger jobs on larger systems almost without cost. They can also be executed on cloud type systems (including commercial cloud systems) if there are capacity constraints and/ or unacceptable wait times on other larger systems. Security will also be critical as Canada s overall needs are assessed and a national technology plan is developed. By managing a portfolio of architectures, an organization can serve a variety of needs. As shown in the chart, some research areas (e.g., chemistry) include all three job types. Some academic researchers need to move between systems during different parts of their research lifecycle. This same feature makes the system attractive to industry users who may wish to test their software on a variety of systems without needing to invest in a wide range of hardware types. The National Advanced Research Computing and Data Management Service 29