Scaling species tree estimation methods to large datasets using NJMerge

Transcription

1 Scaling species tree estimation methods to large datasets using NJMerge Erin Molloy and Tandy Warnow {emolloy2, University of Illinois at Urbana Champaign 2018 Phylogenomics Software Symposium August 17, 2018 Molloy & Warnow NJMerge 1/48

2 Large-scale Phylogeny Estimation is computationally intensive A New View of the Tree of Life [Hug et al., 2016] 3,083 genomes 2,596 AA positions from concatenating 16 protein alignments RAxML with 156 bootstrap replicates 3,840 CPU hours on the CIPRES supercomputer Molloy & Warnow NJMerge 2/48

3 NJMerge NJMerge can enable species tree methods (ASTRAL-III, SVDquartets, and RAxML) to analyze large and/or hetergeneous datasets using a single compute node with 16 cores 64 GB of physical memory 48 hours maximum wall-clock time without sacrificing accuracy. Molloy & Warnow NJMerge 3/48

4 Divide-and-Conquer Methods [e.g., Huson et al., 1999; Nelesen et al., 2012] Molloy & Warnow NJMerge 4/48

5 Divide-and-Conquer Methods [e.g., Huson et al., 1999; Nelesen et al., 2012] Molloy & Warnow NJMerge 5/48

6 Divide-and-Conquer Methods [e.g., Huson et al., 1999; Nelesen et al., 2012] Molloy & Warnow NJMerge 6/48

7 Divide-and-Conquer Methods [e.g., Huson et al., 1999; Nelesen et al., 2012] Molloy & Warnow NJMerge 7/48

8 Divide-and-Conquer Methods Subset trees can be estimated using a highly accurate (but computationally intensive) method and then combined using a supertree method. Molloy & Warnow NJMerge 8/48

9 Divide-and-Conquer Methods [e.g., Huson et al., 1999; Nelesen et al., 2012] Molloy & Warnow NJMerge 9/48

10 Divide-and-Conquer Methods [Steel, 1992; Jiang et al., 2001; Bansal et al., 2010] Molloy & Warnow NJMerge 10/48

11 Divide-and-Conquer Methods [e.g., Huson et al., 1999; Nelesen et al., 2012] Molloy & Warnow NJMerge 11/48

12 New Divide-and-Conquer Strategy [e.g., Huson et al., 1999; Nelesen et al., 2012] Molloy & Warnow NJMerge 12/48

13 New Divide-and-Conquer Strategy Subset trees can be estimated using a highly accurate (but computationally intensive) method and then merged together using a fast (but potentially less accurate) method. Molloy & Warnow NJMerge 13/48

14 New Divide-and-Conquer Strategy [e.g., Huson et al., 1999; Nelesen et al., 2012] Molloy & Warnow NJMerge 14/48

15 Merging disjoint trees A D + E H B C F G Merging G H Joining with single edge A D A H F C B C D E F G B E Molloy & Warnow NJMerge 15/48

16 Merging disjoint trees Given a set T of pairwise disjoint trees with combined leaf set S, we seek a compatibility supertree for T that is close to the true tree on S. NJMerge attempts to do this by using a dissimilarity matrix D. Molloy & Warnow NJMerge 16/48

17 NJMerge NJMerge is a polynomial-time extension of Neighbor-Joining [Saitou and Nei, 1987] takes a set of trees (on pairwise disjoint subsets of the leaf set) as input that operate as constraints on the output tree. Input: Output: Dissimilarity matrix D on species set S Set of k unrooted binary trees T = {T 1, T 2,..., T k } on pairwise disjoint subsets of the species set S Unrooted binary tree T on the species set S that agrees with every tree in T Molloy & Warnow NJMerge 17/48

18 NJMerge NOTE: Distance matrix is additive for (((A, B), (C, D)), E, (F, (G, H))); Molloy & Warnow NJMerge 18/48

19 NJMerge NOTE: Distance matrix is additive for (((A, B), (C, D)), E, (F, (G, H))); Molloy & Warnow NJMerge 19/48

20 NJMerge NJ is an agglomerative technique that joins pairs of nodes at each iteration and reduces the number of remaining nodes by one. NJMerge differs from NJ in that it only accepts a siblinghood proposal (x, y) if and only if making x and y siblings does not violate the pairwise compatibility of the set T of constraint trees. Molloy & Warnow NJMerge 20/48

21 NJMerge Determining the compatibility of a set of unrooted phylogenetic trees is NP-complete [Steel, 1992; Warnow, 1994]. Thus, NJMerge uses a polynomial-time heuristic: Update constraint trees based on siblinghood proposal (x, y) Test whether the set of constraint trees that contain leaf (x, y) are compatible in polynomial time by rooting these trees at leaf (x, y) [Aho et al., 1981] Not every constraint tree will contain the leaf (x, y), so this does not a guarantee that the set of k constraint trees is compatible. It is possible for NJMerge to fail. Molloy & Warnow NJMerge 21/48

22 NJMerge Determining the compatibility of a set of unrooted phylogenetic trees is NP-complete [Steel, 1992; Warnow, 1994]. Thus, NJMerge uses a polynomial-time heuristic: Update constraint trees based on siblinghood proposal (x, y) Test whether the set of constraint trees that contain leaf (x, y) are compatible in polynomial time by rooting these trees at leaf (x, y) [Aho et al., 1981] Not every constraint tree will contain the leaf (x, y), so this does not a guarantee that the set of k constraint trees is compatible. It is possible for NJMerge to fail. Molloy & Warnow NJMerge 22/48

23 NJMerge Advantages No supertree estimation Polynomial Time Parallel, e.g., each siblinghood proposal can be evaluated in parallel Disadvantages Can fail to return a tree Molloy & Warnow NJMerge 23/48

24 Species Tree Estimation Molloy & Warnow NJMerge 24/48

25 Using NJMerge in species tree estimation pipeline Molloy & Warnow NJMerge 25/48

26 Using NJMerge in a pipeline for species tree estimation How should the distance matrix be computed? How large should the maximum subset size be? How should subsets be created? Which species tree method should be used on subsets? Molloy & Warnow NJMerge 26/48

27 Simulated Datasets Dataset Size 100 species; 25, 100, 1000 genes 1000 species; 1000 genes Gene (alignment) length varied from 300 to 1500 bp Level of Incomplete Lineage Sorting (ILS) Moderate: 8-10% AD Very high: 68-69% AD where AD is the Average (RF) Distance between the true species tree and the tree gene trees. Sequence Type Exons: lower phylogenetic signal (not shown) Introns: greater phylogenetic signal Molloy & Warnow NJMerge 27/48

28 How does error in the estimated distance matrix impact NJMerge? 100 taxa, Moderate ILS 100 taxa, Very High ILS 0.30 Species Tree Error Number of Genes NJst NJMerge+True NOTE: NJMerge failed to return a tree on 1 dataset with 100 taxa, 25 genes, and very high ILS; average is on datasets for which both methods completed. Molloy & Warnow NJMerge 28/48

29 Evaluation Subset trees are estimated using ASTRAL-III, SVDquartets, and unpartitioned concatenation using RAxML. Metrics Robinson-Foulds (RF) distance between the true and the estimated species trees Running time All methods limited to 48 hours wall-clock time one compute node with 16 cores 64 GB of physical memory Molloy & Warnow NJMerge 29/48

30 Running time for pipeline using NJMerge T = ( T1 Method + + Tk Method ) + T NJMerge where k = number of subsets T Method i = time to compute tree on subset i for given method T NJMerge = time to run NJMerge Molloy & Warnow NJMerge 30/48

31 ASTRAL [Mirarab et al., 2014; Mirarab and Warnow, 2015; Zhang et al., 2018] Gene tree summary method Finds optimal solution to search problem within a constrained search space Computationally intensive when Constrained search space is large, e.g., large datasets with high levels of gene tree heterogeneity due to ILS or gene tree estimation error Molloy & Warnow NJMerge 31/48

32 ASTRAL-III and NJMerge Species Tree Error species, 1000 introns 1000 species, 1000 introns Moderate Very High Moderate Very High ASTRAL Level of ILS NJMerge+ASTRAL NOTE: ASTRAL completed on 16/20 datasets with 1000 taxa and very high ILS; NJMerge+ASTRAL completed on 20/20 of these datasets; box plots show all datasets on which at least one method completed. Molloy & Warnow NJMerge 32/48

33 ASTRAL-III and NJMerge 100 species, 1000 introns 1000 species, 1000 introns Running Time (m) Moderate Very High Moderate Very High ASTRAL Level of ILS NJMerge+ASTRAL NOTE: ASTRAL completed on 16/20 datasets with 1000 taxa and very high ILS; NJMerge+ASTRAL completed on 20/20 of these datasets; bar graphs show all datasets on which at least one method completed. Molloy & Warnow NJMerge 33/48

34 SVDquartets [Chifman and Kubatko, 2014] Site-based coalescent method Use the concatenated alignment to estimate quartet trees and then combine the quartet trees Computationally intensive when Number of species is large Molloy & Warnow NJMerge 34/48

35 SVDquartets and NJMerge Species Tree Error species, 1000 introns 1000 species, 1000 introns Moderate Very High Moderate Very High SVDquartets Level of ILS NJMerge+SVDquartets NOTE: SVDquartets ran on 0/40 datasets with 1000 taxa due to segmentation faults; NJMerge+SVDquartets completed on 40/40 of these datasets; box plots show all datasets on which at least one method completed. Molloy & Warnow NJMerge 35/48

36 SVDquartets and NJMerge Running Time (m) species, 1000 introns 1000 species, 1000 introns Moderate Very High Moderate Very High SVDquartets Level of ILS NJMerge+SVDquartets NOTE: SVDquartets ran on 0/40 datasets with 1000 taxa due to segmentation faults; NJMerge+SVDquartets completed on 40/40 of these datasets; bar graphs show all datasets on which at least one method completed. Molloy & Warnow NJMerge 36/48

37 Unpartitioned concatenation using RAxML [Stamatakis, 2014] ML tree estimation on unpartitioned concatenated alignment Not statistically consistent under MSC model Computationally intensive when Number of species is large Number of unique site patterns in alignment is large, i.e., cannot effectively compress the alignment Molloy & Warnow NJMerge 37/48

38 RAxML and NJMerge Species Tree Error species, 1000 introns 1000 species, 1000 introns Moderate Very High Moderate Very High RAxML Level of ILS NJMerge+RAxML NOTE: RAxML completed on 1/40 datasets with 1000 taxa due to Out of Memory errors; NJMerge+RAxML completed on 40/40 of these datasets; box plots show all datasets on which at least one method completed. Molloy & Warnow NJMerge 38/48

39 RAxML and NJMerge 100 species, 1000 introns 1000 species, 1000 introns Running Time (m) Moderate Very High Moderate Very High RAxML Level of ILS NJMerge+RAxML NOTE: RAxML completed on 1/40 datasets with 1000 taxa due to Out of Memory errors; NJMerge+RAxML completed on 40/40 of these datasets; bar graphs show all datasets on which at least one method completed. Molloy & Warnow NJMerge 39/48

40 Summary NJMerge is a generic technique for scaling phylogeny estimation methods to large and/or hetergeneous datasets. In our experiments, NJMerge improved running time without sacrificing accuracy enabled all three methods to run on large and/or heterogeneous when constrained by memory (64GB) and running time (48 hours) had a low failure rate (< 1%) Molloy & Warnow NJMerge 40/48

41 Which species tree method do I use on datasets with 1000 taxa and 1000 genes? Species Tree Error Moderate ILS Very High ILS Method NJMerge+ASTRAL NJMerge+RAxML NJMerge+SVDquartets NJst It depends on the dataset! Molloy & Warnow NJMerge 41/48

42 Which species tree method do I use on datasets with 1000 taxa and 1000 genes? Species Tree Error Moderate ILS Very High ILS Method NJMerge+ASTRAL NJMerge+RAxML NJMerge+SVDquartets NJst It depends on the dataset! Molloy & Warnow NJMerge 42/48

43 Conclusions NJMerge enables species tree methods (ASTRAL-III, SVDquartets, and RAxML) to analyze large and/or hetergeneous datasets using a single compute node with 16 cores 64 GB of physical memory 48 hours maximum wall-clock time without sacrificing accuracy. How to best run NJMerge depends on the dataset! Several algorithmic choices are left to the discretion of the user. Molloy & Warnow NJMerge 43/48

44 Funding This work was supported by the National Science Foundation Graduate Research Fellowship Program (DGE ) Blue Waters Sustained-Petascale Computing Project (OCI and ACI ) Graph-Theoretic Algorithms to Improve Phylogenomic Analyses (CCF ) and the Ira & Debra Cohen Graduate Fellowship in Computer Science. Molloy & Warnow NJMerge 44/48

45 Use NJMerge! Paper to appear in proceedings of RECOMB-CG Available on Github: github.com/ekmolloy/njmerge Molloy & Warnow NJMerge 45/48

46 References Hug et al. (2016). A new view of the tree of life. Nature Microbiology 1: Huson et al. (1999). Disk-covering, a fast-converging method for phylogenetic tree reconstruction. Journal of Computational Biology, 6(3-4): Nelesen et al. (2012). DACTAL: Divide-And-Conquer Trees (almost) without Alignments. Bioinformatics 28(12):i274 i282. Steel. (1992). The complexity of reconstructing trees from qualitative characters and subtrees. Journal of Classification 9: Jiang et al. (2001). A polynomial time approximation scheme for inferring evolutationay trees from quartet topologies and its application. SIAM Journal on Computing 30(6): Molloy & Warnow NJMerge 46/48

47 References Bansal et al. (2010). Robinson-Foulds Supertrees. Algorithms for Molecular Biology 5(1). Saitou and Nei. (1987). The neighbor-joining method: a new method for reconstruction of phylogenetic trees. Molecular Biology and Evolution 4: Warnow (1994). Tree compatibility and inferring evolutionary history. Journal of Algorithms, 16(3): Agho et al. (1981). Inferring a Tree from Lowest Common Ancestors with an Application to the Optimization of Relational Expressions. SIAM Journal on Computing, 10(3): Mirarab et al. (2014) ASTRAL: genome-scale coalescent-based species tree estimation. Bioinformatics, 30(17):i541 i548. Molloy & Warnow NJMerge 47/48

48 References Mirarab and Warnow. (2015). ASTRAL-II: coalescent-based species tree estimation with many hundreds of taxa and thousands of genes. Bioinformatics, 31(12):i44 i52. Zhang et al. (2017). ASTRAL-III: Increased Scalability and Impacts of Contract- ing Low Support Branches. In Meidanis and Nakhleh, editors, Comparative Genomics. RECOMB- CG Lecture Notes in Computer Science, volume Springer, Cham. Chifman and Kubatko. (2014). Quartet Inference from SNP Data Under the Coalescent Model. Bioinformatics, 30(23): Stamatakis. (2014). RAxML Version 8: A tool for Phylogenetic Analysis and Post-Analysis of Large Phylogenies. Bioinformatics, 30(9). Molloy & Warnow NJMerge 48/48