Enhancing population pharmacokinetic modeling efficiency and quality using an integrated workflow

Transcription

1 DOI /s ORIGINAL PAPER Enhancing population pharmacokinetic modeling efficiency and quality using an integrated workflow Henning Schmidt Andrijana Radivojevic Received: 16 May 2014 / Accepted: 8 July 2014 Ó Springer Science+Business Media New York 2014 Abstract Population pharmacokinetic (poppk) analyses are at the core of Pharmacometrics and need to be performed regularly. Although these analyses are relatively standard, a large variability can be observed in both the time (efficiency) and the way they are performed (quality). Main reasons for this variability include the level of experience of a modeler, personal preferences and tools. This paper aims to examine how the process of poppk model building can be supported in order to increase its efficiency and quality. The presented approach to the conduct of poppk analyses is centered around three key components: (1) identification of most common and important poppk model features, (2) required information content and formatting of the data for modeling, and (3) methodology, workflow and workflow supporting tools. This approach has been used in several poppk modeling projects and a documented example is provided in the supplementary material. Efficiency of model building is improved by avoiding repetitive coding and other laborintensive tasks and by putting the emphasis on a fit-forpurpose model. Quality is improved by ensuring that the workflow and tools are in alignment with a poppk modeling guidance which is established within an organization. The main conclusion of this paper is that workflow based approaches to poppk modeling are feasible and have significant potential to ameliorate its various aspects. However, the implementation of such an approach in a pharmacometric organization requires openness towards innovation and Electronic supplementary material The online version of this article (doi: /s ) contains supplementary material, which is available to authorized users. H. Schmidt (&) A. Radivojevic Advanced Quantitative Sciences, Novartis Pharma AG, Postfach, 4002 Basel, Switzerland henning.schmidt@novartis.com change the key ingredient for evolution of integrative and quantitative drug development in the pharmaceutical industry. Keywords Population pharmacokinetics Modeling Workflow Efficiency Quality Dataset specification Introduction Population pharmacokinetic modeling constitutes a substantial area of Pharmacometrics [1] and is an integral part of model-based drug development [2]. In general terms, the main purpose of poppk modeling is to identify the central tendencies, the uncertainty and variability in the pharmacokinetic (PK) parameters, and the influence of covariates [3, 4]. PopPK modeling activities need to be performed regularly during drug development from first-in-human dose selection to submission and beyond [5 7]. In this paper it is assumed that the rational for a specific poppk modeling activity is well defined and the focus is set on the efficient, streamlined, and systematic conduct of poppk analyses. The planning and reporting parts are outside of scope. Anecdotal evidence indicates that lack of meeting the time constraints often limits the extent of pharmacometric involvement in the industry [8]. Although this quote is not specifically addressing poppk analyses, it is well known that these analyses can take a considerable amount of time, binding resources that are then not available to support other aspects of model-based drug development [9, 10]. It is not implicitly assumed that each poppk analysis that requires a long time is not done efficiently and it is not in scope of this paper to assess the manifold reasons why sometimes such analyses take longer. However, one of the stated aims of this

2 Table 1 Relative frequency of poppk model elements, based on internal survey and not considering binding models (N = 52) Model features Number of compartments Other 0 Model outputs Single 94 Multiple (e.g., plasma and tissue concentrations, 6 metabolites) Elimination Linear, from central compartment, time 92 independent elimination rate constant Saturable or linear? saturable, from central 8 compartment, time independent elimination rate constant, saturable part modelled by Michaelis Menten term Other 0 Bioavailability Linear (covariates on F are allowed) 44 Nonlinear (F as nonlinear function of dose or 2 concentration, etc.) Not considered in the model 52 Other 2 Absorption First-order, time independent 71 Zero-order, time independent 0 Mixed zero/first order, time independent 8 Not considered in the model 21 Other 0 Absorption delay Not considered in the model 86 Lag time 10 Transit compartment 4 Other 0 Distribution Linear, time independent 100 Nonlinear or time dependent 0 Other nonlinearities None 96 Saturable red blood cell binding 2 Self-induction, self-inhibition 2 Other 0 Time dependent covariates No 100 Yes 0 Inter occasion variability No 85 Yes 15 Relative frequency (%) paper is to address improvement of efficiency in the conduct of poppk analyses. Here, efficiency means not only a potential decrease of time for turn-around of such analyses, but also that the conduct of these analyses should be supported by methodology and tools. This will allow the modeler to focus more on the interpretation of modeling results and to assess which model fits the purpose of interest best, rather than to spend a lot of time on getting the best model and performing overly repetitive tasks. Another challenge with poppk analyses lays in the relatively large inter-modeler variability of how such analyses might be conducted, all due to the modeler s experience, expertise, and employed tools. Byon and colleagues [2] already pointed out the need for standard procedures, at least across the same organization, in order to achieve a more consistent high quality output of poppk analyses. There are many aspects that contribute to the quality of these analyses. Here in this paper a minimum level of quality is considered and defined by the following requirements: All poppk modeling activities within the same organization are performed following the same internal poppk modeling guidance. Analysis results should be independent of the modeler who is conducting the analysis. Code, models, data that were used for the analysis should be easily accessible, self-explaining, and executable by everyone without the need for interaction with the original modeler. In this paper it is assumed that an internal poppk modeling guidance is in place, covering not only all aspects of the published guidances from health authorities [11, 12], but also more in depth methodology and modeling workflow. Efficiency and quality of pharmacometric analyses have already been considered throughout the literature. Bonate and colleagues [13] present detailed guidelines for quality control of population pharmacometric analyses, with a focus on NONMEM [14]. However, many of the discussed elements are equally applicable or adaptable with respect to other tools. An example for the consideration of efficiency is the work by Jönsson and Jonsson [10], which discusses important requirements for efficient conduct of general pharmacometric activities, including planning, execution, and reporting. Focusing more on the execution part of poppk modeling activities, other authors acknowledged the need for flexible integration of the extensive choice of software platforms and available tools (see, e.g., [15] ). Bergsma and colleagues [16] presented an implementation of such an integrated framework in R language [17] and Ridolfi and co-workers [18] proposed a web-based approach [18]. Both [16] and do not cover the aspect of a modeling guidance or

3 modeling methodology, which is critical when considering consistency and/or quality of results across an organization. Automatic poppk model generation is often associated with the term efficiency and some efforts were made in the past to achieve this target. Schaap et al. [19] used a genetic algorithm approach for automatic model building, still preserving the freedom of a modeler to take certain decision in between. However, main drawbacks of general optimization based approaches to searching the space of candidate models are the definition of the cost-function and the definition of the model space to search at a given time. A recent contribution from Byon and colleagues [2] presents a detailed poppk modeling guidance, focusing on systematic, streamlined, and standardized approaches to poppk modeling in order to optimize and harmonize the processes across their organization. However, the authors do not go into more detail regarding the support of their poppk workflow with associated workflow tools. It is important to realize, however, that a workflow based only on poppk guidance and definition of the methodology, might not be the most efficient if not supported by workflow tools. Equally, a platform that integrates all available tools might not lead to consistent high quality modeling results if no guidance exists on how to perform the analysis. This paper presents a general approach that addresses the linking of a modeling guidance with efficient workflow tools, to allow for an increase in efficiency and quality of the conduct of poppk analyses. This approach can be considered not only tool independent, but also independent on a specific modeling methodology or poppk guidance. Implementation is possible in different scripting languages, with different nonlinear mixed effect modeling tools at the back-end. Nevertheless, in order to demonstrate the usefulness of the approach, a specific platform had to be chosen for a first implementation of the workflow tools. In this case the platform of choice consists of MATLAB [20], SBPOP [21], MONOLIX [22] and NONMEM [14]. Depending on the preferences of an organization, modifications can be done on all parts. This paper is structured as follows: section Definition of considered poppk model features summarizes the results of an internal survey about the most common features in poppk models, allowing definition of the features that a workflow minimally needs to cover. A generalized dataset specification is presented in section Generalized dataset format, allowing to capture the modeling relevant information, without being tool or modeling task specific. The modeling workflow and associated workflow tools are presented in section PopPK modeling workflow and workflow tools.this is followed by a discussion and conclusion. A complete example poppk analysis on simulated data, using the presented workflow with both NONMEM and MONOLIX, is made available in the supplementary material, allowing the Table 2 Selected poppk model/modeling features that are considered in the workflow Model feature Type of model Number of compartments Model outputs Elimination Bioavailability Absorption Absorption delay Distribution Individual parameter distributions Residual error models Fixed and random effects Covariance model Covariate model Options considered in workflow Compartmental models (binding models out of scope) 1, 2, 3 Single Linear and linear? saturable (Michaelis Menten) from central compartment, time independent Linear First-order Lag-time Linear Logit-normal for relative bioavailability, lognormal for other PK parameters Additive, proportional, additive/proportional, exponential Allowing for estimation and fixing of values for selected parameters Diagonal, block-diagonal and full correlation matrices for random effects Allowing estimation and fixing of selected covariate coefficients interested reader to repeat the example analysis and to adapt it to own projects. The supplementary material is made available as a zip file. The contents can be accessed by opening the file Supplementary Material.html in a web browser of choice. Definition of considered poppk model features Pharmacometric modeling is often considered to be an art, and a typical argument against a more industrialized form of model building is that all models are different. However, more concrete data on how different poppk models can be is generally not available. Prior to embarking on the development of the presented poppk modeling workflow and associated tools it is important to characterize the poppk model space of interest and to select a certain subspace to be considered by the workflow. The chosen approach to determine the typical elements of poppk models was to conduct an internal poppk model survey by interviews with colleagues in the Advanced Quantitative Sciences group (AQS, former Modellng&Simulation). The limitation to an internal survey, rather than conducting a full literature review, was intentional and allows focusing on fit-for-purpose models that

4 Table 3 Format of generalized dataset Dataset section Column name Definition Type Identification of USUBJID Unique subject identifier String subject STUDY Study number Numerical CENTER Centre number Numerical SUBJECT Subject number Numerical Study information PART Part of study (if parts present, otherwise 0) Numerical INDICATION Indication flag Numerical INDICATION_NAME Indication name String Treatment group TRT Unique treatment identifier Numerical information TRT_NAME Analyst given treatment name String Visit information VISIT Visit number Numerical VISIT_NAME Visit name as coded in the clinical database String BASE Flag indicating assessments at baseline (= 0 for non-baseline, = 1 for first Numerical baseline, 2 = for second baseline, etc.) SCREEN Flag indicating assessments at screening (= 0 for non-screening, = 1 for first Numerical screening, 2 = for second screening, etc.) Event time DATE_DAY Date of event String information DATE_TIME Time of event String NOMINAL_TIME Planned time of event, based on protocol in the time unit, defined in TIME_UNIT Numerical column TIME Actual time of event in the time unit, defined in TIME_UNIT column. Time 0 Numerical defined as time of first active or placebo dose TIME_UNIT Unit of all numerical time definitions in the dataset (e.g., hours or days ) String Event value TYPE See Table 4 Numerical information TYPE_NAME See Table 4 String SUBTYPE See Table 4 Numerical VALUE See Table 4 Numerical VALUE_TEXT Text version of value (e.g. male or female for gender ) String UNIT Unit of the value reported in the VALUE column. For same event the same unit has String to be used across the dataset NAME See Table 4 String DURATION See Table 4 Numerical ULOQ See Table 4 Numerical LLOQ See Table 4 Numerical Dose event additional information ROUTE See Table 4 String INTERVAL Interval of dosing in time units, defined in the TIME_UNIT column allows for Numerical coding repeated dosing more efficiently NR_DOSES Number of doses given with the specified interval allows for coding repeated dosing more efficiently Numerical are developed on a day-to-day basis to support decision making in internal drug development projects. Interviews were conducted either personally, via phone, or . The focus was only on the models and their features and not on the modeling methodology nor on the tools used for the modeling. The survey resulted in data for 58 poppk modeling activities (43 different compounds, otherwise data from different development phases and/or different modeling approaches compartmental/binding), of which 6 were using binding models and 52 were compartmental models. Table 1 summarizes the results of the survey in terms of the relative frequency for poppk model elements for the 52 compartmental models. Additional observations from the survey: All used proportional, additive/proportional, or exponential residual error models. Individual PK parameters are always assumed to be lognormally distributed, except in some cases the bioavailability for which a logit-normal distribution was used.

5 Fixed and random effect parameters and covariate coefficients were estimated or fixed to specific values, based on assumptions or prior knowledge. Transit compartments were used only on data with very rich sampling in the absorption phase, which sometimes was available in phase I studies. Based on the survey, poppk model features to be considered for the development of the workflow and the associated workflow tools were identified and listed in Table 2. Based on the selection in Table 2, % of the possible model features in each category in Table 1 are considered. Although this might still seem somewhat limited, this selection itself does not impact the generality of the presented workflow approach. Focusing on the elements of main importance allowed a first development of the workflow and implementation of associated tools. Extension of the workflow to additional model features is straight-forward and can be considered on a need basis. Generalized dataset format Within AQS, modeling datasets for the conduct of pharmacometric analyses are usually prepared by the modeler him/herself or requested from a supporting programming group. The content and the structure of such datasets are defined in a dataset specification, which typically depends on the modeling activity, the modeling tool, the models to be assessed, and even on the modeler. In the context of model-based drug development often many different analyses, or pharmacometric activities, need to be performed within the same project. This includes, but is not limited to, graphical exploration, doseconcentration and concentration response modeling, where many different biomarkers or endpoints might need to be considered. Requesting a separate modeling dataset for each modeling activity considerably strains the resources of the modeler who prepares these datasets or of the supporting programming group, especially in the case when a certain level of validation is required. In this paper, a generalized dataset format is proposed (Table 3), covering PK as well as pharmacodynamic (PD) data. Requirements for this dataset format have been a welldefined structure that is project independent (same across projects, compounds, and indications), independent of the modeling activity to be performed, and independent of the modeling tool to be used. Additionally, the format had to have a minimum level of redundancy and include information that is typically not found within pharmacometric analysis datasets: names for variables, units, etc. This renders the dataset more understandable without additional documentation, reduces mistakes, and facilitates project hand-over processes. The presented dataset format can be seen as an extension to the notion of a master modeling dataset, discussed in [10]. Each row in this dataset corresponds to an event, i.e., to a given dose or an observation (compound or metabolite concentration, biomarker or endpoint for safety or efficacy, adverse event). Extensions to other types of events are possible but not in scope of this paper. The standardized structure allows for more efficient and less error-prone preparation of datasets, pooling across studies, easier understanding and re-use of the data, automatic generation of the most common graphical exploration plots, and automatic conversion to modeling activity and potentially tool dependent modeling datasets. Many of the data consistency checks that are suggested in [13] can be done automatically based on such a general data format. To seamlessly integrate this format into the presented poppk workflow, a minimum set of standards need to be defined (Table 4). The main drawback of such a master dataset approach is the non-negligible size in large drug development projects. It cannot be excluded that for very specific poppk analyses manual modifications of an automatically generated modeling dataset need to be done. However, all poppk modeling features considered in this paper (Table 2) are captured by the approach of a generalized dataset format. Additional features can be handled when needed, by revising and extending the format, including updating of the relevant workflow tools. PopPK modeling workflow and workflow tools Figure 1 shows a high level poppk workflow. The only difference to traditional approaches is the use of the generalized dataset format, rather than starting directly from a tool and modeling activity dependent modeling dataset. In this section, the different steps of the workflow and their support by workflow tools are discussed in more detail. Due to practicability, the workflow tools cannot be exemplified based on a complete poppk example throughout this paper. Pseudo-code and summary results will be shown instead, where applicable. For a complete overview of the workflow tools and their functioning, the reader is referred to the supplementary material. As with the model features in Table 1, a workflow tool based approach should not and cannot aim at covering 100 % of the possible analyses or features. However, this does not represent a limitation of the approach. At each stage the modeler can perform additional analyses, if these are needed. Over time, if an analysis that is not yet covered becomes more often used, it could be included into the workflow tool.

6 Table 4 Standards in the dataset required for the presented poppk workflow Column name Definitions and conventions Standards for poppk workflow TYPE TYPE_NAME Numerical value for grouping of certain types of events. Required groups are 0 (for dosing events) and 1 (for PK observations). Additional groups could consist of demographics, metabolites, target capture, mechanism of action (MoA) biomarkers, clinical endpoints, etc Textual representation of the corresponding TYPE value. No standard defined. For dosing events Dose might be used and for PK observation events PK TYPE = 0 corresponds to dosing events, and TYPE = 1to PK measurement events SUBTYPE Numerical identifier for a specific event within a group If TYPE = 0: SUBTYPE = 1 corresponds to dosing events of the first compound, SUBTYPE = 2 to the second, etc If TYPE = 1: SUBTYPE = 1 corresponds to the observed PK concentration of the first compound, SUBTYPE = 2 to the second, etc VALUE NAME DURATION ULOQ LLOQ ROUTE Value of the event, defined by TYPE and SUBTYPE. E.g., the given dose, the observed PK concentration, or the value of other readouts. The values need to be in the units, defined in the UNIT column Unique name for the event that is coded by TYPE and SUBTYPE. No standard defined. For dosing events Dose compound X and for PK observations X plasma concentration might be used Numeric value capturing the duration of an event, in the time units defined in the column TIME_UNIT Upper limit of quantification of event defined by TYPE and SUBTYPE (if applicable), empty if not Lower limit of quantification of event defined by TYPE and SUBTYPE (if applicable), empty if not For the conversion to a model specific modeling dataset the route of administration needs to be defined for dosing events None None None If TYPE = 0 (dosing event), this column needs to contain the duration of the administration (e.g., the infusion time). 0 thus means bolus administration None If TYPE = 1: lower limit of quantification for the concentration of compound defined by SUBTYPE The following identifiers are recognized by the current poppk workflow tools: IV Subcut Oral Graphical exploration Prior to any modeling activity the data needs to be explored and understood. As with the model features (Table 1), there are some graphical analyses that should always be carried out and there are others that are more rarely done or are very specific to a project. Modelers typically develop their own scripts which then are adapted to each new modeling dataset to generate the needed graphical exploration plots. Since each modeler has a favorite tool for implementation of these scripts, established coding preferences, personal aesthetic criteria (e.g., how to label the axes, which colors to use), etc., this approach is not the most efficient and leads to a considerable variability of resulting plots. A more efficient approach to graphical exploration is to: (1) define upfront the plots/analyses that at least are needed in every poppk modeling activity, (2) develop a workflow tool that allows producing all these plots with a single function call, including elements such as high quality representation, axes labelling, and storing the resulting output in a reasonably organized structure. A critical requirement to allow for automatic generation of these analyses is the use of a standardized dataset format, which here is the previously defined generalized dataset format. A pseudo-code example for the graphical exploration analysis is shown below: In this example, the input to the workflow tool/function (ExplorePopPKdata) is the dataset in the generalized dataset format (data), the names of components in the dataset that are to be considered as potential covariates (covariates), and a path to where to write the output

7 Fig. 1 High level poppk modeling workflow Generalized dataset format Graphical exploration Data cleaning Modeling dataset generation Model building Final model (outputpath). The output of this workflow tool is several files in the output folder. Examples for the generated information are: Summary plots of the individual data on linear and logarithmic Y axis. Individual plots of the individual data on linear and logarithmic Y axis, indicating dosing times and drug amounts. Each observation record is annotated by the index in the dataset and by its value and time after last dose. Summary statistics for the chosen covariates. Spaghetti diagrams stratified by treatment arm and study, dose normalized and non-dose normalized, and both over time since first dose (TIME) and since last dose (TAD). Histograms and correlation plots for the continuous covariates and correlation information for categorical and continuous covariates. Data cleaning Exploration of PK data often leads to the identification of PK samples that might be inconsistent with the rest of the data, resulting in the exclusion of single PK records or all records for certain subjects from consideration. Since the generalized dataset often does not only contain PK but also PD data, subjects receiving only placebo doses should be removed for poppk modeling. Covariates of interest are not always recorded in all subjects, requiring imputation of missing ones. Depending on the decision of a modeler, records with concentration samples that are below the limit of quantification might be removed from the dataset or treated in a different way. There are many additional elements of data cleaning that are not considered in here. Important is that the step of data cleaning is often done in a modeler-dependent fashion. Some modelers might request from a programmer to flag the records and subjects to be removed from the subsequent analysis within the dataset, leading to multiple iterations on the original dataset. Others might handle it themselves by scripting or manual removal. Some modelers might request imputation of covariates to be done during the dataset preparation; others do it on their own. Rules for imputation of covariates might also vary across modelers and programmers. All these different possible approaches can lead to considerable inter-modeler variability and a reduction of efficiency. A workflow tool that supports a modeling methodology/ guidance allows applying more consistent rules for data cleaning across an organization, leading to increased quality, and can allow for a faster turnaround of the data cleaning. The output of this workflow tool should be a cleaned dataset to which the desired transformations have been applied. Additionally, a record of the applied transformations should be generated for documentation purposes in a well-organized structure. After this step it might be useful to re-run the graphical exploration analyses on the cleaned dataset, for documentation and checking of the results. A pseudo-code example for the data cleaning step is shown below:

8 In this example, the input to the workflow tool/function (CleanPopPKdata) is the dataset in the generalized dataset format (data), the names of components in the dataset that are to be considered as potential covariates (covariates), the rules for imputation of missing covariates (imputationrules), the unique subject identifiers of the subjects to be removed and the reason for removal (removesubjects), the index of records in the dataset that are to be removed and the reason for removal (removerecords), a flag that indicates on how to treat data below the limit of quantification (FLAG_LLOQ) and a path to where to write the log files containing information about the cleaning steps (outputpath). The output argument of this workflow tool is the cleaned dataset (datacleaned). Different options for the FLAG_LLOQ might be available. In the actual implementation of the workflow tool, a value of 0 removes the data below the LLOQ from the dataset, while a value of 1 keeps them and creates a new column with information about censored data. Additional approaches to handle data below the LLOQ can be implemented in a similar manner, allowing efficient adaptation of the data to the needs of a modeler. Base model Fig. 2 Model building workflow Covariance model Covariate model Final model selection software used by the workflow tools. By employing the generalized dataset format this step can be completed transparently to the user, meaning that the modeler does not need to be an expert in creating datasets for specific NLME software. The minimum information required by the workflow tool for this step is the generalized dataset format and a list with names of readouts that should be included in the modeling dataset as candidate covariates. A pseudo-code example for the modeling dataset generation: Modeling dataset generation Up to this point in the poppk modeling workflow, the format of the dataset was independent of any non-linear mixed effect modeling (NLME) software. However, for the purpose of parameter estimation, the dataset needs to be converted into a format that is suitable for the NLME In this example, the input to the workflow tool/function (Convert2popPKdataset) is the cleaned dataset that has been returned by the data cleaning step (data), the names of components in the dataset that are to be added as columns in the modeling dataset (covariates), and a path with filename to where to write the generated modeling dataset (outputfile). The output argument

9 Fig. 3 Efficient approach to evaluate a certain poppk model sub-space of interest User defined model space One step model sub-space evaluation Model generation Parameter estimation Creation of GoF plots Creation of tables for comparison Results for selected model space (convertinfo) of this workflow tool might contain information that is required for model generation step and is passed as an input argument to this step (see next pseudo-code example). In the actual implementation of this workflow tool, the output consists of a string that describes the type of columns in the modeling dataset. Model building: general steps Figure 2 shows the four main steps of poppk model building, considered in this paper. As discussed above, the goal is not to define how these different steps should be done (this should be covered in the modeling guidance), but to show an efficient way of how a workflow tool can support the modeler in the exploration of the model space of interest. The base model building step considers the underlying structural model, the residual error model, and fixed and random effects to be estimated. Certain selected covariates might already be considered if there is a rational for doing so. The covariance model building step focuses on a refinement of the decision regarding the selection of random effects and the correlations between random effects to be estimated. Assessment of statistical significance and potential clinical relevance of candidate covariates is done in the covariate model building step, while the final model selection step might assess clinical relevance of covariates more in detail, resulting in the selection of the final poppk model. The inclusion of covariates into a model might have an impact on the identifiability of the covariance matrix for the random effects. This means that the modeler needs to decide in which order to execute the covariance model building and the covariate model building. The example in the supplementary material demonstrates that the order of these steps indeed matters. Model building: one step model sub-space evaluation Independently of the current model building step of consideration and of the approach (manual or workflow tool supported poppk modeling), the usual steps to be taken are to (1) select a model to test, (2) generate the model code, (3) run the estimation algorithm, and (4) process the results to produce goodness-of-fit assessments, visual predictive checks (VPCs), simulations, etc. Based on the obtained results, alternative models are selected and the whole process is repeated, eventually exploring the space of poppk models to find the model that is best supported by the available data and the prior knowledge about the compound. In comparison to the approach where all models are coded and executed manually, potentially requiring switching between different tools and computational environments to produce goodness of fit plots etc., a workflow tool based approach can considerably enhance this process, as outlined in Fig. 3. The modeler defines a model sub-space of interest to be assessed, using a workflow tool dependent syntax. Then the workflow tool automatically generates the relevant models, performs the parameter estimation, reads the results and generates goodness-of-fit plots and tables, allowing comparing the considered models all in one step. Such a workflow tool should at least cover all model features and options, listed in Table 2. A pseudo-code example for a one-step model sub-space evaluation for a base model:

10 In this example, the input to the workflow tool/function (PopPKModelSpace) has two parts. The first part (input1) defines general settings, where stepidentifier is an identifier that is used, e.g., as part of the path where the generated models and outputs are stored, convertinfo is the output of the modeling dataset generation step, data defines the path and filename of the modeling dataset, and options are a set of user defined options for the underlying estimation software. The second input (input2) defines the poppk model features to be tested. The argument parameters contains information about initial guesses for the fixed and random effects. Additionally, using the parameters argument, the modeler can define different scenarios, each defining which fixed and random effects to be estimated or kept fixed. Once this workflow tool is executed, it will generate a sub-space of poppk models that is defined by the combination of all model features with each other. This means that in the example above, 36 different poppk models multiplied by the number of parameters scenarios, the modeler has chosen, are generated. A good practice in the first base model iteration is to limit the number of scenarios to one, by assuming that all fixed and all random effects are estimated, or to select a reasonable setting through insight based on the available data. Refinement of this assumption can be done after the consideration of the estimation results. After the generation of all models the parameter estimations are performed. Parallel processing is used, if available. Subsequently, the most common goodness-of-fit (GoF) plots are generated for all evaluated models and stored in an ordered fashion. Additionally, tables for comparison are generated as text files, summarizing the results for the models (parameter estimates and some metrics of interest) in the user defined model space, allowing the modeler to get a quick overview of the performance of the different models and support decisions about the next modeling steps. Examples, covering such tables are shown and discussed more in detail in the supplementary material. Model building: technical requirements for the one step evaluation In order to be able to automatically generate models of interest, a model library is required. Since any model library is of finite size, the properties of the models in this library need to be defined upfront. An alternative to a model library is a general model that contains all the considered structural elements, representing the entirety of a model library by switching on and off certain features. A model structure, capturing all structural properties of the models defined by Table 2 is shown in Fig. 4. A dual approach has been taken in the implementation of the workflow tool, in order to improve the computational performance. Models that are linear and analytically solvable are taken from a model library, while non-linear models are implemented by using the general model in the form of ordinary differential equations (ODEs). PK parameters are always assumed to be log-normally distributed, except the bioavailability, which is assumed to be logit-normally distributed. In order to realistically pursue the path of such a workflow tool based exploration of poppk model subspaces, an important key assumption needs to be made: the availability of numerically stable estimation algorithms that, given a reasonably well defined estimation problem, are able to produce the output of interest in a reliable manner. The authors experience suggests that the implementations of the SAEM algorithm in both MONOLIX and NONMEM do meet this requirement. Other methods, such as FOCE in NONMEM do not generally qualify for the workflow based approach. Suitable initial parameter guesses and model parameterizations might need to be found, to allow for convergence of the estimation algorithm. In case of failure of the estimation algorithm, the error messages need to be assessed very carefully in order to determine needed changes in the model. This is a time consuming process which is counterproductive for efficiency and the outcome is often that the modeler needs to focus more on understanding the tool

11 Fig. 4 General structural poppk model Tlag Dosing Dosing 1 Fabs1 Fiv Dosing 2 compartment ka Peripheral compartment 1 (Vp1) Q1 Central compartment (Vc) Q2 Peripheral compartment 2 (Vp2) CL, VMAX, KM and possible workarounds, rather than on the interpretation of estimation results. Model building: hierarchical exploration of the model space An important factor in the workflow based approach is the selection of the model sub-space to be considered. Theoretically, it would be possible to consider the complete poppk model space, spanned by combination of all model features and options, shown in Table 2. However, this would lead to a tremendous combinatorial explosion of models to be evaluated and compared. Even though this might still be technically feasible, given enough computational resources and/or time, it is out of scope of this paper. Instead, the approach advocated here is to use a hierarchical/modular approach, following the steps in Fig. 2. After each step the modeler assesses the results and refines the model features to be considered in the analysis. In a first step, a base model is sought that adequately describes the data. A good guess about a possible model structural elements and to include selected covariates only if a good rational exists. An example for the first base model assessment would be to consider different numbers of compartments, selected error models, possible lag time on absorption, and linear and linear? saturable elimination (see previous pseudo-code example). All fixed and random effects could be estimated. The base model is then refined in a second step. Based on the results from the first step, the main structural elements for which there is support in the data are better understood (e.g., number of compartments, type of elimination) and the model space with respect to these elements can be reduced. Random effects for which the relative standard errors and/or the shrinkage are very large might be excluded from further consideration. Several different parameter scenarios could be considered in this second step, comparing the impact of different combinations of parameters for which to estimate fixed and random effects. This steps leads to the selection of the final base model. A pseudo-code example for a one-step model sub-space evaluation for a base model refinement: space for the base model should be already available based on the graphical exploration of the available data and/or from prior knowledge about the compound. It is useful to limit the number of model features and options to the main In this example, the same workflow function (PopPK- ModelSpace) is used as in the first base model assessment, but with a reduced selection of possible model features to be

12 tested. Using the parameters argument, three different scenarios are defined, testing different sets of random effects in the models (parameter names defined in Fig. 4). In a third step, the covariate model is addressed. Several possible methods are available in the literature and employed in day-to-day work. In the context of an efficient poppk workflow it is of less importance which methods are used, as long as they identify the clinically relevant covariates. What is more important is that this method is/ these methods are implemented and available as workflow tool(s) in a user-friendly and seamless way. The method used in the actual implementation of the workflow is the Full Model Estimation Approach [23]. A pseudo-code example for a one-step model sub-space evaluation for a covariate model: discussed here) to obtain an idea about the impact of the selected covariates on the PK parameters. Covariate coefficients can be estimated or fixed to initial guesses (default initial guesses are 0). Another important aspect is the transformation of covariates. Since lognormal distribution of individual PK parameters is assumed by the workflow tools, the covariates are automatically log transformed and centered by the median. The covariate step is discussed more in detail in the supplementary material. The covariance model is addressed in the fourth step of model building. As before, the best model is retained from the previous steps and the search space is expanded towards different structures of the random effect covariance matrix. A first guess about a suitable structure of the covariance matrix might be available, based on the information from the Again, the same workflow function (PopPKModel- Space) is used as in the previous examples. The model sub-space of consideration is spanned by the final base model and four different covariate models (covariate) to be assessed independently. The estimation results can be processed further with additional workflow tools (not previous models (i.e., correlation of individual parameter estimates). Additional scenarios should be considered, including a full covariance matrix. This step will lead to the selection of the final poppk model. A pseudo-code example for a one-step model sub-space evaluation for a covariance model:

13 The workflow tool/function in this step (PopPK- ModelSpace) is the same as in the previous example. The model sub-space of consideration is spanned by the final covariate model and 3 different covariance models (covariance) to be assessed independently. The total number of models, evaluated by the pseudocode examples above, is 46. The interpretation of the results by the modeler was needed in each step to select the model sub-space to be considered in the next step, resulting in a relatively small number of evaluated models. Alternatively, all the above tested model features could have been combined in a single call to the PopPKModelSpace workflow tool: Model building: additional analyses Additional analyses are of use and should be carried out at certain stages of model building. Often used ones are VPCs, both on data used for modeling and on other data for external validation, and bootstrapping to obtain a measure for uncertainty in the estimated parameters. For these analyses, user-friendly implementations should be available within the workflow supporting tools. For the purpose of model selection and rationale based decisions during the model building process, additional model based analyses are of interest and should be an integral part of the model building workflow. Two such analyses are: This single call generates 1,728 different candidate poppk models. On one hand this shows that such a workflow tool allows efficiently generating and evaluating a large number of models. On the other hand, it also shows that most of these models are not relevant and the modeler has the tremendous task of filtering them. It is important to note that the number 1,728 is still not the final, since some of the definitions above have been chosen based on the knowledge gained in previous steps. For example, the number of scenarios considered for the random effects might need to be increased in such a brute-force approach. Even though such an approach might have some academic value, the modeler guided step-wise model building approach is to be preferred. Robustness analysis Re-estimation of parameters for the same model but starting from randomized initial guesses. This supports the assessment of the actual identifiability of the estimated parameters. A model should be rejected if repeated estimation from different initial guesses leads to significant differences in the parameter estimates. In practical applications, this analysis has already given valuable information about the parameters that have most contribution to the over-parameterization by showing drifts. Such drifts are represented by correlations of two or more parameters with the objective function value, which can be explained by a shallow objective function surface in certain directions of the parameter space. The optimization algorithm might get stuck or shows a slower convergence in these directions.

14 In the actual implementation of the workflow tools, this analysis is available as option to the PopPKModel- Space function. Sensitivity analysis Here it is understood as using simulation based parametric sensitivity analysis to predict the expected information content in data that is not yet modelled (e.g., because a phase III study has not read out yet but planning for the poppk modeling needs to be done). Often, late phase studies have a very sparse PK sampling and re-estimation of all parameters might not be the best approach. Sensitivity analysis can be used as a scientifically valid motivation for fixing some parameters on results from previous analyses and only the parameters for which information in the new data is expected to be available might be re-estimated. Methods from the area of optimal design might be used for such an analysis, but with a focus on efficiency they would need to be made available as workflow tools, not requiring the recoding of the models into a different format. In the actual implementation of the workflow tools a simple parametric sensitivity analysis based on the population mean parameters has been chosen. Discussion and conclusion The landscape of available tools and the way on how population pharmacokinetic and pharmacodynamic (PKPD) modeling is often taught and performed supports the conclusion that the area of Pharmacometrics is very much tool-centric. Tremendous effort has been and is put into understanding and sharing tips-and-tricks on how to use certain tools rather than on modeling methodology and guidance. So far, the tool centricity might have limited the investigation of more streamlined and standard approaches to population modeling, associated workflows, and supporting these workflows by efficient integrated workflow tools. It is clear that a general approach to all possible pharmacometric population modeling activities is complex/ impossible to realize. However, when limiting the scope from all possible to very specific modeling activities, such as poppk, it can be shown that the type of models, model features and options are often quite similar. In this paper an attempt has been made to approach poppk model building in a workflow oriented way, supporting the workflow by efficient workflow tools. The center-piece of the presented approach is a workflow tool that, based on the definition of a poppk model sub-space of interest, generates all the corresponding models, performs the parameter estimation, and summarizes the results in a form that is easily accessible and interpretable by a modeler. One advantage of this approach is that all NLME parameter estimation tool specific issues can be hidden from the modeler, shifting the focus to defining the right model space to investigate and interpreting the results of the parameter estimation in order to select the next model features to test. Another key component of the presented work is the definition of a generalized dataset format. Such a format provides the basis for an efficient implementation of the poppk workflow. Not only can the most important graphical exploration plots of poppk data be generated automatically, but it also allows for more efficient dataset preparation, easier re-use of the data due to the additional annotation, and supports newcomers in the area of Pharmacometrics to get their dataset specification right the first time. The presented workflow has been used on a large number of poppk modeling activities, ranging from phase I to submission. In all these activities the workflow approach eliminated labor-intensive repetitive coding tasks, allowed a better focus on the modeling rather than the tools used, and led to a larger degree of trust in the final model and parameter estimates. The fact that the workflow tools ensure organized storage of models, model results, generated tables and graphics is a benefit not to be underestimated when collaborating with other modelers and model reviewers. To provide an approximate measure that can be linked to efficiency metrics, the example poppk (one single dose study and one multiple dose study, 86 subjects on active treatment, 1,331 observation and 1,166 dose records) in the supplementary material can be completed in the presented form in less than a day (without all the detailed documentation that was added for the purpose of this publication). In terms of quality metrics, the example poppk has been conducted, following the assumed underlying modeling guidance. Furthermore, the required code for the complete poppk is available as a single script that would be executable by every modeler without the need for interaction with the original author, ensuring reproducibility. All models and outputs are stored in an organized structure that allows finding the information of interest immediately. Although the presented work is limited to the poppk model features shown in Table 2, the approach is general and easily extensible. This means that, if needed, the approach can be extended to cover inter-occasion variability, zero-order and mixed absorption, nonlinear bioavailability, nonlinear inter-compartmental distribution, etc. For the proof-of-concept implementation and to gain experience with this approach a subset of model features had to be chosen. Additionally, when basing the workflow tools on a scripting language, it is possible to step out of the