pypesto.hierarchical.ordinal

Ordinal and censored data integration

Contains the implementation of the optimal scaling approach, applied for integration of ordinal and censored data in ODE modeling. Each ordinal datapoint is defined as being part of a category, where the mutual ordering of the categories of the same observable is known. The category interval bounds are numerically optimized and quantitative surrogate measurements are calculated to represent the ordinal measurements. This constitutes the inner subproblem of the hierarchical optimization problem. For censored data, as the category interval bounds are known, the surrogate measurements are directly calculated.

An example of parameter estimation with ordinal data can be found in pypesto/doc/examples/ordinal_data.ipynb. An example of parameter estimation with censored data can be found in pypesto/doc/examples/censored_data.ipynb.

Details on the optimal scaling approach can be found in Shepard, 1962 (https://doi.org/10.1007/BF02289621). Details on the application of the gradient-based optimal scaling approach to mechanistic modeling with ordinal data can be found in Schmiester et al. 2020 (https://doi.org/10.1007/s00285-020-01522-w) and Schmiester et al. 2021 (https://doi.org/10.1093/bioinformatics/btab512).

class pypesto.hierarchical.ordinal.OrdinalCalculator

Bases: AmiciCalculator

A calculator is passed as calculator to the pypesto.AmiciObjective.

The object is called by pypesto.AmiciObjective.call_unprocessed() to calculate the current objective function values and gradient.

__call__(x_dct, sensi_orders, mode, amici_model, amici_solver, edatas, n_threads, x_ids, parameter_mapping, fim_for_hess, rdatas=None)

Perform the actual AMICI call.

Parameters:

• x_dct (dict) – Parameters for which to compute function value and derivatives.

• sensi_orders (tuple[int, ...]) – Tuple of requested sensitivity orders.

• mode (str) – Call mode (function value or residual based).

• amici_model (Union[Model, ModelPtr]) – The AMICI model.

• amici_solver (Union[Solver, SolverPtr]) – The AMICI solver.

• edatas (list[ExpData]) – The experimental data.

• n_threads (int) – Number of threads for AMICI call.

• x_ids (Sequence[str]) – Ids of optimization parameters.

• parameter_mapping (ParameterMapping) – Mapping of optimization to simulation parameters.

• fim_for_hess (bool) – Whether to use the FIM (if available) instead of the Hessian (if requested).

• rdatas (list[ReturnData]) – AMICI simulation return data. In case the calculator is part of the pypesto.objective.amici.InnerCalculatorCollector, it will already simulate the model and pass the results here.

Returns:

inner_result – A dict containing the calculation results: FVAL, GRAD, RDATAS.

__init__(inner_problem, inner_solver=None)

Initialize the calculator from the given problem.

Parameters:

• inner_problem (OrdinalProblem) – The optimal scaling inner problem.

• inner_solver (OrdinalInnerSolver) – A solver to solve inner_problem. Defaults to OptimalScalingInnerSolver.

initialize()

Initialize.

class pypesto.hierarchical.ordinal.OrdinalInnerSolver

Bases: InnerSolver

Solve the inner subproblem of the optimal scaling approach for ordinal data.

Options

method:

The method to use for the inner optimization problem. Can be ‘standard’ or ‘reduced’. The latter is more efficient.

reparameterized:

Whether to use reparameterized optimization.

intervalConstraints:

The type of interval constraints to use. Can be ‘max’ or ‘maxmin’.

minGap:

The minimum gap between two consecutive categories.

__init__(options=None)

Construct.

Parameters:

options (dict) –

calculate_gradients(problem, x_inner_opt, sim, sy, sigma, ssigma, parameter_mapping, par_opt_ids, par_sim_ids, par_edatas_indices, snllh)

Calculate gradients of the inner objective function.

Calculates gradients of the objective function with respect to outer (dynamical) parameters.

Parameters:

• problem (OrdinalProblem) – Optimal scaling inner problem.

• x_inner_opt (list[dict]) – List of optimization results of the inner subproblem.

• sim (list[ndarray]) – Model simulations.

• sy (list[ndarray]) – Model sensitivities.

• sigma (list[ndarray]) – Model noise parameters.

• ssigma (list[ndarray]) – Model sensitivity of noise parameters.

• parameter_mapping (ParameterMapping) – Mapping of optimization to simulation parameters.

• par_opt_ids (list) – Ids of outer otimization parameters.

• par_sim_ids (list) – Ids of outer simulation parameters, includes fixed parameters.

• par_edata_indices – Indices of parameters from amici_model.getParameterIds() that are needed for sensitivity calculation. Comes from edata.plist for each condition.

• snllh (ndarray) – A zero-initialized vector of the same length as par_opt_ids to store the gradients in. Will be modified in-place.

• par_edatas_indices (list) –

Returns:

The gradients with respect to the outer parameters.

static calculate_obj_function(x_inner_opt)

Calculate the inner objective function value.

Calculates the inner objective function value from a list of inner optimization results returned from OptimalScalingInnerSolver.solve.

Parameters:

x_inner_opt (list) – List of optimization results of the inner subproblem.

Returns:

Inner objective function value.

static get_default_options()

Return default options for solving the inner problem.

Return type:

dict

solve(problem, sim, sigma)

Get results for every group (inner optimization problem).

Parameters:

• problem (OrdinalProblem) – Optimal scaling inner problem.

• sim (list[ndarray]) – Model simulations.

• sigma (list[ndarray]) – Standard deviation of the noise.

Return type:

list

Returns:

List of optimization results of the inner subproblem.

validate_options()

Validate the current options dictionary.

class pypesto.hierarchical.ordinal.OrdinalParameter

Bases: InnerParameter

Inner parameter of the optimal scaling inner optimization problem for ordinal data.

dummy_value

Value to be used when the optimal parameter is not yet known (in particular to simulate unscaled observables).

inner_parameter_id

The inner parameter ID.

inner_parameter_type

The inner parameter type.

ixs

A mask (boolean matrix) that indicates the measurements that this parameter is used in.

lb

The lower bound, for optimization.

scale

Scale on which to estimate this parameter.

ub

The upper bound, for optimization.

observable_id

The id of the observable whose measurements are in the category with this inner parameter.

category

Category index.

group

Group index.

value

Current value of the inner parameter.

estimate

Whether to estimate inner parameter in inner subproblem.

censoring_type

The censoring type of the measurements in the category with this inner parameter. In case of ordinal measurements, this is None.

__init__(*args, observable_id=None, group=None, category=None, estimate=False, censoring_type=None, **kwargs)

Construct.

Parameters:

• attributes. (See class) –

• observable_id (str) –

• group (int) –

• category (int) –

• estimate (bool) –

• censoring_type (Literal[None, 'left-censored', 'right-censored', 'interval-censored']) –

initialize()

Initialize.

class pypesto.hierarchical.ordinal.OrdinalProblem

Bases: AmiciInnerProblem

Inner optimization problem for ordinal or censored data.

The ordinal inner problem contains the following parameters: surrogate data, lower bounds, and upper bounds. All parameters are optimized to minimize the distance between the surrogate data and the simulated data while satisfying the ordering constraints of the problem. Depending on the method, the problem is re-formulated to reduce the number of parameters to estimate.

xs

Mapping of (inner) parameter ID to InnerParameters.

data

Measurement data. One matrix (num_timepoints x num_observables) per simulation condition. Missing observations as NaN.

edatas

AMICI ExpDatas for each simulation condition.

groups

A dictionary of the groups of the subproblem.

method

A string representing the method of the Optimal Scaling approach, either reduced or standard.

__init__(method, **kwargs)

Construct.

Parameters:

method (str) –

static from_petab_amici(petab_problem, amici_model, edatas, method=None)

Construct the inner problem from the petab_problem.

Return type:

OrdinalProblem

Parameters:

• petab_problem (Problem) –

• amici_model (Model) –

• edatas (list[ExpData]) –

• method (str) –

get_cat_lb_parameters_for_group(group)

Get OptimalScalingParameters that are category lower boundaries and belong to the given group.

Return type:

list[OrdinalParameter]

Parameters:

group (int) –

get_cat_ub_parameters_for_group(group)

Get OptimalScalingParameters that are category upper boundaries and belong to the given group.

Return type:

list[OrdinalParameter]

Parameters:

group (int) –

get_censored_group_quantitative_ixs(xs)

Return a list of boolean masks indicating which data points are quantitative.

For a given group with censored data, return a list of boolean masks indicating which data points are not censored, and therefore quantitative.

Parameters:

xs (list[OrdinalParameter]) – List of OptimalScalingParameters of a group with censored data.

Return type:

list[ndarray]

Returns:

quantitative_ixs – List of boolean masks indicating which data points are quantitative.

get_d(group, xs, y_sim_all, eps)

Return vector of minimal gaps and ranges.

Return type:

ndarray

Parameters:

• xs (list[OrdinalParameter]) –

• y_sim_all (ndarray) –

• eps (float) –

get_dd_dtheta(group, xs, y_sim_all, sy_all)

Return the derivative of vector of minimal gaps and ranges with respect to an outer parameter.

Return type:

ndarray

Parameters:

• group (int) –

• xs (list[OrdinalParameter]) –

• y_sim_all (ndarray) –

• sy_all (ndarray) –

get_fixed_xs_for_group(group)

Get OptimalScalingParameters that are fixed and belong to the given group.

Return type:

list[OrdinalParameter]

Parameters:

group (int) –

get_free_xs_for_group(group)

Get OptimalScalingParameters that are free and belong to the given group.

Return type:

list[OrdinalParameter]

Parameters:

group (int) –

get_groups_for_xs(inner_parameter_type)

Get unique list of OptimalScalingParameter.group values.

Return type:

list[int]

Parameters:

inner_parameter_type (str) –

get_inner_parameter_dictionary()

Return a dictionary with inner parameter ids and their values.

Return type:

dict

get_interpretable_x_ids()

Get IDs of interpretable inner parameters.

There are no interpretable inner parameters for the ordinal problem.

Return type:

list[str]

get_w(group, y_sim_all)

Return the weight matrix of the group.

Return type:

ndarray

Parameters:

• group (int) –

• y_sim_all (ndarray) –

get_wdot(group, y_sim_all, sy_all)

Return the derivative of the weight matrix of a group with respect to an outer parameter.

Return type:

ndarray

Parameters:

• group (int) –

• y_sim_all (ndarray) –

• sy_all (ndarray) –

get_xs_for_group(group)

Get OptimalScalingParameters that belong to the given group.

Return type:

list[OrdinalParameter]

Parameters:

group (int) –

initialize()

Initialize the subproblem.

Return type:

None

initialize_c(group)

Initialize the constraints matrix for the group.

The structure of the constraints matrix is the following: Each row C_i of the matrix C represents one optimization constraint as C_i * xi + d(theta, sim) <= 0, where xi is the vector of inner paramters (surrogate data, lower bounds, upper bounds)^T, and d is the vector of minimal category interval ranges and gaps.

First self.groups[group][NUM_DATAPOINTS] rows constrain the surrogate data to stay larger than lower category bounds. Then another self.groups[group][NUM_DATAPOINTS] rows constrain the surrogate data to stay smaller than upper category bounds. Then self.groups[group][NUM_CATEGORIES] rows constrain the ordering of the categories. And lastly, the remaining self.groups[group][NUM_CATEGORIES] constrain the lower bound to be smaller than the upper bound for each category.

Return type:

ndarray

Parameters:

group (int) –

initialize_w(group)

Initialize the weight matrix for the group.

Return type:

ndarray

Parameters:

group (int) –