pypesto.hierarchical.optimal_scaling
Optimal scaling
Contains the implementation of the optimal scaling approach, applied for integration of ordinal and censored data in ODE modeling. Each ordinal datapoint is defined as being part of a category, where the mutual ordering of the categories of the same observable is known. The category interval bounds are numerically optimized and quantitative surrogate measurements are calculated to represent the ordinal measurements. This constitutes the inner subproblem of the hierarchical optimization problem. For censored data, as the category interval bounds are known, the surrogate measurements are directly calculated.
An example of parameter estimation with ordinal data can be found in pypesto/doc/examples/example_ordinal.ipynb. An example of parameter estimation with censored data can be found in pypesto/doc/examples/example_censored.ipynb.
Details on the optimal scaling approach can be found in Shepard, 1962 (https://doi.org/10.1007/BF02289621). Details on the application of the gradient-based optimal scaling approach to mechanistic modeling with ordinal data can be found in Schmiester et al. 2020 (https://doi.org/10.1007/s00285-020-01522-w) and Schmiester et al. 2021 (https://doi.org/10.1093/bioinformatics/btab512).
- class pypesto.hierarchical.optimal_scaling.OptimalScalingAmiciCalculator(inner_problem: OptimalScalingProblem, inner_solver: OptimalScalingInnerSolver | None = None)[source]
Bases:
AmiciCalculator
A calculator is passed as calculator to the pypesto.AmiciObjective.
The object is called by
pypesto.AmiciObjective.call_unprocessed()
to calculate the current objective function values and gradient.- __call__(x_dct: Dict, sensi_orders: Tuple[int, ...], mode: str, amici_model: Model | ModelPtr, amici_solver: Solver | SolverPtr, edatas: List[ExpData], n_threads: int, x_ids: Sequence[str], parameter_mapping: ParameterMapping, fim_for_hess: bool, rdatas: List[ReturnData] | None = None)[source]
Perform the actual AMICI call.
- Parameters:
x_dct – Parameters for which to compute function value and derivatives.
sensi_orders – Tuple of requested sensitivity orders.
mode – Call mode (function value or residual based).
amici_model – The AMICI model.
amici_solver – The AMICI solver.
edatas – The experimental data.
n_threads – Number of threads for AMICI call.
x_ids – Ids of optimization parameters.
parameter_mapping – Mapping of optimization to simulation parameters.
fim_for_hess – Whether to use the FIM (if available) instead of the Hessian (if requested).
rdatas – AMICI simulation return data. In case the calculator is part of the
pypesto.objective.amici.InnerCalculatorCollector
, it will already simulate the model and pass the results here.
- Returns:
A dict containing the calculation results: FVAL, GRAD, RDATAS and X_INNER_OPT.
- Return type:
inner_result
- __init__(inner_problem: OptimalScalingProblem, inner_solver: OptimalScalingInnerSolver | None = None)[source]
Initialize the calculator from the given problem.
- Parameters:
inner_problem – The optimal scaling inner problem.
inner_solver – A solver to solve
inner_problem
. Defaults toOptimalScalingInnerSolver
.
- class pypesto.hierarchical.optimal_scaling.OptimalScalingInnerSolver(options: Dict | None = None)[source]
Bases:
InnerSolver
Solve the inner subproblem of the optimal scaling approach for ordinal data.
Options
- method:
The method to use for the inner optimization problem. Can be ‘standard’ or ‘reduced’. The latter is more efficient.
- reparameterized:
Whether to use reparameterized optimization.
- intervalConstraints:
The type of interval constraints to use. Can be ‘max’ or ‘maxmin’.
- minGap:
The minimum gap between two consecutive categories.
- calculate_gradients(problem: OptimalScalingProblem, x_inner_opt: List[Dict], sim: List[ndarray], sy: List[ndarray], sigma: List[ndarray], ssigma: List[ndarray], parameter_mapping: ParameterMapping, par_opt_ids: List, par_sim_ids: List, par_edatas_indices: List, snllh: Dict)[source]
Calculate gradients of the inner objective function.
Calculates gradients of the objective function with respect to outer (dynamical) parameters.
- Parameters:
problem – Optimal scaling inner problem.
x_inner_opt – List of optimization results of the inner subproblem.
sim – Model simulations.
sy – Model sensitivities.
sigma – Model noise parameters.
ssigma – Model sensitivity of noise parameters.
parameter_mapping – Mapping of optimization to simulation parameters.
par_opt_ids – Ids of outer otimization parameters.
par_sim_ids – Ids of outer simulation parameters, includes fixed parameters.
par_edata_indices – Indices of parameters from amici_model.getParameterIds() that are needed for sensitivity calculation. Comes from edata.plist for each condition.
snllh – Empty dictionary with optimization parameters as keys.
- Return type:
Filled in snllh dictionary with objective function gradients.
- static calculate_obj_function(x_inner_opt: list)[source]
Calculate the inner objective function value.
Calculates the inner objective function value from a list of inner optimization results returned from OptimalScalingInnerSolver.solve.
- Parameters:
x_inner_opt – List of optimization results of the inner subproblem.
- Return type:
Inner objective function value.
- solve(problem: OptimalScalingProblem, sim: List[ndarray], sigma: List[ndarray]) list [source]
Get results for every group (inner optimization problem).
- Parameters:
problem – Optimal scaling inner problem.
sim – Model simulations.
sigma – Standard deviation of the noise.
- Return type:
List of optimization results of the inner subproblem.
- class pypesto.hierarchical.optimal_scaling.OptimalScalingParameter(*args, observable_id: str | None = None, group: int | None = None, category: int | None = None, estimate: bool = False, censoring_type: Literal[None, 'left-censored', 'right-censored', 'interval-censored'] | None = None, **kwargs)[source]
Bases:
InnerParameter
Inner parameter of the optimal scaling hierarchical optimization problem.
- dummy_value
Value to be used when the optimal parameter is not yet known (in particular to simulate unscaled observables).
- inner_parameter_id
The inner parameter ID.
- inner_parameter_type
The inner parameter type.
- ixs
A mask (boolean matrix) that indicates the measurements that this parameter is used in.
- lb
The lower bound, for optimization.
- scale
Scale on which to estimate this parameter.
- ub
The upper bound, for optimization.
- observable_id
The id of the observable whose measurements are in the category with this inner parameter.
- category
Category index.
- group
Group index.
- value
Current value of the inner parameter.
- estimate
Whether to estimate inner parameter in inner subproblem.
- censoring_type
The censoring type of the measurements in the category with this inner parameter. In case of ordinal measurements, this is None.
- class pypesto.hierarchical.optimal_scaling.OptimalScalingProblem(xs: List[OptimalScalingParameter], data: List[ndarray], method: str)[source]
Bases:
InnerProblem
Inner optimization problem for optimal scaling.
- xs
Mapping of (inner) parameter ID to
InnerParameters
.
- data
Measurement data. One matrix (num_timepoints x num_observables) per simulation condition. Missing observations as NaN.
- groups
A dictionary of the groups of the subproblem.
- method
A string representing the method of the Optimal Scaling approach, either ‘reduced’ or ‘standard’.
- static from_petab_amici(petab_problem: Problem, amici_model: Model, edatas: List[ExpData], method: str | None = None) OptimalScalingProblem [source]
Construct the inner problem from the petab_problem.
- get_cat_lb_parameters_for_group(group: int) List[OptimalScalingParameter] [source]
Get
OptimalScalingParameter
s that are category lower boundaries and belong to the given group.
- get_cat_ub_parameters_for_group(group: int) List[OptimalScalingParameter] [source]
Get
OptimalScalingParameter
s that are category upper boundaries and belong to the given group.
- get_censored_group_quantitative_ixs(xs: List[OptimalScalingParameter]) List[ndarray] [source]
Return a list of boolean masks indicating which data points are quantitative.
For a given group with censored data, return a list of boolean masks indicating which data points are not censored, and therefore quantitative.
- Parameters:
xs – List of
OptimalScalingParameter
s of a group with censored data.- Returns:
List of boolean masks indicating which data points are quantitative.
- Return type:
quantitative_ixs
- get_d(group, xs: List[OptimalScalingParameter], y_sim_all: ndarray, eps: float) ndarray [source]
Return vector of minimal gaps and ranges.
- get_dd_dtheta(group: int, xs: List[OptimalScalingParameter], y_sim_all: ndarray, sy_all: ndarray) ndarray [source]
Return the derivative of vector of minimal gaps and ranges with respect to an outer parameter.
- get_fixed_xs_for_group(group: int) List[OptimalScalingParameter] [source]
Get
OptimalScalingParameter
s that are fixed and belong to the given group.
- get_free_xs_for_group(group: int) List[OptimalScalingParameter] [source]
Get
OptimalScalingParameter
s that are free and belong to the given group.
- get_groups_for_xs(inner_parameter_type: str) List[int] [source]
Get unique list of
OptimalScalingParameter.group
values.
- get_inner_parameter_dictionary() Dict [source]
Return a dictionary with inner parameter ids and their values.
- get_wdot(group: int, y_sim_all: ndarray, sy_all: ndarray) ndarray [source]
Return the derivative of the weight matrix of a group with respect to an outer parameter.
- get_xs_for_group(group: int) List[OptimalScalingParameter] [source]
Get
OptimalScalingParameter
s that belong to the given group.
- initialize_c(group: int) ndarray [source]
Initialize the constraints matrix for the group.
The structure of the constraints matrix is the following: Each row C_i of the matrix C represents one optimization constraint as C_i * xi + d(theta, sim) <= 0, where xi is the vector of inner paramters (surrogate data, lower bounds, upper bounds)^T, and d is the vector of minimal category interval ranges and gaps.
First self.groups[group][NUM_DATAPOINTS] rows constrain the surrogate data to stay larger than lower category bounds. Then another self.groups[group][NUM_DATAPOINTS] rows constrain the surrogate data to stay smaller than upper category bounds. Then self.groups[group][NUM_CATEGORIES] rows constrain the ordering of the categories. And lastly, the remaining self.groups[group][NUM_CATEGORIES] constrain the lower bound to be smaller than the upper bound for each category.