Limit Regret in Binary Treatment Choice with Misspecified Plug-In Predictors and Decision Thresholds
Abstract
We study the population limit maximum regret (MR) of plug-in prediction when the decision problem is to choose between two treatments for the members of a population with observed covariates x. In this setting, the optimal treatment for persons with covariate value x is B if the conditional probability P(y = 1|x) of a binary outcome y exceeds an x-specific known threshold and is A otherwise. This structure is common in medical decision making, as well as non-medical contexts. Plug-in prediction uses data to estimate P(y|x) and acts as if the estimate is accurate. We are concerned that the model used to estimate P(y|x) may be misspecified, with true conditional probabilities being outside the model space. In practice, plug-in prediction has been performed with a wide variety of prediction models that commonly are misspecified. Further, applications often use a conventional x-invariant threshold, whereas optimal treatment choice uses x-specific thresholds. The main contribution of this paper is to shed new light on limit MR when plug-in prediction is performed with misspecified models. We use a combination of algebraic and computational analysis to study limit MR, demonstrating how it depends on the limit estimate and on the thresholds used to choose treatments. We recommend that a planner who wants to use plug-in prediction to achieve satisfactory MR should jointly choose a predictive model, estimation method, and x-specific thresholds to accomplish this objective.
Summary
This paper investigates the problem of choosing between two treatments for individuals in a population based on observed covariates, using plug-in prediction. The optimal treatment depends on whether the conditional probability of a binary outcome exceeds a known, potentially covariate-specific threshold. The research focuses on the limit maximum regret (MR) when the model used to estimate these probabilities is misspecified and when a conventional x-invariant threshold is used instead of x-specific thresholds. The authors use a combination of algebraic and computational analysis to understand how limit MR depends on the limit estimate and chosen thresholds. The key recommendation is that planners should jointly select a predictive model, estimation method, and x-specific thresholds to minimize MR when using plug-in prediction. The paper builds upon previous work by the authors, expanding the scope to consider population-wide welfare optimization and relaxing the assumption of using pre-specified thresholds. They show that while correctly specified models lead to zero limit MR with appropriate thresholds, misspecification leads to positive limit MR that depends on both the estimation method and chosen thresholds. The paper explores elementary predictor models algebraically and then uses computational analysis to study the limit behavior of approaches commonly applied in practice, such as parametric models estimated by least squares. Preliminary findings reveal that misspecified plug-in predictors and decision thresholds can have harmful effects on welfare, emphasizing the need for careful joint selection.
Key Insights
- •The paper formally defines the problem of treatment choice under misspecified prediction models, considering the population-wide welfare objective.
- •It demonstrates that correct model specification, coupled with appropriate covariate-specific thresholds, leads to zero limit maximum regret.
- •The paper reveals that using a marginal event probability to predict conditional probabilities, even with a covariate-invariant threshold, can lead to suboptimal treatment assignment and positive limit maximum regret.
- •A K-dimensional parametric model used to correctly predict K conditional probabilities still results in positive limit maximum regret for the remaining covariates, bounded by expressions involving the welfare values and covariate probabilities.
- •Preliminary computational results suggest that misspecified plug-in predictors and incorrect threshold selection can significantly increase maximum regret, potentially exceeding the regret associated with making decisions without any data.
- •The paper highlights the suboptimality of using x-invariant thresholds when the optimal thresholds (1-U_xB) are significantly disparate.
- •The paper shows algebraically that, for the case of using the marginal event probability to predict conditional probabilities, regret is maximized when the marginal probability takes the threshold value, and the conditional probabilities take opposite extreme values.
Practical Implications
- •The research has direct implications for medical decision-making and clinical practice guidelines, where plug-in prediction is commonly used.
- •Healthcare practitioners and policy makers should be aware of the potential for increased regret when using misspecified models and non-optimal thresholds, and should carefully consider the joint selection of models, estimation methods, and thresholds.
- •The findings suggest the need for further research into robust estimation methods that minimize maximum regret, even under model misspecification.
- •Future research could explore the use of x-specific thresholds or modifications to the predictor to mitigate the limitations of x-invariant thresholds.
- •The computational framework developed in the paper can be extended to evaluate the performance of different prediction models and threshold selection strategies in various application domains.