Machine learning models are increasingly deployed in high-stakes domains such as healthcare and autonomous systems, where decisions carry significant risks. Probabilistic machine learning is valuable in these settings, as it quantifies predictive uncertainty, notably by generating probabilistic predictions. We focus on regression, where the goal is to predict one or more continuous outputs given a set of inputs. In this context, we consider two main forms of uncertainty representation: predictive distributions, which assign probabilities to possible output values, and prediction sets, which are designed to contain the true output with a pre-specified probability. For these predictions to be reliable and informative, they must be calibrated and sharp, i.e., statistically consistent with observed data and concentrated around the true value.
In this thesis, we develop distribution-free regression methods to produce calibrated and sharp probabilistic predictions using neural network models. We consider both single-output and the less-explored multi-output regression settings. Specifically, we develop and study recalibration, regularization, and conformal prediction (CP) methods. The first adjusts predictions after model training, the second augments the training objective, and the last produces prediction sets with finite-sample coverage guarantees.
For single-output regression, we conduct a large-scale experimental study to provide a comprehensive comparison of these methods. The results reveal that post-hoc approaches consistently achieve superior calibration. We explain this finding by establishing a formal link between recalibration and CP, showing that recalibration also benefits from finite-sample coverage guarantees. However, the separate training and recalibration steps typically lead to degraded negative log-likelihood. To address this issue, we develop an end-to-end training procedure that incorporates the recalibration objective directly into learning, resulting in improved negative log-likelihood while maintaining calibration.
For multi-output regression, we conduct a comparative study of CP methods and introduce new classes of approaches that offer novel trade-offs between sharpness, compatibility with generative models, and computational efficiency. A key challenge in CP is achieving conditional coverage, which ensures that coverage guarantees hold for specific inputs rather than only on average. To address this, we propose a method that improves conditional coverage using conditional quantile regression, thereby avoiding the need to estimate full conditional distributions. Finally, for tasks requiring a full predictive density, we introduce a recalibration technique that operates in the latent space of invertible generative models such as conditional normalizing flows. This approach yields an explicit, calibrated multivariate probability density function. Collectively, these contributions advance the theory and practice of uncertainty quantification in machine learning, facilitating the development of more reliable predictive systems across diverse applications.
@phdthesis{Dheur2025-distribution,title={Distribution-Free and Calibrated Predictive Uncertainty in Probabilistic Machine Learning},author={Dheur, Victor},school={University of Mons},address={Mons, Belgium},month=dec,year={2025},}
Reliably characterizing the full conditional distribution of a multivariate response variable given a set of covariates is crucial for trustworthy decision-making. However, misspecified or miscalibrated multivariate models may yield a poor approximation of the joint distribution of the response variables, leading to unreliable predictions and suboptimal decisions. Furthermore, standard recalibration methods are primarily limited to univariate settings, while conformal prediction techniques, despite generating multivariate prediction regions with coverage guarantees, do not provide a full probability density function. We address this gap by first introducing a novel notion of latent calibration, which assesses probabilistic calibration in the latent space of a conditional normalizing flow. Second, we propose latent recalibration (LR), a novel post-hoc model recalibration method that learns a transformation of the latent space with finite-sample bounds on latent calibration. Unlike existing methods, LR produces a recalibrated distribution with an explicit multivariate density function while remaining computationally efficient. Extensive experiments on both tabular and image datasets show that LR consistently improves latent calibration error and the negative log-likelihood of the recalibrated models.
@inproceedings{Dheur2025-multivariate,title={Multivariate latent recalibration for conditional normalizing flows},author={Dheur, Victor and Ben Taieb, Souhaib},booktitle={The 39th Annual Conference on Neural Information Processing Systems},year={2025},}
Conformal prediction provides a powerful framework for constructing distribution-free prediction regions with finite-sample coverage guarantees. While extensively studied in univariate settings, its extension to multi-output problems presents additional challenges, including complex output dependencies and high computational costs, and remains relatively underexplored. In this work, we present a unified comparative study of nine conformal methods with different multivariate base models for constructing multivariate prediction regions within the same framework. This study highlights their key properties while also exploring the connections between them. Additionally, we introduce two novel classes of conformity scores for multi-output regression that generalize their univariate counterparts. These scores ensure asymptotic conditional coverage while maintaining exact finite-sample marginal coverage. One class is compatible with any generative model, offering broad applicability, while the other is computationally efficient, leveraging the properties of invertible generative models. Finally, we conduct a comprehensive empirical evaluation across 13 tabular datasets, comparing all the multi-output conformal methods explored in this work. To ensure a fair and consistent comparison, all methods are implemented within a unified code base.
@inproceedings{Dheur2025-unified,title={A unified comparative study with generalized conformity scores for multi-output conformal regression},author={Dheur, Victor and Fontana, Matteo and Estievenart, Yorick and Desobry, Naomi and Ben Taieb, Souhaib},booktitle={The 42nd International Conference on Machine Learning},year={2025},}
We present a new method for generating confidence sets within the split conformal prediction framework. Our method performs a trainable transformation of any given conformity score to improve conditional coverage while ensuring exact marginal coverage. The transformation is based on an estimate of the conditional quantile of conformity scores. The resulting method is particularly beneficial for constructing adaptive confidence sets in multi-output problems where standard conformal quantile regression approaches have limited applicability. We develop a theoretical bound that captures the influence of the accuracy of the quantile estimate on the approximate conditional validity, unlike classical bounds for conformal prediction methods that only offer marginal coverage. We experimentally show that our method is highly adaptive to the local data structure and outperforms existing methods in terms of conditional coverage, improving the reliability of statistical inference in various applications.
@inproceedings{Plassier2025-rectifying,title={Rectifying conformity scores for better conditional coverage},author={Plassier*, Vincent and Fishkov*, Alexander and Dheur*, Victor and Guizani, Mohsen and Ben Taieb, Souhaib and Panov, Maxim and Moulines, Eric},booktitle={The 42nd International Conference on Machine Learning},year={2025},note={(<em>* denotes equal contribution.</em>)}}
With the growing utilization of machine learning models in real-world applications, improving neural network calibration has become a primary concern. Various methods have been proposed, including post-hoc methods that adjust predictions after training and regularization methods that act during training. In regression, post-hoc methods have shown superior calibration performance compared to regularization methods. However, the base model is trained without any direct connection to the subsequent post-hoc step. To address this limitation, we introduce a novel end-to-end method called Quantile Recalibration Training, integrating post-hoc calibration directly into the training process. We also propose an algorithm unifying our method with other post-hoc and regularization methods. We demonstrate the performance of Quantile Recalibration Training in a large-scale experiment involving 57 tabular regression datasets, showcasing improved predictive accuracy. Additionally, we conduct an ablation study, revealing the significance of different elements in our method. Aiming for reproducibility and fair comparisons, we have implemented our experiments in a common code base.
@inproceedings{Dheur2024-probabilistic,title={Probabilistic Calibration by Design for Neural Network Regression},booktitle={The 27th International Conference on Artificial Intelligence and Statistics},author={Dheur, Victor and Ben Taieb, Souhaib},year={2024},}
Sequences of labeled events observed at irregular intervals
in continuous time are ubiquitous across various fields.
Temporal Point Processes (TPPs) provide a mathematical
framework for modeling these sequences, enabling inferences
such as predicting the arrival time of future events and
their associated label, called mark. However, due to model
misspecification or lack of training data, these
probabilistic models may provide a poor approximation of the
true, unknown underlying process, with prediction regions
extracted from them being unreliable estimates of the
underlying uncertainty. This paper develops more reliable
methods for uncertainty quantification in neural TPP models
via the framework of conformal prediction. A primary
objective is to generate a distribution-free joint
prediction region for the arrival time and mark, with a
finite-sample marginal coverage guarantee. A key challenge
is to handle both a strictly positive, continuous response
and a categorical response, without distributional
assumptions. We first consider a simple but overly
conservative approach that combines individual prediction
regions for the event arrival time and mark. Then, we
introduce a more effective method based on bivariate highest
density regions derived from the joint predictive density of
event arrival time and mark. By leveraging the dependencies
between these two variables, this method exclude unlikely
combinations of the two, resulting in sharper prediction
regions while still attaining the pre-specified coverage
level. We also explore the generation of individual
univariate prediction regions for arrival times and marks
through conformal regression and classification techniques.
Moreover, we investigate the stronger notion of conditional
coverage. Finally, through extensive experimentation on both
simulated and real-world datasets, we assess the validity
and efficiency of these methods.
@inproceedings{Dheur2024-distribution,title={Distribution-Free Conformal Joint Prediction Regions for Neural Marked Temporal Point Processes},booktitle={Machine Learning, Call for Papers: Conformal Prediction and Distribution-Free Uncertainty Quantification},author={Dheur*, Victor and Bosser*, Tanguy and Izbicki, Rafael and Ben Taieb, Souhaib},year={2024},note={(<em>* denotes equal contribution.</em>)}}
Accurate estimation of predictive uncertainty is essential for
optimal decision making. However, recent works have shown that
current neural networks tend to be miscalibrated, sparking
interest in different approaches to calibration. In this paper,
we conduct a large-scale empirical study of the probabilistic
calibration of neural networks on 57 tabular regression
datasets. We consider recalibration, conformal and
regularization approaches, and investigate the trade-offs they
induce on calibration and sharpness of the predictions. Based on
kernel density estimation, we design new differentiable
recalibration and regularization methods, yielding new insights
into the performance of these approaches. Furthermore, we find
conditions under which recalibration and conformal prediction
are equivalent. Our study is fully reproducible and implemented
in a common code base for fair comparison.
@inproceedings{Dheur2023-large,title={A Large-Scale Study of Probabilistic Calibration in Neural Network Regression},booktitle={The 40th International Conference on Machine Learning},author={Dheur, Victor and Ben Taieb, Souhaib},year={2023},}