go-v.orgdoi

Using Expert Interpretation and Reasoning to Guide Model Selection in Machine Learning

Jiaxuan Wang · Deep Blue (University of Michigan) · 2022

  • DOI 10.7302/4606
  • Publisher University of Michigan
  • Type dissertation
  • Issued 2022

Not cached yet. This record was fetched live for this request and is stored after the lookup. Reopening the page should serve the cached copy.

Deep Blue (University of Michigan) · University of Michigan · dissertation · 2022

Abstract

In using machine learning to train predictive models, training data often under-specify solutions due to limited sample size and/or the lack of diversity in samples. When selecting a solution (or model) for deployment, we must consider metrics beyond statistics calculated in distribution (i.e., statistics on samples available in model learning such as training and validation error) so that we can identify and correct potential pitfalls of the learned model when applied to out of distribution scenarios (e.g., the deployment environment). To address model underspecification, in this thesis, we develop several methods that leverage domain knowledge during model selection.

First, to select among solutions, one must understand the learned model. We demonstrate how one can achieve a holistic understanding by including system level knowledge about the problem. Our approach, Shapley Flow, takes a user defined causal graph on the features as input and summarizes the attribution to model prediction along the causal edges. Shapley Flow unifies three widely used Shapley-value based model interpretation methods and elucidates the need to consider the data generation procedure, capturing both the direct and indirect impact of features. Second, domain knowledge is not only useful in model interpretation, it is also crucial in training.

If one knows what features experts rely on, one can incorporate such knowledge to avoid using proxy features that are spuriously correlated with the outcome. We propose a novel regularization technique to learn a credible model, one that is both accurate and is aligned with domain experts. Using our approach, Expert Yielded Estimate (EYE), we demonstrate on two large scale clinical datasets that one can significantly increase alignment with expert knowledge without sacrificing accuracy, while outperforming an approach based on expert knowledge alone.

Finally, we connect credible learning with shortcut learning, identifying sufficient assumptions for credible models to eliminate the dependence on spurious correlation. In this process, we extend the EYE penalty to work with nonlinear models and work on tasks with domain knowledge not expressed on the input space. Applied to two benchmark datasets, our approach successfully mitigates shortcut learning, even when assumptions are moderately violated. By leveraging domain knowledge, our proposed approaches help build trustworthy systems that can be safely applied in practice.

Open the resolved locator · 10.7302/4606

Metadata licence: OpenAlex metadata (CC0)

Registry
OpenAlex DOI index + API
Fetched
2026-11-20 02:28 UTC
Cache state
not cached yet (bulk-index+openalex)
Locator
https://hdl.handle.net/2027.42/172577
Metadata licence
OpenAlex metadata (CC0)
Scope
Metadata as returned by the resolver; abstracts are never generated here.