← Back to Sessions

Calibrated Uncertainty Estimates for Deep Learning-Based Phase Arrival Time Estimates

Description: 

Few techniques exist for estimating the prediction uncertainty of deep learning models. Nominally, estimates of prediction uncertainty are obtained by sampling from the predictive distribution, but this task is computationally intractable for deep learning models with millions of parameters. Consequently, a Bayesian approximation method must be used. Dropout at inference time is a common approach but generally fails to approximate the true posterior effectively. Here, we use a method called Stochastic Weight Averaging with Gaussian posterior (SWAG). This method explores a mode of the model posterior during training using stochastic gradient descent with a large learning rate. It computes and stores a mean for each model parameter and approximates a low-rank covariance matrix. At inference time, many model realizations can efficiently be sampled from the distribution of model parameters. We produce an ensemble of SWAG models, known as MultiSWAG, to compute phase arrival times and their uncertainties. We then compare the uncertainty estimates to those from dropout. We find the two methods are comparable when an appropriate dropout rate is selected. MultiSWAG produces a broader range of predicted pick distributions than dropout without overly restricting the minimum potential width. Additionally, MultiSWAG uncertainties show a stronger correlation with signal-to-noise ratio. However, MultiSWAG requires more hyperparameter tuning and is ~6-7 times slower. Both methods produce broader prediction distributions when shown noise examples. In either case, deep learning models are often over-confident, so we utilize an efficient, model-agnostic method of empirically calibrating the uncertainties to produce meaningful credible intervals. Before calibration, only 47% of the predicted MultiSWAG standard deviations for a test set contain the analyst pick. Afterward, 66% of analyst picks are within the predicted 68% credible interval bounds.

Session: Opportunities and Challenges for Machine Learning Applications in Seismology

Type: Oral

Date: 4/19/2023

Presentation Time: 11:00 AM (local time)

Presenting Author: Alysha D. Armstrong

Student Presenter: Yes

Invited Presentation: Yes

Authors