Skip to content

Evaluation Metrics for Volcanic Ash Cloud Forecasts

Hannah C.M. Susorney1 Frances M. Beckett1  Charlie E. Bates1

Abstract

Volcanic ash dispersion simulations can be evaluated with observations, particularly satellite imagery and retrievals, to inform how well the dispersion models simulate volcanic ash cloud location and concentration. In both volcanic ash dispersion and the wider community many evaluation metrics have been proposed and used, and effective evaluation would benefit from standardization of the metrics.    Evaluation metrics can be useful for both model development and in-event decision making. They allow decisions to be made about whether to implement new developments into dispersion models, by comparing simulations to historic observations with different model setups. During an event, metrics can support decision making, for example at Volcanic Ash Advisory Centres (VAACs), to assess whether the forecast is optimal and determine if changes to the simulation e.g., source terms or meteorological data used, would deliver forecasts which better match recent ash observations.  To explore using evaluation metrics in a systematic way we calculate a range of evaluation metrics for 5-day NAME ash dispersion simulations of the Raikoke 2019 eruption compared to satellite observations re-gridded with the same horizontal scale. We find that no one metric captures all that we are interested in for both model development and in-event decision making. Different metrics quantify different aspects of the ash cloud and are therefore useful in different ways. Evaluation of volcanic ash forecasts for both model development and in-event decision making may require a range of metrics to fully evaluate the model output.