Interpretations of Ensemble-Based Forecast Guidance from the 2017 Spring Forecast Experiment Survey (167)

Pam Heinselman, NOAA NSSL, Norman, OK

Katie Wilson, OU CIMMS

Pat Skinner, OU CIMMS

Kim Klockow-McClain, OU CIMMS

Jessica Choate, OU CIMMS



Operational and experimental convection-allowing model (CAM) ensembles and probabilistic forecast guidance are becoming increasingly available to forecasters. As a result, the paradigm for interpreting forecast guidance is evolving from one that is deterministic to one that is probabilistic. The strengths and limitations of CAMs are being tested and evaluated by model developers and users within NOAA National Centers for Environmental Prediction, NOAA Weather Forecast Offices, and within NOAA testbeds. The focus of these evaluations is mostly on verification statistics. While verification statistics provide information useful for evidence-based decision making, they do not provide insight into how probabilistic forecast guidance is interpreted or used by meteorologists.The goal of this study is to sample and document meteorologists' interpretations of probabilistic forecast guidance via a survey. This survey was administered to meteorologists as a part of the first evaluation of the experimental Warn-on-Forecast (WoF) system within the 2017 Spring Forecast Experiment, held at the NOAA Hazardous Weather Testbed in Norman, Oklahoma 1 May - 2 June. The WoF system is a frequently updated, regional-scale, on-demand convection-allowing ensemble analysis and prediction system, nested within an hourly convection-allowing ensemble forecast system (currently HRRRe). This system produces 0-3-h predictions of individual convective storms and mesoscale environments that provide probabilistic forecast guidance, such as exceedance probabilities (e.g., probability of simulated reflectivity above a threshold reflectivity value at a grid point) and ensemble percentile values (e.g., 90th) of accumulated rainfall, 2-5-km updraft helicity, and 0-2-km vertical vorticity, among others.Meteorologists were presented 12 open-ended and multiple choice questions that queried their interpretation of probabilistic and percentile-based products in spatial, temporal, and severity space using ensemble forecast output from the WoF system. For example, &ldquoIn an ensemble-based probabilistic forecast, what do you think the 70th percentile value of accumulated rainfall represents?&rdquo and &ldquoGiven the information presented, what is the probability of exceeding 0.5&rdquo of rainfall within box A?&rdquo Questions on the joint interpretation of probabilistic and percentile products are also included. The analysis is conducted using survey responses from 60 meteorologists, classified as researchers (n=38), forecasters (n=12), and student or post-doc (n=10). Within this sample, there is a mix of correct and incorrect interpretations of ensemble-based probabilistic forecast guidance. Some common issues include: mistaking probability for amount, inconsistent interpretation of trajectory uncertainty at the end of forecast swaths, and misunderstanding of the underlying probability distribution function of an ensemble forecast. Representative results from a subset of questions will be shown in the presentation. Findings indicate a need for training on ensemble-based forecast guidance across the meteorological community and amplify the importance of investigating and understanding meteorologists' mental models a priori of a potential paradigm shift in forecast operations.