Verification of AI-models in Operational Centers
Verification of AI models in Operational Centers
Montreal and online, 22-23 October 2025
Contact and Organizer: barbara.casati@ec.gc.ca
Workshop background information
The recent rapid improvement of AI-based weather prediction has prompted
national meteorological centres to begin operationalizing pure-AI and hybrid models,
integrating AI systems with the traditional physically-based Numerical Weather Prediction (NWP) systems.
Robust, fair, and transparent verification methods are essential to understand
the characteristics and capabilities of these new systems in comparison to traditional NWP,
to guide their effective use, and to pave the way toward optimal hybrid approaches to weather forecasting.
This workshop brings together international scientists and researchers
from the WWRP Joint Working Group on Forecast Verification Research
(JWGFVR),
from the WCRP Working Group on Numerical Experiment (WGNE),
and from the Meteorological Service of Canada
(MSC) and the
Meteorological Research Division
(MRD)
of Environment and Climate Change Canada
(ECCC),
to address emerging challenges in the operational verification of AI models alongside traditional NWP systems.
We aim to examine how established verification practices can be adapted to account for the distinct
characteristics of AI-based forecasts, explore metrics and novel diagnostics for meaningful comparisons,
and promote best practices to ensure that WMO Members and forecast users can trust and interpret outputs
from both paradigms with confidence. Building on discussions from the two preceding
(2024,
2025)
VerAI workshops hosted at the University of Reading (UK), this event also aims to
establish a baseline for verification activities within the WGNE Weather Predition Model
Intercomparison Projet (WP-MIP),
and will inform the ongoing revision of the
WIPPS standardized verification.
Contributions and Registration
Contributions on verification approaches for comparing AI, hybrid and traditional NWP models are welcome.
Topics of interest include (but are not limited to) the following verification research topics:
- Developing best practices for fair comparison between AI, hybrid and physical NWP models.
- Verification observation reference and incestuousness of verification against own analysis.
- Spatial methods: neighborhood, filtering and scale-separation approaches; field morphing and displacement methods.
- Process diagnostics and physical coherence assessment.
- Verification of extreme and rare events.
Contributions and registration deadline is the 26th of September.
To register and/or to propose a contribution, please write an email to barbara.casati@ec.gc.ca.
Please note that solely registered participants will receive the link for online attendance.
Program
The workshop will consist of three open sessions (Wednesday and Thursday morning),
with in-person and online presentations followed by Questions and Answers (Q&A).
The workshop final session will also include an open discussion to summarize the
most promising venues for operational verification of AI-models.
Thursday afternoon a small group of representatives of the JWGFVR, WGNE, WP-MIP and ECCC
will meet to outline the WP-MIP verification activities, leveraging on the workshop major outcomes.
- Wednesday 22nd October, 9h00-12h00: Open Session 1 -
Novel verification approaches for AI-models
- Wednesday 22nd October, 13h00-16h00: Open Session 2 -
Verification practices in operational centers
- Thursday 23rd October, 9h00-12h00: Open Session 3 -
AI-model verification in operational practices
- Thursday 23rd October, 14h00-15h00: Closed Meeting (JWGFVR,WGNE,WP-MIP and ECCC) - Outline of the WP-MIP verification activities
Confirmed Speakers:
- Barbara Casati (JWGFVR and ECCC, Canada)
- Ron McTaggart-Cowan (WGNE and ECCC, Canada)
- Jochen Broecker (JWGFVR and Reading Univ, UK)
- Romain Pic (Geneva Univ, Switzerland)
- Sebastian Lerch (KIT, Germany)
- Christopher Subich (ECCC, Canada)
- Zied BenBouallegue (JWGFVR and ECMWF, UK)
- Sabrina Whal (JWGFVR and DWD, Germany)
- Nick Loveday (JWGFVR and BoM, Australia)
- Leo Separovic (ECCC, Canada)
The program is still taking shape, if you wish to contribute feel free to contact the organizer.
Logistic
- In-person attendees:
the workshop will take place
in the meeting room Salle des Vents (107),
at the Canadian Meteorological Center (CMC),
2121 Transcanada hwy (Dorval), H9P 1J3, Canada.
The CMC can be reached by the bus 215, from metro station Cote-Vertu (orange line),
or by taxi (most convenient, approximately 50$-60$ from downtown Montreal).
If you plan to attend in person, bring a personal ID for security clearance.
- Local Arrangements:
more detailed information on transport and accomodation in downtown Montreal and near CMC
can be found in this Local Arrangements (pdf) document.
- Online attendance:
please note that the agenda refers to local Montreal times (UTC-4), and that
solely registered attendees will receive the link for online participation
- Costs: there is no registration fee associated to this workshop;
there is also no hospitality nor funds (participants are expected to cover their own travelling costs).
- Participation (online and in-person) is limited,
and contributing participants
(scientists and researchers associated with the JWGFVR,WGNE,WP-MIP and ECCC) will be prioritized.
Useful links