About this project

Many changes during the model development cycle can result in non-bit-for-bit changes. These include porting to a new compute environment, compiler changes, code modifications and optimizations, and running on different hardware. This project aims to ensure that these changes do not adversely affect the results and to determine whether the differences are due to a bug in the code. This is done by evaluating the model’s variability through ensembles and hypothesis testing based on Principal Component Analysis (PCA). This work involves developing a tool (PyCECT) that provides a simple PASS or FAIL result and integrating it within the automated testing frameworks of the Community Atmosphere Model (CAM), MPAS-A, and MOM6.

csf-correctness-2.png

csf-correctness-1.png

Why this work is important

To ensure that changes to the code, compiler, and computer architecture that result in non-bit-for-bit answers do not adversely affect simulation results and scientific outcomes, ensuring software excellence and confidence in our results.

How does this fit within the CSF

  1. Fits within Better Practices and automated testing
  2. Tests for code portability across different architectures

NSF NCAR Labs involved in this project

CISL
CGD
MMM

External partners

Colorado School of Mines
University of Colorado

More information

Github repository: https://github.com/NCAR/PyCECT
Documentation: https://pycect.readthedocs.io/en/latest/

Publications:

Price-Broncucia, T., Baker, A., Hammerling, D., Duda, M., and Morrison, R.: The ensemble consistency test: from CESM to MPAS and beyond, Geosci. Model Dev., 18, 2349–2372, https://doi.org/10.5194/gmd-18-2349-2025, 2025.

Milroy, D. J., Baker, A. H., Hammerling, D. M., and Jessup, E. R.: Nine time steps: ultra-fast statistical consistency testing of the Community Earth System Model (pyCECT v3.0), Geosci. Model Dev., 11, 697–711, https://doi.org/10.5194/gmd-11-697-2018, 2018.

Baker, A. H., Hammerling, D. M., Levy, M. N., Xu, H., Dennis, J. M., Eaton, B. E., Edwards, J., Hannay, C., Mickelson, S. A., Neale, R. B., Nychka, D., Shollenberger, J., Tribbia, J., Vertenstein, M., and Williamson, D.: A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0), Geosci. Model Dev., 8, 2829–2840, https://doi.org/10.5194/gmd-8-2829-2015, 2015.