WarmWorld Easier – An overview of the improvements to the lifecycle of km scale Earth System Model simulations¶
Easier ICON Simulations¶
-
Workflow management with mkexp
mkexp creates model experiment workflows from simple configuration files. It generates shell scripts to set up storage, prepare input data, define parameter files and complementing processes, and orchestrate model run and processing. The same configuration may be used across HPC environments.
-
Analysis ready datasets with Hiopy
Hiopy is a tool to create multi-resolution analysis ready datasets. It uses YAC as its data provider. This means any climate model that uses YAC can use Hiopy as long as the model also exposes its fields to YAC. It would execute with its MPI subcommunicator in a hybrid/het MPI job. The primary driver of the interface is the ICON climate model.
-
ComIn insitu PAMTRA plugin
The developments prototyped the use of an instrument simulator (PAMTRA) during an ICON simulation coupled via ComIn. This allows a more flexible and reduced output - e.g. to mimic a satellite observation, only the satellite track and the observable quantity (e.g. radar reflectivity) has to be saved instead of the full 3D fields. This enables an easier and more suitable comparison.
Easier Data System: Findability, Accessibility and Storage¶
-
Freva:
A unified data access platform for models, reanalysis data, and observations that integrates a fast, user-friendly search engine with DKRZ's High-Performance Computing system. It provides a low-barrier environment for web and shell/Python users, streamlining data retrieval and simplifying the development of sophisticated analysis tools.
-
SpatioTemporal Asset Catalog (STAC):
STAC is a standard for metadata catalogs in Earth System Sciences. Within WarmWorld-Easier, we are developing a federated catalog that combines datasets stored at various institutions. The STAC browser provides a graphical user interface (GUI) for human readability. STAC itself is based on the JSON format, making it perfectly suited to machine access (catalog endpoint).
-
User Guide for STAC and Data Access Services:
User guides covering a brief tutorial on browsing the STAC catalog and accessing data through services deployed within WarmWorld-Easier. Data access services include the direct hosting of Zarr over HTTP, as well as serving data from a Fields DataBase (FDB) as Zarr via a web service utilising the Zarr-FDB interface. The latter allows for user-defined datasets. Data access also includes direct disc access where applicable.
-
User Documentation for FDB, PyFDB and ZFDB
WarmWorld Easier provides user-focused documentation for the services built around the FDB, a domain-specific object store for GRIB data. It explains how to access and work with data through the FDB's Python interface as well as through virtual Zarr stores, enabling users to efficiently create and manage new datasets derived from existing FDB resources.
-
Handling Zarr data movement across storage tiers:
The zarr data is characteristically stored in a large number of small files (chunks). So before it can be stored on long term storage tiers like tape archive it needs some reformatting into aggregates of larger size. An approach has been developed to handle this data movement across the storage tiers to systematically pack the zarr stores into tarballs and index their content for faster access.
Easier Knowlege Gain¶
-
ProImpact
ProImpact is a WarmWorld-Smarter project that aims at using the tools developed within Easier to transfer classical user-analysis and data-extraction workflows to data streams. A Dashboard is currently being developed with the features of accessing stored data and performing analysis, as well as streaming data from models as they run. Another aspect is the creation of ComIn plugins that build on the capabilities of Hiopy for efficiently storing high temporal resolution data on the disk.
