{ "cells": [ { "cell_type": "markdown", "id": "3a8c4d8a", "metadata": {}, "source": [ "# Browsing a STAC Catalog with the `pystac_client`\n", "\n", "The `pystac_client` provide a python integration and command line tools to interact with a STAC catalog.\n", "The full documentation can be found [here](https://pystac-client.readthedocs.io/en/latest/). \n", "We will focus on examples to illustrate access to data within the WarmWorld catalog." ] }, { "cell_type": "markdown", "id": "ed955afe", "metadata": {}, "source": [ "## Opening a known Dataset\n", "\n", "For a known dataset, e.g. we have found the collection ID and item ID by browsing through the catalog with the STAC browser, we can quite easy get a way to open a dataset.\n", "\n", "Afterwards we need to follow the required procedure to actually open the dataset, which will be part of a different guide." ] }, { "cell_type": "code", "execution_count": 1, "id": "c2249888", "metadata": {}, "outputs": [], "source": [ "import pystac_client\n", "stac_url = \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/\"\n", "collection_id = \"ngc3028\"\n", "item_id = \"ngc3028_P1D_3\"\n", "catalog = pystac_client.Client.open(stac_url)\n", "collection = catalog.get_collection(collection_id)\n", "item = collection.get_item(item_id)\n" ] }, { "cell_type": "markdown", "id": "053a8998", "metadata": {}, "source": [ "Now we want to find out, which asset we can use to access the data. Let's therefore loop over the available entries and decide if there is something we can use." ] }, { "cell_type": "code", "execution_count": 2, "id": "3db3defe", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "asset name: disk\n", "{\n", " \"href\": \"file:///work/bm1235/k203123/nextgems_cycle3/experiments/ngc3028/outdata/ngc3028_P1D_3.zarr\",\n", " \"type\": \"application/vnd+zarr\"\n", "}\n", "\n" ] } ], "source": [ "import json\n", "for name, asset in item.assets.items():\n", " print(f\"asset name: {name}\")\n", " print( json.dumps(asset.to_dict(), indent=2))\n", " print()" ] }, { "cell_type": "markdown", "id": "66a6522b", "metadata": {}, "source": [ "We only find a single asset called disk. The type indicated, that we would open a zarr dataset. Having a closer look at the `href` shows the prefix `file://`, so we have a local file. We know that this dataset is stored at DKRZ. So accessing the data is only possible, if we have access to the file system and the permissions to this particular directory and files." ] }, { "cell_type": "markdown", "id": "8b609884", "metadata": {}, "source": [ "## A deeper look into a STAC catalog\n", "\n", "This is a good point to inspect the underlying entries closer. Let's print the content of the collection as well as the content of the item.\n", "For a nicer formatting, we weill use the json package." ] }, { "cell_type": "code", "execution_count": 3, "id": "2c1ab449", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"type\": \"Collection\",\n", " \"id\": \"ngc3028\",\n", " \"stac_version\": \"1.1.0\",\n", " \"description\": \"ngc3028 collection\",\n", " \"links\": [\n", " {\n", " \"rel\": \"aggregate\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/collections/ngc3028/aggregate\",\n", " \"type\": \"application/json\"\n", " },\n", " {\n", " \"rel\": \"aggregations\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/collections/ngc3028/aggregations\",\n", " \"type\": \"application/json\"\n", " },\n", " {\n", " \"rel\": \"items\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/collections/ngc3028/items\",\n", " \"type\": \"application/geo+json\"\n", " },\n", " {\n", " \"rel\": \"parent\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/\",\n", " \"type\": \"application/json\"\n", " },\n", " {\n", " \"rel\": \"queryables\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/collections/ngc3028/queryables\",\n", " \"type\": \"application/schema+json\"\n", " },\n", " {\n", " \"rel\": \"root\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/\",\n", " \"type\": \"application/json\",\n", " \"title\": \"Warm World Data Catalog\"\n", " },\n", " {\n", " \"rel\": \"self\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/collections/ngc3028\",\n", " \"type\": \"application/json\"\n", " }\n", " ],\n", " \"title\": \"\",\n", " \"extent\": {\n", " \"spatial\": {\n", " \"bbox\": [\n", " [\n", " -180,\n", " -90,\n", " 180,\n", " 90\n", " ]\n", " ]\n", " },\n", " \"temporal\": {\n", " \"interval\": [\n", " [\n", " \"2024-07-22T11:50:29.668363Z\",\n", " \"2024-07-23T11:50:29.668363Z\"\n", " ]\n", " ]\n", " }\n", " },\n", " \"license\": \"CC-BY-SA-4.0\"\n", "}\n" ] } ], "source": [ "import json\n", "print( json.dumps(collection.to_dict(), indent=2))" ] }, { "cell_type": "markdown", "id": "22f54e6b", "metadata": {}, "source": [ "We can see a clear structure on the output. This includes a nested tree with a number of different entries.\n", "A collection also has a number of links, to navigate through the catalog. For example, the child link gives all the items belonging to this collection.\n", "\n", "The queryables link gives an overview on possible metadata we can use in the search function. This also includes detailed descriptions. The extent has a bounding box that covers all included items. The temporal extend covers the first and last date and time part of the dataset. \n" ] }, { "cell_type": "code", "execution_count": 4, "id": "e176f361", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"type\": \"Feature\",\n", " \"stac_version\": \"1.1.0\",\n", " \"stac_extensions\": [],\n", " \"id\": \"ngc3028_P1D_3\",\n", " \"geometry\": {\n", " \"type\": \"Polygon\",\n", " \"coordinates\": [\n", " [\n", " [\n", " -180.0,\n", " -90.0\n", " ],\n", " [\n", " -180.0,\n", " 90.0\n", " ],\n", " [\n", " 180.0,\n", " 90.0\n", " ],\n", " [\n", " 180.0,\n", " -90.0\n", " ],\n", " [\n", " -180.0,\n", " -90.0\n", " ]\n", " ]\n", " ]\n", " },\n", " \"bbox\": [\n", " -180.0,\n", " -90.0,\n", " 180.0,\n", " 90.0\n", " ],\n", " \"properties\": {\n", " \"datetime\": \"2024-07-22T11:50:29.809918Z\",\n", " \"experiment\": \"ngc3028\",\n", " \"frequency\": \"P1D\",\n", " \"healpixLevel\": \"3\",\n", " \"variables\": \"-\",\n", " \"created\": \"2025-10-22T13:32:51.193696Z\",\n", " \"updated\": \"2025-10-22T13:32:51.193696Z\"\n", " },\n", " \"links\": [\n", " {\n", " \"rel\": \"self\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/collections/ngc3028/items/ngc3028_P1D_3\",\n", " \"type\": \"application/json\"\n", " },\n", " {\n", " \"rel\": \"parent\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/collections/ngc3028\",\n", " \"type\": \"application/json\"\n", " },\n", " {\n", " \"rel\": \"collection\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/collections/ngc3028\",\n", " \"type\": \"application/json\"\n", " },\n", " {\n", " \"rel\": \"root\",\n", " \"href\": \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/collections/ngc3028\",\n", " \"type\": \"application/json\",\n", " \"title\": \"\"\n", " }\n", " ],\n", " \"assets\": {\n", " \"disk\": {\n", " \"href\": \"file:///work/bm1235/k203123/nextgems_cycle3/experiments/ngc3028/outdata/ngc3028_P1D_3.zarr\",\n", " \"type\": \"application/vnd+zarr\"\n", " }\n", " },\n", " \"collection\": \"ngc3028\"\n", "}\n" ] } ], "source": [ "import json\n", "print( json.dumps(item.to_dict(), indent=2))" ] }, { "cell_type": "markdown", "id": "c42ed133", "metadata": {}, "source": [ "The general structure of the entry is similar to a collection, e.g. links, id, general information. An item has also additional entries. Most important is the assets block, with the possible ways to access the dataset. The geometry can be a polygon for a detailed description of the covered region. \n", "\n", "A new part is the properties block, that contains fields for the creation and update of the item as well as general metadata, like the frequency of the data or the healpixLevel that corresponds to the spatial resolution." ] }, { "cell_type": "markdown", "id": "adcdb698", "metadata": {}, "source": [ "## Browsing through a Catalog\n", "\n", "With this basic understanding of a STAC catalog, it should be clearer, that we can browse through the different layers, even without a package like the `pystac_client`. \n", "Nevertheless the `pystac_clients` simplifies a few things, so we will stick to it.\n", "\n", "Next, lets browse through the full catalog and search if we find something interesting:\n", "\n", "### Browsing through Collections:\n", "\n", "Let's first have a look at all the available collections:" ] }, { "cell_type": "code", "execution_count": 5, "id": "c59d5fef", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Title: ERA5, with collection ID: ERA5\n", "Title: IFS, with collection ID: IFS\n", "Title: MERRA-2, with collection ID: JSC-M2I3NVASM\n", "Title: IMERG v1, with collection ID: JSC-anemoi-4d2a51d7_6763_4b15_93a8_d8090405fec9\n", "Title: CERRA v2, with collection ID: JSC-anemoi-afeaa2db_5a20_421d_a5e6_270c7fa1bdd7\n", "Title: ERA5 (FDB), with collection ID: JSC-class~ea_domain~g_expver~0001_stream~oper_type~an\n", "Title: SEVIRI, with collection ID: SEVIRI\n", "Title: , with collection ID: ngc3028\n", "Title: , with collection ID: ngc4008\n", "Title: , with collection ID: ngcX001\n", "Title: WarmWorldEasierTestRun, with collection ID: stac_test\n" ] } ], "source": [ "import pystac_client\n", "stac_url = \"https://wwestac.cloud.dkrz.de/stac-fastapi-es/\"\n", "catalog = pystac_client.Client.open(stac_url)\n", "collections = catalog.get_collections()\n", "for collection in collections:\n", " print(f\"Title: {collection.title}, with collection ID: {collection.id}\")\n" ] }, { "cell_type": "markdown", "id": "eaa81ec4", "metadata": {}, "source": [ "We can now select one of these collections for a further inspection, like `JSC-class~ea_domain~g_expver~0001_stream~oper_type~an` \n", "Let us have a look at the available fields in the entry." ] }, { "cell_type": "code", "execution_count": 6, "id": "5e3105d0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inspecting keys in collection: JSC-class~ea_domain~g_expver~0001_stream~oper_type~an\n", "-- type\n", "-- id\n", "-- stac_version\n", "-- description\n", "-- links\n", "-- stac_extensions\n", "-- cube:dimensions\n", "-- cube:variables\n", "-- title\n", "-- extent\n", "-- license\n", "-- keywords\n", "-- providers\n", "-- summaries\n" ] } ], "source": [ "collection_id = \"JSC-class~ea_domain~g_expver~0001_stream~oper_type~an\"\n", "collection = catalog.get_collection(collection_id)\n", "print(f\"Inspecting keys in collection: {collection_id}\")\n", "for key in collection.to_dict().keys():\n", " print(\"-- \", key)" ] }, { "cell_type": "markdown", "id": "5b88ba6a", "metadata": {}, "source": [ "We see a number of keys, that we have already seen in the STAC browser, like title, providers, summaries, keywords, extend etc. \n", "We can also see keys with the prefix `cube:` those are dictionaries for the data cube extension, which gives a quite nice overview on the data cube.\n", "Let us for example figure out, which variables are inside this datacube:" ] }, { "cell_type": "code", "execution_count": 7, "id": "ef1c2509", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Variable in datacube:\n", "-- geopotential (ml)\n", "-- temperature (ml)\n", "-- u component of wind (ml)\n", "-- v component of wind (ml)\n", "-- specific humidity (ml)\n", "-- vertical velocity (ml)\n", "-- vorticity (relative) (ml)\n", "-- logarithm of surface pressure (ml)\n", "-- divergence (ml)\n", "-- ozone mass mixing ratio (ml)\n", "-- specific cloud liquid water content (ml)\n", "-- specific cloud ice water content (ml)\n", "-- fraction of cloud cover (ml)\n", "-- sea ice area fraction\n", "-- snow density\n", "-- sea surface temperature\n", "-- ice temperature layer 1\n", "-- volumetric soil water layer 1\n", "-- volumetric soil water layer 2\n", "-- volumetric soil water layer 3\n", "-- volumetric soil water layer 4\n", "-- convective available potential energy\n", "-- leaf area index, low vegetation\n", "-- leaf area index, high vegetation\n", "-- geopotential\n", "-- surface pressure\n", "-- soil temperature level 1\n", "-- snow depth\n", "-- mean sea level pressure\n", "-- boundary layer height\n", "-- total cloud cover\n", "-- 10 metre u wind component\n", "-- 10 metre v wind component\n", "-- 2 metre temperature\n", "-- 2 metre dewpoint temperature\n", "-- soil temperature level 2\n", "-- land-sea mask\n", "-- soil temperature level 3\n", "-- low cloud cover\n", "-- medium cloud cover\n", "-- high cloud cover\n", "-- skin reservoir content\n", "-- instantaneous moisture flux\n", "-- skin temperature\n", "-- soil temperature level 4\n", "-- temperature of snow layer\n" ] } ], "source": [ "print(\"Variable in datacube:\")\n", "for var_name in collection.extra_fields[\"cube:variables\"].keys():\n", " print(\"-- \", var_name)" ] }, { "cell_type": "markdown", "id": "87d14d74", "metadata": {}, "source": [ "Here, we see something interesting: `2 metre temperature`. We now want to plot this. The next step is to find the corresponding item." ] }, { "cell_type": "code", "execution_count": 8, "id": "4f54c7a9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Item Title : ERA5 (FDB) on ml - model levels, with item ID: JSC-class~ea_domain~g_expver~0001_stream~oper_type~an_levtype~ml\n" ] } ], "source": [ "itmes = collection.get_items()\n", "for item in itmes:\n", " print( f\"Item Title : {item.properties['title']}, with item ID: {item.id}\")" ] }, { "cell_type": "markdown", "id": "f136882d", "metadata": {}, "source": [ "We know that 2 metre temperature is a surface type variable, which is encoded as `levtype=sfc` in the ECMWF metadata standard.\n", "The catalog is structure, that surface variables are put together with each 3D field dataset. So we need to have a look at the results at model levels.\n", "\n", "In a case, where a dataset has only surface variables, those would be presented as an individual item.\n", "\n", "\n", "We will expect the assets. Here, we will focus on a few metadata entries. You can also plot the complete entry with\n", "```python3\n", "import json\n", "for name, asset in item.assets.items():\n", " print(json.dumps(asset.to_dict(), indent=2))\n", "```\n", "We do not show this output here as it is quite long due to the inclusion of the data cube extension." ] }, { "cell_type": "code", "execution_count": 9, "id": "347741e9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "zarr-Stream Request\n", "-- type: application/json\n", "-- roles: ['API', 'data', 'ZARR', 'ready to use']\n", "-- description: Opening of complete dataset as zarr. Included values are given in the metadata of this asset\n", "-- Volume: 81.69 GB\n", "-- Number of fields: 70932\n", "\n", "incomplete zarr-Stream Request\n", "-- type: application/json\n", "-- roles: ['API', 'data', 'zarr', 'requires specification']\n", "-- description: Opening of the dataset as zarr. You need to add the missing metadata to the request. We use a MARS based format. The following keys are required: datetime, date, levelist, param, step, time, param_sfc. Possible values are given in the metadata of this asset.\n", "-- Volume: 81.69 GB\n", "-- Number of fields: 70932\n", "\n", "grib-Download\n", "-- type: application/json\n", "-- roles: ['API', 'data', 'GRIB', 'ready to use']\n", "-- description: Link to request download of full dataset. The preparation of the dataset can require some time. The download might require a large amount of disc space. Included values are given in the metadata of this asset.\n", "-- Volume: 79.59 GB\n", "-- Number of fields: 69414\n", "\n", "incomplete grib-Download\n", "-- type: application/json\n", "-- roles: ['API', 'data', 'GRIB', 'requires specification']\n", "-- description: Link to request download of full dataset. The preparation of the dataset can require some time. The download might require a large amount of disc space. You need to add the missing metadata to the request. We use a MARS based format. The following keys are required: datetime, date, levelist, param, step, time. Possible values are given in the metadata of this asset.\n", "-- Volume: 79.59 GB\n", "-- Number of fields: 69414\n", "\n", "surface grib-Download\n", "-- type: application/json\n", "-- roles: ['API', 'data', 'GRIB', 'ready to use']\n", "-- description: Link to request download of full dataset. The preparation of the dataset can require some time. The download might require a large amount of disc space. Included values are given in the metadata of this asset.\n", "-- Volume: 2.1 GB\n", "-- Number of fields: 1518\n", "\n", "incomplete surface-grib-Download\n", "-- type: application/json\n", "-- roles: ['API', 'data', 'GRIB', 'requires specification']\n", "-- description: Link to request download of full dataset. The preparation of the dataset can require some time. The download might require a large amount of disc space. You need to add the missing metadata to the request. We use a MARS based format. The following keys are required: datetime, date, param, step, time. Possible values are given in the metadata of this asset.\n", "-- Volume: 2.1 GB\n", "-- Number of fields: 1518\n", "\n" ] } ], "source": [ "item = collection.get_item(\"JSC-class~ea_domain~g_expver~0001_stream~oper_type~an_levtype~ml\")\n", "for asset_name, asset in item.assets.items():\n", " print(f\"{asset.title}\")\n", " print(f\"-- type: {asset.media_type}\")\n", " print(f\"-- roles: {asset.roles}\")\n", " print(f\"-- description: {asset.description}\")\n", " print(f\"-- Volume: {asset.extra_fields['Volume']}\")\n", " print(f\"-- Number of fields: {asset.extra_fields['number of fields']}\")\n", " print()" ] }, { "cell_type": "markdown", "id": "30fb3288", "metadata": {}, "source": [ "We see four different entries. The descriptions reveal, that all of them are requests to an API. Two of them are marked with \"required specification\". Those are base requests, that need some details in case a user does only want to access a part of the data cube. \n", "The other two requests grant access or download the full data cube. Using those assets is a topic for another guide.\n", "\n" ] }, { "cell_type": "markdown", "id": "30986e8b", "metadata": {}, "source": [ "# Searching a Catalog with `pystac_client`\n", "As a last point in this section, we want to search for specific items with the `pystac_client`\n", "This should only serve as a small illustration. More sophisticated guides on filtering can be found in the [`pystac_client` tutorials](https://pystac-client.readthedocs.io/en/latest/tutorials.html)." ] }, { "cell_type": "code", "execution_count": 10, "id": "844a99d0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "We found the following items:\n", "\n", " item Title: IFS run started at 20220315-00:00 on pl - pressure levels\n", " -- Item ID: JSC-class~od_date~20220315_domain~g_expver~0001_stream~oper_time~0000_type~fc_levtype~pl\n", " -- Collection Info:\n", " ---- Collection Title: IFS\n", " ---- Collection ID: IFS\n", "\n", " item Title: IFS run started at 20220315-00:00 on ml - model levels\n", " -- Item ID: JSC-class~od_date~20220315_domain~g_expver~0001_stream~oper_time~0000_type~fc_levtype~ml\n", " -- Collection Info:\n", " ---- Collection Title: IFS\n", " ---- Collection ID: IFS\n", "\n", " item Title: ERA5 (FDB) on ml - model levels\n", " -- Item ID: JSC-class~ea_domain~g_expver~0001_stream~oper_type~an_levtype~ml\n", " -- Collection Info:\n", " ---- Collection Title: ERA5 (FDB)\n", " ---- Collection ID: JSC-class~ea_domain~g_expver~0001_stream~oper_type~an\n" ] } ], "source": [ "stac_api_url = \"https://esm-data.fz-juelich.de/catalog/api/v1\"\n", "client = pystac_client.Client.open(stac_api_url)\n", "search = client.search(\n", " datetime=[\"2022-03-15\"],\n", "# query=[\"levtype=ml - model levels\"]\n", ")\n", "print(\"We found the following items:\")\n", "for item in search.item_collection().items:\n", " coll = item.get_parent()\n", " print(\"\")\n", " print(f\" item Title: {item.properties['title']}\")\n", " print(f\" -- Item ID: {item.id}\")\n", " print( \" -- Collection Info:\")\n", " print(f\" ---- Collection Title: {coll.title}\")\n", " print(f\" ---- Collection ID: {item.collection_id}\")" ] }, { "cell_type": "markdown", "id": "3596c13d", "metadata": {}, "source": [ "The result of the search is are all items, that match the search. \n", "For the output we first get access to the parent collection of each item `coll = item.get_parent()`. \n", "This allows us to print the tree and access metadata from both levels." ] } ], "metadata": { "kernelspec": { "display_name": ".venv (3.11.13)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.13" } }, "nbformat": 4, "nbformat_minor": 5 }