Introduction to products and measurements
Products used: ls8_sr
Prerequisites: Users of this notebook should have a basic understanding of:
How to run a Jupyter notebook
Keywords beginner’s guide; products, beginner’s guide; measurements, products; beginner’s guide, measurements; beginner’s guide, data used; landsat 8
Background
A “datacube” is a digital information architecture that specialises in hosting and cataloguing spatial information. Digital Earth Africa (DE Africa) is based on the Open Data Cube infrastructure, and specialises in storing remotely sensed data, particularly from Earth Observation satellites such as Landsat and Sentinel.
The Digital Earth Africa datacube contains both raw satellite data and derivative data “products”. These data products are often composed of a range of “measurements” such as the suite of remote sensing band values or statistical product summaries. Before running a query to load data from the datacube, it is useful to know what it contains. This notebook demonstrates several straightforward ways to inspect the product and measurement contents of a datacube.
Description
This notebook demonstrates how to connect to the Digital Earth Africa datacube and interrogate the available products and measurements stored within. Topics covered include:
How to connect to a datacube
How to list all the products
How to list a selected product’s measurements
How to interactively visualise data in the datacube
Getting started
To run this introduction to products and measurements, run all the cells in the notebook starting with the “Load packages” cell. For help with running notebook cells, refer back to the Jupyter Notebooks notebook.
Load packages
The datacube package is required to access and work with available data. The pandas package is required to format tables. The DcViewer utility will allow us to interactively explore the products available in the datacube.
[1]:
import datacube
import pandas as pd
from odc.ui import DcViewer
# Set some configurations for displaying tables nicely
pd.set_option('display.max_colwidth', 200)
pd.set_option('display.max_rows', None)
Connect to the datacube
After importing the datacube package, users need to specify a name for their session, known as the app name.
This name is generated by the user and is used to track down issues with database queries. It does not have any effect on the analysis. Use a short name that is consistent with the purpose of your notebook such as the way 02_Products_and_measurements has been used as the app name in this notebook.
The resulting dc object is what we use to access all the data contained within the Digital Earth Africa datacube.
[2]:
dc = datacube.Datacube(app="02_Products_and_measurements")
List products
Once a datacube instance has been created, users can explore the products and measurements stored within.
The following cell lists all product attributes currently available in the Digital Earth Africa datacube by using the dc.list_products().columns function.
[3]:
dc.list_products().columns
[3]:
Index(['name', 'description', 'license', 'default_crs', 'default_resolution'], dtype='object')
Any of these can be used to customise the product information returned by the dc.list_products() function, as shown in the next cell.
Additionally, the next cell lists all products that are currently available in the Digital Earth Africa datacube by using the dc.list_products() function.
Products listed under name in the following table represent the product options available when querying the datacube. The table below provides some useful information about each product, including a brief product description, the instrument and platform the data originated from (e.g. Landsat 8 OLI), and the product’s default crs (coordinate reference system) and resolution if applicable.
[4]:
products = dc.list_products()
display_columns = ["name",
"description",
"default_crs",
"default_resolution"]
products[display_columns].sort_index()
[4]:
| name | description | default_crs | default_resolution | |
|---|---|---|---|---|
| name | ||||
| alos_palsar_mosaic | alos_palsar_mosaic | ALOS/PALSAR and ALOS-2/PALSAR-2 annual mosaic tiles generated for use in the Data Cube - 25m pixel spacing, WGS84. These tiles are derived from the orignal JAXA mosaics with conversion to GeoTIFF. | None | None |
| cci_landcover | cci_landcover | ESA Climate Change Initiative Land Cover | None | None |
| cgls_landcover | cgls_landcover | Copernicus Global Land Service, Land Use/Land Cover at 100 m | None | None |
| crop_mask | crop_mask | Annual cropland extent map produced by Digital Earth Africa. | None | None |
| crop_mask_central | crop_mask_central | Annual cropland extent map for Central Africa produced by Digital Earth Africa. | None | None |
| crop_mask_eastern | crop_mask_eastern | Annual cropland extent map for Eastern Africa produced by Digital Earth Africa. | None | None |
| crop_mask_indian_ocean | crop_mask_indian_ocean | Annual cropland extent map for Indian Ocean Africa produced by Digital Earth Africa. | None | None |
| crop_mask_northern | crop_mask_northern | Annual cropland extent map for Northern Africa produced by Digital Earth Africa. | None | None |
| crop_mask_sahel | crop_mask_sahel | Annual cropland extent map for Sahel Africa produced by Digital Earth Africa. | None | None |
| crop_mask_southeast | crop_mask_southeast | Annual cropland extent map for Southeast Africa produced by Digital Earth Africa. | None | None |
| crop_mask_southern | crop_mask_southern | Annual cropland extent map for Southern Africa produced by Digital Earth Africa. | None | None |
| crop_mask_western | crop_mask_western | Annual cropland extent map for Western Africa produced by Digital Earth Africa. | None | None |
| dem_cop_30 | dem_cop_30 | Copernicus DEM 30 m | None | None |
| dem_cop_90 | dem_cop_90 | Copernicus DEM 90 m | None | None |
| dem_srtm | dem_srtm | 1 second elevation model | None | None |
| dem_srtm_deriv | dem_srtm_deriv | 1 second elevation model derivatives | None | None |
| esa_worldcover | esa_worldcover | ESA World Cover, global 10 m land use/land cover data from 2020. | None | None |
| esa_worldcover_2020 | esa_worldcover_2020 | ESA World Cover, global 10 m land use/land cover data from 2020. | None | None |
| esa_worldcover_2021 | esa_worldcover_2021 | ESA World Cover, global 10 m land use/land cover data from 2021. | None | None |
| fc_ls | fc_ls | Landsat Fractional Cover Observations from Space | None | None |
| fc_ls_summary_annual | fc_ls_summary_annual | DE Africa Landsat Fractional Cover Percentiles | None | None |
| gm_ls5_ls7_annual | gm_ls5_ls7_annual | Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 5 and Landsat 7 | None | None |
| gm_ls5_ls7_annual_lowres | gm_ls5_ls7_annual_lowres | Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 5 and Landsat 7. Low resolution version used for visualisations. | None | None |
| gm_ls8_annual | gm_ls8_annual | Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 8 | None | None |
| gm_ls8_annual_lowres | gm_ls8_annual_lowres | Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 8. Low resolution version used for visualisations. | None | None |
| gm_ls8_ls9_annual | gm_ls8_ls9_annual | Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 8 and Landsat 9 | None | None |
| gm_ls8_ls9_annual_lowres | gm_ls8_ls9_annual_lowres | Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 8 and Landsat 9. Low resolution version used for visualisations. | None | None |
| gm_s2_annual | gm_s2_annual | Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Sentinel-2 | None | None |
| gm_s2_annual_lowres | gm_s2_annual_lowres | Annual Geometric Median, Sentinel-2 - Low Resolution mosaic | None | None |
| gm_s2_rolling | gm_s2_rolling | Surface Reflectance 3 Monthly Rolling Geometric Median and Median Absolute Deviations, Sentinel-2 | None | None |
| gm_s2_semiannual | gm_s2_semiannual | Surface Reflectance Semiannual Geometric Median and Median Absolute Deviations, Sentinel-2 | None | None |
| gm_s2_semiannual_lowres | gm_s2_semiannual_lowres | Surface Reflectance Semiannual Geometric Median and Median Absolute Deviations, Sentinel-2. Low resolution version used for visualisations. | None | None |
| gmw | gmw | Global Mangrove Watch data sourced from the UN Environment Program at https://data.unep-wcmc.org/datasets/45 | None | None |
| io_lulc | io_lulc | Impact Observatory (ESRI) Landcover Classification 9 Class | None | None |
| isda_soil_bedrock_depth | isda_soil_bedrock_depth | Soil bedrock depth predictions made by iSDA Africa | None | None |
| isda_soil_bulk_density | isda_soil_bulk_density | Soil bulk density predictions made by iSDA Africa | None | None |
| isda_soil_carbon_total | isda_soil_carbon_total | Soil total carbon predictions made by iSDA Africa | None | None |
| isda_soil_clay_content | isda_soil_clay_content | Soil clay content predictions made by iSDA Africa | None | None |
| isda_soil_sand_content | isda_soil_sand_content | Soil sand content predictions made by iSDA Africa | None | None |
| isda_soil_silt_content | isda_soil_silt_content | Soil silt content predictions made by iSDA Africa | None | None |
| jers_sar_mosaic | jers_sar_mosaic | JERS-1 SAR annual mosaic tiles generated for use in the Data Cube 25m pixel spacing, WGS84. These tiles are derived from the orignal JAXA mosaics with conversion to GeoTIFF. | None | None |
| landsat_c2l2_ar | landsat_c2l2_ar | USGS Landsat Collection 2 Level-2 Provisional Aquatic Reflectance | None | None |
| ls5_sr | ls5_sr | USGS Landsat 5 Collection 2 Level-2 Surface Reflectance | None | None |
| ls5_st | ls5_st | USGS Landsat 5 Collection 2 Level-2 Surface Temperature | None | None |
| ls7_sr | ls7_sr | USGS Landsat 7 Collection 2 Level-2 Surface Reflectance | None | None |
| ls7_st | ls7_st | USGS Landsat 7 Collection 2 Level-2 Surface Temperature | None | None |
| ls8_sr | ls8_sr | USGS Landsat 8 Collection 2 Level-2 Surface Reflectance | None | None |
| ls8_st | ls8_st | USGS Landsat 8 Collection 2 Level-2 Surface Temperature | None | None |
| ls9_sr | ls9_sr | USGS Landsat 9 Collection 2 Level-2 Surface Reflectance | None | None |
| ls9_st | ls9_st | USGS Landsat 9 Collection 2 Level-2 Surface Temperature | None | None |
| nasadem | nasadem | NASADEM from Microsoft's Planetary Computer | None | None |
| ndvi_anomaly | ndvi_anomaly | Monthly NDVI Anomalies produced by Digital Earth Africa. | None | None |
| ndvi_climatology_ls | ndvi_climatology_ls | Monthly NDVI Climatologies produced by Digital Earth Africa. | None | None |
| pc_s2_annual | pc_s2_annual | Surface Reflectance Annual Clear Pixel Count, Sentinel-2 | None | None |
| rainfall_chirps_daily | rainfall_chirps_daily | Rainfall Estimates from Rain Gauge and Satellite Observations | None | None |
| rainfall_chirps_monthly | rainfall_chirps_monthly | Rainfall Estimates from Rain Gauge and Satellite Observations | None | None |
| s1_rtc | s1_rtc | Sentinel 1 Gamma0 normalised radar backscatter | None | None |
| s2_l2a | s2_l2a | Sentinel-2a and Sentinel-2b imagery, processed to Level 2A (Surface Reflectance) and converted to Cloud Optimized GeoTIFFs | None | None |
| wofs_ls | wofs_ls | Historic Flood Mapping Water Observations from Space | None | None |
| wofs_ls_summary_alltime | wofs_ls_summary_alltime | Water Observations from Space Alltime Statistics | None | None |
| wofs_ls_summary_annual | wofs_ls_summary_annual | Water Observations from Space Annual Statistics | None | None |
| wsf_2015 | wsf_2015 | The World Settlement Footprint WSF 2015 version 2 (WSF2015 v2) is a 10m resolution binary maskoutlining the extent of human settlements globally for the year 2015 based on multiple datasets. | None | None |
| wsf_2019 | wsf_2019 | The World Settlement Footprint (WSF) 2019 is a 10m resolution binary mask outlining the extent of human settlements globally derived by means of 2019 multitemporal Sentinel-1 (S1) and Sentinel-2 (... | None | None |
| wsf_evolution | wsf_evolution | The World Settlement Footprint (WSF) Evolution is a 30m resolution dataset derived from Landsat-5/7 outlining the global settlement extent on a yearly basis from 1985 to 2015. | None | None |
List measurements
Most products are associated with a range of available measurements. These can be individual satellite bands (e.g. Landsat’s near-infrared band) or statistical product summaries.
Using the name column of products listed above, let’s interrogate the measurements associated with the ls8_usgs_sr_scene product using the dc.list_measurements() function. This product name refers to the US Geological Survey’s Landsat 8 Analysis-ready data product.
The table below includes a range of technical information about each band in the dataset, including any aliases which can be used to load the data, the data type or dtype, any flags_definition that are associated with the measurement (this information is used for tasks like cloud masking), and the measurement’s nodata value.
Change the product name below and re-run the following cell to explore available measurements associated with other products.
[5]:
product = "ls8_sr"
measurements = dc.list_measurements()
measurements.loc[product]
[5]:
| name | dtype | units | nodata | aliases | flags_definition | |
|---|---|---|---|---|---|---|
| measurement | ||||||
| SR_B1 | SR_B1 | uint16 | 1 | 0.0 | [band_1, coastal_aerosol] | NaN |
| SR_B2 | SR_B2 | uint16 | 1 | 0.0 | [band_2, blue] | NaN |
| SR_B3 | SR_B3 | uint16 | 1 | 0.0 | [band_3, green] | NaN |
| SR_B4 | SR_B4 | uint16 | 1 | 0.0 | [band_4, red] | NaN |
| SR_B5 | SR_B5 | uint16 | 1 | 0.0 | [band_5, nir] | NaN |
| SR_B6 | SR_B6 | uint16 | 1 | 0.0 | [band_6, swir_1] | NaN |
| SR_B7 | SR_B7 | uint16 | 1 | 0.0 | [band_7, swir_2] | NaN |
| QA_PIXEL | QA_PIXEL | uint16 | bit_index | 1.0 | [pq, pixel_quality] | {'snow': {'bits': 5, 'values': {'0': 'not_high_confidence', '1': 'high_confidence'}}, 'clear': {'bits': 6, 'values': {'0': False, '1': True}}, 'cloud': {'bits': 3, 'values': {'0': 'not_high_confid... |
| QA_RADSAT | QA_RADSAT | uint16 | bit_index | 0.0 | [radsat, radiometric_saturation] | {'nir_saturation': {'bits': 4, 'values': {'0': False, '1': True}}, 'red_saturation': {'bits': 3, 'values': {'0': False, '1': True}}, 'blue_saturation': {'bits': 1, 'values': {'0': False, '1': True... |
| SR_QA_AEROSOL | SR_QA_AEROSOL | uint8 | bit_index | 1.0 | [qa_aerosol, aerosol_qa] | {'water': {'bits': 2, 'values': {'0': False, '1': True}}, 'nodata': {'bits': 0, 'values': {'0': False, '1': True}}, 'aerosol_level': {'bits': [6, 7], 'values': {'0': 'climatology', '1': 'low', '2'... |
Visualising available data
For a more visual way of exploring the data that is available within the Digital Earth Africa datacube, we can use the interactive DcViewer utility or the online DE Africa Explorer website. We will use the DcViewer utility in this exiercise. Select a product from the drop-down menu on the top-left of the map to show the areas data is available for in blue. You can also use the back and forward buttons above the map to toggle
through time.
The utility is only able to visualise a limited number of datasets at one time. If the available data footprints do not appear, either press the “show” button on the top right, or zoom further in on the map.
[6]:
DcViewer(dc=dc,
time='2017',
center=(0.565, 38.007),
zoom=6)
[6]:
Recommended next steps
For more advanced information about working with Jupyter Notebooks or JupyterLab, you can explore JupyterLab documentation page.
To continue working through the notebooks in this beginner’s guide, the following notebooks are designed to be worked through in the following order:
Products and measurements (this notebook)
Once you have you have completed the above six tutorials, join advanced users in exploring:
The “Datasets” directory in the repository, where you can explore DE Africa products in depth.
The “Frequently used code” directory, which contains a recipe book of common techniques and methods for analysing DE Africa data.
The “Real-world examples” directory, which provides more complex workflows and analysis case studies.
Additional information
License: The code in this notebook is licensed under the Apache License, Version 2.0. Digital Earth Africa data is licensed under the Creative Commons by Attribution 4.0 license.
Contact: If you need assistance, please post a question on the Open Data Cube Slack channel or on the GIS Stack Exchange using the open-data-cube tag (you can view previously asked questions here). If you would like to report an issue with this notebook, you can file one on
Github.
Compatible datacube version:
[7]:
print(datacube.__version__)
1.8.15
Last Tested:
[8]:
from datetime import datetime
datetime.today().strftime('%Y-%m-%d')
[8]:
'2023-08-11'