Introduction to products and measurements

  • Products used: ls8_sr

  • Prerequisites: Users of this notebook should have a basic understanding of:

Keywords beginner’s guide; products, beginner’s guide; measurements, products; beginner’s guide, measurements; beginner’s guide, data used; landsat 8

Background

A “datacube” is a digital information architecture that specialises in hosting and cataloguing spatial information. Digital Earth Africa (DE Africa) is based on the Open Data Cube infrastructure, and specialises in storing remotely sensed data, particularly from Earth Observation satellites such as Landsat and Sentinel.

The Digital Earth Africa datacube contains both raw satellite data and derivative data “products”. These data products are often composed of a range of “measurements” such as the suite of remote sensing band values or statistical product summaries. Before running a query to load data from the datacube, it is useful to know what it contains. This notebook demonstrates several straightforward ways to inspect the product and measurement contents of a datacube.

Description

This notebook demonstrates how to connect to the Digital Earth Africa datacube and interrogate the available products and measurements stored within. Topics covered include:

  • How to connect to a datacube

  • How to list all the products

  • How to list a selected product’s measurements

  • How to interactively visualise data in the datacube


Getting started

To run this introduction to products and measurements, run all the cells in the notebook starting with the “Load packages” cell. For help with running notebook cells, refer back to the Jupyter Notebooks notebook.

Load packages

The datacube package is required to access and work with available data. The pandas package is required to format tables. The DcViewer utility will allow us to interactively explore the products available in the datacube.

[1]:
import datacube
import pandas as pd
from odc.ui import DcViewer

# Set some configurations for displaying tables nicely
pd.set_option('display.max_colwidth', 200)
pd.set_option('display.max_rows', None)

Connect to the datacube

After importing the datacube package, users need to specify a name for their session, known as the app name.

This name is generated by the user and is used to track down issues with database queries. It does not have any effect on the analysis. Use a short name that is consistent with the purpose of your notebook such as the way 02_Products_and_measurements has been used as the app name in this notebook.

The resulting dc object is what we use to access all the data contained within the Digital Earth Africa datacube.

[2]:
dc = datacube.Datacube(app="02_Products_and_measurements")

List products

Once a datacube instance has been created, users can explore the products and measurements stored within.

The following cell lists all product attributes currently available in the Digital Earth Africa datacube by using the dc.list_products().columns function.

[3]:
dc.list_products().columns
[3]:
Index(['name', 'description', 'license', 'default_crs', 'default_resolution'], dtype='object')

Any of these can be used to customise the product information returned by the dc.list_products() function, as shown in the next cell.

Additionally, the next cell lists all products that are currently available in the Digital Earth Africa datacube by using the dc.list_products() function.

Products listed under name in the following table represent the product options available when querying the datacube. The table below provides some useful information about each product, including a brief product description, the instrument and platform the data originated from (e.g. Landsat 8 OLI), and the product’s default crs (coordinate reference system) and resolution if applicable.

[4]:
products = dc.list_products()

display_columns = ["name",
                   "description",
                   "default_crs",
                   "default_resolution"]

products[display_columns].sort_index()
[4]:
name description default_crs default_resolution
name
alos_palsar_mosaic alos_palsar_mosaic ALOS/PALSAR and ALOS-2/PALSAR-2 annual mosaic tiles generated for use in the Data Cube - 25m pixel spacing, WGS84. These tiles are derived from the orignal JAXA mosaics with conversion to GeoTIFF. None None
cci_landcover cci_landcover ESA Climate Change Initiative Land Cover None None
cgls_landcover cgls_landcover Copernicus Global Land Service, Land Use/Land Cover at 100 m None None
crop_mask crop_mask Annual cropland extent map produced by Digital Earth Africa. None None
crop_mask_central crop_mask_central Annual cropland extent map for Central Africa produced by Digital Earth Africa. None None
crop_mask_eastern crop_mask_eastern Annual cropland extent map for Eastern Africa produced by Digital Earth Africa. None None
crop_mask_indian_ocean crop_mask_indian_ocean Annual cropland extent map for Indian Ocean Africa produced by Digital Earth Africa. None None
crop_mask_northern crop_mask_northern Annual cropland extent map for Northern Africa produced by Digital Earth Africa. None None
crop_mask_sahel crop_mask_sahel Annual cropland extent map for Sahel Africa produced by Digital Earth Africa. None None
crop_mask_southeast crop_mask_southeast Annual cropland extent map for Southeast Africa produced by Digital Earth Africa. None None
crop_mask_southern crop_mask_southern Annual cropland extent map for Southern Africa produced by Digital Earth Africa. None None
crop_mask_western crop_mask_western Annual cropland extent map for Western Africa produced by Digital Earth Africa. None None
dem_cop_30 dem_cop_30 Copernicus DEM 30 m None None
dem_cop_90 dem_cop_90 Copernicus DEM 90 m None None
dem_srtm dem_srtm 1 second elevation model None None
dem_srtm_deriv dem_srtm_deriv 1 second elevation model derivatives None None
esa_worldcover esa_worldcover ESA World Cover, global 10 m land use/land cover data from 2020. None None
esa_worldcover_2020 esa_worldcover_2020 ESA World Cover, global 10 m land use/land cover data from 2020. None None
esa_worldcover_2021 esa_worldcover_2021 ESA World Cover, global 10 m land use/land cover data from 2021. None None
fc_ls fc_ls Landsat Fractional Cover Observations from Space None None
fc_ls_summary_annual fc_ls_summary_annual DE Africa Landsat Fractional Cover Percentiles None None
gm_ls5_ls7_annual gm_ls5_ls7_annual Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 5 and Landsat 7 None None
gm_ls5_ls7_annual_lowres gm_ls5_ls7_annual_lowres Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 5 and Landsat 7. Low resolution version used for visualisations. None None
gm_ls8_annual gm_ls8_annual Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 8 None None
gm_ls8_annual_lowres gm_ls8_annual_lowres Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 8. Low resolution version used for visualisations. None None
gm_ls8_ls9_annual gm_ls8_ls9_annual Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 8 and Landsat 9 None None
gm_ls8_ls9_annual_lowres gm_ls8_ls9_annual_lowres Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Landsat 8 and Landsat 9. Low resolution version used for visualisations. None None
gm_s2_annual gm_s2_annual Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Sentinel-2 None None
gm_s2_annual_lowres gm_s2_annual_lowres Annual Geometric Median, Sentinel-2 - Low Resolution mosaic None None
gm_s2_rolling gm_s2_rolling Surface Reflectance 3 Monthly Rolling Geometric Median and Median Absolute Deviations, Sentinel-2 None None
gm_s2_semiannual gm_s2_semiannual Surface Reflectance Semiannual Geometric Median and Median Absolute Deviations, Sentinel-2 None None
gm_s2_semiannual_lowres gm_s2_semiannual_lowres Surface Reflectance Semiannual Geometric Median and Median Absolute Deviations, Sentinel-2. Low resolution version used for visualisations. None None
gmw gmw Global Mangrove Watch data sourced from the UN Environment Program at https://data.unep-wcmc.org/datasets/45 None None
io_lulc io_lulc Impact Observatory (ESRI) Landcover Classification 9 Class None None
isda_soil_bedrock_depth isda_soil_bedrock_depth Soil bedrock depth predictions made by iSDA Africa None None
isda_soil_bulk_density isda_soil_bulk_density Soil bulk density predictions made by iSDA Africa None None
isda_soil_carbon_total isda_soil_carbon_total Soil total carbon predictions made by iSDA Africa None None
isda_soil_clay_content isda_soil_clay_content Soil clay content predictions made by iSDA Africa None None
isda_soil_sand_content isda_soil_sand_content Soil sand content predictions made by iSDA Africa None None
isda_soil_silt_content isda_soil_silt_content Soil silt content predictions made by iSDA Africa None None
jers_sar_mosaic jers_sar_mosaic JERS-1 SAR annual mosaic tiles generated for use in the Data Cube 25m pixel spacing, WGS84. These tiles are derived from the orignal JAXA mosaics with conversion to GeoTIFF. None None
landsat_c2l2_ar landsat_c2l2_ar USGS Landsat Collection 2 Level-2 Provisional Aquatic Reflectance None None
ls5_sr ls5_sr USGS Landsat 5 Collection 2 Level-2 Surface Reflectance None None
ls5_st ls5_st USGS Landsat 5 Collection 2 Level-2 Surface Temperature None None
ls7_sr ls7_sr USGS Landsat 7 Collection 2 Level-2 Surface Reflectance None None
ls7_st ls7_st USGS Landsat 7 Collection 2 Level-2 Surface Temperature None None
ls8_sr ls8_sr USGS Landsat 8 Collection 2 Level-2 Surface Reflectance None None
ls8_st ls8_st USGS Landsat 8 Collection 2 Level-2 Surface Temperature None None
ls9_sr ls9_sr USGS Landsat 9 Collection 2 Level-2 Surface Reflectance None None
ls9_st ls9_st USGS Landsat 9 Collection 2 Level-2 Surface Temperature None None
nasadem nasadem NASADEM from Microsoft's Planetary Computer None None
ndvi_anomaly ndvi_anomaly Monthly NDVI Anomalies produced by Digital Earth Africa. None None
ndvi_climatology_ls ndvi_climatology_ls Monthly NDVI Climatologies produced by Digital Earth Africa. None None
pc_s2_annual pc_s2_annual Surface Reflectance Annual Clear Pixel Count, Sentinel-2 None None
rainfall_chirps_daily rainfall_chirps_daily Rainfall Estimates from Rain Gauge and Satellite Observations None None
rainfall_chirps_monthly rainfall_chirps_monthly Rainfall Estimates from Rain Gauge and Satellite Observations None None
s1_rtc s1_rtc Sentinel 1 Gamma0 normalised radar backscatter None None
s2_l2a s2_l2a Sentinel-2a and Sentinel-2b imagery, processed to Level 2A (Surface Reflectance) and converted to Cloud Optimized GeoTIFFs None None
wofs_ls wofs_ls Historic Flood Mapping Water Observations from Space None None
wofs_ls_summary_alltime wofs_ls_summary_alltime Water Observations from Space Alltime Statistics None None
wofs_ls_summary_annual wofs_ls_summary_annual Water Observations from Space Annual Statistics None None
wsf_2015 wsf_2015 The World Settlement Footprint WSF 2015 version 2 (WSF2015 v2) is a 10m resolution binary maskoutlining the extent of human settlements globally for the year 2015 based on multiple datasets. None None
wsf_2019 wsf_2019 The World Settlement Footprint (WSF) 2019 is a 10m resolution binary mask outlining the extent of human settlements globally derived by means of 2019 multitemporal Sentinel-1 (S1) and Sentinel-2 (... None None
wsf_evolution wsf_evolution The World Settlement Footprint (WSF) Evolution is a 30m resolution dataset derived from Landsat-5/7 outlining the global settlement extent on a yearly basis from 1985 to 2015. None None

List measurements

Most products are associated with a range of available measurements. These can be individual satellite bands (e.g. Landsat’s near-infrared band) or statistical product summaries.

Using the name column of products listed above, let’s interrogate the measurements associated with the ls8_usgs_sr_scene product using the dc.list_measurements() function. This product name refers to the US Geological Survey’s Landsat 8 Analysis-ready data product.

The table below includes a range of technical information about each band in the dataset, including any aliases which can be used to load the data, the data type or dtype, any flags_definition that are associated with the measurement (this information is used for tasks like cloud masking), and the measurement’s nodata value.

Change the product name below and re-run the following cell to explore available measurements associated with other products.

[5]:
product = "ls8_sr"

measurements = dc.list_measurements()
measurements.loc[product]
[5]:
name dtype units nodata aliases flags_definition
measurement
SR_B1 SR_B1 uint16 1 0.0 [band_1, coastal_aerosol] NaN
SR_B2 SR_B2 uint16 1 0.0 [band_2, blue] NaN
SR_B3 SR_B3 uint16 1 0.0 [band_3, green] NaN
SR_B4 SR_B4 uint16 1 0.0 [band_4, red] NaN
SR_B5 SR_B5 uint16 1 0.0 [band_5, nir] NaN
SR_B6 SR_B6 uint16 1 0.0 [band_6, swir_1] NaN
SR_B7 SR_B7 uint16 1 0.0 [band_7, swir_2] NaN
QA_PIXEL QA_PIXEL uint16 bit_index 1.0 [pq, pixel_quality] {'snow': {'bits': 5, 'values': {'0': 'not_high_confidence', '1': 'high_confidence'}}, 'clear': {'bits': 6, 'values': {'0': False, '1': True}}, 'cloud': {'bits': 3, 'values': {'0': 'not_high_confid...
QA_RADSAT QA_RADSAT uint16 bit_index 0.0 [radsat, radiometric_saturation] {'nir_saturation': {'bits': 4, 'values': {'0': False, '1': True}}, 'red_saturation': {'bits': 3, 'values': {'0': False, '1': True}}, 'blue_saturation': {'bits': 1, 'values': {'0': False, '1': True...
SR_QA_AEROSOL SR_QA_AEROSOL uint8 bit_index 1.0 [qa_aerosol, aerosol_qa] {'water': {'bits': 2, 'values': {'0': False, '1': True}}, 'nodata': {'bits': 0, 'values': {'0': False, '1': True}}, 'aerosol_level': {'bits': [6, 7], 'values': {'0': 'climatology', '1': 'low', '2'...

Visualising available data

For a more visual way of exploring the data that is available within the Digital Earth Africa datacube, we can use the interactive DcViewer utility or the online DE Africa Explorer website. We will use the DcViewer utility in this exiercise. Select a product from the drop-down menu on the top-left of the map to show the areas data is available for in blue. You can also use the back and forward buttons above the map to toggle through time.

The utility is only able to visualise a limited number of datasets at one time. If the available data footprints do not appear, either press the “show” button on the top right, or zoom further in on the map.

[6]:
DcViewer(dc=dc,
         time='2017',
         center=(0.565, 38.007),
         zoom=6)
[6]:

Additional information

License: The code in this notebook is licensed under the Apache License, Version 2.0. Digital Earth Africa data is licensed under the Creative Commons by Attribution 4.0 license.

Contact: If you need assistance, please post a question on the Open Data Cube Slack channel or on the GIS Stack Exchange using the open-data-cube tag (you can view previously asked questions here). If you would like to report an issue with this notebook, you can file one on Github.

Compatible datacube version:

[7]:
print(datacube.__version__)
1.8.15

Last Tested:

[8]:
from datetime import datetime
datetime.today().strftime('%Y-%m-%d')
[8]:
'2023-08-11'