Introduction to products and measurements

  • Products used: ls8_sr

  • Prerequisites: Users of this notebook should have a basic understanding of:

Keywords beginner’s guide; products, beginner’s guide; measurements, products; beginner’s guide, measurements; beginner’s guide, data used; landsat 8

Background

A “datacube” is a digital information architecture that specialises in hosting and cataloguing spatial information. Digital Earth Africa (DE Africa) is based on the Open Data Cube infrastructure, and specialises in storing remotely sensed data, particularly from Earth Observation satellites such as Landsat and Sentinel.

The Digital Earth Africa datacube contains both raw satellite data and derivative data “products”. These data products are often composed of a range of “measurements” such as the suite of remote sensing band values or statistical product summaries. Before running a query to load data from the datacube, it is useful to know what it contains. This notebook demonstrates several straightforward ways to inspect the product and measurement contents of a datacube.

Description

This notebook demonstrates how to connect to the Digital Earth Africa datacube and interrogate the available products and measurements stored within. Topics covered include:

  • How to connect to a datacube

  • How to list all the products

  • How to list a selected product’s measurements

  • How to interactively visualise data in the datacube

Getting started

To run this introduction to products and measurements, run all the cells in the notebook starting with the “Load packages” cell. For help with running notebook cells, refer back to the Jupyter Notebooks notebook.

Load packages

The datacube package is required to access and work with available data. The pandas package is required to format tables. The DcViewer utility will allow us to interactively explore the products available in the datacube.

[1]:

import datacube
import pandas as pd
from odc.ui import DcViewer

# Set some configurations for displaying tables nicely
pd.set_option('display.max_colwidth', 200)
pd.set_option('display.max_rows', None)

Connect to the datacube

After importing the datacube package, users need to specify a name for their session, known as the app name.

This name is generated by the user and is used to track down issues with database queries. It does not have any effect on the analysis. Use a short name that is consistent with the purpose of your notebook such as the way 02_Products_and_measurements has been used as the app name in this notebook.

The resulting dc object is what we use to access all the data contained within the Digital Earth Africa datacube.

[2]:

dc = datacube.Datacube(app="02_Products_and_measurements")

List products

Once a datacube instance has been created, users can explore the products and measurements stored within.

The following cell lists all product attributes currently available in the Digital Earth Africa datacube by using the dc.list_products().columns function.

[3]:

dc.list_products().columns

[3]:

Index(['name', 'description', 'license', 'default_crs', 'default_resolution'], dtype='object')

Any of these can be used to customise the product information returned by the dc.list_products() function, as shown in the next cell.

Additionally, the next cell lists all products that are currently available in the Digital Earth Africa datacube by using the dc.list_products() function.

Products listed under name in the following table represent the product options available when querying the datacube. The table below provides some useful information about each product, including a brief product description, the instrument and platform the data originated from (e.g. Landsat 8 OLI), and the product’s default crs (coordinate reference system) and resolution if applicable.

[4]:

products = dc.list_products()

display_columns = ["name",
                   "description",
                   "default_crs",
                   "default_resolution"]

products[display_columns].sort_index()

[4]:
name description default_crs default_resolution
name
alos_palsar_mosaic alos_palsar_mosaic ALOS/PALSAR and ALOS-2/PALSAR-2 annual mosaic tiles generated for use in the Data Cube - 25m pixel spacing, WGS84. These tiles are derived from the orignal JAXA mosaics with conversion to GeoTIFF. None None
crop_mask_eastern crop_mask_eastern Annual cropland extent map for Eastern Africa produced by Digital Earth Africa. None None
crop_mask_western crop_mask_western Annual cropland extent map for Western Africa produced by Digital Earth Africa. None None
dem_srtm dem_srtm 1 second elevation model EPSG:4326 (-0.00027777777778, 0.00027777777778)
fc_ls fc_ls Landsat Fractional Cover Observations from Space None None
ga_ls8c_wofs_2 ga_ls8c_wofs_2 Historic Flood Mapping Water Observations from Space None None
ga_ls8c_wofs_2_annual_summary ga_ls8c_wofs_2_annual_summary Water Observations from Space Annual Statistics None None
ga_ls8c_wofs_2_summary ga_ls8c_wofs_2_summary Water Observations from Space Full History Statistics None None
gm_s2_annual gm_s2_annual Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Sentinel-2 None None
gm_s2_annual_lowres gm_s2_annual_lowres Annual Geometric Median, Sentinel-2 - Low Resolution mosaic None None
gm_s2_semiannual gm_s2_semiannual Surface Reflectance Semiannual Geometric Median and Median Absolute Deviations, Sentinel-2 None None
io_lulc io_lulc ESRI Landcover Classification None None
jers_sar_mosaic jers_sar_mosaic JERS-1 SAR annual mosaic tiles generated for use in the Data Cube 25m pixel spacing, WGS84. These tiles are derived from the orignal JAXA mosaics with conversion to GeoTIFF. None None
ls5_sr ls5_sr USGS Landsat 5 Collection 2 Level-2 Surface Reflectance None None
ls5_st ls5_st USGS Landsat 5 Collection 2 Level-2 Surface Temperature None None
ls7_sr ls7_sr USGS Landsat 7 Collection 2 Level-2 Surface Reflectance None None
ls7_st ls7_st USGS Landsat 7 Collection 2 Level-2 Surface Temperature None None
ls8_sr ls8_sr USGS Landsat 8 Collection 2 Level-2 Surface Reflectance None None
ls8_st ls8_st USGS Landsat 8 Collection 2 Level-2 Surface Temperature None None
nasadem nasadem NASADEM from Microsoft's Planetary Computer EPSG:4326 (-0.0002777777777777778, 0.0002777777777777778)
pc_ls8_annual pc_ls8_annual Landsat-8 Annual Clear Pixel Count None None
pc_s2_annual pc_s2_annual Surface Reflectance Annual Clear Pixel Count, Sentinel-2 None None
s1_rtc s1_rtc Sentinel 1 Gamma0 normalised radar backscatter EPSG:4326 (-0.0002, 0.0002)
s2_l2a s2_l2a Sentinel-2a and Sentinel-2b imagery, processed to Level 2A (Surface Reflectance) and converted to Cloud Optimized GeoTIFFs None None
wofs_ls wofs_ls Historic Flood Mapping Water Observations from Space None None
wofs_ls_summary_alltime wofs_ls_summary_alltime Water Observations from Space Alltime Statistics None None
wofs_ls_summary_annual wofs_ls_summary_annual Water Observations from Space Annual Statistics None None

List measurements

Most products are associated with a range of available measurements. These can be individual satellite bands (e.g. Landsat’s near-infrared band) or statistical product summaries.

Using the name column of products listed above, let’s interrogate the measurements associated with the ls8_usgs_sr_scene product using the dc.list_measurements() function. This product name refers to the US Geological Survey’s Landsat 8 Analysis-ready data product.

The table below includes a range of technical information about each band in the dataset, including any aliases which can be used to load the data, the data type or dtype, any flags_definition that are associated with the measurement (this information is used for tasks like cloud masking), and the measurement’s nodata value.

Change the product name below and re-run the following cell to explore available measurements associated with other products.

[5]:

product = "ls8_sr"

measurements = dc.list_measurements()
measurements.loc[product]

[5]:
name dtype units nodata aliases flags_definition
measurement
SR_B1 SR_B1 uint16 1 0.0 [band_1, coastal_aerosol] NaN
SR_B2 SR_B2 uint16 1 0.0 [band_2, blue] NaN
SR_B3 SR_B3 uint16 1 0.0 [band_3, green] NaN
SR_B4 SR_B4 uint16 1 0.0 [band_4, red] NaN
SR_B5 SR_B5 uint16 1 0.0 [band_5, nir] NaN
SR_B6 SR_B6 uint16 1 0.0 [band_6, swir_1] NaN
SR_B7 SR_B7 uint16 1 0.0 [band_7, swir_2] NaN
QA_PIXEL QA_PIXEL uint16 bit_index 1.0 [pq, pixel_quality] {'snow': {'bits': 5, 'values': {'0': 'not_high_confidence', '1': 'high_confidence'}}, 'clear': {'bits': 6, 'values': {'0': False, '1': True}}, 'cloud': {'bits': 3, 'values': {'0': 'not_high_confid...
QA_RADSAT QA_RADSAT uint16 bit_index 0.0 [radsat, radiometric_saturation] {'nir_saturation': {'bits': 4, 'values': {'0': False, '1': True}}, 'red_saturation': {'bits': 3, 'values': {'0': False, '1': True}}, 'blue_saturation': {'bits': 1, 'values': {'0': False, '1': True...
SR_QA_AEROSOL SR_QA_AEROSOL uint8 bit_index 1.0 [qa_aerosol, aerosol_qa] {'water': {'bits': 2, 'values': {'0': False, '1': True}}, 'nodata': {'bits': 0, 'values': {'0': False, '1': True}}, 'aerosol_level': {'bits': [6, 7], 'values': {'0': 'climatology', '1': 'low', '2'...

Visualising available data

For a more visual way of exploring the data that is available within the Digital Earth Africa datacube, we can use the interactive DcViewer utility or the online DE Africa Explorer website. We will use the DcViewer utility in this exiercise. Select a product from the drop-down menu on the top-left of the map to show the areas data is available for in blue. You can also use the back and forward buttons above the map to toggle through time.

The utility is only able to visualise a limited number of datasets at one time. If the available data footprints do not appear, either press the “show” button on the top right, or zoom further in on the map.

[6]:

DcViewer(dc=dc,
         time='2017',
         center=(0.565, 38.007),
         zoom=6)

Additional information

License: The code in this notebook is licensed under the Apache License, Version 2.0. Digital Earth Africa data is licensed under the Creative Commons by Attribution 4.0 license.

Contact: If you need assistance, please post a question on the Open Data Cube Slack channel or on the GIS Stack Exchange using the open-data-cube tag (you can view previously asked questions here). If you would like to report an issue with this notebook, you can file one on Github.

Compatible datacube version:

[7]:

print(datacube.__version__)

1.8.5

Last Tested:

[8]:

from datetime import datetime
datetime.today().strftime('%Y-%m-%d')

[8]:

'2021-09-13'