Introduction to products and measurements

  • Products used: ls8_sr

  • Prerequisites: Users of this notebook should have a basic understanding of:

Keywords beginner’s guide; products, beginner’s guide; measurements, products; beginner’s guide, measurements; beginner’s guide, data used; landsat 8

Background

A “datacube” is a digital information architecture that specialises in hosting and cataloguing spatial information. Digital Earth Africa (DE Africa) is based on the Open Data Cube infrastructure, and specialises in storing remotely sensed data, particularly from Earth Observation satellites such as Landsat and Sentinel.

The Digital Earth Africa datacube contains both raw satellite data and derivative data “products”. These data products are often composed of a range of “measurements” such as the suite of remote sensing band values or statistical product summaries. Before running a query to load data from the datacube, it is useful to know what it contains. This notebook demonstrates several straightforward ways to inspect the product and measurement contents of a datacube.

Description

This notebook demonstrates how to connect to the Digital Earth Africa datacube and interrogate the available products and measurements stored within. Topics covered include:

  • How to connect to a datacube

  • How to list all the products

  • How to list a selected product’s measurements

  • How to interactively visualise data in the datacube


Getting started

To run this introduction to products and measurements, run all the cells in the notebook starting with the “Load packages” cell. For help with running notebook cells, refer back to the Jupyter Notebooks notebook.

Load packages

The datacube package is required to access and work with available data. The pandas package is required to format tables. The DcViewer utility will allow us to interactively explore the products available in the datacube.

[17]:
import datacube
import pandas as pd
from odc.ui import DcViewer

# Set some configurations for displaying tables nicely
pd.set_option('display.max_colwidth', 200)
pd.set_option('display.max_rows', None)

Connect to the datacube

After importing the datacube package, users need to specify a name for their session, known as the app name.

This name is generated by the user and is used to track down issues with database queries. It does not have any effect on the analysis. Use a short name that is consistent with the purpose of your notebook such as the way 02_Products_and_measurements has been used as the app name in this notebook.

The resulting dc object is what we use to access all the data contained within the Digital Earth Africa datacube.

[18]:
dc = datacube.Datacube(app="02_Products_and_measurements")

List products

Once a datacube instance has been created, users can explore the products and measurements stored within.

The following cell lists all product attributes currently available in the Digital Earth Africa datacube by using the dc.list_products().columns function.

[19]:
dc.list_products().columns
[19]:
Index(['name', 'description', 'instrument', 'platform', 'creation_time',
       'format', 'product_family', 'product_type', 'dataset_maturity', 'lon',
       'time', 'lat', 'region_code', 'label', 'crs', 'resolution', 'tile_size',
       'spatial_dimensions'],
      dtype='object')

Any of these can be used to customise the product information returned by the dc.list_products() function, as shown in the next cell.

Additionally, the next cell lists all products that are currently available in the Digital Earth Africa datacube by using the dc.list_products() function.

Products listed under name in the following table represent the product options available when querying the datacube. The table below provides some useful information about each product, including a brief product description, the instrument and platform the data originated from (e.g. Landsat 8 OLI), and the product’s default crs (coordinate reference system) and resolution if applicable.

[20]:
products = dc.list_products()

display_columns = ["name",
                   "description",
                   "platform",
                   "instrument",
                   "crs",
                   "resolution"]

products[display_columns].sort_index()
[20]:
name description platform instrument crs resolution
id
2 s2_l2a Sentinel-2a and Sentinel-2b imagery, processed to Level 2A (Surface Reflectance) and converted to Cloud Optimized GeoTIFFs None None NaN NaN
3 s1_rtc Sentinel 1 Gamma0 normalised radar backscatter None None NaN NaN
4 pc_s2_annual Surface Reflectance Annual Clear Pixel Count, Sentinel-2 None None EPSG:6933 (-20, 20)
5 gm_s2_annual Surface Reflectance Annual Geometric Median and Median Absolute Deviations, Sentinel-2 None None EPSG:6933 (-10, 10)
6 gm_s2_annual_lowres Annual Geometric Median, Sentinel-2 - Low Resolution mosaic None None NaN NaN
34 alos_palsar_mosaic ALOS/PALSAR and ALOS-2/PALSAR-2 annual mosaic tiles generated for use in the Data Cube - 25m pixel spacing, WGS84. These tiles are derived from the orignal JAXA mosaics with conversion to GeoTIFF. ALOS/ALOS-2 PALSAR/PALSAR-2 EPSG:4326 (-0.000222222222222, 0.000222222222222)
35 crop_mask_eastern Annual cropland extent map for Eastern Africa produced by Digital Earth Africa. None None epsg:6933 (-20, 20)
36 dem_srtm 1 second elevation model SRTM SIR NaN NaN
37 ga_ls8c_wofs_2 Historic Flood Mapping Water Observations from Space None None NaN NaN
38 ga_ls8c_wofs_2_annual_summary Water Observations from Space Annual Statistics None None EPSG:6933 (-30, 30)
39 ga_ls8c_wofs_2_summary Water Observations from Space Full History Statistics None None EPSG:6933 (-30, 30)
40 gm_s2_semiannual Surface Reflectance Semiannual Geometric Median and Median Absolute Deviations, Sentinel-2 None None EPSG:6933 (-10, 10)
41 jers_sar_mosaic JERS-1 SAR annual mosaic tiles generated for use in the Data Cube 25m pixel spacing, WGS84. These tiles are derived from the orignal JAXA mosaics with conversion to GeoTIFF. JERS-1 SAR EPSG:4326 (-0.000222222222222, 0.000222222222222)
42 ls5_sr USGS Landsat 5 Collection 2 Level-2 Surface Reflectance None None NaN NaN
43 ls5_st USGS Landsat 5 Collection 2 Level-2 Surface Temperature None None NaN NaN
44 ls7_sr USGS Landsat 7 Collection 2 Level-2 Surface Reflectance None None NaN NaN
45 ls7_st USGS Landsat 7 Collection 2 Level-2 Surface Temperature None None NaN NaN
46 ls8_sr USGS Landsat 8 Collection 2 Level-2 Surface Reflectance None None NaN NaN
47 ls8_st USGS Landsat 8 Collection 2 Level-2 Surface Temperature None None NaN NaN

List measurements

Most products are associated with a range of available measurements. These can be individual satellite bands (e.g. Landsat’s near-infrared band) or statistical product summaries.

Using the name column of products listed above, let’s interrogate the measurements associated with the ls8_usgs_sr_scene product using the dc.list_measurements() function. This product name refers to the US Geological Survey’s Landsat 8 Analysis-ready data product.

The table below includes a range of technical information about each band in the dataset, including any aliases which can be used to load the data, the data type or dtype, any flags_definition that are associated with the measurement (this information is used for tasks like cloud masking), and the measurement’s nodata value.

Change the product name below and re-run the following cell to explore available measurements associated with other products.

[21]:
product = "ls8_sr"

measurements = dc.list_measurements()
measurements.loc[product]
[21]:
name dtype units nodata aliases flags_definition
measurement
SR_B1 SR_B1 uint16 1 0 [band_1, coastal_aerosol] NaN
SR_B2 SR_B2 uint16 1 0 [band_2, blue] NaN
SR_B3 SR_B3 uint16 1 0 [band_3, green] NaN
SR_B4 SR_B4 uint16 1 0 [band_4, red] NaN
SR_B5 SR_B5 uint16 1 0 [band_5, nir] NaN
SR_B6 SR_B6 uint16 1 0 [band_6, swir_1] NaN
SR_B7 SR_B7 uint16 1 0 [band_7, swir_2] NaN
QA_PIXEL QA_PIXEL uint16 bit_index 1 [pq, pixel_quality] {'snow': {'bits': 5, 'values': {'0': 'not_high_confidence', '1': 'high_confidence'}}, 'clear': {'bits': 6, 'values': {'0': False, '1': True}}, 'cloud': {'bits': 3, 'values': {'0': 'not_high_confid...
QA_RADSAT QA_RADSAT uint16 bit_index 0 [radsat, radiometric_saturation] {'nir_saturation': {'bits': 4, 'values': {'0': False, '1': True}}, 'red_saturation': {'bits': 3, 'values': {'0': False, '1': True}}, 'blue_saturation': {'bits': 1, 'values': {'0': False, '1': True...
SR_QA_AEROSOL SR_QA_AEROSOL uint8 bit_index 1 [qa_aerosol, aerosol_qa] {'water': {'bits': 2, 'values': {'0': False, '1': True}}, 'nodata': {'bits': 0, 'values': {'0': False, '1': True}}, 'aerosol_level': {'bits': [6, 7], 'values': {'0': 'climatology', '1': 'low', '2'...

Visualising available data

For a more visual way of exploring the data that is available within the Digital Earth Africa datacube, we can use the interactive DcViewer utility or the online DE Africa Explorer website. We will use the DcViewer utility in this exiercise. Select a product from the drop-down menu on the top-left of the map to show the areas data is available for in blue. You can also use the back and forward buttons above the map to toggle through time.

The utility is only able to visualise a limited number of datasets at one time. If the available data footprints do not appear, either press the “show” button on the top right, or zoom further in on the map.

[22]:
DcViewer(dc=dc,
         time='2017',
         center=(0.565, 38.007),
         zoom=6)

Additional information

License: The code in this notebook is licensed under the Apache License, Version 2.0. Digital Earth Africa data is licensed under the Creative Commons by Attribution 4.0 license.

Contact: If you need assistance, please post a question on the Open Data Cube Slack channel or on the GIS Stack Exchange using the open-data-cube tag (you can view previously asked questions here). If you would like to report an issue with this notebook, you can file one on Github.

Compatible datacube version:

[23]:
print(datacube.__version__)
1.8.4.dev52+g07bc51a5

Last Tested:

[24]:
from datetime import datetime
datetime.today().strftime('%Y-%m-%d')
[24]:
'2021-05-19'