Load in Precursor Coincident Dataset sites On-Demand#

coincident supports both reading into memory and downloading the Precursor Coincident Dataset (PCD) sites explained in https://coincident.readthedocs.io/en/latest/datasets

You can read in individual sites by memory given their PCD ID, or you can download the respective spatiotemporal metadata for ALL sites

from coincident import pcd_fixtures
import matplotlib.pyplot as plt

/home/docs/checkouts/readthedocs.org/user_builds/coincident/checkouts/v0.5/src/coincident/io/download.py:27: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  from tqdm.autonotebook import tqdm

pcd_fixtures.read_pcd_site?

Remember, our sites are below:

Provider	PCD Site Identifier	Description	Fourway Overlap Area (km²)	Aerial Lidar Start Date	Aerial Lidar End Date
USGS	`CA_SanFrancisco_1_B23`	Urban over San Francisco	55	2023-04-20	2023-04-20
USGS	`AZ_PimaCo_2_2021`	Desert / Mine in southern Arizona	53	2021-09-27	2021-11-17
NEON	`REDB`	Deciduous / Conifer in northern Utah	25	2021-05-20	2021-05-21
USGS	`NE_Northeast_Phase2_2_2020`	Cropland in eastern Nebraska	540	2020-11-16	2020-12-09
USGS	`WI_Brown_2_2020`	Urban / Wetlands in Green Bay, Wisconsin	89	2020-05-07	2020-05-07
USGS	`GA_Central_3_2019`	Mixed LULC in southern Georgia	745	2020-02-02	2020-03-28
NCALM	`OTLAS.092021.32611.1`	Southern San Andreas fault line	12	2020-02-15	2020-02-18
USGS	`CA_YosemiteNP_2019`	Coniferous / Mountainous in northern Yosemite National Park	84	2019-10-07	2019-10-23
USGS	`TX_DesertMountains_B1_2018`	Shrubland / Grassland in western Texas	165	2019-09-11	2019-10-20
NEON	`BART`	Mixed hardwood forest in eastern New Hampshire	32	2019-08-25	2019-08-25
USGS	`CO_WestCentral_2019`	Coniferous / Mountainous in the Colorado Rockies	184	2019-08-21	2019-09-19
USGS	`WY_FEMA_East_B9_2019`	Glaciers / Mountainous in western Wyoming	681	2019-07-26	2019-09-22
NEON	`WREF`	Conifer forest in southern Washington state	18	2019-07-12	2019-07-15

Note

The NCALM site’s “ground truth” is not aerial lidar, but rather dense aerial SfM from a drone

%%time

site = "GA_Central_3_2019"

dict_GA_Central_3_2019 = pcd_fixtures.read_pcd_site(site)

CPU times: user 299 ms, sys: 82.7 ms, total: 382 ms
Wall time: 9.8 s

dict_GA_Central_3_2019.keys()

dict_keys(['als', 'maxar', 'is2', 'gedi', 'overlap'])

Note

Some sites take a longer time to read in than others (ranging from a few seconds to ~1 minute). This is mainly dependent on the overlap area and length of date range.

gf_als = dict_GA_Central_3_2019["als"]
gf_maxar = dict_GA_Central_3_2019["maxar"]
gf_is2 = dict_GA_Central_3_2019["is2"]
gf_gedi = dict_GA_Central_3_2019["gedi"]
gf_overlap = dict_GA_Central_3_2019["overlap"]

gf_als.head()

	workunit	workunit_id	project	project_id	start_datetime	end_datetime	ql	spec	p_method	dem_gsd_meters	...	seamless_category	seamless_reason	lpc_link	sourcedem_link	metadata_link	geometry	collection	datetime	dayofyear	duration
0	GA_Central_3_2019	217457	GA_Central_2019_B19	183179	2020-02-02	2020-03-28	QL 2	USGS Lidar Base Specification 1.3	linear-mode lidar	1.0	...	Meets	Meets 3DEP seamless DEM requirements	https://rockyweb.usgs.gov/vdelivery/Datasets/S...	https://prd-tnm.s3.amazonaws.com/index.html?pr...	https://prd-tnm.s3.amazonaws.com/index.html?pr...	MULTIPOLYGON (((-84.21686 31.22133, -84.21603 ...	3DEP	2020-02-29 12:00:00	60	55

1 rows × 33 columns

gf_maxar.head()

	type	stac_version	stac_extensions	id	target_to_spacecraft_elevation_start	target_to_spacecraft_elevation_min	links	assets	collection	datetime	...	target_to_spacecraft_azimuth_max	target_to_spacecraft_azimuth_min	spacecraft_to_target_azimuth_start	target_to_spacecraft_azimuth_start	target_to_spacecraft_elevation_avg	target_to_spacecraft_elevation_end	target_to_spacecraft_elevation_max	bbox	geometry	dayofyear
0	Feature	1.1.0	[https://stac-extensions.github.io/eo/v1.1.0/s...	10300100A26FD700	59.181292	59.181292	[{'href': 'https://api.maxar.com/discovery/v1/...	{'browse': {'href': 'https://api.maxar.com/dis...	wv02	2020-02-09 16:26:28.697000+00:00	...	191.169335	187.57504	7.57504	187.575040	60.862231	62.496991	62.496991	{'xmin': -84.135038, 'ymin': 30.978549, 'xmax'...	POLYGON ((-84.13481 31.95756, -84.13457 31.902...	40
1	Feature	1.1.0	[https://stac-extensions.github.io/eo/v1.1.0/s...	10300100A132F900	79.741386	77.626514	[{'href': 'https://api.maxar.com/discovery/v1/...	{'browse': {'href': 'https://api.maxar.com/dis...	wv02	2020-02-09 16:25:11.447000+00:00	...	30.488295	17.26067	210.36903	30.488295	78.655265	77.626514	79.741386	{'xmin': -84.124106, 'ymin': 30.989507, 'xmax'...	POLYGON ((-84.12347 31.03196, -84.12349 30.989...	40

2 rows × 69 columns

gf_is2.head()

	type	stac_version	stac_extensions	id	storage:schemes	end_datetime	links	assets	collection	datetime	start_datetime	bbox	geometry	dayofyear
0	Feature	1.1.0	[https://stac-extensions.github.io/storage/v2....	ATL03_20200203102127_05910606_007_01.h5	{'aws': {'type': 'aws-s3', 'platform': 'https:...	2020-02-03 10:29:56.914000+00:00	[{'href': 'https://cmr.earthdata.nasa.gov/clou...	{'browse': {'href': 'https://data.nsidc.earthd...	ATL03_007	2020-02-03 10:21:26.442000+00:00	2020-02-03 10:21:26.442000+00:00	{'xmin': -84.67408, 'ymin': 26.95991, 'xmax': ...	POLYGON ((-79.92492 59.53417, -80.14715 59.545...	34
1	Feature	1.1.0	[https://stac-extensions.github.io/storage/v2....	ATL03_20200316201642_12390602_007_01.h5	{'aws': {'type': 'aws-s3', 'platform': 'https:...	2020-03-16 20:25:13.144000+00:00	[{'href': 'https://cmr.earthdata.nasa.gov/clou...	{'browse': {'href': 'https://data.nsidc.earthd...	ATL03_007	2020-03-16 20:16:42.935000+00:00	2020-03-16 20:16:42.935000+00:00	{'xmin': -88.23844, 'ymin': 26.95964, 'xmax': ...	POLYGON ((-88.01777 59.54564, -88.23844 59.534...	76

gf_gedi.head()

	type	stac_version	stac_extensions	id	storage:schemes	end_datetime	links	assets	collection	datetime	start_datetime	bbox	geometry	dayofyear
0	Feature	1.1.0	[https://stac-extensions.github.io/storage/v2....	GEDI02_A_2020040225749_O06582_03_T02744_02_003...	{'aws': {'type': 'aws-s3', 'platform': 'https:...	2020-02-10 00:30:43+00:00	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	{'browse': {'href': 'https://data.lpdaac.earth...	GEDI02_A_002	2020-02-09 22:57:49+00:00	2020-02-09 22:57:49+00:00	{'xmin': -143.2615406, 'ymin': 0.4409358, 'xma...	POLYGON ((-143.26111 51.77555, -137.28425 51.6...	40
1	Feature	1.1.0	[https://stac-extensions.github.io/storage/v2....	GEDI02_A_2020044212416_O06643_03_T05590_02_003...	{'aws': {'type': 'aws-s3', 'platform': 'https:...	2020-02-13 22:57:10+00:00	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	{'browse': {'href': 'https://data.lpdaac.earth...	GEDI02_A_002	2020-02-13 21:24:16+00:00	2020-02-13 21:24:16+00:00	{'xmin': -143.1507574, 'ymin': 0.4364369, 'xma...	POLYGON ((-143.1504 51.77648, -137.17463 51.64...	44
2	Feature	1.1.0	[https://stac-extensions.github.io/storage/v2....	GEDI02_A_2020048195041_O06704_03_T01321_02_003...	{'aws': {'type': 'aws-s3', 'platform': 'https:...	2020-02-17 21:23:34+00:00	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	{'browse': {'href': 'https://data.lpdaac.earth...	GEDI02_A_002	2020-02-17 19:50:41+00:00	2020-02-17 19:50:41+00:00	{'xmin': -143.0124222, 'ymin': 0.406638, 'xmax...	POLYGON ((-143.01216 51.777, -137.03573 51.636...	48
3	Feature	1.1.0	[https://stac-extensions.github.io/storage/v2....	GEDI02_A_2020052181703_O06765_03_T04320_02_003...	{'aws': {'type': 'aws-s3', 'platform': 'https:...	2020-02-21 19:49:56+00:00	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	{'browse': {'href': 'https://data.lpdaac.earth...	GEDI02_A_002	2020-02-21 18:17:03+00:00	2020-02-21 18:17:03+00:00	{'xmin': -142.9432462, 'ymin': 0.4701104, 'xma...	POLYGON ((-142.94325 52.0317, -136.9351 51.725...	52
4	Feature	1.1.0	[https://stac-extensions.github.io/storage/v2....	GEDI02_A_2020056164321_O06826_03_T04167_02_003...	{'aws': {'type': 'aws-s3', 'platform': 'https:...	2020-02-25 18:16:15+00:00	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	{'browse': {'href': 'https://data.lpdaac.earth...	GEDI02_A_002	2020-02-25 16:43:21+00:00	2020-02-25 16:43:21+00:00	{'xmin': -142.921822, 'ymin': 0.3679382, 'xmax...	POLYGON ((-142.92161 51.77505, -136.94421 51.6...	56

gf_overlap

	pcd_id	geometry
0	GA_Central_3_2019	POLYGON ((-84.12672 31.01941, -84.1333 31.5686...

Visualize the acquisition times of the PCD site

fig, ax = plt.subplots(figsize=(12, 5))

als_start = gf_als["start_datetime"].iloc[0]
als_end = gf_als["end_datetime"].iloc[0]
ax.axvspan(als_start, als_end, color="gray", alpha=0.3, label="ALS Window")

ax.scatter(
    gf_maxar["datetime"],
    ["Maxar"] * len(gf_maxar),
    marker="D",
    s=80,
    label="Maxar Stereo",
)
ax.scatter(
    gf_is2["datetime"], ["ICESat-2"] * len(gf_is2), marker="D", s=80, label="ICESat-2"
)
ax.scatter(gf_gedi["datetime"], ["GEDI"] * len(gf_gedi), marker="D", s=80, label="GEDI")

ax.set_title("Data Availability for Site: CA_SanFrancisco_1_B23")
ax.set_xlabel("Date")
ax.set_ylabel("Data Collection")
ax.legend(loc="best")
ax.grid(axis="x", linestyle=":", alpha=0.4)
fig.autofmt_xdate()
plt.tight_layout();

../_images/c6926bbbf72b453edcc849f096cfefcccfda394d94a9b0ddf89be1fb3f09b414.png

See the spatial extent

style_args = {"fillOpacity": 0.15, "weight": 2.5}

m = gf_als.explore(
    name="ALS", color="gray", style_kwds=style_args, tiles="Esri.WorldImagery"
)
m = gf_maxar.explore(m=m, name="Maxar", color="blue", style_kwds=style_args)
m = gf_is2.explore(m=m, name="ICESat-2", color="orange", style_kwds=style_args)
m = gf_gedi.explore(m=m, name="GEDI", color="green", style_kwds=style_args)
m = gf_overlap.explore(m=m, name="Overlap Area", color="black", style_kwds=style_args)
m

Make this Notebook Trusted to load map: File -> Trust Notebook

Downloading#

You can also download the PCD files for all sites using the coincident.pcd_fixtures module. coincident.pcd_fixtures.download_pcd_files() supports this by streaming calls to the respective STAC catalogs and ALS endpoints.

Source code in coincident/scripts as seen below supports the downloading of the latest released GitHub assets of these PCD dataset files.

pixi run python src/coincident/scripts/generate_pcd.py

What’s the difference between the two?

coincident.pcd_fixtures.download_pcd_files() grabs the metadata for all PCD sites as provided by the respective STAC catalogs and API endpoints. src/coincident/scripts/generate_pcd.py pulls from the latest GitHub assets, which includes more complex overlap area geometries, LULC and elevation statistics over these geometries, and extended ALS metadata. The difference exists at the PCD sites from the latest GitHub assets have this extra metadata manually determined uniquely for each site (via reading individual lidar metadata reports, manually defining overlap geometries based on filtered data, using code that exists outside of coincident, etc.).

Because of this, running coincident.pcd_fixtures.download_pcd_files() will take minutes and running src/coincident/scripts/generate_pcd.py will take seconds.

pcd_fixtures.download_pcd_files?

# pcd_fixtures.download_pcd_files("/tmp")

Note

This takes ~4 minutes to run and the total output size for all files is 31mb (parquet files sum to ~8.5mb and geojsons sum to ~22mb)