Skip to content

Download the Jupyter notebook

elapid's Geospatial Features

This notebook reviews common patterns for working with vector and raster data using elapid.

We'll review the core tools used for extracting raster data from points and polygons, assigning sample weights based on sample density, and creating spatially-explicit train/test data, while also showing general patterns for working with the GeoDataFrame and GeoSeries classes from geopandas, which are frequently used.

The sample data we'll use includes species occurrence records from the genus Ariolimax, a group of North American terrestrial slugs, as well as some climate and vegetation data.

Several datasets used here are hosted on a web server. These files can be accessed locally with ela.download_sample_data(), but we'll work with them directly on the web to demonstrate that reading remote data is nearly identical to reading local data thanks to the amazing libraries elapid is built on top of.

pip install elapid rasterio scikit-learn

Packages, paths & preferences

# packages
import elapid as ela

import os
from pprint import pprint
import numpy as np
import pandas as pd
import geopandas as gpd
import rasterio as rio
from rasterio import plot as rioplot
from sklearn import metrics
import matplotlib as mpl
import matplotlib.pyplot as plt

from sklearn.metrics import RocCurveDisplay
## Load the prepared data
amphibians = gpd.read_file("../data/amphibians.geojson")

amphibians["species"].value_counts().head(12)
species
Pseudacris crucifer          1839
Aquarana clamitans           1779
Anaxyrus americanus          1689
Boreorana sylvatica          1617
Lithobates catesbeianus      1487
Ambystoma maculatum          1317
Plethodon cinereus           1283
Lithobates palustris         1049
Dryophytes versicolor         790
Notophthalmus viridescens     611
Lithobates pipiens            326
Hemidactylium scutatum        233
Name: count, dtype: int64

Pickerel frog (Lithobates palustris)

image.png

lithobates = amphibians[amphibians["species"] == "Lithobates palustris"]
lithobates.head()
gbifID species decimalLongitude decimalLatitude coordinateUncertaintyInMeters year geometry
2 6520721493 Lithobates palustris -68.636788 44.860466 112.0 2022.0 POINT (-68.63679 44.86047)
3 6520700408 Lithobates palustris -69.865662 44.361917 4.0 2026.0 POINT (-69.86566 44.36192)
5 6520663387 Lithobates palustris -69.811887 43.815353 13.0 2026.0 POINT (-69.81189 43.81535)
21 6520437434 Lithobates palustris -70.447664 43.766105 2.0 2026.0 POINT (-70.44766 43.7661)
26 6520345349 Lithobates palustris -70.886383 43.230412 3.0 2026.0 POINT (-70.88638 43.23041)
bioclim_names = {
    'wc2.1_2.5m_bio_1.tif':  'Annual mean temperature',
    'wc2.1_2.5m_bio_2.tif':  'Mean diurnal temperature range',
    'wc2.1_2.5m_bio_3.tif':  'Isothermality',
    'wc2.1_2.5m_bio_4.tif':  'Temperature seasonality',
    'wc2.1_2.5m_bio_5.tif':  'Max temperature, warmest month',
    'wc2.1_2.5m_bio_6.tif':  'Min temperature, coldest month',
    'wc2.1_2.5m_bio_7.tif':  'Annual temperature range',
    'wc2.1_2.5m_bio_8.tif':  'Mean temperature, wettest quarter',
    'wc2.1_2.5m_bio_9.tif':  'Mean temperature, driest quarter',
    'wc2.1_2.5m_bio_10.tif': 'Mean temperature, warmest quarter',
    'wc2.1_2.5m_bio_11.tif': 'Mean temperature, coldest quarter',
    'wc2.1_2.5m_bio_12.tif': 'Annual precipitation',
    'wc2.1_2.5m_bio_13.tif': 'Precipitation, wettest month',
    'wc2.1_2.5m_bio_14.tif': 'Precipitation, driest month',
    'wc2.1_2.5m_bio_15.tif': 'Precipitation seasonality',
    'wc2.1_2.5m_bio_16.tif': 'Precipitation, wettest quarter',
    'wc2.1_2.5m_bio_17.tif': 'Precipitation, driest quarter',
    'wc2.1_2.5m_bio_18.tif': 'Precipitation, warmest quarter',
    'wc2.1_2.5m_bio_19.tif': 'Precipitation, coldest quarter',
}

rasters = [f"../data/bioclim_ME/{var}" for var in bioclim_names]
labels = list(bioclim_names.values())

Reading vector files

Point-format data are most commonly provided in formats like GeoPackage, GeoJSON, or Shapefiles. You can read these into memory easily with geopandas.

# plot the points geographically by species
fig, ax = plt.subplots(figsize=(6,6))
sp = lithobates.plot(
    column='species',
    ax=ax,
    legend=True,
    legend_kwds={"bbox_to_anchor": (1, 1)}
)

png

Working with raster data

This demo includes 6 raster files defined at the start of the notebook, corresponding to the average and standard deviation in annual cloud cover, temperature, and vegetation growth across California.

Following a brief overview of the rasters, the sections below will review how to extract data from them using point and polygon vector data.

# first we'll just read and display the surface temperature data
fig, ax = plt.subplots(figsize=(6,6))
surfacetemp_raster = rasters[0]
with rio.open(surfacetemp_raster, 'r') as src:

    # this plots the mean annual temperature data
    # low temps are dark grey, high temps are light grey
    rioplot.show(src, ax=ax, cmap="gray")

    # overlay the ariolimax data
    lithobates.to_crs(src.crs).plot(column='species', ax=ax, legend=True)
/Users/isaac/miniconda3/envs/meadow/lib/python3.11/site-packages/matplotlib/colors.py:778: RuntimeWarning: overflow encountered in multiply
  xa *= self.N

png

Drawing random pseudoabsence locations

In addition to species occurrence records (where y = 1), species distributions models often require a set of random pseudo-absence/background points (y = 0). These are a random geographic sampling of where you might expect to find a species across the target landscape.

The background point distribution is extremely important for presence-only models, and should be selected with care. elapid provides a range of methods for sample selection.

For the unintiated, these papers by Morgane Barbet-Massin et al. and Phillips et al. are excellent resources to consult for considering how background point samples should be selected.

From a raster's extent

You can use elapid to create a uniform random geographic sample from unmasked locations within a raster's extent:

# sampling
count = 1000
pseudoabsence = ela.sample_raster(surfacetemp_raster, count)

# plotting
pseudoabsence.plot(markersize=0.5)
pseudoabsence.describe()
count                                              1000
unique                                              924
top       POINT (-70.72916666666667 44.020833333333336)
freq                                                  3
dtype: object

png

# some pixel centroids were sampled multiple times
# you can drop them if you'd like.
pseudoabsence.drop_duplicates(inplace=True)
pseudoabsence.describe()
count                          920
unique                         920
top       POINT (-70.4375 44.9375)
freq                             1
dtype: object

From a bias raster

You could also provide a "bias" file to the ela.sample_bias_file() function, where the raster grid cells contain information on the probability of sampling an area. Bias adjustments are recommended because species occurrence records are often biased towards certain locations (near roads, parks, or trails).

The grid cells can be an arbitrary range of values. What's important is that the values encode a linear range of numbers that are higher where you're more likely to draw a sample. The probability of drawing a sample is dependent on two factors: the range of values provided and the frequency of values across the dataset.

Because amphibians are ectotherms, we'll guess that people are more likely to look for them in warmer areas (in Maine). We'll use the mean annual temperature covariate to select more background samples from warmer areas.

mean_temp_raster = rasters[0]
pseudoabsence_bias = ela.sample_bias_file(mean_temp_raster, count)
pseudoabsence_bias.plot(markersize=0.5)
pseudoabsence_bias.describe()
<Geographic 2D CRS: EPSG:4326>
Name: WGS 84
Axis Info [ellipsoidal]:
- Lat[north]: Geodetic latitude (degree)
- Lon[east]: Geodetic longitude (degree)
Area of Use:
- undefined
Datum: World Geodetic System 1984
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich

png

# again, duplicates can be dropped
# (this can take some time to run)
pseudoabsence_bias.drop_duplicates(inplace=True)
pseudoabsence_bias.describe()
count                                               929
unique                                              929
top       POINT (-70.14583333333333 44.979166666666664)
freq                                                  1
dtype: object

We can see that we are selecting a disproportionately high number of points in the south and disproportionately few samples in the colder north.

Point annotation

Annotation refers to extracting raster values at the each point occurrence record and storing those records together in a GeoDataFrame. Once you have your presence and pseudo-absence records, you can annotate these records with the corresponding covariate data.

This function, since it's geographically indexed, doesn't require the point data and the raster data to be in the same projection. elapid handles reprojection and resampling on the fly.

It also allows you to pass multiple raster files, which can be in different projections, extents, or grid sizes. This means you don't have to explicitly re-sample your raster data prior to analysis, which is always a chore.

presence = ela.annotate(
    lithobates.geometry,
    rasters,
    labels=labels,
    drop_na=True,
    quiet=True,
)

# we'll use the biased sample locations
background = ela.annotate(
    pseudoabsence_bias,
    rasters,
    labels=labels,
    drop_na=True,
    quiet=True,
)

Setting drop_na=True works with raster datasets that have nodata values assigned in their metadata. These point locations will be dropped from the output dataframe.

Next let's plot some histograms to understand the similarities and differences between the conditions at presence locations and at background locations.

fig, axs = plt.subplots(nrows=3, ncols=2, figsize=(6,9))

for label, ax in zip(labels, axs.ravel()):
    pvar = presence[label]
    bvar = background[label]
    ax.hist(
        [pvar, bvar],
        density=True,
        alpha=0.7,
        label=['presence', 'background'],
    )
    ax.set_title(label)

handles, lbls = ax.get_legend_handles_labels()
fig.legend(handles, lbls, loc=(0.6, 0.9))
plt.tight_layout()

png

This checks out. Lithobes are commonly found in areas with higher mean annual temperature, and lower temperature seasonality (even after adjusting for bias and oversampling areas with high mean annual temperature).

Geographic sample weights

Despite ingesting spatial data, many statistically-driven SDMs are not spatial models in a traditional sense: they don't handle geographic information during model selection or in model scoring.

One way to add spatial information to a model is to compute geographically-explicit sample weights. elapid does this by calculating weights based on the distance to the nearest neighbor. Points nearby other points receive lower weight scores; far-away points receive higher weight scores.

You can compute weights based on the distance to the nearest point by setting n_neighbors=1.

presence['SampleWeight'] = ela.distance_weights(presence, n_neighbors=1)
presence.plot(column='SampleWeight', legend=True, cmap='YlOrBr')
/Users/isaac/miniconda3/envs/meadow/lib/python3.11/site-packages/elapid/geo.py:921: UserWarning: Computing distances using geographic coordinates is bad
  distances = nearest_point_distance(points, n_neighbors=n_neighbors, cpu_count=cpu_count)





<Axes: >

png

The warning here is telling us something important, namely that it is not good to calculate distance using decimal Lat/Long values (i.e. EPSG:4326) We will use EPSG:26919, UTM Zone 19N, for Maine, which is a projected coordinate system with units in meters.

presence['SampleWeight'] = ela.distance_weights(presence.to_crs('EPSG:26919'), n_neighbors=1)
presence.plot(column='SampleWeight', legend=True, cmap='YlOrBr')
<Axes: >

png

Alternatively, you could compute the average distance from each point to all other points, computing sample weights from point density instead of the distance to the single nearest point. This may be useful if you have small clusters of a few points far away from large, densely populated regions, like the above.

This is the default parameter setting (and specified by setting n_neighbors=-1).

background['SampleWeight'] = ela.distance_weights(background.to_crs('EPSG:26919'), n_neighbors=-1)
presence['SampleWeight'] = ela.distance_weights(presence.to_crs('EPSG:26919'), n_neighbors=-1)
presence.plot(column='SampleWeight', legend=True, cmap='YlOrBr')
<Axes: >

png

As you can see, there are tradeoffs for finding the best sample weights depending on the spatial distributions of each occurrence dataset. We'll use the sample density weights here to upweight the samples in the northeastern and southwestern regions.

Stacking dataframes

Now that we have our presence points and our pseudoabsence points, we can merge them together into a single dataset for analysis.

elapid provides a convenience function for merging GeoDataFrames, designed for stacking presence/background data. It will reproject geometries on the fly if needed, and you can optionally add a 'class' column to the output GeoDataFrame.

merged = ela.stack_geodataframes(
    presence,
    background,
    add_class_label=True,
).to_crs('EPSG:26919')
merged.head()
Annual mean temperature Mean diurnal temperature range Isothermality Temperature seasonality Max temperature, warmest month Min temperature, coldest month Annual temperature range Mean temperature, wettest quarter Mean temperature, driest quarter Mean temperature, warmest quarter ... Precipitation, wettest month Precipitation, driest month Precipitation seasonality Precipitation, wettest quarter Precipitation, driest quarter Precipitation, warmest quarter Precipitation, coldest quarter geometry SampleWeight class
0 6.544000 11.580667 28.693426 1010.674011 26.608000 -13.752000 40.360001 1.951333 -5.095333 18.765999 ... 100.0 70.0 8.947875 288.0 237.0 263.0 255.0 POINT (528696.105 4967514.248) 0.962385 1
1 6.777333 11.249333 28.270338 1008.412781 26.364000 -13.428000 39.792000 2.227333 -4.852000 18.994667 ... 106.0 71.0 9.855001 303.0 251.0 270.0 252.0 POINT (431019.46 4912435.936) 0.858192 1
2 7.709868 9.223246 26.563841 900.671448 24.947369 -9.773685 34.721054 3.848246 18.163158 18.678070 ... 123.0 77.0 14.432403 340.0 245.0 249.0 297.0 POINT (434705.98 4851685.25) 0.996496 1
3 7.198833 11.659000 29.897938 970.147827 26.420000 -12.576000 38.996002 2.836667 18.161333 18.936666 ... 121.0 82.0 12.655907 334.0 251.0 255.0 286.0 POINT (383479.146 4846913.693) 1.184145 1
4 6.873090 9.896180 27.359558 919.006592 24.825001 -11.345833 36.170834 -1.854861 17.963888 17.963888 ... 155.0 75.0 19.003300 424.0 263.0 263.0 375.0 POINT (556646.745 4908463.83) 0.992957 1

5 rows × 22 columns

Model setup

We won't get into much detail on model parameters, leaving that for another in-depth notebook. This section will focus on fitting models using default settings, then evaluate the effects of different forms of train/test splitting strategies.

# initialize and print the model defaults
maxent = ela.MaxentModel()
maxent.set_params(n_cpus=4)
pprint(maxent.get_params())
{'beta_categorical': 1.0,
 'beta_hinge': 1.0,
 'beta_lqp': 1.0,
 'beta_multiplier': 1.5,
 'beta_threshold': 1.0,
 'clamp': True,
 'class_weights': 100,
 'convergence_tolerance': 2e-06,
 'feature_types': ['linear', 'hinge', 'product'],
 'max_iter': 1000,
 'n_cpus': 10,
 'n_hinge_features': 10,
 'n_lambdas': 100,
 'n_threshold_features': 10,
 'random_state': None,
 'scorer': 'roc_auc',
 'tau': 0.5,
 'transform': 'cloglog',
 'use_lambdas': 'best',
 'use_sklearn': True}
# set up the x/y data for model fitting
X = merged.drop(columns=['class', 'SampleWeight'])
y = merged['class']
sample_weight = merged['SampleWeight']

# fit and evaluate the model under naive conditions
maxent.fit(X, y, sample_weight=sample_weight)
ypred = maxent.predict(X)

print(f"Unweighted naive AUC score: {metrics.roc_auc_score(y, ypred):0.3f}")
RocCurveDisplay.from_predictions(y, ypred)
print(f"Weighted naive AUC score  : {metrics.roc_auc_score(y, ypred, sample_weight=sample_weight):0.3f}")
Unweighted naive AUC score: 0.881
Weighted naive AUC score  : 0.870

png

Train / test splits

Uniformly random train/test splits are generally discouraged in spatial modeling because of the strong spatial structure inherent in many datasets. The non-independence of these data is referred to as spatial autocorrelation. Using distance- or density-based sample weights is one way to mitigate these effects. Another is to split the data into geographically distinct train/test regions to try and prioritize model generalization.

Checkerboard splits

With a "checkerbox" train/test split, points are intersected along a regular grid, and every other grid is used to split the data into train/test sets.

The height and width of the grid used to split the data is controlled by the grid_size parameter. This should specify distance in the units of the point data's CRS. In our example, the units are in meters.

The black and white structure of the checkerboard means this method can only generate one train/test split.

# let's use a 50km grid
grid_size = 50_000
train, test = ela.checkerboard_split(merged, grid_size=grid_size)

# we'll re-merge them for plotting purposes
train['split'] = 'train'
test['split'] = 'test'
checker = ela.stack_geodataframes(train, test)
checker.plot(column='split', markersize=0.75, legend=True)
<Axes: >

png

# set up model fitting
Xtrain = train.drop(columns=['class', 'SampleWeight', 'split'])
ytrain = train['class']
sample_weight_train = train['SampleWeight']

Xtest = test.drop(columns=['class', 'SampleWeight', 'split'])
ytest = test['class']
sample_weight_test = test['SampleWeight']

maxent.fit(Xtrain, ytrain, sample_weight=sample_weight_train)
ypred = maxent.predict(Xtest)

print(f"Unweighted checkerboard AUC score: {metrics.roc_auc_score(ytest, ypred):0.3f}")
print(f"Weighted checkerboard AUC score: {metrics.roc_auc_score(ytest, ypred, sample_weight=sample_weight_test):0.3f}")
Unweighted checkerboard AUC score: 0.707
Weighted checkerboard AUC score: 0.702

This lower AUC score is to be expected and should be interpreted as a good thing. It indicates that the naive model was overfitting to nearby points, and the inflated scores were likely the result of spatial autocorrelation.

Apply the trained model to the ME clipped rasters

output_raster = "lithobates-maxent-predictions.tif"
ela.apply_model_to_rasters(maxent, rasters, output_raster, quiet=True)

# and read into memory
with rio.open(output_raster, 'r') as src:
    pred = src.read(1, masked=True)
fig, ax = plt.subplots(1, 1, figsize=(7, 6), dpi=100)
plot = ax.imshow(pred, vmin=0, vmax=1, cmap='GnBu')
ax.set_title('$Lithobates\ palustris$ suitability')
ax.set_xticks([])
ax.set_yticks([])
cbar = plt.colorbar(plot, ax=ax, label="relative occurrence probability", pad=0.04)
plt.tight_layout()

png

Buffered leave-one-out

Leave-one-out cross-validation refers to training on n-1 points and testing on a single test point, looping over the full dataset to test on each point separately.

The buffered leave-one-out strategy, as described by Ploton et al., modifies this approach. While each point is iterated over for testing, the pool of available training points is reduced at each iteration by dropping training points located within a certain distance of the test data. The purpose of this method is to evaluate how a model performs far away from where it was trained.

In SDM contexts, however, standard leave-one-out strategies may not be appropriate. Model performance is best evaluated on presence-only data; model performance on background points may not be meaningful.

To only run the leave-one-out analysis on presence-only points, specify the column with the 0/1 class labels during train/test splitting:

# store the individual sample predictions and evaluate at the end
yobs_scores = []
ypred_scores = []

# buffered leave-one-out, 5km buffer radius
distance = 50_000

# Downsample the merged presence/absence data to 500 random rows because LOO is slow
merged_subset = merged.sample(500)

bloo = ela.BufferedLeaveOneOut(distance=distance)
idx = 0
for train_idx, test_idx in bloo.split(merged_subset, class_label="class"):
    idx += 1
    if not idx % 10:
        print(idx, end=" ")
    train = merged_subset.iloc[train_idx]
    test = merged_subset.iloc[test_idx]

    # set up model fitting
    xtrain = train.drop(columns=['class'])
    ytrain = train['class']
    xtest = test.drop(columns=['class'])
    ytest = test['class']

    # evaluation
    maxent.fit(xtrain, ytrain)
    ypred = maxent.predict(xtest)
    ypred_scores.append(ypred)
    yobs_scores.append(ytest)
10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 210 220 230 240 250 260
# get model predictions for just the background
bg = merged_subset[merged_subset['class'] == 0]
xbg = bg.drop(columns='class')
ybg = bg['class'].to_numpy()
bg_pred = maxent.predict(xbg)

# concatenate the background and leave-one-out points
bl_pred = np.array(ypred_scores)
bl_obs = np.array(yobs_scores)
ypred = np.vstack((bg_pred.reshape(-1,1), bl_pred))
yobs = np.vstack((ybg.reshape(-1,1), bl_obs))

# evaluate
auc = metrics.roc_auc_score(yobs, ypred)
print(f"Buffered leave-one-out AUC: {auc:0.3f}")
RocCurveDisplay.from_predictions(yobs, ypred)

# no buffer
x = merged_subset.drop(columns=['class'])
y = merged_subset['class']
maxent.fit(x, y)
ypred = maxent.predict(x)
auc = metrics.roc_auc_score(y, ypred)
print(f"Naive model AUC: {auc:0.3f}")
RocCurveDisplay.from_predictions(y, ypred)
Buffered leave-one-out AUC: 0.869
Naive model AUC: 0.975





<sklearn.metrics._plot.roc_curve.RocCurveDisplay at 0x18d586d90>

png

png

#RocCurveDisplay.from_predictions(yobs, ypred)
#RocCurveDisplay.from_estimator(maxent, x, y)
ypred
array([0.12757082])