Gradient forest: P. guttatus
Fit SNP-specific forests using the example VCF, sample coordinates and 19 bioclim rasters, then map environmental turnover.
Install pyGF
The pyGF package is only available in my github repo, so the installation will need to be
'by hand'.
- Open a new terminal and
cd ~to go to your home directory git clone https://github.com/isaacovercast/pygf.gitto pull down this repositorypip install -e pygfto install this package and all requirements- Restart the kernel for this notebook w/ the 'circle arrow' button in the notebook toolbar
- Now you should be able to do the imports in the next cell
from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pygf import GradientForest, read_coordinates, read_vcf, sample_environment
Read samples and genotypes
read_coordinates returns a GeoDataFrame from the headerless sample, longitude, latitude file. read_vcf returns an xarray Dataset of alternate-allele dosages (0, 1, 2), with missing calls as NaN.
Keep SNPs with call rate ≥ 80% and minor allele frequency ≥ 5%, then sample 256 across the file. one_per_contig=True keeps one eligible SNP per RAD contig; use False for whole-chromosome data. Set max_variants=None to keep all eligible SNPs. Plain and gzipped VCFs are supported.
samples=sites returns genotypes in coordinate-file order. Climate rows preserve the same order, so sites, X and y stay aligned.
sites = read_coordinates("/home/jovyan/shared/EasternSnakeGenetics/Pguttatus_coords.txt")
genotypes = read_vcf(
"/home/jovyan/shared/EasternSnakeGenetics/Pguttatus.vcf",
samples=sites,
min_call_rate=0.8,
min_maf=0.05,
one_per_contig=True,
max_variants=256,
random_state=42,
)
sites.plot()
Sample climate
sample_environment takes the bioclim directory, opens a bounding box around the sites with 20% of its width/height added on each side, and extracts native-resolution values in coordinate-file order. It reprojects sites to the raster CRS and closes the rasters after sampling. Use variables=["bio1", "bio12"] to select layers or buffer=0.1 for a smaller region.
Missing climate values remain NaN. drop_missing=True removes incomplete rows; if used, align the other inputs to those remaining IDs once before fitting. The example retains all 36 sites.
X = sample_environment("/home/jovyan/shared/bioclim", sites)
y = genotypes.dosage.to_pandas()
X.head()
sites.plot(column=X.bio1, legend=True);
Fit and inspect
Each SNP uses its observed genotypes. Only responses with positive out-of-bag (OOB) R² and permutation importance contribute to turnover.
max_level=2 uses approximate conditional permutations for correlated predictors; max_level=0 gives marginal permutations. corr_threshold sets the correlation cutoff. Other options control tree size (max_features, min_samples_leaf, max_depth), curve resolution (n_bins), permutation repeats (n_permutations), minimum skill (min_oob_r2), density correction (density_correction) and tree parallelism (n_jobs).
gf = GradientForest(
n_estimators=100, max_level=2, random_state=42, keep_response_curves=True,
).fit(X, y)
gf.response_scores_.sort_values("oob_r2", ascending=False).head()
response_scores_ records OOB skill and inclusion for every SNP; oob_predictions_ holds its OOB predictions. permutation_importances_ contains raw MSE increases and response_importances_ contains allocated R². Set keep_forests=True to retain fitted trees in forests_.
importance_ ranks predictors by mean allocated R²; feature_importances_ contains the same values in input order.
gf.importance_.sort_values().plot.barh(figsize=(6, 5), xlabel="Mean allocated OOB R²");
Turnover curves and transformed samples
turnover_ contains a cumulative curve for each predictor. Steep sections indicate rapid turnover; endpoints give predictor importance. transform returns these turnover coordinates, preserving sample IDs. predict is an alias; fit_transform(X, y) combines fitting and transformation.
fig, axes = plt.subplots(2, 3, figsize=(11, 6), layout="constrained")
for ax, name in zip(axes.flat, gf.importance_.head(6).index):
curve = gf.turnover_[name]
ax.plot(curve.value, curve.cumulative_importance)
ax.set(title=name, xlabel="Environmental value", ylabel="Cumulative importance")
plt.show()
turnover = gf.transform(X)
turnover.head()
Map turnover
plot_turnover() uses the directory and sites remembered from sample_environment, opens the same buffered region, transforms it, and overlays the sites. It defaults to the most important predictor; pass a name such as "bio1" to choose another. Directory maps use every fourth cell by default (stride=1 keeps native resolution).
The result exposes .figure, .ax and a computed xarray .turnover Dataset for export. transform_raster remains available for custom xarray grids; missing pixels are masked and values outside training ranges are clamped.
turnover_map = gf.plot_turnover() # Or gf.plot_turnover("bio1")
Spatial turnover and environmental biplots
The map and biplot share colours from the first three PCs of GF-transformed environments. Pass the bioclim directory; site bounds, the 20% buffer and downsampling are handled internally. PCA uses the resulting region without standardizing predictors, so GF importance weights are preserved. Labelled arrows show the largest environmental covariance loadings in PC1/2: direction indicates increasing values and relative length indicates contribution to the displayed pattern.
response=None uses all contributing SNPs. Pass a SNP name for its own turnover curves (requires keep_response_curves=True when fitting). n_vectors controls labelled arrows; label_sites=True adds sample IDs; inset=True places the biplot inside the map. Omit environment for a PCA of sample locations alone.
biplot = gf.plot_biplot(X, sites, environment="/home/jovyan/shared-readwrite/bioclim", n_vectors=3)
snp = gf.response_scores_.oob_r2.idxmax()
snp_biplot = gf.plot_biplot(X, sites, environment="../envs", response=snp, n_vectors=3)
biplot.scores, .vectors and .colors contain the numerical results; .rgb is the georeferenced map colour array. Each SNP has its own PCA/colour scale. These plots summarize predicted turnover, not predicted allele frequencies. This example uses individuals; for population-level composition, fit population allele frequencies against population-level environments and supply population locations.
Export
Save diagnostics, sample coordinates, curves and a GeoTIFF with one named band per predictor.
Path("outputs").mkdir(exist_ok=True)
gf.response_scores_.to_csv("outputs/snp_oob_scores.csv")
gf.importance_.to_csv("outputs/predictor_importance.csv")
turnover.to_csv("outputs/sample_turnover.csv")
pd.concat(gf.turnover_, names=["predictor", "bin"]).to_csv("outputs/turnover_curves.csv")
bands = turnover_map.turnover.to_array("band").rio.write_nodata(np.nan)
bands.attrs["long_name"] = tuple(X.columns)
bands.rio.to_raster("outputs/genomic_turnover.tif", compress="deflate", dtype="float32")
turnover_map.figure.savefig("outputs/turnover_map.png", dpi=150, bbox_inches="tight")
biplot.figure.savefig("outputs/turnover_biplot.png", dpi=150, bbox_inches="tight")
snp_biplot.figure.savefig("outputs/snp_biplot.png", dpi=150, bbox_inches="tight")
These are individual genotype responses, not population allele frequencies. OOB skill and turnover alone do not establish adaptation or spatial transfer accuracy. Conditional permutations and smoothing differ from R gradientForest; see the README for method details and Ellis et al. (2012) for the original method.