Assignment 09: Gradient Forest and Genomic Turnover
BIO597 Spatial Analysis of Biodiversity
Repeat the Lab09 workflow for a snake species other than P. guttatus. Use SNP genotypes and climate predictors to investigate where modeled genetic turnover occurs along environmental gradients. You may reuse code from the lab.
Deliverable
Submit this completed notebook with sample maps, SNP diagnostics, predictor importance, turnover curves, an aggregate PCA map/biplot, a single-SNP map/biplot, and short written answers. Save your diagnostic tables, transformed sample coordinates, curves, figures, and a GeoTIFF with one named band per predictor. Use the lab's filtering and model settings; no parameter search, additional algorithm, or future-climate projection is required.
Install pyGF
Because this notebook environment is a bit wonky you need to redo this install every time you run a new notebook process.
The pyGF package is only available in my github repo, so the installation will need to be
'by hand'.
- Open a new terminal and
cd ~to go to your home directory - You should already have cloned the pygf repo so you don't need to do this again (but if you need to the command is
git clone https://github.com/isaacovercast/pygf.git) pip install -e pygfto install this package and all requirements- Restart the kernel for this notebook w/ the 'circle arrow' button in the notebook toolbar
- Now you should be able to do the imports in the next cell
1. Import packages and set up paths
Run this notebook from docs/assignments using the same Python environment as Lab09, with pygf and its notebook dependencies installed. The imports and JupyterHub paths are provided below.
The assignment's genetic data are in /home/jovyan/shared/EasternSnakeGenetics. The climate layers use the same /home/jovyan/shared-readwrite/bioclim directory as Lab09. Keep all input paths in these two variables so later calls use the same files. Save your results in your own repository, not in a shared input directory.
For local work, the genetic files are in ../labs/EasternSnakes; you must also point bioclim_dir to a local directory containing the 19 regional bioclim rasters. The Maine-only climate layers from earlier labs do not cover these samples.
from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pygf import GradientForest, read_coordinates, read_vcf, sample_environment
genetics_dir = Path("/home/jovyan/shared/EasternSnakeGenetics")
bioclim_dir = Path("/home/jovyan/shared-readwrite/bioclim")
2. Choose another snake species
Choose one of the three species with both a VCF and a headerless coordinate file. Use the file stem exactly as shown; do not choose the lab's Pguttatus.
| File stem | Species | Genetic files |
|---|---|---|
Acontortrix |
Agkistrodon contortrix | Acontortrix.vcf, Acontortrix_coords.txt |
Lgetula |
Lampropeltis getula | Lgetula.vcf, Lgetula_coords.txt |
Sdekayi |
Storeria dekayi | Sdekayi.vcf, Sdekayi_coords.txt |
The .txt coordinate files contain sample ID, longitude, and latitude without a header, as expected by read_coordinates(). Use these files rather than the similarly named .csv tables.
target_species = "YOUR_SPECIES_HERE"
species_names = {
"Acontortrix": "Agkistrodon contortrix",
"Lgetula": "Lampropeltis getula",
"Sdekayi": "Storeria dekayi",
}
assert target_species in species_names, "Choose one of the three file stems above."
species_name = species_names[target_species]
vcf_path = genetics_dir / f"{target_species}.vcf"
coords_path = genetics_dir / f"{target_species}_coords.txt"
assert vcf_path.is_file(), f"VCF not found: {vcf_path}"
assert coords_path.is_file(), f"Coordinates not found: {coords_path}"
output_dir = Path("Assignment09_outputs") / target_species
output_dir.mkdir(parents=True, exist_ok=True)
print(species_name)
print(vcf_path)
print(coords_path)
Why this species? Give one biological reason for your choice.
Your answer:
3. Read coordinates and filtered genotypes
Read coords_path into sites with read_coordinates(). Read vcf_path into genotypes with read_vcf(), passing samples=sites so genotype rows follow coordinate-file order. Keep the lab settings:
min_call_rate=0.8min_maf=0.05one_per_contig=Truemax_variants=256random_state=42
Display the number of individuals and retained SNPs. Map the sample locations with a title naming your species. These points describe sampling, not the species' entire range.
# Read coordinates as sites.
# Read the VCF as genotypes, using samples=sites and all five settings above.
# Use genotypes.dosage.to_pandas().shape to inspect individuals × SNPs.
# Print the dimensions and plot sites with a species title.
Interpretation:
- What do dosage values 0, 1, and 2 represent? How does a missing call differ from a zero?
- What do call rate and minor allele frequency filter out? Why might the chosen SNP subset affect your conclusions?
- Why does taking one SNP per RAD contig reduce repeated local information without guaranteeing independent loci?
Your answers:
4. Sample climate and align individuals
Extract all 19 climate variables at the sample locations. The provided code removes rows with missing climate and aligns all three inputs to the same IDs. Missing genotypes remain NaN; each SNP's forest uses its observed calls. Do not replace missing genotypes with zeros.
sample_environment() reprojects points to the raster CRS and samples raster values at those locations. It also remembers the source directory and sampling region for mapping.
n_before = len(sites)
X = sample_environment(bioclim_dir, sites, drop_missing=True)
y = genotypes.dosage.to_pandas().loc[X.index]
sites = sites.loc[X.index].copy()
assert sites.index.equals(X.index)
assert X.index.equals(y.index)
assert np.isfinite(X.to_numpy()).all()
assert X.shape[1] == 19, "Use the full regional bioclim stack."
print("Individuals before climate extraction:", n_before)
print("Individuals with complete climate:", len(X))
print("Climate predictors:", X.shape[1])
print("Filtered SNPs:", y.shape[1])
X.head()
# Map sites colored by X["bio1"] with a legend and species title.
# Inspect y.head() to confirm that SNPs are columns and individuals are rows.
Interpretation: Were any individuals removed for missing climate? Why must sites, X, and y retain the same IDs and row order? Explain why these responses are individual genotypes rather than population allele frequencies.
Your answer:
5. Fit SNP-specific forests
Create gf = GradientForest(...) using n_estimators=100, max_level=2, random_state=42, and keep_response_curves=True, then fit it with X and y.
These settings match the lab. max_level=2 controls approximate conditional permutations for correlated predictors; it does not limit tree depth. Retaining response curves lets us inspect one SNP later.
# Create gf with the four settings above and fit it to X and y.
# Display gf.response_scores_ sorted by descending oob_r2.
6. Inspect OOB skill and contributing SNPs
Use gf.response_scores_ to report the number of SNPs evaluated, the number contributing to turnover (retained=True), and the fraction contributing. Display the counts of each reason, and plot the distribution of oob_r2 values with a vertical line at zero.
Only SNPs with positive OOB R² and positive permutation importance contribute. Include excluded SNPs in your diagnostic summary, not just the best-scoring ones. If no SNPs contribute, report that outcome and consult the instructor before changing filters or model settings.
# Store gf.response_scores_ as scores.
# Print evaluated and retained counts and the retained fraction.
# Display scores["reason"].value_counts().
# Plot scores["oob_r2"].dropna() as a histogram, with a reference line at zero.
Interpretation: What does an OOB R² below zero mean? Why does a positive OOB score not establish either adaptation or accurate prediction in a new geographic region?
Your answer:
7. Rank environmental predictors
Plot gf.importance_ as a horizontal bar chart, as in the lab. Label the axis Mean allocated OOB R². Identify your top three predictors and look up their meanings in the WorldClim variable definitions.
These values average allocated skill over contributing SNPs. They are not percentages of adaptive variation and need not sum to one.
# Make a labeled horizontal bar plot of gf.importance_.
# Display the three highest-ranked predictors and their importance values.
Interpretation: Name your top three climate variables. Suggest a biological explanation for one association, and one reason that predictor importance does not isolate a causal effect. How could correlation among the 19 climate predictors complicate the ranking?
Your answer:
8. Plot cumulative turnover curves
Run the lab's plotting recipe below to show the six leading predictors. Each x axis uses that predictor's environmental values; check the raster units or scaling before assigning physical units. Each endpoint equals the predictor's importance.
fig, axes = plt.subplots(2, 3, figsize=(11, 6), layout="constrained")
for ax, name in zip(axes.flat, gf.importance_.head(6).index):
curve = gf.turnover_[name]
ax.plot(curve.value, curve.cumulative_importance)
ax.set(title=name, xlabel="Environmental value", ylabel="Cumulative importance")
fig.suptitle(f"{species_name}: cumulative turnover")
plt.show()
curves_figure = fig
Interpretation: Choose one curve and describe the environmental interval where it rises most rapidly, using approximate x-axis values. What does a high but flat section mean? Why can the cumulative curve increase even when some SNP dosages decrease along the gradient?
Your answer:
9. Transform sampled environments and map a turnover layer
Use gf.transform(X) to create turnover, then display its first rows and dimensions. This returns one turnover coordinate per environmental predictor for each individual. In pyGF, predict() aliases transform(); it does not return predicted allele frequencies.
The map call below uses the leading predictor and the same regional bioclim directory. It samples every fourth raster cell in a region buffered 20% around the sites, as in the lab. Missing predictor pixels are masked; values beyond a fitted predictor range are clamped to its curve endpoint.
# Create turnover with gf.transform(X).
# Display turnover.head() and turnover.shape.
turnover_map = gf.plot_turnover(environment=bioclim_dir, points=sites)
Interpretation: Which predictor is mapped? Describe one geographic transition in its transformed values. Why is a high map value not necessarily a place with rapid local turnover? What happens outside the predictor's training range, and why does this limit interpretation?
Your answer:
10. Make an aggregate spatial PCA biplot
Create biplot with gf.plot_biplot(X, sites, environment=bioclim_dir, n_vectors=3). The map and ordination share colors from the first three PCs of the GF-transformed environments. PC1 and PC2 locate points in the biplot; PC1/2/3 supply the color channels.
The PCA uses transformed raster environments without standardizing columns to unit variance, preserving GF importance weights. Labelled arrows are environmental covariance loadings: direction indicates increasing transformed environmental values, and relative length indicates contribution to the displayed PC1/2 pattern.
# Create biplot with the call above.
# Display biplot.vectors to inspect the environmental arrows.
Interpretation: Describe the main geographic color transition and an environmental arrow associated with it. What does similarity in color summarize? Why do neither the colors nor the arrows measure fitness or prove local adaptation?
Your answer:
11. Compare with a single-SNP biplot
Select the highest-OOB-R² SNP among contributing responses using the provided code. Create snp_biplot with the same call as before, adding response=snp. Use environment=bioclim_dir in both biplot calls, rather than an upstream-relative ../envs path.
Each response has its own PCA basis and color calibration. Compare geographic organization and arrows; matching colors across separate plots have no shared genetic meaning. Selecting the best-scoring SNP is an illustration, not independent validation or discovery of an adaptive locus.
contributing_scores = gf.response_scores_.loc[gf.response_scores_["retained"]]
snp = contributing_scores["oob_r2"].idxmax()
print("Selected SNP:", snp)
print(contributing_scores.loc[snp])
# Create snp_biplot using gf.plot_biplot(), environment=bioclim_dir,
# response=snp, and n_vectors=3.
Interpretation: How does this SNP's geographic pattern differ from the aggregate pattern? Can you interpret a red region in both plots as the same genetic composition? Explain.
Your answer:
12. Export your results
The export recipe below follows the lab. It saves outputs under Assignment09_outputs/<your file stem>/, keeping runs for different species separate. The GeoTIFF contains one named band per environmental predictor, not per SNP.
Run the cell after completing the earlier steps. Keep the sample IDs in exported tables and preserve the raster's CRS and NoData mask. The small summary records the species, data paths, filtering choices, and model settings used in this analysis.
gf.response_scores_.to_csv(output_dir / "snp_oob_scores.csv")
gf.importance_.to_csv(output_dir / "predictor_importance.csv")
turnover.to_csv(output_dir / "sample_turnover.csv")
sites[["longitude", "latitude"]].to_csv(output_dir / "sample_coordinates.csv")
pd.concat(gf.turnover_, names=["predictor", "bin"]).to_csv(output_dir / "turnover_curves.csv")
summary = pd.DataFrame([{
"species": species_name,
"file_stem": target_species,
"vcf": str(vcf_path),
"coordinates": str(coords_path),
"bioclim_directory": str(bioclim_dir),
"n_individuals": len(X),
"n_predictors": X.shape[1],
"n_snps": y.shape[1],
"n_contributing": int(gf.response_scores_["retained"].sum()),
"min_call_rate": 0.8,
"min_maf": 0.05,
"one_per_contig": True,
"max_variants": 256,
"n_estimators": 100,
"max_level": 2,
"random_state": 42,
}])
summary.to_csv(output_dir / "analysis_summary.csv", index=False)
bands = turnover_map.turnover.to_array("band").rio.write_nodata(np.nan)
bands.attrs["long_name"] = tuple(X.columns)
bands.rio.to_raster(output_dir / "genomic_turnover.tif", compress="deflate", dtype="float32")
curves_figure.savefig(output_dir / "turnover_curves.png", dpi=150, bbox_inches="tight")
turnover_map.figure.savefig(output_dir / "turnover_map.png", dpi=150, bbox_inches="tight")
biplot.figure.savefig(output_dir / "turnover_biplot.png", dpi=150, bbox_inches="tight")
snp_biplot.figure.savefig(output_dir / "snp_biplot.png", dpi=150, bbox_inches="tight")
print("Saved results to:", output_dir.resolve())
summary
Final reflection
Answer each question in two or three sentences, referring to your figures or diagnostic results.
- How does this genotype-based analysis differ from the species presence/background SDMs in Labs07–08? Why is a turnover coordinate different from a suitability score or allele frequency?
- Which predictor and environmental interval showed a clear turnover pattern for your species? What evidence supports your description, and what sampling or modeling choice could change it?
- Describe one alternative to local adaptation that could produce a genetic–environmental association. What additional data or analysis would help distinguish the explanations?
- What would you withhold to test prediction in a new region? Explain why OOB evaluation alone does not answer that question.
Your answers:
Submit your work
Restart the kernel and run the notebook from top to bottom after completing the code prompts. Check that your figures appear, written answers are filled in, diagnostic counts are reported, and all exports exist. Commit and push this notebook and your species' output folder to your class GitHub repository. The original VCF and climate rasters remain in shared storage and should not be included in your submission.
Run these Git commands from the repository root, replacing YOUR_SPECIES with your chosen file stem:
git add docs/assignments/Assignment-09-GradientForest.ipynb docs/assignments/Assignment09_outputs/YOUR_SPECIES/
git commit -m "Complete Gradient Forest assignment"
git push