# CropCast

Seq2Seq LSTM forecaster for NDVI time-series with online fine-tuning. Pretrained on historical multi-year satellite vegetation data, then adapted to unseen sites via a causally-gated online fine-tuning loop.

## Overview

The pipeline has four stages:

1. **Pretraining** — A two-layer encoder-decoder LSTM is trained on multi-year NDVI curves using variable-length windows, blended teacher-forcing / autoregressive loss, and slope + curvature regularization with cosine-decayed weights.

2. **Validation** — The trained model is evaluated in pure autoregressive mode on held-out data. Metrics (RMSE, R²) are computed per timestep horizon.

3. **Streaming online fine-tuning** — Simulates real-time deployment. At each DAP cycle:
   - New raw satellite observations are ingested up to the current DAP
   - Sparse observations are smoothed to a daily grid via RBF-kernel SVR
   - The model is fine-tuned on the causal window with:
     - Frozen encoder, trainable decoder + prediction heads
     - Trust-region regularization against pretrained weights
     - Trajectory and flow-continuation priors (slope/curvature persistence from history)
     - Rollout-gated acceptance with EMA shadow
     - Replay buffer across DAP cycles

4. **Evaluation** — Per-DAP results (predictions, ground truth, RMSE, R², slope diagnostics) are saved as CSVs for analysis.

## Quick Start

The included notebook uses a **pretrained checkpoint** and sample test data, so you can run the online fine-tuning pipeline without training from scratch.

1. Install the package: `pip install -e .`
2. Place test data in `data/`:
   - `test_site_raw_2024.csv` — raw satellite observations (date-named columns)
   - `test_site_NDVI_2024.csv` — pre-smoothed ground truth
3. Place `pretrained.pt` in `checkpoints/`
4. Open `notebooks/online_finetuning_pipeline.ipynb` and run all cells

Training from scratch is available in the notebook appendix, using the full training data.

## Project Structure

```
crop_cast/
├── ndvi_forecast/          # Python package
│   ├── __init__.py         # Public API
│   ├── seed.py             # Reproducibility
│   ├── _utils.py           # Shared helpers
│   ├── data.py             # CSV loading, group stats, train/val split
│   ├── datasets.py         # VarSeqNDVIDataset, FrozenVarSeqNDVIDataset, BatchCollator
│   ├── model.py            # Seq2SeqNDVI (encoder-decoder LSTM)
│   ├── losses.py           # Masked MSE, slope, curvature losses
│   ├── trainer.py          # Pretraining loop
│   ├── finetune.py         # OnlineFineTuner + driver loop
│   ├── forecast.py         # Autoregressive inference
│   ├── offset.py           # Pre/post fine-tuning vertical shift
│   ├── evaluate.py         # Per-DAP metrics and CSV export
│   ├── viz.py              # NDVI curve plots, prediction GIFs
│   └── streaming.py        # Streaming driver: data ingestion, RBF smoothing, incremental FT
├── checkpoints/            # Pretrained model checkpoint
│   └── README.md
├── notebooks/
│   └── online_finetuning_pipeline.ipynb
├── data/                   # Place CSV files here (see data/README.md)
│   └── README.md
├── pyproject.toml
├── requirements.txt
├── setup.cfg
└── setup.py
```

## Installation

Requires Python >= 3.9.

```bash
cd crop_cast
pip install -e .
```

Or install dependencies directly:

```bash
pip install -r requirements.txt
```

### Dependencies

- PyTorch >= 2.0
- NumPy >= 1.24
- pandas >= 2.0
- scikit-learn >= 1.3
- matplotlib >= 3.7
- SciPy >= 1.11
- Pillow >= 10.0

## Data Setup

### For running the notebook (streaming online fine-tuning)

Place the test site data and pretrained checkpoint:

| File | Location | Description |
|------|----------|-------------|
| `test_site_raw_2024.csv` | `data/` | Raw satellite NDVI with date-named columns (e.g. `avND20240325` or `20240325`) |
| `test_site_NDVI_2024.csv` | `data/` | Pre-smoothed ground truth NDVI (127 DAPs) |
| `pretrained.pt` | `checkpoints/` | Pretrained model weights |

### For training from scratch (optional)

Additionally place the full training CSVs:

| File | Location | Description |
|------|----------|-------------|
| `train_site_NDVI_2020.csv` | `data/` | Training data, year 1 |
| `train_site_NDVI_2021.csv` | `data/` | Training data, year 2 |
| `train_site_NDVI_2022.csv` | `data/` | Training data, year 3 |

### CSV format

Each CSV has one row per spatial grid cell and 128 columns: a grid cell identifier followed by 127 NDVI values (DAP 0–126). NDVI values should be floats in [0, 1], preprocessed and smoothed prior to use. See `data/README.md` for details.

## Key Hyperparameters

### Pretraining

| Parameter | Value | Description |
|-----------|-------|-------------|
| Hidden dim | 64 | LSTM state size |
| Encoder layers | 2 | Stacked LSTM layers |
| Decoder layers | 2 | With dropout 0.1 |
| Learning rate | 1e-3 | Adam optimizer |
| Batch size | 16 | Training batches |
| Input length | 60–125 | Variable window (DAPs) |
| Target length | 39 | Fixed prediction horizon |
| Group stats DAP | 45 | DAP index for year-level grouping |
| Early stopping | patience 5 | Based on validation loss |

### Streaming Online Fine-Tuning

| Parameter | Value | Description |
|-----------|-------|-------------|
| Learning rate | 8e-5 | Fine-tuning rate |
| DAP start | 45 | First fine-tuning step |
| DAP step | 3 | Interval between cycles |
| FT window | 8 DAPs | Short causal target window |
| Replay buffer | 12 | Past FT batches retained |
| Trust-region | 3e-4 | L2 penalty to pretrained weights |
| Trajectory prior | 0.20 | Slope persistence weight |
| RBF smoothing | C=500, gamma=1e-4 | SVR kernel parameters |

## Outputs

Results are saved under `results/`:

- `online_ft_*/csv/dap_XXX/` — Per-DAP directories containing:
  - `inputs.csv`, `preds.csv`, `ground_truth.csv`
  - `metrics.csv` — Per-grid RMSE and R²
  - `diagnostics.csv` — Slope MAE, peak offsets
  - `overall.txt` — Aggregate metrics and diversity
- `metrics_vs_dap.svg` — Summary plot of RMSE and R² across DAPs

## Model Architecture

```
Encoder (2-layer LSTM)
  Input: [NDVI, RAW_DAP] → (B, L_in, 2)
  Output: hidden state (h, c)

Season Head (MLP)
  Input: top encoder hidden state → (B, 64)
  Output: latent season descriptor → (B, 32)

Decoder (2-layer LSTM, autoregressive)
  Input per step: [last_ndvi, group_mean, group_std, dap_t, season_desc] → (B, 1, 36)
  Prediction: Δ_NDVI via 3-layer MLP (tanh-bounded ±0.25)
  Stitch: hard C0 (pred[0] = last obs) + soft C1 (slope blending via learned gate)
  Output: last_ndvi + Δ_NDVI + affine correction → clamped to [0, 1]
```
