18 lines
842 B
Markdown
18 lines
842 B
Markdown
# LoRA Training on AMD
|
|
|
|
## Overview
|
|
|
|
### `flake.nix`
|
|
Provides a pure development environment via Nix Flakes. It handles:
|
|
- Configures Python 3.13 with ROCm support enabled.
|
|
- Provides PyTorch and specialized ML libraries (transformers, accelerate).
|
|
- Packages `unsloth` and `unsloth-zoo` with necessary patches for compatibility.
|
|
- Includes `marimo` and `ipython` for interactive development.
|
|
- Automatically managed via `direnv` (`use flake`) or `nix develop`.
|
|
|
|
### `train.py`
|
|
A `marimo` script that executes the fine-tuning process:
|
|
- Loads the `unsloth/Qwen3.5-0.8B` model in 16-bit.
|
|
- Prepares a sample dataset (`laion/OIG/unified_chip2.jsonl`).
|
|
- Configures Parameter-Efficient Fine-Tuning (PEFT) using LoRA (Rank 16).
|
|
- Sets up an `SFTTrainer` (using 8-bit AdamW) and trains the model for 100 steps, saving results to `outputs_qwen35`.
|