aland/unsloth-lora-8060S
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Minimal example training a Qwen3.5 0.8B lora on AMD RYZEN AI MAX+ 395 APU
3 commits 1 branch 0 tags 35 KiB
Find a file
Exact
Alan c0a36eaa2c made dependancies pure
2026-03-29 12:06:04 +11:00
.envrc made dependancies pure 2026-03-29 12:06:04 +11:00
.gitignore made dependancies pure 2026-03-29 12:06:04 +11:00
flake.lock made dependancies pure 2026-03-29 12:06:04 +11:00
flake.nix made dependancies pure 2026-03-29 12:06:04 +11:00
README.md made dependancies pure 2026-03-29 12:06:04 +11:00
train.py init 2026-03-26 20:40:59 +11:00

README.md

LoRA Training on AMD

Overview

flake.nix

Provides a pure development environment via Nix Flakes. It handles:

  • Configures Python 3.13 with ROCm support enabled.
  • Provides PyTorch and specialized ML libraries (transformers, accelerate).
  • Packages unsloth and unsloth-zoo with necessary patches for compatibility.
  • Includes marimo and ipython for interactive development.
  • Automatically managed via direnv (use flake) or nix develop.

train.py

A marimo script that executes the fine-tuning process:

  • Loads the unsloth/Qwen3.5-0.8B model in 16-bit.
  • Prepares a sample dataset (laion/OIG/unified_chip2.jsonl).
  • Configures Parameter-Efficient Fine-Tuning (PEFT) using LoRA (Rank 16).
  • Sets up an SFTTrainer (using 8-bit AdamW) and trains the model for 100 steps, saving results to outputs_qwen35.