unsloth-lora-8060S/README.md

LoRA Training on AMD

Overview

flake.nix

Provides a pure development environment via Nix Flakes. It handles:

• Configures Python 3.13 with ROCm support enabled.
• Provides PyTorch and specialized ML libraries (transformers, accelerate).
• Packages unsloth and unsloth-zoo with necessary patches for compatibility.
• Includes marimo and ipython for interactive development.
• Automatically managed via direnv (use flake) or nix develop.

train.py

A marimo script that executes the fine-tuning process:

• Loads the unsloth/Qwen3.5-0.8B model in 16-bit.
• Prepares a sample dataset (laion/OIG/unified_chip2.jsonl).
• Configures Parameter-Efficient Fine-Tuning (PEFT) using LoRA (Rank 16).
• Sets up an SFTTrainer (using 8-bit AdamW) and trains the model for 100 steps, saving results to outputs_qwen35.