LoRA Training on AMD

Overview

shell.nix

Provides a development environment compatible within Nix. It handles:

• Installing PyTorch optimized for AMD ROCm (rocm7.2).
• Installing unsloth and unsloth-zoo for efficient fine-tuning.
• Installing marimo and ipython as QOL.

train.py

A marimo script that executes the fine-tuning process:

• Loads the unsloth/Qwen3.5-0.8B model in 16-bit.
• Prepares a sample dataset (laion/OIG/unified_chip2.jsonl).
• Configures Parameter-Efficient Fine-Tuning (PEFT) using LoRA (Rank 16).
• Sets up an SFTTrainer (using 8-bit AdamW) and trains the model for 100 steps, saving results to outputs_qwen35.