Result Analysis of Low-Rank Adaptation Hyperparameters for PolyCC LLM League on AWS SageMaker

Abstract

This experiment investigates the optimization of Parameter-Efficient Fine-Tuning (PEFT) for foundations models in a cloud-native environment. Specifically, it analyses the performance of Meta’s Llama 3.2 3B Instruct model when adapted to a specialized dataset focused on the Malaysian vocational education landscape, industry partnerships, and institutional roles of polytechnics and community colleges. Using Low-Rank Adaptation (LoRA) on Amazon SageMaker, fifteen experimental iterations were conducted to identify the critical thresholds for learning rates, epoch counts, and rank configurations. The findings indicate that Llama 3.2 3B is highly sensitive to the scaling ratio between LoRA Alpha and Rank. The study identifies that Lim-Model-13, utilizing a 10^-4 learning rate and a 4:1 Alpha-to-Rank ratio over 12 epochs, provides the most balanced performance by minimizing evaluation loss (1.0017) while preventing the catastrophic forgetting observed at higher learning rates. A detailed gap analysis further reveals critical boundaries between training fit and generalization, highlighting the risks of aggressive learning in specialized domains.