Systematic Hyperparameter Optimization of LoRA-Based Fine-Tuning in a Large Language Model Competition

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has become a critical approach for adapting Large Language Models (LLMs) under computational and data constraints. Among these methods, Low-Rank Adaptation (LoRA) enables efficient model specialization with minimal parameter updates; however, its performance is highly sensitive to hyperparameter configuration, particularly in small, domain-specific datasets. Despite its widespread adoption, systematic empirical evidence on optimal hyperparameter settings in constrained and competitive environments remains limited. This study presents a structured empirical investigation of LoRA-based fine-tuning applied to the Llama 3.2 (3B) model within a domain-specific LLM competition setting. A total of 73 controlled experiments were conducted to evaluate the impact of key hyperparameters, including learning rate (1×10⁻⁵ to 5×10⁻⁴), training epochs (1–8), and validation split ratios (0.1 and 0.2), alongside dataset engineering strategies. Model performance was assessed using evaluation loss and perplexity to quantify generalization capability. The results demonstrate that learning rate is the dominant factor influencing model stability and performance. The optimal configuration—learning rate of 1×10⁻⁵, 3 training epochs, and a validation split of 0.2—achieved the lowest evaluation loss (1.1020) and perplexity (3.0101), outperforming the final competition submission models. Findings further reveal that extended training beyond early convergence leads to performance saturation or degradation, while dataset engineering techniques contributed marginal improvements within the tested scope. This study contributes a reproducible experimental framework for systematic hyperparameter optimization in PEFT-based LLM adaptation. The findings provide actionable guidelines for achieving stable and generalizable performance in resource-constrained environments, particularly in educational and competition-based settings. These insights highlight the importance of controlled training dynamics over heuristic dataset manipulation, offering practical implications for efficient LLM deployment in real-world applications.