Hyperparameter Optimization for LoRA-Based Fine-Tuning in Domain-Specific Large Language Models: A Case Study of POLYCC LLM League 2025

Abstract

Parameter-Efficient Fine-Tuning (PEFT), particularly Low-Rank Adaptation (LoRA), enables efficient adaptation of large language models (LLMs) to domain-specific tasks, yet practical hyperparameter optimization (HPO) remains challenging for non-expert users. This study investigates LoRA HPO within the POLYCC LLM League 2025, a cloud-based, gamified AI competition using AWS SageMaker. A structured grid search was conducted across learning rates (1×10⁻⁵–3×10⁻⁴), ranks (r∈{8,10,14,16}), scaling factors (α∈[16,64]), dropout (0.05–0.1), and training epochs (1–40), using fine-tuning logs from domain-specific datasets. Performance was evaluated using a standardized win rate metric reflecting comparative generative quality. The results reveal strong interdependencies among rank, scaling, and learning rate, where moderate configurations (r=16, α=32, LR=5×10⁻⁵, dropout=0.05) consistently outperform more aggressive settings, while excessive scaling and higher dropout reduce stability and degrade performance. These findings challenge the assumption that larger LoRA adapters yield better outcomes. This study provides empirically grounded HPO guidelines tailored to resource-constrained, educational environments, enabling users to achieve competitive performance without advanced expertise or extensive computational resources.