The Hidden Cost of Fine-Tuning: Trade-offs Between Training Time, Win Rate, and Model Generalization

Abstract

The fine-tuning of the Meta Llama 3.2 3B Instruct model within the Amazon SageMaker JumpStart environment presents significant challenges in balancing computational expenditure with empirical performance, particularly in high-stakes settings such as the POLYCC LLM League 2025. This manuscript explicitly explores the "Hidden Cost" trade-off inherent in Parameter-Efficient Fine-Tuning (PEFT) through a systematic analysis of hyperparameter configurations. By evaluating a comprehensive grid derived from official league performance logs, we investigate the impact of learning rates, LoRA rank, and training duration on model efficacy. Our results demonstrate how excessive training time and aggressive parameter scaling, specifically in Epochs and LoRA Rank, frequently lead to diminishing returns in competitive Win Rates and severe degradations in model generalization. Empirical data identifies an optimal performance "sweet spot" at a Learning Rate of 5 x , a LoRA Rank of 16, and 20 Epochs, yielding a peak Win Rate of 62.0%. Conversely, extending training to 30 Epochs resulted in a drastic performance decline to 24.0%, while the discovery of the "Perplexity Paradox" highlights that low evaluation perplexity does not consistently correlate with generative task alignment. These findings mandate a principle of "Engineering Restraint" as a primary guideline for preserving core model intelligence in resource-constrained AI deployments.