Robust Federated Fine-Tuning in Heterogeneous Networks with Unreliable Connections: An Aggregation View

Abstract

Federated Fine-Tuning (FFT) has attracted growing interest as it leverages both server- and client-side data to enhance global model generalization while preserving privacy, and significantly reduces the computational burden on edge devices by avoiding training from scratch. Despite these advantages, FFT performance is often degraded by unreliable server-client connections and heterogeneous client data distributions. Most existing methods assume homogeneous network conditions or require prior knowledge of connection failures. However, these assumptions are impractical in real-world networks characterized by diverse communication standards (e.g., wired, Wi-Fi, 4G, and 5G) and heterogeneous failure patterns. To address these limitations, we propose FedAuto, a novel FFT framework that mitigates the combined effects of connection failures and data heterogeneity via adaptive aggregation. FedAuto operates without prior knowledge of network conditions or modifications to existing infrastructure, enabling seamless plug-and-play deployment. Moreover, we establish a rigorous convergence guarantee for FedAuto. By adopting a novel per-round aggregation perspective, our analysis removes the need for assumptions on connection failures probabilities or client selection strategies commonly imposed in prior work, and guarantees convergence of FedAuto for each individual realization, providing a stronger theoretical assurance. Extensive experiments demonstrate that FedAuto consistently outperforms state-of-the-art baselines under diverse connection failure scenarios for both full-parameter and partial-parameter fine-tuning (e.g., LoRA), and even surpasses strategies that rely on complex communication resource optimization.

Summary

This paper tackles the challenge of Federated Fine-Tuning (FFT) in realistic, heterogeneous network environments where unreliable connections and data heterogeneity can significantly degrade performance. The authors identify that existing FFT methods often assume ideal communication conditions or require knowledge of connection failure probabilities, which is impractical in real-world settings. To address this, they propose FedAuto, a novel FFT framework that adaptively aggregates client updates to mitigate the combined effects of connection failures and data heterogeneity. FedAuto requires no prior knowledge of network conditions and can be easily deployed on existing infrastructure. The core of FedAuto lies in its server-side adaptive aggregation strategy, which includes two modules: compensatory training and aggregation weight optimization. Compensatory training uses the server's public dataset to compensate for missing classes due to client disconnections. Aggregation weight optimization then balances the contribution of each class to the global model based on its proportion in the global dataset, ensuring that no class is disproportionately affected by connection issues. The authors provide a rigorous convergence analysis of FedAuto, demonstrating its ability to converge for each individual realization, a stronger guarantee than existing methods that only guarantee convergence in expectation. Experimental results across multiple datasets and connection failure scenarios show that FedAuto consistently outperforms state-of-the-art baselines, even surpassing methods that rely on complex communication resource optimization. This work matters to the field of federated learning because it provides a practical and robust solution for deploying FFT in realistic network environments, paving the way for wider adoption of FFT in various applications.

Key Insights

• •FedAuto introduces a novel server-side compensatory training mechanism that leverages the public dataset to compensate for missing classes due to connection failures, enhancing robustness in unreliable networks.
• •The paper proposes a novel aggregation weight optimization formulation (equation 8) which aims to balance each class's contribution to the global aggregation, addressing the impact of both connection unreliability and data heterogeneity.
• •FedAuto's convergence analysis provides a stronger theoretical guarantee by ensuring convergence for each individual realization, eliminating the need for assumptions on connection failure probabilities or client selection strategies.
• •Theorem 1 provides the first theoretical characterization of how aggregation weights influence FFT convergence under unreliable connections, offering valuable insights into the design of robust FFT strategies.
• •Experimental results show that FedAuto consistently outperforms state-of-the-art baselines under diverse connection failure scenarios, achieving up to 10% improvement in testing accuracy compared to FedAvg in non-i.i.d. settings with mixed connection failures (as implied from Fig. 2).
• •The paper explicitly addresses the joint effects of data heterogeneity and unreliable connections, a challenge often overlooked in existing FFT methods, which significantly improves performance in practical scenarios.
• •The constraint on the server's aggregation weight (β r s) in equation (9) ensures proportional contribution from the server's public dataset while allowing the global model to continuously integrate knowledge from private datasets, enhancing generalization.

Practical Implications

• •FedAuto can be readily deployed in real-world applications such as healthcare, finance, and edge intelligence, where data privacy and unreliable network connections are major concerns.
• •Practitioners and engineers can use FedAuto to improve the performance and robustness of FFT models in heterogeneous commercial networks without requiring modifications to existing network infrastructure.
• •The adaptive aggregation strategy in FedAuto can be easily integrated into existing FL systems, providing a plug-and-play solution for mitigating the effects of connection unreliability and data heterogeneity.
• •The theoretical analysis provides valuable guidelines for designing more robust FFT strategies under unreliable connections, enabling researchers to develop new algorithms that address the limitations of existing methods.
• •Future research directions include exploring privacy-preserving mechanisms for sharing local class distributions, relaxing the assumption that the server has access to all client-owned classes, and investigating the application of FedAuto in personalized federated learning settings.

Links & Resources

Authors