Ning Wang a, Hamid R. Parsaei b,Yali Ren c, 28 June 2026
https://www.sciencedirect.com/science/article/abs/pii/S0925231226008660
Abstract
This paper presents a novel Deep Q-Learning (DQL) framework for multi-robot navigation that addresses the critical problem of target congestion in swarm robotics systems. The framework employs a Centralized Training with Decentralized Execution (CTDE) paradigm, where a single Deep Q-Network (DQN) agent learns a global coordination policy during offline training, and individual robots execute learned policies using only local Radio-Frequency Identification (RFID) based observations during deployment. Scalability refers to decentralized execution, while training is conducted under fixed swarm configurations. Unlike traditional vision-based obstacle-avoidance methods, we propose an RFID-based information-processing module combined with Robot Constraint Rules (RCR) that provide explicit heuristic guidance for flow regulation, while the DQN learns to optimize its application. The Probabilistic Finite State Machines (PFSM) and RCR modules define structured safety constraints, while coordination strategies are learned within this constrained action space. The system is modeled as a Discrete-time Markov Decision Process (DTMDP), which is subsequently integrated with PFSM to capture both reactive behaviors and learned coordination strategies. The PFSM framework enables the formalization of human-traffic-inspired behaviors (such as following, lane-changing, etc.) that arise within the constrained decision-making framework through reward-driven learning rather than hard-coded rules. The framework demonstrates empirically stable convergence under structured constraints. Simulation results demonstrate that our method significantly reduces the average robot workload by up to 66% and improves system efficiency by 51% compared to baseline reactive collision-avoidance methods without congestion control, while maintaining target occupancy density below 50% and ensuring deadlock-free operation even with 100 robots.
Introduction
In recent years, swarm robotics has emerged as a transformative paradigm for accomplishing complex tasks through the coordinated operation of large numbers of simple, low-cost robots [1], [2], [3]. Unlike traditional single-robot systems that rely on sophisticated sensing, computation, and actuation capabilities concentrated in individual units, swarm robotics distributes functionality across many agents, offering significant advantages in robustness (graceful degradation under failures), flexibility (adaptation to diverse tasks), and scalability (performance improvement with additional robots) [4]. These characteristics make swarm robotics particularly attractive for applications in warehouse automation [5], search and rescue operations [6], environmental monitoring [7], and precision agriculture, where tasks can be naturally decomposed and parallelized across multiple agents.
However, as swarm size grows, target congestion, where multiple robots simultaneously converge on shared goal locations, becomes a critical bottleneck that can degrade throughput by up to 70% and induce system-wide deadlock [8], [9], [10]. This problem is distinct from general collision avoidance: it is spatially concentrated at predictable locations, temporally dynamic with stochastic arrival patterns, and exhibits non-linear degradation with sharp phase transitions beyond critical density thresholds. Existing approaches—centralized planning, reactive avoidance, spatial partitioning, and learning-based methods—each address only a subset of these characteristics, leaving a significant gap in scalable, proactive congestion prevention.
This paper bridges that gap by presenting a Deep Q-Learning (DQL) framework that integrates Radio-Frequency Identification (RFID) based infrastructure with adaptive reinforcement learning to regulate robot flow into congested regions before congestion occurs. Our main contributions are:
- 1.A hybrid Centralized Training and Decentralized Execution (CTDE)–RFID architecture enabling scalable deployment without centralized control;
- 2.A Probabilistic Finite State Machine (PFSM) constrained Discrete-time Markov Decision Process (DTMDP) formulation combining interpretable finite-state behaviors with learned coordination;
- 3.Flow regulated congestion aware reward shaping through Robot Constraint Rules (RCR) that enforce safety while permitting adaptive optimization;
- 4.Comprehensive empirical validation against Multi-Agent Reinforcement Learning (MARL) baselines (QMIX, Counterfactual Multi-Agent Policy Gradient (COMA), Independent Deep Q-Network (I-DQN)) demonstrating 87.6% efficiency at 100 robots with deadlock-free operation.
The resulting framework is theoretically grounded and empirically validated, rather than provably optimal.
Our proposed framework is inspired by human traffic management systems, such as traffic lights, flow metering, and lane control, as well as transportation theory, including queuing models and flow dynamics. These concepts inform our method for regulating robot entry into congested regions while maintaining smooth bidirectional flows. Specifically, we address key limitations in existing approaches as follows. (i) Flow regulation is implemented through an RFID-based entry and exit control module, which manages robot admission based on real-time density measurements rather than relying on post-hoc collision avoidance. (ii) Predictive coordination is achieved using a DQN that learns anticipatory policies, such as proactively slowing before arrival when target occupancy approaches capacity, by optimizing cumulative discounted rewards over a planning horizon. (iii) Entry and exit balance is maintained through zone geometry design, which features separated entry and exit sectors and distributed exit angles. (iv) Scalability is ensured by the CTDE architecture: training utilizes the global state, while execution is fully decentralized with constant per-robot computation and linear communication complexity. (v) Automatic parameter tuning replaces manual threshold selection, as the DQN jointly optimizes admission thresholds, detour triggers, and speed-adjustment parameters through reward maximization. (vi) Theoretical grounding is established through formal DTMDP convergence analysis under tabular assumptions (Theorem 1, Remark 1), with explicit acknowledgment of the gap to practical DQN and supporting empirical convergence evidence.
The central insight of this framework is the modeling of target congestion as a DTMDP, in which system states represent robot distributions across behavioral modes (waiting, navigating, following, detouring, agitated) and target-area density. Concurrently, actions regulate admission policies and behavioral triggers via an RFID-based coordination module. By employing DQN to learn high-quality policies, the system adaptively balances competing objectives: maximizing throughput, minimizing waiting times, reducing energy consumption, and preventing deadlocks. Fig. 1 presents an overview of the complete system architecture.
The remainder of this paper is organized as follows. Section 2 reviews the related work. Section 3 introduces the DQL framework for multi-robot anti-congestion navigation. Section 4 presents a comprehensive simulation-based evaluation. Section 5 discusses the theoretical contributions and practical implications. Finally, Section 6 concludes the paper and outlines directions for future research.