The recent development of connected and automated vehicle (CAV) technologies has spurred investigations to optimize dense urban traffic, maximizing vehicle speed and throughput. This article explores advisory autonomy, in which real-time driving advisories are issued to human drivers, thus achieving near-term performance of automated vehicles. Due to the complexity of traffic systems, recent studies of coordinating CAVs have leveraged deep reinforcement learning (RL). Coarse-grained advisory is formalized as zero-order holds, and we consider a range of hold durations from 0.1 to 40 s. However, despite the similarity of the higher frequency tasks for CAVs, a direct application of deep RL fails to generalize to advisory autonomy tasks. To overcome this, we employ zero-shot transfer, training policies on a set of source tasks—specific traffic scenarios with designated hold durations—and then evaluating the efficacy of these policies on different target tasks. We introduce temporal transfer learning (TTL) algorithms to select source tasks for zero-shot transfer, systematically leveraging the temporal structure to solve the full range of tasks. TTL selects the most suitable source tasks to maximize the performance of the range of tasks. We validate our algorithms on diverse mixed-traffic scenarios, demonstrating that TTL more reliably solves the tasks than baselines. This article underscores the potential of coarse-grained advisory autonomy with TTL in traffic flow optimization.
With continuously increasing requirements for physical human–robot interaction (pHRI), cable-driven parallel robots (CDPRs) have emerged as outstanding systems for its implementation due to the sufficient motion workspace and inherent cable flexibility. In particular, their modular structure facilitates straightforward reconfiguration. Inspired by this, this article aims to enhance the dynamic characteristics of CDPRs during pHRI through online reconfiguration, so as to achieve the interaction performance improvement based on human intent. A novel metric, the mixed interaction wrench margin, is first proposed to determine the optimal reconfiguration. This metric is devised by integrating the interaction force characteristics with CDPR inherent workspace properties, while explicitly considering the leading role of human intent. Subsequently, an adaptive-interaction-based reconfiguration strategy is established that the configuration can be arbitrarily changed by cable anchors, enabling compliant and adaptive pHRI. Informed by the actual interaction frequency, the strategy implements an asynchronous adjustment with different periods for configuration change and platform movement to achieve online optimization for reconfiguration. Finally, simulations and experiments conducted on different CDPR configurations with multiple pHRI tasks indicate that this strategy provides humans with more freedom, allowing them to exert more casual interaction forces, receive a quicker interactive response, and operate in a larger workspace.
Learning from demonstrations, the field that proposes to learn robot behavior models from data, is gaining popularity with the emergence of deep generative models. Although the problem has been studied for years under names, such as imitation learning, behavioral cloning, and inverse reinforcement learning, classical methods have relied on models that do not capture complex data distributions well or do not scale well to large numbers of demonstrations. In recent years, the robot learning community has shown increasing interest in using deep generative models to capture the complexity of large datasets. In this survey, we aim to provide a unified and comprehensive review of the last year’s progress in the use of deep generative models in robotics. We present the different types of models that the community has explored, such as energy-based models, diffusion models, action value maps, and generative adversarial networks. We also present the different types of applications in which deep generative models have been used, from grasp generation to trajectory generation or cost learning. One of the most important elements of generative models is the generalization out of distributions. In our survey, we review the different decisions the community has made to improve the generalization of the learned models. Finally, we highlight the research challenges and propose a number of future directions for learning deep generative models in robotics.
User intention estimation is essential in shared control systems for powered wheelchairs. It enables seamless navigation assistance that enhances safety, efficiency, and usability, while preserving user autonomy and reducing effort. This article presents diffusion-based wheelchair user intention estimation (DIWIE), a novel multimodal learning framework that leverages a denoising diffusion probabilistic model to forecast multiple plausible future trajectories, addressing uncertainty in human behavior. DIWIE conditions on diverse inputs, including obstacle information, user attention cues from eye gaze and head pose, semantic context, wheelchair kinematics, and joystick commands, operating without predefined maps or target destinations. Evaluated on a large new dataset of natural navigation by multiple drivers, DIWIE outperforms state-of-the-art methods, achieving lower displacement errors and collision rates, making it a valuable component for integration into shared control systems. This work also analyzes the relevance of different data sources for intention estimation and aligns evaluation metrics with related fields to foster reproducibility.
We present a neural radiance field (NeRF)-based large-scale reconstruction system that fuses lidar and vision data to generate high-quality reconstructions that are geometrically accurate and capture photorealistic texture. Our system adopts the state-of-the-art NeRF representation to additionally incorporate lidar. Adding lidar data adds strong geometric constraints on the depth and surface normals, which is particularly useful when modeling uniform texture surfaces which contain ambiguous visual reconstruction cues. A key contribution of this work is a novel method to quantify the epistemic uncertainty of the lidar-visual NeRF reconstruction by estimating the spatial variance of each point location in the radiance field given the sensor observations from the cameras and lidar. This provides a principled approach to evaluate the contribution of each sensor modality to the final reconstruction. In this way, reconstructions that are uncertain (due to, e.g., uniform visual texture, limited observation viewpoints, or little lidar coverage) can be identified and removed. Our system is integrated with a real-time pose-graph lidar simultaneous localisation and mapping (SLAM) system, which is used to bootstrap a structure-from-motion reconstruction procedure. It also helps to properly constrain the overall metric scale, which is essential for the lidar depth loss. The refined SLAM trajectory can then be divided into submaps using spectral clustering to group sets of covisible images together. This submapping approach is more suitable for visual reconstruction than distance-based partitioning. Our uncertainty estimation is particularly effective when merging submaps, as their boundaries often contain artifacts due to limited observations. We demonstrate the reconstruction system using a multicamera, lidar sensor suite in experiments involving both robot-mounted and handheld scanning. Our test datasets cover a total area of more than $\text{20 000 m}^{2}$, including multiple university buildings and an aerial survey of a multistorey building. Quantitative evaluation is provided by comparing with maps produced by a commercial tripod scanner.
As robots integrate into human society, ensuring safe robot-environment interaction, particularly in the event of collisions, has emerged as a growing design priority. A promising solution is introducing proper compliance into existing rigid robots, akin to musculoskeletal systems, to absorb impacts in various directions. However, mimicking longitudinal compliance seen in biological joints to absorb compressive impacts along limbs, such as the role of cartilages like menisci, remains a challenge shadowed by the complexity of joint architecture. Here, exploring and adapting the elastic longitudinal movement structure of knee, we incorporated traditional mechanical hinges with a compact buffer structure to enable both simple rotational motion and effective longitudinal impact absorption. Under longitudinal loading, the buffer structure functions as a mechanism transmitting the limited compression to amplified deformations of elastic elements, thus producing resistance against load. The resistive load-displacement relationship is tailorable by tuning diverse elastic components, allowing for a high-static-low-dynamic stiffness to improve energy absorption efficiency. Drop tests and walking robot demonstrations confirm that the proposed knee-inspired hinge not only mitigates acceleration transmitted to the robot’s main body but also reduces ground reaction forces, thereby improving robot-environment interaction safety. This work highlights the design paradigm of adapting natural solutions to mechanical systems, and holds potential for direct integration into diverse robots, exoskeletons, and prostheses.
Humans regularly throw projectiles with high speed and accuracy; some animals, including chimpanzees and elephants, also throw objects occasionally. In comparison, robots are currently lagging behind, despite having lower communication latency and more accurate motor control. To understand this paradox and ultimately achieve ubiquitous throwing robots, one of the major obstacles is the lack of high-fidelity and tractable physical models of the transient release dynamics, where the momentum exchange between the hand and the object occurs within tens of milliseconds via the frictional interface. In this work, we try to establish a physical model for the release dynamics. We first demonstrate that the conventional model, which combines rigid-body dynamics and patch friction [limit surface (LS)], struggles to capture the release dynamics and exhibits pathological behaviors, such as Zeno-like oscillations, leading to poor accuracy in predicting throwing outcomes. To mitigate this, we formulate a viscous-smoothed variant of the limit surface model solved via implicit integration (ILS), which achieves high predictive fidelity but incurs significant computational cost. On the other hand, motivated by the dominant effect of in-hand pivoting in release dynamics, we propose a sliding pivot model that simplifies the contact dynamics by capturing the sticking–pivoting–sliding behavior emerging under vanishing normal force. This model achieves accuracy comparable to ILS, with only 10% higher error while offering over 20× faster computation. Compared to conventional LS models, our method reduces horizontal velocity prediction error by 40% and angular velocity prediction error by 63%, achieving 2.4-cm mean absolute error (MAE) for landing position and 15.4$^\circ$ MAE for landing orientation. These results provide a robust physically grounded foundation for future scalable robot throwing systems.
Automating the manipulation of granular materials poses significant challenges due to complex contact dynamics, unpredictable material properties, and intricate system states. Existing approaches often fail to achieve efficiency and accuracy in such tasks. To fill the research gap, this article studies the small-scale and high-precision granular material digging task with unknown physical properties. A key scientific problem addressed is the feasibility of applying first-order gradient-based optimization to complex differentiable granular material simulation and overcoming associated numerical instability. A new framework, named differentiable digging robot (DDBot), is proposed to manipulate granular materials, including sand and soil. Specifically, we equip DDBot with a differentiable physics-based simulator, tailored for granular material manipulation, powered by GPU-accelerated parallel computing and automatic differentiation. DDBot can perform efficient differentiable system identification and high-precision digging skill optimization for unknown granular materials, which is enabled by a differentiable skill-to-action mapping, a task-oriented demonstration method, gradient clipping and line search-based gradient descent. Experimental results show that DDBot can efficiently (converge within 5 to 20 minutes) identify unknown granular material dynamics and optimize digging skills, with high-precision results in zero-shot real-world deployments, highlighting its practicality. Benchmark results against state-of-the-art baselines also confirm the robustness and efficiency of DDBot in such digging tasks.
This study presents an innovative solution for integrating a human and a robotic knee prosthesis symbiotically for walking. Achieving this requires human–robot cross-joint coordination to provide personalized walking assistance. Our approach uses inverse reinforcement learning to identify control objectives for reinforcement learning controller. Unlike existing methods that optimize performance of human or robot alone, our approach considers both human (thigh segmental angle) and robot (knee joint kinematics) aspects. This bilevel optimization method was evaluated on three nondisabled participants and two people with amputation. Results showed that the approach personalized the objective function and resulted in a robust policy, completing optimization within a duration of 3.5 min. Compared to previous approaches focusing only on robot states, this symbiotic approach increased stance time and step length on the prosthesis side for most participants. Our results highlight the potential of integrating human state into prosthesis control personalization, enhancing the functionality and health of people with amputation.
The agility of visual servo systems is essential for efficient operation in dynamic environments. However, this agility is limited by uncertainties, perceptual limitations, and the failure to integrate perception and control in conventional frameworks. In this article, a bioinspired reinforced active-inference-driven agile visual servo (RAID-AgiVS) technology is presented to systematically elude the above limitations. Central to RAID-AgiVS is the reinforced sensing mechanism that augments sparse visual measurements with fast soft measurements generated by alternating prediction and observation; thereby producing high-bandwidth data that surpasses the native sampling rate of the visual sensor. In addition, the reinforced data are employed by an active inference controller—embedded within a reciprocal perceptual control structure—to introduce perceptual inference and active control actions that bolster adaptability and agility in visual servo tasks. By fusing biological adaptability with advanced data generation and analytic modeling, the RAID-AgiVS framework markedly improves performance, particularly under sensory limitations and dynamic uncertainties. Comparative evaluations on a 6-DoF manipulator—and an indoor quadrotor unmanned aerial vehicle (UAV)—demonstrate the superior accuracy, agility, and adaptability of the proposed method.
Imitation-learning-based visuomotor policies excel at manipulation tasks but often produce suboptimal action trajectories compared to model-based methods. Directly mapping camera data to actions via neural networks can result in jerky motions and difficulties in meeting critical constraints, compromising safety and robustness in real-world deployment. For tasks that require high robustness or strict adherence to constraints, ensuring trajectory quality is crucial. However, the lack of interpretability in neural networks makes it challenging to generate constraint-compliant actions in a controlled manner. This article introduces differentiable policy trajectory optimization with generalizability (DiffOG), a learning-based trajectory optimization framework designed to enhance visuomotor policies. By leveraging the proposed differentiable formulation of trajectory optimization with transformer, DiffOG seamlessly integrates policies with a generalizable optimization layer. DiffOG refines action trajectories to be smoother and more constraint-compliant while maintaining alignment with the original demonstration distribution, thus avoiding degradation in policy performance. We evaluated DiffOG across 11 simulated tasks and two real-world tasks. The results demonstrate that DiffOG significantly enhances the trajectory quality of visuomotor policies while having minimal impact on policy performance, outperforming trajectory processing baselines such as greedy constraint clipping and penalty-based trajectory optimization. Furthermore, DiffOG achieves superior performance compared to existing constrained visuomotor policy.
With the promise of greater safety and adaptability, modular reconfigurable uncrewed air vehicles have been proposed as unique, versatile platforms holding the potential to replace multiple types of monolithic vehicles at once. State-of-the-art rigidly assembled modular vehicles are generally 2-D configurations in the shape of a “flight array,” where the rotors are coplanar. We introduce the Dodecacopter, a new type of modular rotorcraft where all modules take the shape of a regular dodecahedron, allowing the creation of richer sets of configurations beyond flight arrays. In particular, we show how the chosen module design can be used to create 3-D and fully actuated configurations. We justify the relevance of these types of configurations in terms of their structural and actuation properties with various performance indicators. Given the broad range of configurations and capabilities that can be achieved with our proposed design, we formulate tractable optimization programs to find optimal configurations given structural and actuation constraints. Finally, a prototype of such a vehicle is presented along with results of performed flights in multiple configurations.
Kinodynamic motion planning for a multirobot system with different dynamics and actuation limits is a challenging problem. The difficulty increases with the presence of aerodynamic interaction forces that occur when aerial robots fly in close proximity. Due to these complexities, existing planners either rely on simplified assumptions (like ignoring robot dynamics and interaction forces) or produce highly suboptimal solutions. This article presents a kinodynamic motion planner for a heterogeneous team of robots that respects robot dynamics, scales well to 16 robots, and directly reasons about interaction forces between aerial robots operating in close proximity. Our method, db-ECBS, generalizes the multiagent path-finding method Enhanced Conflict-Based Search (ECBS) to the continuous domain by using the single-robot kinodynamic motion planner discontinuity-bounded A*. The planner db-ECBS operates on three levels. Initially, individual robot trajectories are computed using a graph search that allows bounded discontinuities between precomputed motion primitives. The second level identifies interrobot collisions or interaction force violations and resolves them by imposing constraints on the first level. The third and final level uses the resulting solution with discontinuities as an initial guess for a joint-space trajectory optimization. The procedure is repeated with a reduced discontinuity bound, resulting in an anytime, probabilistically complete, and asymptotically bounded suboptimal planner. We provide a benchmark of 65 problems with six different dynamics. We demonstrate that db-ECBS produces trajectories that are less than half the cost of existing planners. We show that the interaction-awareness is particularly important for very dense scenarios.
In continuum robotics, real-time robust shape estimation is crucial for planning and control tasks that involve physical manipulation in complex environments. In this article, we present a novel stochastic observer-based shape estimation framework designed specifically for continuum robots. The shape state space is uniquely represented by the modal coefficients of a polynomial, enabled by leveraging polynomial curvature kinematics to describe the curvature distribution along the arclength. Our framework processes noisy measurements from limited discrete position, orientation, or pose sensors to estimate the shape state robustly. We derive a novel noise-weighted observability matrix, providing a detailed assessment of observability variations under diverse sensor configurations. To overcome the limitations of a single model, our observer employs the interacting multiple model (IMM) method, coupled with extended Kalman filters, to mix polynomial curvature models of different orders. The IMM approach, rooted in Markov processes, effectively manages multiple model scenarios by dynamically adapting to different polynomial orders based on real-time model probabilities. This adaptability is key to ensuring robust shape estimation of the robot’s behaviors under various conditions. Our comprehensive analysis, supported by both simulation studies and experimental validations, confirms the robustness and accuracy of our methods.
Heterogeneous Bayesian decentralized data fusion captures the set of problems in which two or more robots must combine probability density functions over nonequal, but overlapping sets of random variables. In the context of multirobot dynamic systems, this enables robots to take a “divide and conquer” approach to reason and share data over complementary tasks instead of over the full joint state space. For example, in a target tracking application, this allows robots to track different subsets of targets and share data on only common targets. This article presents a system by which robots can each use a local factor graph to represent relevant partitions of a complex global joint probability distribution, thus allowing them to avoid reasoning over the entirety of a more complex model and saving communication as well as computation costs. From a theoretical point of view, this article makes contributions by casting the heterogeneous decentralized fusion problem in terms of factor graphs, analyzing the challenges that arise due to dynamic filtering, and then developing a new conservative filtering algorithm that ensures statistical correctness. From a practical point of view, we show how this system can be used to represent different multirobot applications and then test it with simulations and hardware experiments to validate and demonstrate its statistical conservativeness, applicability, and robustness to real-world challenges.
Soft robotic devices, known for their high compliance, are increasingly being used in assistance and rehabilitation. However, the limited force output of soft actuators has hindered their broader adoption. In this study, a lobster-tail-inspired high-force-output soft pneumatic bending actuator (SPBA) is developed, featuring a soft deformable body and a rigid kirigami limiting shell. The SPBA, with a radius of 10 mm, can generate forces of approximately 22 N at an internal pressure of 0.1 MPa and 36.43 N at 0.16 MPa. An analytical model based on the Euler–Bernoulli beam theory, incorporating a hyperelastic material model, has been constructed to predict the deformation and force of the actuated SPBA. This model demonstrates good agreement with simulated and experimental results. For assistance, a soft robotic gripper with four SPBAs can lift a weight of 5.38 kg at 0.26 MPa. For rehabilitation, an SPBA-based hand exoskeleton has been developed, demonstrating significant effectiveness in mitigating hand spasticity following strokes. This study introduces a novel SPBA design with promising potential for future applications in grasping, assistance, and rehabilitation.
Narrow passage path planning is a prevalent problem from industrial to household sites, often facing difficulties in finding feasible paths or requiring excessive computational resources. In this article, we propose a homotopy optimization method tailored for the narrow passage problem, utilizing a novel collision constraint interpolation method using signed distance functions. The framework begins by decomposing the environment into convex objects and representing it as a simplicial complex based on their connectivity. This representation enables topological analysis to induce an easy-to-hard sequence of collision constraint interpolation that preserves homotopy equivalence. Using this collision constraint interpolation, the optimization proceeds through a series of subproblems, gradually guiding the path to the final solution. Several examples are presented to demonstrate how the proposed framework addresses narrow passage path planning problems.
Surface finishing is a time consuming, dangerous task, difficult to automate despite its necessity in many manufacturing processes. Its automation, particularly through robotics, increases productivity and relieves workers from health-critical tasks. However, challenges remain, as automated offline planning tools can result in certain areas being either neglected or overly processed. Ergodic control offers the possibility to cover target probability distributions in an online manner, by taking into account the observed coverage history. However, existing ergodic control approaches provide little flexibility in designing and adapting coverage strategies. Moreover, they come with simplifying assumptions, such as point-based dynamics, which are no longer valid for tasks where the robot is in contact with strongly varying curvatures on nontrivial surface geometries. In this work, we introduce a closed-form ergodic control framework that includes the tool imprint in the system modeling while simultaneously permitting the intuitive transfer of finish strategies, namely preferred motion directions. We build on the Spectral Multiscale Coverage approach, augmenting it with a tool imprint model, as well as both target distributions and state-dependent movement directions extracted from human demonstrations. Through evaluations in a surface finishing task using a torque-controlled, 7-DoF, robot arm we show that our approach optimally covers surfaces according to the tool contact area, with robust error convergence.
This study informs the design of future multiagent pathfinding and multirobot motion planning (MRMP) algorithms by guiding choices based on constraint classification for constraint-based search algorithms. We categorize constraints as conservative or aggressive and provide insights into their search behavior, focusing specifically on vanilla conflict-based search and conflict-based search with priorities. Under a hybrid grid-roadmap representation with varying resolution, we observe that aggressive (priority constraint) formulations tend to solve more instances as agent count or resolution increases, whereas conservative (motion constraint) formulations yield stronger solution quality when both succeed. Findings are synthesized in a decision flowchart, aiding users in selecting suitable constraints. Recommendations extend to MRMP, emphasizing the importance of considering topological features alongside problem, solution, and representation features. A comprehensive exploration of the study, including raw data and map performance, is available in our public GitHub Repository.
Inspired by soft creatures and structures in nature, continuum robots exhibit remarkable flexibility, safe interaction, and ease of miniaturization, showcasing vast application potential. However, their flexible structure renders analytical methods inadequate for precise modeling and control, while existing data-driven approaches suffer from low data efficiency and unproven theoretical control performance. This article aims to achieve data-efficient modeling and reliable control of continuum robots through innovative algorithms, exploring the performance of the new method from both theoretical and experimental perspectives. Specifically, we utilize neural ordinary differential equations (NODE) to achieve data-efficient modeling of continuum robots and investigate the performance of the modeling method. Then, we propose a novel predefined-time-synchronized stable zeroing neurodynamics (PTSS-ZND) model. By combining the NODE method and the PTSS-ZND method, we propose a reliable data-driven control system. Through rigorous theoretical analysis, we prove the stability and predefined-time convergence of the data-driven control system. Finally, through simulations and physical experiments, we validate the feasibility and convergence of the novel method and its advantages over existing data-driven methods. Experiments on one-segment and three-segment continuum robots indicate that the proposed method achieves a root mean square position error (e.g., 2.5 mm for the three-segment robot) of less than 1% of the robot length using fewer than 100 data samples. Our method also demonstrates robust performance under various external and internal disturbances. In addition, it can potentially be extended for end-effector pose control.
The enhanced mobility brought by legged locomotion empowers quadrupedal robots to navigate through complex and unstructured environments. However, optimizing agile locomotion while accounting for the varying energy costs of traversing different terrains remains an open challenge. Most previous work focuses on planning trajectories with traversability cost estimation based on human-labeled environmental features. This human-centric approach is insufficient because it does not account for the varying capabilities of the robot locomotion controllers over challenging terrains. To address this, we introduce a novel real-world learning pipeline that unifies offline demonstrations, online reinforcement learning, and multimodal perception to achieve robust legged navigation. The framework employs multiple training stages to develop a planner that guides the robot in avoiding obstacles and hard-to-traverse terrains while reaching its goals. We first develop a novel traversability estimator in a robot-centric manner. The training of the navigation planner is directly performed in the real world using a sample efficient reinforcement learning method. With the proposed method, a quadrupedal robot learns to perform traversability-aware navigation through real-world interactions in diverse offroad and unstructured environments. Moreover, the robot demonstrates the ability to generalize the learned navigation skills to unseen scenarios.
This article addresses survey missions involving multiple uncrewed aerial vehicles (UAVs) over complex, varying terrain. The methodology integrates a probabilistic model of target’s position uncertainty with UAV flight dynamics, camera properties, and a machine learning-based detection system. It estimates undetected target probability and overall search performance, feeding into a feedback loop that combines 2-D ergodic search with model predictive control (MPC) of UAV altitude and velocity. Trial trajectory optimization accounts for sensing characteristics and operational constraints, producing terrain-aware, collision-free trajectories that balance area coverage with target detection. Simulations demonstrate the integration of MPC and ergodic search, enabling dynamic altitude adjustments to enhance the search performance. The control algorithm operates in real time and performs reliably under uncertainty. Field experiments provided training data, validated the method, and confirmed compliance with motion constraints. Detection rates closely match model predictions, demonstrating stable performance even under significant deviations from ideal conditions. The framework, thus, offers a reliable solution for autonomous multi-UAV search operations in real-world environments.
Passivity-based control is a cornerstone of control theory and an established design approach in robotics. Its strength is based on the passivity theorem, which provides a powerful interconnection framework for robotics. However, the design of passivity-based controllers and their optimal tuning remain challenging. We propose here an intuitive design approach for fully actuated robots, where the control action is determined by a “virtual-mechanism” as in classical virtual model control. The result is a robot whose controlled behavior can be understood in terms of physics. We achieve optimal tuning by applying algorithmic differentiation to ordinary differential equation simulations of the rigid body dynamics. Overall, this leads to a flexible design and optimization approach: stability is proven by passivity of the virtual mechanism, while performance is obtained by optimization using algorithmic differentiation.
Vector fields are advantageous in handling nonholonomic motion planning, as they provide the robot with reference orientation across the workspace. However, additionally incorporating curvature constraints presents challenges due to the interconnection between the design of the curvature-bounded vector field and the tracking controller under limited actuation. In this article, we present a novel framework to co-develop the vector field and the control law, guiding the nonholonomic robot to the target configuration with curvature-bounded trajectory. First, we formulate the problem by introducing the target positive limit set, which allows the robot to either converge to or pass through the target configuration, depending on its dynamics and the specific tasks. Next, we construct a curvature-constrained vector field (CVF) via blending and embedding elementary flows in the workspace. To track such a CVF, a saturated control law with dynamic gains is proposed, under which the tracking error’s magnitude decreases even when saturation occurs. Under the control law, the kinematically constrained nonholonomic robot is guaranteed to track the reference CVF and converge to the target positive limit set with bounded trajectory curvature. Numerical simulations show that the proposed CVF method outperforms other vector-field-based algorithms. Experiments on Ackermann uncrewed ground vehicles and semiphysical fixed-wing uncrewed aerial vehicles demonstrate that the method can be effectively implemented in real-world scenarios.
Magnetic compression anastomosis (MCA) offers a promising solution for minimally invasive anastomosis surgery. However, current MCA schemes lack safe, real-time localization, and guidance for compression magnets, hindering surgeons' ability to control the compression magnets effectively in complex, unstructured endoluminal environments. To address these limitations, this article introduces the MagsL-HUD endoscopic system, a novel solution that enables multimagnetic six-degree-of-freedom (six-DoF) localization and head-up display (HUD) guidance within the endoscopic view (EV). Specifically, the system integrates a developed Endo-MagCap device with an orthogonal magnet configuration, along with a magnetic sensor array, to achieve real-time full-pose localization. An endoscopic camera model is incorporated for HUD visualization, enhancing intuitive interaction for surgeons' better-informed decisions. Eventually, the effectiveness of the MagsL-HUD endoscopic system is validated through laboratory experiments and ex vivo animal trials. The system demonstrates six-DoF tracking accuracy with average errors of 0.0070 m and 0.1437 rad, and 0.0071 m and 0.1721 rad in the designed trajectory cases for two compression magnets, respectively. Additionally, ex vivo porcine tests confirm the system's feasibility and applicability, successfully performing a stomach-colon MCA surgery with a final compression gap of approximately 0.00247 m. Further comparative studies demonstrate that the MagsL-HUD method has a compression success rate of 71.4$\%$ versus 42.9$\%$ of the non-HUD approaches in the designed tests. This work represents a significant step toward the clinical adoption of magnetic-assisted endoscopy for minimally invasive anastomosis surgeries, holding substantial practical significance for improving the safety and efficacy of MCA procedures in complex, unstructured endoluminal environments.
Social robots are increasingly used in healthcare and education, but technological gaps, fears of human replacement, and moral or social beliefs can limit their acceptance. In collaborative settings, the activities to complete may influence users’ willingness to participate, raising the question of how moral and social attitudes shape human–robot interaction. This article studies the effect of the social judgment theory on social robotics to analyze which factors affect the users’ willingness to complete the robot’s requests. The methodology classifies the activities requested by the robot into assimilation (activities people typically accept), noncommitment (activities people usually reject), and the contrast (activities some people accept) groups. We conducted a user study with 63 participants interacting with the Mini social robot in a collaborative session where it requested some actions from the user. We analyze whether the kind of activity requested by the robot, its expressiveness, and demographic, moral, social, and robot factors influence the user behavior. Results show that the social judgment theory can be extended to social robotics since the kind of activity affects the user’s willingness to complete it. Besides, the results indicate that an expressive robot convinced users more than a nonexpressive robot and that participants who lied about their completed activities were more easily persuaded. We also found that participants with moderate knowledge of robotics completed more activities than those with low knowledge, and individuals with previous experience interacting with Mini were more likely to comply with its requests. However, demographic factors such as age or gender do not seem to influence robot persuasion despite previous studies suggesting they are important in human–robot collaboration.
Underactuated robotic hands are extensively used in remote manipulation due to their ability to adapt to various object sizes and shapes. Their structural simplicity and small number of actuators required for operation make them highly versatile and responsive, which is crucial for effective teleoperation. In addition to grasping performance, haptic feedback, which integrates force and tactile sensing, is essential for dexterous manipulation. This study proposes a solution using fiber-optic tendons embedded with fiber Bragg gratings (FBGs), combining sensing and actuation to simultaneously perform power transmission, along with force and tactile sensing. Each finger employs a fiber-optic tendon with three FBGs: one measures tendon tension, and the other two at the fingertip detect contact force and temperature. The tendon is placed on the volar side of the finger and routed to an actuation module with a servomotor at the wrist for power transmission. This tendon enables finger flexion, while a passive extension mechanism with linear springs on the dorsal side facilitates extension. Experimental results demonstrate the feasibility of this approach, showing the hand’s multifunctional capabilities, including haptic feedback and power transmission, as well as its potential for teleoperation. This approach improves the robotic hand’s ability to provide real-time feedback, improving dexterity in remote manipulation.
Underwater acoustic communication, characterized by limited bandwidth, high latency, and low reliability, poses significant challenges for data exchange in bathymetric collaborative simultaneous localization and mapping (CSLAM). In this article, we introduce a novel vector quantization (VQ) method called ID(O) for mapping data compression in bathymetric CSLAM. ID(O) encodes the map into an index map ($\mathbb {I}$), a central depth map ($\mathbb {D}$), and an orientation map ($\mathbb {O}$). To accommodate strict communication constraints, orientations can be partially or fully excluded from transmission, and we propose a method to estimate these orientations during map restoration. Moreover, we integrate ID(O) within a feature-based bathymetric CSLAM framework named TTT CSLAM. Extensive experiments on two large-scale sea trial datasets demonstrate that ID(O) achieves about 40$\%$ higher restoration accuracy than the baseline method using principal component analysis. TTT CSLAM with ID(O) can match that with lossless compression regarding mapping accuracy and efficiency, and it is robust against 40$\%$ packet loss and large dead reckoning drift errors across diverse environments. To the best of the authors’ knowledge, ID(O) is the first VQ method for bathymetric data compression, and TTT CSLAM with ID(O) is the first bathymetric CSLAM tested within an underwater communication network employed by acoustic modems.
As learning-based methods for legged robots rapidly grow in popularity, it is important that we can provide safety assurances efficiently across different controllers and environments. Existing works either rely on a priori knowledge of the environment and safety constraints to ensure system safety or provide assurances for a specific locomotion policy. To address these limitations, we propose an observation-conditioned reachability-based (OCR) safety-filter framework. Our key idea is to use an OCR value network (OCR-VN) that predicts the optimal control-theoretic safety value function for new failure regions and dynamic uncertainty during deployment time. Specifically, the OCR-VN facilitates rapid safety adaptation through two key components: a LiDAR-based input that allows the dynamic construction of safe regions in light of new obstacles and a disturbance estimation module that accounts for dynamics uncertainty in the wild. The predicted safety value function is used to construct an adaptive safety filter that overrides the nominal quadruped controller when necessary to maintain safety. Through simulation studies and hardware experiments on a Unitree Go1 quadruped in agile planar navigation tasks, we demonstrate that the proposed framework can automatically safeguard a wide range of hierarchical quadruped controllers, adapts to novel environments, and is robust to unmodeled dynamics without a priori access to the controllers or environments—hence, “One Filter to Deploy Them All.”
A standardized robot-testing algorithm should satisfy accuracy, efficiency, and very importantly, repeatability—consistently yielding similar outcomes across multiple executions by different stakeholders. Achieving repeatability grows challenging with increasing complexity, intelligence, diversity, and inherent stochasticity in testing methods, robotic platforms, and environments. While existing efforts address repeatability through ethical, hardware, or procedural means, this study specifically targets algorithm-level repeatability, focusing on statistical query (SQ) algorithms commonly used in standardized evaluations. We propose a lightweight, adaptive modification for any SQ-based routine, including Monte Carlo, importance sampling, and adaptive sampling, guaranteeing provable repeatability with bounded accuracy and efficiency. Effectiveness is demonstrated across three cases: standardized manipulator testing, intelligent risk assessment for automated vehicles, and performance evaluation of humanoid robot locomotion.
Many robots are not equipped with a manipulator and many objects are not suitable for prehensile manipulation (such as large boxes and cylinders). In these cases, pushing is a simple yet effective nonprehensile skill for robots to interact with and further change the environment. Existing work often assumes a set of predefined pushing modes and fixed-shape objects. This work tackles the general problem of controlling a robotic fleet to push collaboratively numerous arbitrary objects to respective destinations, within complex environments of cluttered and movable obstacles. It incorporates several characteristic challenges for multirobot systems, such as online task coordination under large uncertainties of cost and duration, and for contact-rich tasks, such as hybrid switching among different contact modes, and under actuation due to constrained contact forces. The proposed method is based on combinatorial hybrid optimization over dynamic task assignments and hybrid execution via sequences of pushing modes and associated forces. It consists of the following three main components: first, the decomposition, ordering, and rolling assignment of pushing subtasks to robot subgroups; second, the keyframe guided hybrid search to optimize the sequence of parameterized pushing modes for each subtask; third, the hybrid control to execute these modes and transit among them. Last but not least, a diffusion-based accelerator is adopted to predict the keyframes and pushing modes that should be prioritized during hybrid search; and further improve planning efficiency. The framework is complete under mild assumptions. Its efficiency and effectiveness under different numbers of robots and general-shaped objects are validated extensively in simulations and hardware experiments, as well as generalizations to heterogeneous robots, planar assembly, and 6-D pushing.
The aspiration to replicate the capabilities of the human hand has driven innovations in the design of soft robotic hands. Despite these advancements, many existing designs of soft hands still lack effective in-hand vision and the ability for each finger to achieve active multidegree-of-freedom motion. This article proposes a cable-driven soft robotic hand that can achieve dexterous grasping and manipulation, vision-guided grasping, vision-based slip detection and compensation, as well as visually servoed in-hand manipulation. The hand has five soft fingers, each capable of independent flexion/extension motion and bidirectional ad/abduction motion. A red–green–blue-depth (RGB-D) camera is integrated into the palm of the soft hand to enable in-hand vision capability. Modeling of the soft hand is established to analyze its kinematics, statics, and manipulability. A series of experiments are conducted to demonstrate its dexterous grasping and manipulation capabilities on a variety of objects. Using 3-D point cloud data from the in-palm camera, an effective vision-guided grasping strategy is developed to grasp objects on a table. The in-hand vision also enables slip detection and compensation during grasping to maintain the grasp stability. Furthermore, a hierarchical, visually servoed controller is developed to perform closed-loop in-hand object manipulation. With its high dexterity and visual feedback capabilities, the soft hand will find important applications such as household object manipulation and food picking/sorting, and may also be used as a prosthetic hand or an auxiliary hand for humans.
Soft robots can revolutionize several applications with high demands on dexterity and safety. When operating these systems, real-time estimation and control require fast and accurate models. However, prediction with first-principles (FP) models is slow, and learned black-box models have poor generalizability. Physics-informed machine learning offers excellent advantages here, but it is currently limited to simple, often simulated systems without considering changes after training. We propose physics-informed neural networks (PINNs) for articulated soft robots (ASRs) with a focus on data efficiency. The amount of expensive real-world training data is reduced to a minimum—one dataset in one system domain. Two hours of data in different domains are used for a comparison against two gold-standard approaches: In contrast to a recurrent neural network, the PINN provides a high generalizability. The prediction speed of an accurate FP model is exceeded with the PINN by up to a factor of 467 at slightly reduced accuracy. This enables nonlinear model predictive control of a pneumatic ASR. Accurate position tracking with the MPC running at 47 Hz is achieved in six dynamic experiments.
Conforming to safety standards often limits collaborative robots’ performance and size, restricting their applications despite their capabilities. Planning their motions in human environments involves a tradeoff between optimal trajectory planning and quick adaptation to dynamic, unstructured spaces. Traditional trajectory planning methods either use simplified robot models and sacrifice robot’s abilities for computational efficiency, or exploit robots’ abilities fully but have high computational complexity and rely on substantial precomputation. This article introduces an approach for trajectory planning that exploits robot’s full motion abilities while planning on-the-fly. In each step of the trajectory execution, it evaluates robot’s movement ability using polytope algebra and calculates a time-optimal Trapezoidal Acceleration Profile (TAP) on the remaining trajectory. The method is shown to be near time-optimal (around 5% slower trajectories) by benchmarking it against the state-of-the-art time-optimal method TOPP-RA. The method allows reaching higher velocities (able to plan up to 100% of the robot’s kinematic limits) while at the same time lowering the tracking error (under 4 mm) than traditional Cartesian Space planning methods. A mock-up experiment demonstrates its efficiency in collaborative waste sorting using a Franka Emika Panda robot.
This article presents a correspondence-free, function-based sim-to-real learning method for controlling deformable freeform surfaces. Unlike traditional sim-to-real transfer methods that strongly rely on marker points with full correspondences, our approach simultaneously learns a deformation function space and a confidence map—both parameterized by a neural network (NN)—to map simulated shapes to their real-world counterparts. As a result, the sim-to-real learning can be conducted by input from either a 3-D scanner as point clouds (without correspondences) or a motion capture system as marker points (tolerating missed markers). The resultant sim-to-real transfer can be seamlessly integrated into a NN-based computational pipeline for inverse kinematics and shape control. We demonstrate the versatility and adaptability of our method on two vision devices and across four pneumatically actuated soft robots: a deformable membrane, a robotic mannequin, and two soft manipulators.
A key open challenge in agile quadrotor flight is how to combine the flexibility and task-level generality of model-free reinforcement learning (RL) with the structure and online replanning capabilities of model predictive control (MPC), aiming to leverage their complementary strengths in dynamic and uncertain environments. This article provides an answer by introducing a new framework called Actor–Critic MPC. The key idea is to embed a differentiable MPC within an actor–critic RL framework. This integration allows for short-term predictive optimization of control actions through MPC, while leveraging RL for end-to-end learning and exploration over longer horizons. Through various ablation studies, conducted in the context of agile quadrotor racing, we expose the benefits of the proposed approach: it achieves better out-of-distribution behavior, better robustness to changes in the quadrotor's dynamics and improved sample efficiency. In addition, we conduct an empirical analysis using a quadrotor platform that reveals a relationship between the critic's learned value function and the cost function of the differentiable MPC, providing a deeper understanding of the interplay between the critic's value and the MPC cost functions. Finally, we validate our method in a drone racing task on different tracks, in both simulation and the real world. Our method achieves the same superhuman performance as state-of-the-art model-free RL, showcasing speeds of up to 21 m/s. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the predictive properties of the MPC to better handle out-of-distribution behavior.
Despite extensive investigations into the multiuser haptic-enabled robotic system (M-Hers), achieving scalable control design in the presence of nonpassive human operators remains a key challenge. This is primarily due to the increasing complexity of stability conditions and interaction coupling as the number of operators grows. In this study, we address this challenge in two steps. First, we introduce the individual interaction environment (IIE) to isolate the passivity violations, which facilitates the independent control design for each human–robot subsystem, thereby enhancing the scalability with respect to the number of subsystems. Second, within the IIE framework, we identify passivity-violating components caused by partners' active behaviors and propose a novel augmented tank-based controller (ATBC) to guarantee passive IIE while maintaining high rendering accuracy. Specifically, the ATBC employs an energy-related power regulation strategy to enhance interaction safety and a time-varying control gain to mitigate the negative effects of power regulation on rendering fidelity. We validated the proposed method through collaborative haptic tasks on a customized M-Hers composed of three robots in four different scenarios. Comparative studies demonstrate that our approach effectively ensures IIE passivity in the presence of active human behaviors, while ensuring high reproducibility and achieving a favorable balance between passivity and rendering accuracy.
Efficient rigid-body dynamics algorithms are instrumental in enabling high-frequency dynamics evaluation for resource-intensive applications (e.g., model-predictive control, large-scale simulation, and reinforcement learning), potentially on resource-constrained hardware. Existing recursive algorithms with low computational complexity are mostly restricted to kinematic trees with external contact constraints or are sensitive to singular cases (e.g., linearly dependent constraints and kinematic singularities), severely impacting their practical usage in existing simulators. This article introduces two original low-complexity recursive algorithms: the loop-constrained articulated body algorithm and proximal BBO (Brandl, Bae, and others), both based on a proximal dynamics formulation for forward simulation of closed-loop mechanisms. These algorithms are derived from first principles using nonserial dynamic programming, exhibit linear complexity in practical scenarios, and are numerically robust in the face of singular cases. They extend the existing constrained articulated body algorithm to handle internal loops and the pioneering BBO algorithm from the 1980s to singular cases. Both algorithms have been implemented by leveraging the open-source Pinocchio library, benchmarked in detail, and demonstrate state-of-the-art performance for various robot topologies, including over $6\times$ speed-ups compared to existing nonrecursive algorithms for high-degree-of-freedom systems with internal loops, such as recent humanoid robots.
Due to their stability and robustness properties, stable dynamical systems (SDSs) have received considerable attention as a means of representing motions in learning from demonstration tasks. Designing vector fields that fit complex trajectories while ensuring stability still remains a key challenge; although recent deep-learning-based methods have shown substantial progress in this direction, their tendency to overfit to demonstration trajectories often leads to undesirable behaviors, particularly as tasks deviate from demonstrations. At a fundamental level, the only reliable way to address this lack of generalization is to provide supervision in out-of-demonstration regions. Focusing on two types of general behaviors, mimicking and contracting, we propose a behavior-controllable stable dynamics model (BCSDM), a one-parameter family of SDS that allows users to adjust the system’s overall behavior depending on user intent. We show how to extend the BCSDM to accommodate demonstrations of multiple tasks, and also propose a deep operator vector field for memory-efficient encoding of multiple dynamical systems. Extensive experiments on tasks that involve mimicking or contracting behaviors demonstrate the advantages of BCSDMs over existing state-of-the-art SDS learning methods.
Due to their continuous electromechanical deformation, rate-dependent viscoelasticity, and complex mechanical vibration, dynamic modeling and high-speed tracking control of dielectric elastomer actuators (DEAs) remain elusive, significantly limiting their working bandwidth. In this work, we propose a physics-informed token prediction (PITP) that enables accurate modeling of DEA dynamics and high-speed feedforward tracking control. The PITP framework consists of two key components: a physics-informed encoder and a dynamic decoder. The physics-informed encoder is designed based on a simplified equivalent linear model and trained through the hierarchical optimization training method, which embeds the global dynamic characteristics into tokens, minimizing the need for extensive data and training. Then, the dynamic decoder is developed by using these tokens as state-dependent parameters, capable of describing complex dynamic responses through the autoregressive solution. Finally, by taking advantage of the model’s reversibility, a direct inverse compensator is established to linearize the input–output relationship. Experimental results of several DEAs with different configurations and payloads demonstrate that, based on our PITP framework, the complex nonlinear dynamic responses of all DEAs can be precisely described and eliminated within their natural frequency, validating its generality and versatility. By leveraging fast modeling ($< $30 min) and high-speed feedforward tracking control, our PITP framework may accelerate DEAs’ practical applications.
There exist numerous ways to achieve multitasking control in kinematically redundant robots to accomplish several goals simultaneously. In all approaches, regardless of the specific type of controller, one has to make a choice about the closed-loop inertia and consequently the dynamic task couplings. Here, we introduce a new control strategy that combines two fundamentally different properties that have not yet been brought together. First, we fully and dynamically decouple all individual subtasks, which cannot be achieved with classical passivity-based or hierarchical approaches. Second, we provide high robustness in practice, which is structurally not possible with any inverse dynamics approaches enforcing a decoupled but constant closed-loop inertia. Besides formal proofs of stability and passivity, we compare our approach with the other categories in various simulations and experiments. Since the proposed controller is grounded in the fundamental property of full natural task-space decoupling, this underlying strategy and its benefits can also be transferred to other design methods such as quadratic programming, model-predictive control, or learning-based approaches.
In safety-critical robot planning or control, manually specifying safety constraints or learning them from demonstrations can be challenging. In this article, we propose a certifiable alignment method for a robot to learn a safety constraint in its model predictive control policy with human online directional feedback. To the best of authors’ knowledge, it is the first method to learn safety constraints from human feedback. The proposed method is based on an empirical observation: human directional feedback, when available, tends to guide the robot toward safer regions. The method only requires the direction of human feedback to update the learning hypothesis space. It is certifiable, providing an upper bound on the total number of human feedback in the case of successful learning, or declaring the hypothesis misspecification, i.e., the true implicit safety constraint cannot be found within the specified hypothesis space. We evaluated the proposed method using numerical examples and user studies in two simulation games. In addition, we implemented and tested the proposed method on a real-world Franka robot arm performing mobile water-pouring tasks. The results demonstrate the efficacy and efficiency of our method, showing that it enables a robot to successfully learn safety constraints with a small handful (tens) of human directional corrections.
Sim-to-real transfer in robotic manipulation tasks has emerged as a crucial area of research, addressing the challenge of bridging the gap between simulated environments and real-world applications. This article presents a brief review of current methodologies and introduces novel approaches to enhance the efficacy of sim-to-real transfer. We propose a new framework, adaptive domain randomization (DR) with progressive neural architecture search, which combines adaptive DR techniques with neural architecture search to optimize both the simulation parameters and the neural network architecture for improved transfer. We evaluate our approach against multiple state-of-the-art baselines, including standard DR, adaptive DR, model-agnostic meta-learning, randomized-to-canonical adaptation networks, and ensemble policy learning. Our experiments on a diverse set of manipulation tasks demonstrate significant improvements in the transfer performance, with up to 35% reduction in reality gap compared to state-of-the-art methods. Furthermore, we introduce a novel metric, the transfer efficiency index, to quantify the effectiveness of sim-to-real transfer across different tasks and methodologies.
This article describes a solution for pick-and-place tasks that do not require high precision throughout the execution, using a robotized gantry crane. Two common issues that occur when using a crane are addressed, and solutions are provided. First, the lack of rotational controllability is overcome by designing a gripper that passively aligns itself with the handle of the payload using a simple, robust control strategy. Second, the active workspace is expanded by using controlled, dynamic motions, based on a variable-length pendulum model. Thus, the workspace is no longer limited to positions directly accessible from above, as is the case with quasi-static control methods. The robustness and effectiveness of the proposed solution was validated by a grasping, and a shelf insertion experiment. The robot was able to grasp the payload during all 48 trials. Failures were detected and recovery strategies were implemented. The handle could also be reliably ungrasped. During the shelf insertion, imperfect trajectory tracking caused a significant error during the unobservable part of the trajectory. Nevertheless, the actual placement position was always close to the desired position.
Robotic dexterous manipulation is a challenging problem due to high degrees of freedom (DoFs) and complex contacts of multifingered robotic hands. Many existing deep reinforcement learning-based methods aim at improving sample efficiency in high-dimensional output action spaces. However, existing works often overlook the role of representations in achieving generalization of a manipulation policy in the complex input space during the hand-object interaction. In this article, we propose DexRep, a novel hand-object interaction representation to capture object surface features and spatial relations between hands and objects for dexterous manipulation skill learning. Based on DexRep, policies are learned for three dexterous manipulation tasks, i.e., grasping, in-hand reorientation, bimanual handover, and extensive experiments are conducted to verify the effectiveness. In simulation, for grasping, the policy learned with 40 objects achieves a success rate of 87.9% on more than 5000 unseen objects of diverse categories, significantly surpassing existing work trained with thousands of objects; for the in-hand reorientation and handover tasks, the policies also boost the success rates and other metrics of existing hand-object representations by 20% to 40%. The grasp policies with DexRep are deployed to the real world under multicamera and single-camera setups and demonstrate a small sim-to-real gap.