Illustration of the proposed Lifelong Robotic Skills Learning (LRSL) task. It requires robotic agents to continually learn new robotic skills, including manipulation and navigation, across different robotic bodies within the dynamic open world. Similar to humans, as agents acquire various skills through lifelong learning, their skill knowledge continually consolidates and accumulates, ultimately achieving a universal robotic agent.
Abstract
Recent large-scale vision-language models have significantly advanced robotic skill learning, promoting progress in both vision-and-language navigation and vision-language-action manipulation. However, existing robotic agents are typically trained under static multi-task settings, which limits their ability to continually acquire new skills in dynamic open-world environments without catastrophically forgetting previously learned ones. To address this limitation, we formulate a new problem, termed Lifelong Robotic Skills Learning (LRSL), where an embodied agent is required to continuously learn a sequence of navigation and manipulation skills across different robotic embodiments, while preserving prior knowledge and performing task-agnostic inference without access to task identities. To solve LRSL, we propose Uni-SkillEvolver, a unified lifelong robotic agent that explicitly decomposes robotic skill knowledge into skill-shared and skill-specific components. Specifically, we design an Orthogonal Decoder-Extension LoRA (ODE-LoRA) to model transferable shared knowledge with a common encoder subspace and progressively expand skill-specific knowledge with orthogonal decoder subspaces. We further design a Task-aware Interlayer Sparse Injection strategy to selectively inject skill-specific adaptations into the most appropriate layers, and a Task Semantic-aware Decoder Activation strategy to retrieve and activate relevant decoder experts for task-agnostic skill execution based on instruction semantics. Extensive experiments across both simulator environments and real-world robotic platforms demonstrate that Uni-SkillEvolver consistently outperforms state-of-the-art baselines.
Method
Illustration of the proposed Uni-SkillEvolver pipeline. It includes (a) an Orthogonal Decoder Extension LoRA (ODE-LoRA) to explicitly decompose the robotic skill knowledge into skill-shared and skill-specific components; (b) a Task-aware Interlayer Sparse Injection (TISI) strategy to dynamically allocate skill-specific intermediate-layer injections for different robotic skills, improving the representation of specific skill knowledge; (c) a Task Semantic-aware Decoder Activation (TSDA) strategy to aggregate the learned knowledge to achieve any id-agnostic skill inference.
Uni-SkillEvolver Algorithms
Overall, under the LRSL setting, we summarize the lifelong training and task-agnostic inference pipelines of Uni-SkillEvolver in Algorithm 1 and Algorithm 2, respectively.
Task Setting
Illustration of the established lifelong robotic skill learning benchmark. We establish a total of 27 robotic skills, including the first 23 robotic skill tasks for lifelong learning and the last 4 skill tasks for evaluating the generalization performance. The established benchmarks encompass a diverse range of robotic skills, including manipulation using the UR-5 robotic arm, manipulation using the Franka robotic arm, Vision-Language Navigation (VLN) in the Habitat simulator or using DeepRobotDog lite2, Object Localization Navigation (OLN), and Dialogue Understanding Navigation (DUN).
Robotic Platforms
Illustration of robotic navigation and manipulation platforms.
We deploy both real-world navigation and manipulation systems to evaluate the continual learning performance of Uni-SkillEvolver in physical environments. As shown in Fig.6 a), the robotic manipulation platform is constructed based on a UR-5 robotic arm. The system integrates a RealSense SR300 camera for scene perception, a control cabinet for low-level robotic actuation computing, and a GPU computation platform with an NVIDIA A6000 Ada GPU for high-level act reasoning. Given a user instruction, the Uni-SkillEvolver agent deployed on the computation platform interprets the task and generates corresponding manipulation actions conditioned on visual observations. The perception data are captured and transmitted to the agent, which outputs control commands that are executed by the UR-5 arm through standard communication interfaces (i.e., RS485 and TCP). This perception–reasoning–action loop enables the robot to perform diverse manipulation tasks in real-world settings. We follow the LeRobot dataset format for data collection, organizing synchronized visual observations, robot states, actions, timestamps, and task annotations into standardized episodes.
As shown in Fig.6 b), the navigation platform is built upon a DeepRobotDog Lite2 quadruped robot. The system is equipped with an onboard Hikvision DS-E112 camera for visual perception and a Wi-Fi-based wireless communication module for real-time data transmission. During deployment, the robot continually captures first-person visual observations from the environment, which are streamed to a remote computation server with a GPU computation platform with an NVIDIA A6000 Ada GPU. The server runs the Uni-SkillEvolver to process visual inputs together with language instructions and predict navigation actions. These actions are then transmitted back to the robot via the wireless channel, enabling closed-loop interaction between perception, reasoning, and control in real-world scenes. Following the previous methods, we collect 200-300 data samples for each skill task as the training set for lifelong learning. In addition, we collect 25 completely independent data samples as the test set for evaluation. Each sample is manually annotated with the corresponding skill instructions.
Experimental Results
Continual Robotic Skills Learning
UR-5 Manipulation
grab toys
Speed 1X
Task1
Pick up the yellow toy on the table.
Scene: 8WUmhLawc2A
Vision-Language Navigation
Speed 1X
Task2
Please walk straight down the hallway, turn left at the portrait to enter the bedroom, continue straight ahead, then turn right and stop in front of the TV.
UR-5 Manipulation
grab bottles
Speed 1X
Task3
Pick up the bottle with the white cap.
Franka Manipulation
LIBERO-Object
Speed 1X
Task4
Pick up the blue box and put it in the basket.
Scene: ur6pFq6Qu1A
Object Localization Navigation
Speed 1X
Task5
I'm feeling a little cold. Please take me to the air conditioner control panel so I can turn up the temperature.
Scene: real-world 1
Vision-Language Navigation
Speed 1X
Task6
Walk straight ahead to the yellow barrier, then turn left to the green barrier, turn right to the red barrier, turn left to the yellow railing, and turn left to stop in front of the red ball.
UR-5 Manipulation
put away toys
Speed 1X
Task7
Put the yellow rubber duck toy in the box.
Franka Manipulation
LIBERO-Spatial
Speed 1X
Task8
Pick up the black bowl between the plate and the small bowl and place it on the plate.
Scene: r47D5H71a5s
Dialogue Understanding Navigation
Speed 1X
Task9
A: Do you remember where my backpack is? B: I think I left it in the storage room at the far west end of the hallway. A: All right, please go get it for me.
Scene: real-world 2
Vision-Language Navigation
Speed 1X
Task10
Walk straight to the pink cylinder, turn right at the green barrier, then turn left, pass the blue barrier, turn right, and stop in front of the red barrier.
Scene: ac26ZMwG7aT
Object Localization Navigation
Speed 1X
Task11
Please take me to the table at the other end of the hallway.
UR-5 Manipulation
wipe the dirty
Speed 1X
Task12
Wipe the stains off the tabletop with a cloth.
Scene: real-world 3
Vision-Language Navigation
Speed 1X
Task13
Walk toward the orange barrier, then turn right and walk to the pink cylinder. Turn left and walk to the yellow barrier, then turn right to the yellow railing. Continue to the right, then turn left past the red barrier and walk to the pink cylinder, where you should stop.
UR-5 Manipulation
open drawers
Speed 1X
Task14
Close the bottom drawer.
Scene: 2n8kARJN3HM
Dialogue Understanding Navigation
Speed 1X
Task15
A: What’s the weather like outside today? B: It should be pretty nice. You can take a look out the front door in the living room. A: Sure, please take me there.
UR-5 Manipulation
stack blocks
Speed 1X
Task16
Please stack the small square blocks on top of the large square blocks.
Scene: mJXqzFtmKg4
Vision-Language Navigation
Speed 1X
Task17
Turn left and exit the room, then walk straight down the hallway until you reach the white door. Turn right there, enter the next room, and stop at the doorway.
Scene: E9uDoFAP3SH
Dialogue Understanding Navigation
Speed 1X
Task18
A: Do you know where the keys are? B: They should be on the round table in the living room. A: Okay, I'll go look for them right now.
Franka Manipulation
LIBERO-Long
Speed 1X
Task19
Put the alphabet soup and the tomato sauce in the basket.
Scene: B6ByNegPMKs
Vision-Language Navigation
Speed 1X
Task20
Turn around, walk straight down the hallway, continue straight past the intersection, and stop in front of the sink on the right.
Franka Manipulation
LIBERO-Goal
Speed 1X
Task21
Open the middle drawer of the cabinet.
Scene: real-world 4
Vision-Language Navigation
Speed 1X
Task22
Walk forward until you reach the purple cylinder, turn left until you reach the red sphere, then turn right and walk toward the small basketball until you stop.
Scene: sT4fr6TAbpF
Object Localization Navigation
Speed 1X
Task23
I'm a little tired. Please take me to the bedroom door on the north side.
Scene: S9hNv5qa7GM
Vision-Language Navigation
Speed 1X
Task24
Please leave the bedroom, turn left into the living room, go out through the other door, and continue straight ahead until you reach the sofa.
UR-5 Manipulation
press the button
Speed 1X
Task25
Please press the red button.
Scene: real-world 5
Vision-Language Navigation
Speed 1X
Task26
Walk toward the red barrier, then turn right toward the purple pillar, turn left toward the yellow barrier, turn right, and stop in front of the orange pillar.
Scene: 5LpN3gDmAk7
Object Localization Navigation
Speed 1X
Task27:
I'd like to change my clothes. Please take me to the closet next to the bedroom.
