lerobot/Cosmos3-Nano-Policy-LIBERO_10

Cosmos3-Nano fine-tuned on the LIBERO-10 as a LeRobot policy. From the task instruction and the third person and wrist camera images, the policy predicts a chunk of 16 actions together with the future video of the scene. This repository holds the EMA weights at optimizer update 2000.

Results on LIBERO

Closed-loop success rates: 50 episodes for each of the 10 tasks of a suite, 500 episodes per suite, with the evaluation setup below.

Evaluation

Command

The results were measured with MuJoCo 2.3.7. The LeRobot lock file installs MuJoCo 3.8.1, so install the older version first:

uv pip install mujoco==2.3.7

lerobot-eval \
  --policy.path=lerobot/Cosmos3-Nano-Policy-LIBERO_10 \
  --policy.dtype=bfloat16 \
  --tf32=false \
  --seed=0 \
  --env.type=libero \
  --env.task=libero_10 \
  --env.obs_type=pixels \
  --env.observation_height=256 \
  --env.observation_width=256 \
  --env.episode_length=510 \
  --eval.n_episodes=50 \
  --eval.batch_size=4 \
  --eval.use_async_envs=true \
  --output_dir=outputs/eval/cosmos3-nano-policy-libero_10/libero_10

Evaluation setup

• MuJoCo 3.x changes the outcome of contact rich LIBERO tasks, so results measured with it are not comparable.
• The policy budgets are those of the native Cosmos3 LIBERO evaluation. The LeRobot defaults are longer.
• The video canvas step of the preprocessor ran on CPU, and cuDNN benchmarking was disabled. lerobot-eval runs the preprocessor on the policy device and enables cuDNN benchmarking, which changes the inputs and the sampling by small numerical amounts.

Evaluation hardware and software

Training

Training setup

Keep the global batch at 2048 samples on other GPUs: batch per GPU x number of GPUs x gradient accumulation steps = 2048. LeRobot counts --steps, --save_freq, --policy.scheduler_warmup_steps and --policy.scheduler_decay_steps in micro batches, so multiply the optimizer update counts by the gradient accumulation steps (2 here). Every rank builds the float32 policy on the CPU before FSDP shards it (about 66 GB for Cosmos3-Nano), so a node needs that much host memory for each of its ranks that load at the same time.

Training command

# Run on each of the 4 nodes with NODE_RANK=0 to 3.
torchrun \
  --nnodes=4 --nproc_per_node=8 --node_rank=${NODE_RANK} \
  --master_addr=${MASTER_ADDR} --master_port=${MASTER_PORT} \
  $(which lerobot-train) \
  --policy.type=cosmos3_nano \
  --policy.pretrained_path=lerobot/Cosmos3-Nano \
  --policy.load_action_modules=false \
  --policy.gradient_checkpointing=true \
  --policy.push_to_hub=false \
  --dataset.repo_id=lerobot/LIBERO_10-Cosmos3 \
  --dataset.video_backend=pyav \
  --parallelism.dp_replicate=4 \
  --parallelism.dp_shard=8 \
  --accelerator.fsdp.mixed_precision_policy.param_dtype=bfloat16 \
  --accelerator.fsdp.mixed_precision_policy.reduce_dtype=bfloat16 \
  --accelerator.fsdp.mixed_precision_policy.cast_forward_inputs=false \
  --accelerator.gradient_accumulation.steps=2 \
  --accelerator.gradient_accumulation.sync_each_batch=true \
  --tf32=false \
  --ema.type=power \
  --ema.sigma_rel=0.1 \
  --batch_size=32 \
  --steps=4000 \
  --save_freq=500 \
  --policy.scheduler_warmup_steps=1000 \
  --policy.scheduler_decay_steps=32000 \
  --num_workers=8 \
  --prefetch_factor=1 \
  --seed=42 \
  --output_dir=outputs/train/cosmos3_nano_policy_libero_10 \
  --job_name=cosmos3_nano_policy_libero_10

The weights of this repository are outputs/train/cosmos3_nano_policy_libero_10/checkpoints/004000/pretrained_model_ema of this run.

Loading and fine-tuning

--policy.path=lerobot/Cosmos3-Nano-Policy-LIBERO_10 loads the configuration, the weights and the processors. config.json sets dtype: float32 on purpose: fine-tuning from this checkpoint keeps float32 master weights. For inference and evaluation pass --policy.dtype=bfloat16.

License and provenance

The NVIDIA model materials use OpenMDW 1.1. Retain the license and applicable copyright and origin notices when redistributing them. NOTICE records the original model and the attribution of this distribution. Separately installed runtime software retains its own licenses.

Consult the original NVIDIA model card and its safety, bias, privacy, and explainability notes for upstream limitations.