LoopQ: Fair, Closed-Loop Evaluation of Quantized Vision-Language-Action Models
LoopQ: Fair, Closed-Loop Evaluation of Quantized Vision-Language-Action Models
This paper introduces LoopQ, a benchmark for fairly comparing post-training quantization methods on vision-language-action (VLA) models. The study evaluates five quantization policies (RTN, AWQ, GPTQ, ActQuant, and QuantVLA) across four VLA models using more than 180,000 episodes of LIBERO and LIBERO-plus datasets. Testing was conducted on NVIDIA H100, L4, and Jetson AGX Thor hardware. Results indicate that while quantization reduces memory usage by 2-3x for low-bit models, VLA-specific PTQ methods generally underperform simpler generic baselines like RTN and AWQ at 4 bits. Speed improvements are inconsistent; most configurations on H100 run slower than their 16-bit counterparts, with gains only observed on L4 with int8 kernels. Profiling reveals that compute-bound backbones and memory-bound action experts (kept in 16-bit) limit performance gains.
The authors present LoopQ, a closed-loop benchmark evaluating post-training quantization methods (RTN, AWQ, GPTQ, ActQuant, QuantVLA) on four VLA models using over 180,000 episodes of LIBERO and LIBERO-plus data across NVIDIA H100, L4, and Jetson AGX Thor hardware.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
training episodes
>180,000 episodes
View original evidence
p benchmark that compares multiple quantization policies on the same checkpoint, layers and engine, varying only the method and the kernels. We evaluate RTN, AWQ, GPTQ, ActQuant and QuantVLA on four VLAs with more than 180,000 episodes of LIBERO and LIBERO-plus, compare against QVLA's reported results on the same layers, and benchmark whether the four public VLA-quantization releases can be reproduced. We benchmark t
Open source S3latency change factor
2.1
View original evidence
hanges latency by 2.1x. Profiling explains why: pi0.5's quantized backbone is compute-bound, while its memory-bound action expert, 63% of GPU time and 13% of the parameters, stays in 16 bits.
Open source S6speed slowdown factor
1.06–1.58
View original evidence
s the losses that do exist: configurations with no significant LIBERO loss lose 1.5-4.3 points under paired LIBERO-plus perturbations. Speed is a weaker story. At batch 1 on the H100, no quantized pi0.5 or OpenVLA-OFT configuration is faster than its 16-bit model (1.06-1.58x slower on library kernels); only the L4 with int8 kernels gains (4-14%), the Thor is slower on every 4-bit configuration, and the engine alone c
Open source S5
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
p benchmark that compares multiple quantization policies on the same checkpoint, layers and engine, varying only the method and the kernels. We evaluate RTN, AWQ, GPTQ, ActQuant and QuantVLA on four VLAs with more than 180,000 episodes of LIBERO and LIBERO-plus, compare against QVLA's reported results on the same layers, and benchmark whether the four public VLA-quantization releases can be reproduced. We benchmark t
Open source S3
he methods on an NVIDIA H100, an L4, and a Jetson AGX Thor. Our results show that while quantization reliably saves memory, 2-3x for real low-bit models, VLA-specific PTQ methods underperform compared to simpler generic baselines. At 4 bits, RTN, AWQ and GPTQ match or exceed ActQuant and QuantVLA, plain RTN matches the calibrated methods, and plain AWQ matches the success QVLA reports for itself. Standard LIBERO hide
Open source S4
hanges latency by 2.1x. Profiling explains why: pi0.5's quantized backbone is compute-bound, while its memory-bound action expert, 63% of GPU time and 13% of the parameters, stays in 16 bits.
Open source S6
s the losses that do exist: configurations with no significant LIBERO loss lose 1.5-4.3 points under paired LIBERO-plus perturbations. Speed is a weaker story. At batch 1 on the H100, no quantized pi0.5 or OpenVLA-OFT configuration is faster than its 16-bit model (1.06-1.58x slower on library kernels); only the L4 with int8 kernels gains (4-14%), the Thor is slower on every 4-bit configuration, and the engine alone c
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org