Abstract

Training a reinforcement learning agent by letting it explore freely is standard practice in simulation, but it is plainly unacceptable when the training environment is a patient. Surgical robotics has consequently turned toward offline reinforcement learning - learning a policy entirely from pre-collected data, with no online interaction during training. The approach leverages the large archives of kinematic teleoperation logs accumulated by systems like the da Vinci Surgical System, turning recorded expert demonstrations and suboptimal trial runs into supervisory signal without requiring any additional physical execution.

Why Offline RL and Not Imitation Learning

Behavioral cloning - the simplest form of imitation learning - copies expert actions directly and works well when the data distribution is dense and the expert is consistent. Surgical data is neither. Expert surgeons adapt mid-task based on tissue state, instrument drift, and patient-specific anatomy, producing a dataset where similar observations lead to different actions depending on context that the telemetry does not fully capture. Naive cloning learns an averaged, blurry policy that fails on the tail cases that surgeons handle intuitively.

Offline RL methods, including Conservative Q-Learning (CQL) developed at UC Berkeley and the Decision Transformer approach from Google Brain, extract a richer behavioral model by reasoning about which actions lead to good long-run outcomes across the full dataset. The Q-function provides counterfactual signal: even if the dataset contains no example of a particular tool path, the learned value function can assign it a conservative but informative score, guiding the extracted policy away from regions the data suggests are risky.

Dataset Curation and the Distribution Shift Problem

The central difficulty in offline RL is distributional shift. A policy that takes an action never seen in the training data will query the Q-function in a region where it was never supervised, potentially receiving wildly optimistic value estimates and taking catastrophically bad actions when finally deployed. Techniques such as CQL penalize out-of-distribution actions during training by adding a regularization term that lowers the estimated value of actions the dataset’s behavioral policy would not take.

For surgical applications, dataset curation matters as much as algorithm choice. Research groups at Johns Hopkins and at NVIDIA’s medical robotics division have argued for stratified dataset collection: mixing expert trajectories with deliberately perturbed recoveries, so the model sees not only the nominal task but also the corrective behavior that follows a deviation. This enriches the data distribution near the boundary of the expert manifold, exactly where distributional shift is most dangerous.

Evaluation Without a Live Environment

Validating offline-trained surgical policies before clinical trials requires proxy evaluation environments. Physics-based tissue simulators such as SOFA Framework and SurRoL provide deformable tissue models that are increasingly calibrated against cadaveric and phantom data. Off-policy evaluation (OPE) estimators - including doubly-robust importance sampling methods - offer a complementary route, estimating policy performance directly from the offline dataset without any simulation.

Regulatory acceptance of offline-trained systems remains a central open question. The FDA’s Software as a Medical Device framework requires pre-market evidence of safety and effectiveness, and the field does not yet have agreed-upon evaluation protocols that regulators and roboticists both consider sufficient. Working groups under ASTM International are actively drafting test method standards for autonomous surgical robot evaluation, with expected publication in 2026.