DrivingBench connects general-purpose vision-language agents to a Toyota Corolla through a narrow interface: camera observations and three tools controlling steering or speed. The car can continue to move while a model reasons, so a delayed response can change the physical state before the next command. The benchmark compares complete model and harness…
Inspect the ClaimsCase · DiggingBeagle record
DrivingBench exposes limits of direct model control in a cone course
Scope
An instrumented Toyota Corolla on a controlled low-speed cone course with a human safety driver and limited runs. This was not public-road driving, generalized autonomous-vehicle certification or a recorded road-traffic injury.
Assessment profile
Assessment method- Setting
- controlled live
- Exploitation
- not established
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- BY AI
Basis and provenance
A primary paper abstract and retained scoped Claims support the supervised real-car closed-course evaluation and narrow completion result. Full telemetry and safety-driver logs were not independently retrieved or reproduced. Impact remains ungraded because the available evidence does not establish realized harm or a defensible general severity for the assessed deployment.
Assessed Oct 16, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-DRIVINGBENCH-CONTROLLED-AUTHORITY-01 · CLM-DRIVINGBENCH-CONTROLLED-AUTHORITY-02 · CLM-DRIVINGBENCH-CONTROLLED-AUTHORITY-03 · DrivingBench: evaluating general-purpose models in real vehicle control
At a glance
The operational boundary is the vehicle controller, not the model response. Physical speed limits, emergency intervention and the observation-action timing should be evaluated separately from route-completion scores. A single successful run is evidence of feasibility within the test, not of a safe unattended deployment.
Read the implicationsThe trials do not establish independent repeatability across routes, weather, vehicle configurations or unseen obstacles. Model capability cannot be separated cleanly from differences in harness, prompt framing and tool latency in this comparison. No conclusion about real-road safety, injury prevalence or production vehicle autonomy follows from the…
Limits and uncertaintyFull account
DrivingBench connects general-purpose vision-language agents to a Toyota Corolla through a narrow interface: camera observations and three tools controlling steering or speed. The car can continue to move while a model reasons, so a delayed response can change the physical state before the next command. The benchmark compares complete model and harness combinations, not isolated neural-network weights.
Across the reported attempts, only GPT-6 Astra finished the course, on its second attempt. The other evaluated systems did not pass the halfway mark. Eight of the eleven plotted attempts ended before the first turn, and the runs were terminated at completion or by safety-driver intervention. That intervention matters: the evidence establishes limited closed-course performance under supervised conditions, not reliable unsupervised road authority. Differences in tool representations, refusal behavior, latency and retained conversation context are possible contributors that cannot be attributed to model architecture alone.
Timeline
- Sep 30, 2026disclosure
Public disclosure
Claims & evidence
CLM-DRIVINGBENCH-CONTROLLED-AUTHORITY-01DrivingBench tests four model and harness combinations using screenshots and a small control interface in a real Toyota Corolla on a controlled low-speed course. Only Astra completed the course, on its second attempt.supported
Basis: reported finding
- supportsDrivingBench: evaluating general-purpose models in real vehicle controlprimary disclosure
Abstract and experimental results
CLM-DRIVINGBENCH-CONTROLLED-AUTHORITY-02Eight of the eleven plotted attempts ended before section B, the first turn; attempts ended after completion or safety-driver intervention.supported
Basis: reported finding
- supportsDrivingBench: evaluating general-purpose models in real vehicle controlprimary disclosure
Section 4.1 and Figure 3, trajectory endpoints
CLM-DRIVINGBENCH-CONTROLLED-AUTHORITY-03The study compares GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol and Grok 4.6 using vendor-native coding-agent harnesses, allowing up to three attempts per system with retained conversational context.supported
Basis: reported finding
- supportsDrivingBench: evaluating general-purpose models in real vehicle controlprimary disclosure
Paper abstract, named model and harness cohort and attempt protocol
Implications
The operational boundary is the vehicle controller, not the model response. Physical speed limits, emergency intervention and the observation-action timing should be evaluated separately from route-completion scores. A single successful run is evidence of feasibility within the test, not of a safe unattended deployment.
Controls and mitigations
- Preserve an independently operable emergency stop or safety-driver override when evaluating vehicle agents.
- Record timestamps for observations, tool commands and actual vehicle motion to distinguish delayed model decisions from controller behavior.
- Repeat the same course with matched interface affordances and fresh sessions before comparing model capabilities or authorizing broader vehicle operation.
Unknowns and contradictions
- The trials do not establish independent repeatability across routes, weather, vehicle configurations or unseen obstacles.
- Model capability cannot be separated cleanly from differences in harness, prompt framing and tool latency in this comparison.
- No conclusion about real-road safety, injury prevalence or production vehicle autonomy follows from the closed-course completion rate.
Sources and citation
Material revision history
- Oct 16, 2026 · Canonical change recorded · new in release · revision 160