What We Can Learn About Autonomous Progress From California Disengagement Reports
By Pete Bigelow, CAR AND DRIVER
February 5, 2018
Let’s start with the caveats. The annual reports on autonomous testing in California required by the state’s Department of Motor Vehicles are far from a perfect measure of any company’s self-driving competence.
These reports, including the latest batch, released last week, provide some fresh details on the circumstances surrounding when and why a self-driving system is disengaged during testing. What they don’t offer is much context.
Using only these reports, it can be problematic to make comparisons of autonomous progress, because one company’s low number of disengagements may occur during testing on empty highways, while another company’s high number may have occurred during testing in busy urban areas.
Comparisons are problematic because some companies place more value on testing in real-world scenarios while others put more emphasis on simulation, and sometimes engineers might be purposely disengaging to validate their systems. Some companies might concentrate their autonomous development efforts in states other than California, the only one that mandates disengagement information from these companies that are bent on altering the future of vehicular travel.
Further, interpretation of the reports can be thorny because the definition of what constitutes a reportable disengagement has some degree of latitude, and some companies may choose to report certain events while others opt not to disclose nearly identical ones. California regulators and consumers who scrutinize these reports are essentially at the mercy of what companies choose to report.
Laundry list of disclaimers now aside, the California DMV’s latest disengagement reports cover the 19 companies required to file them for the testing period from December 1, 2016, through November 30, 2017. And there are noteworthy insights to be gleaned from the reports, which detail both computer glitches that cause systems to disengage on their own and interventions that occur when human safety drivers wrest control from their robot masters.
Real-World Miles Take a Dip
Among the broader takeaways: California’s reputation as the epicenter for autonomous testing may be fading.
Fewer autonomous miles were logged on the state’s public roads during the most recent reporting period. In the previous time frame, nine companies traveled 656,503 miles, while in the more recent one, 12 companies collectively accrued 507,126 miles. That decline is largely caused by a 55 percent decrease in Waymo’s public-road testing in the state, which has occurred as the company ramped up its testing in Phoenix, Arizona, and elsewhere. Nationwide, Waymo says it accumulated nearly two million real-world autonomous miles in 2017, more than its seven previous years combined.
“It’s only because of California’s rules that the public can find out what’s happening when companies use public roads as their private laboratories.”
– John Simpson, Consumer Watchdog
Seven companies permitted to test in the state didn’t do so during the reporting period, and they included Tesla Motors, Ford, BMW, Honda, Volkswagen, Nio, and Wheego. Still, those numbers are an imperfect barometer of the autonomous miles traveled in California. Fifty companies currently hold testing permits, a number that’s steadily rising, but only 19 were required this year to report their disengagement figures.
Of the dozen companies that logged those 507,126 miles, they collectively reported 2304 disengagements, which equates to one every 220 miles. Although there’s much enthusiasm for autonomous-driving technology, that’s not nearly safe enough for actual deployment anytime soon, says John Simpson, privacy and technology director at Consumer Watchdog, a California nonprofit that keeps tabs on automated-vehicle developments.
“Despite the self-serving hype of the manufacturers, robot technology simply isn’t ready for our roads without hands-on, behind-the-wheel engagement and supervision by a human driver,” he said.
Your Mileage May Vary
Waymo and General Motors’ Cruise Automation subsidiary, widely considered two of the frontrunners in the race to launch automated vehicles, accounted for more than 95 percent of the total California testing miles.
Though testing complexity is not measured, Waymo has maintained a lead apparent in three years’ worth of data available since California started collecting these reports. The company reported 63 disengagements over 352,545 miles of testing, a rate of 1 every 5596 miles, or, examined another way, 0.18 disengagements per 1000 miles. That shows continued improvement from the company’s 0.2 disengagements per every 1000 miles reported in last year’s disengagement reports and the 0.8 mark established in 2015.
“Disengagements provide important feedback that allows us to continually improve our software, introduce new driving features and upgrade the sensors and hardware on our vehicles,” a Waymo spokesperson wrote. “During public-road testing, our objective is to gather as much data as possible to enable us to improve our self-driving system while operating safely.”
Meanwhile, Cruise Automation reported 105 disengagements over 131,676 miles, a rate of one disengagement for every 1254 miles logged, or 0.79 disengagement for every 1000 miles traveled. That’s about the same as the 0.8 rate per 1000 miles that Waymo logged in 2015.
But that doesn’t necessarily reflect Cruise’s progress. The company’s testing ambitious grew throughout 2017 as it added autonomous Chevrolet Bolt EVs to a fleet that now numbers 94 in California. In the first three months covered by the report, Cruise accumulated 11,839 miles and recorded 30 disengagements, a rate of one every 395 miles. In the final three months covered, it logged 62,689 miles and reported 12 disengagements, one every 5224 miles.
An analysis issued earlier this month by Navigant Research ranked the leaders in bringing autonomous-driving systems to the market, and placed General Motors and Cruise in first place, based largely on the all-electric Bolt EVs and the company’s ability to quickly scale. GM narrowly finished ahead of Waymo. The analysis noted that Cruise’s “automation system was operating well, though the overall actuation control lacked refinement. However, this is an area that is far easier to fix and falls well within the scope of GM engineers’ knowledge.”
Crash Records Provide Further Context
If the disengagement reports show Waymo once again setting a standard and Cruise adding vehicles and lowering its disengagement rate, other California documents paint a more disconcerting picture. Manufacturers are required to file reports of any crashes involving their autonomous vehicles in the state, and although Cruise’s 131,676 miles represented 26 percent of the total autonomous miles reported by companies in California during the testing period, Cruise vehicles were involved in 21 of the 27 statewide autonomous-vehicle crashes reported during the same time frame. In both September and October, Cruise reported more crashes than disengagements.
Cruise, which intends to launch self-driving service in 2019, attributes this at least in part to testing almost exclusively within San Francisco city limits, a complex traffic environment. The company’s report to the California DMV states that more than 125,000 of its 131,676 miles came within the city, and a spokesperson said: “Cruise’s rapid rate of improvement, in the most complex testing environments, is why we’re confident in our ability to safely deploy self-driving cars in 2019. For example, in November 2017 alone, our reported disengagement rate is equivalent to driving from San Francisco to Detroit and back, in dense traffic the entire time.”
With 21 collisions, including one in December with a motorcyclist that has prompted a lawsuit, Cruise averaged one crash every 6270 miles. With four crashes over 352,545 miles driven, Waymo averaged one crash every 88,136 miles.
Zoox and Uber each reported one crash during the testing period.
Disengagements Can Be a Misleading Measure
The volume of miles driven and crash reports may provide some general guidance on where Waymo and Cruise stand. It’s harder to distill insight on the remaining companies testing in California.
On the surface, it might appear that Bosch and Mercedes-Benz are lagging. Bosch reported 598 disengagements over 1454 miles tested and Mercedes notched 842 over 1088 miles. By contrast, Bosch and Daimler, Mercedes-Benz’s parent company, ranks third on the Navigant autonomous leader board.
Both companies are testing autonomous technology around the world. In Bosch’s case, a spokesperson said the company’s three cars in California are devoted exclusively toward testing systems that are not intended for any sort of near-term plans and that engineers are concentrating specifically on testing edge-case scenarios.
“The goal of these tests is not to complete an endurance run, but further development of software and hardware solutions that may be offered for future customer automated driving projects,” the company said in a statement.
Such background provides some of the nuance that can be hard to gather from the disengagement reports alone. But it’s important to remember that federal safety regulators have mandated no such reporting, instead meekly asking manufacturers to submit voluntary safety assessments that include no requirement for similar numbers. Nor have other states that host testing, among them Arizona, Nevada, and Michigan, demanded figures that contribute to the vetting of safety of autonomous vehicles.
“We have no idea what’s happening in the other states,” Simpson said. “It’s only because of California’s rules that the public can find out what’s happening when companies use public roads as their private laboratories. It’s only a partial picture, because other state regulators aren’t protecting their citizens.”
In other words, California’s disengagement numbers have their shortcomings for use as accurate measures of autonomous progress. But they’re one puzzle piece, and they’re perhaps the best one we’ve got.
