HomestoryDigital Privacy › Otto’s Self-Driving-Truck Tests on California Roads May Run Afoul of State Regulations

News Story

Otto’s Self-Driving-Truck Tests on California Roads May Run Afoul of State Regulations

Car and Driver

In December, Uber began testing autonomous vehicles on California public roads without obtaining the necessary permit required by state regulations. A public standoff ensued. The ride-hailing service argued that its autonomous technologies fell outside the scope of the state’s regulations and proceeded with its project. California’s Department of Motor Vehicles disagreed and responded by revoking the registrations of the 16 vehicles involved in the project. Uber executives finally capitulated and relocated testing to Arizona. That move ended, at least for a while, the battle between California regulators and the high-profile tech company. But it didn’t end the company’s testing of autonomous technology on the state’s public roads. Otto, a company based in San Francisco developing autonomous technology for trucks, has tested on California roadways since its founding in January 2016. Uber acquired the company for $680 million in August 2016, and testing continues today.

A spokesperson for the subsidiary said Otto does not need a permit because it only tests driver-assist technologies that are similar to vehicle safety systems sold by auto manufacturers today, features that include collision-avoidance systems, adaptive cruise control, and lane-keeping assist. Those systems do not fall within the purview of California’s autonomous-vehicle regulations, which were established in 2014. But an Otto software engineer tells a different story. In a company document obtained by Car and Driver via a public records request, he reveals the company’s procedures for testing a more extensive self-driving system on California public roads. This system experiences anomalies, commonly known as disengagements, that Otto says carry varying levels of safety threats. The document describes Otto’s testing as concerning not simpler driver-assist technologies, but rather a “self-driving system” that operates in “self-driving mode” on California public roads. Without a permit, this testing may run afoul of the state’s regulations, setting up a potential second confrontation between Otto’s parent company and DMV regulators. California’s DMV is examining whether this testing violates state regulations. “Otto has assured the department that their trucks are not capable of operating in autonomous mode in California,” Jessica Gonzalez, an agency spokesperson, told Car and Driver. “We will look into it further.

Scott Ryvola, an Otto software quality engineer responsible for the safe operation of the self-driving system in real-world environments, wrote in the document that the company’s trucks are driving the highways surrounding San Francisco on a daily basis. In these tests, he described, the company analyzes disengagements of its autonomous technology, how advanced machine learning is used to train a “self-driving vehicle,” and how two human test drivers monitor the performance of a “self-driving system.” Ryvola’s document itself does not make reference to the National Highway Traffic Safety Administration’s (NHTSA) widely accepted levels of autonomy—0 is assigned to vehicles with no automated capability whatsoever, while 5 is for full-time autonomous vehicles—when describing the technologies Otto is testing on California’s public roads. The company maintains it tests different levels of semi-autonomous and autonomous technologies in different states in accordance with state laws and/or regulations. John Simpson, privacy director at Consumer Watchdog, a nonprofit that closely follows autonomous testing in California and advocates for more stringent oversight, said Ryvola’s document shows that Otto is violating California’s autonomous-driving regulations. “It seems obvious Otto is following the renegade and illegal practices of its parent company in testing self-driving technologies in California without a permit,” he said. “This is the same thing as driving without a license, and the enforcement folks should be pursuing this with vigor. I think someone should be in jail.”

Tracking Disengagements of a “Self-Driving System”

California requires all companies that hold autonomous-testing permits in the state to file annual reports that contain, among other things, the number of times their autonomous technology disengages during public-road testing, either because of a software flaw or because a safety driver intervened with a vehicle’s road actions. Otto holds no permit, nor is it divulging any numbers to regulators. But its engineers are tracking that metric. A map included in Ryvola’s document, entitled “Otto–Testing Operations,” shows dozens of locations along Bay Area highways where Otto’s self-driving system experienced disengagements over an undated two-week time frame. While most of the disengagements, according to a color-coordination scale on the map, initially are assigned a status of “problem” by the company’s road testers, a handful, perhaps half a dozen, are designated as presumably more serious “software kickout” or “team priority.”

Once all of those logs are analyzed, Otto employees then assign the disengagements a severity level among four internal categories: • Comfort, when an operator felt uncomfortable with a decision the “self-driving system” made, but it posed a “near-zero safety threat” to the truck or other motorists. • Public Perception, a case in which the behavior of the truck would have looked “odd” from another motorist’s perspective but, again, had “near-zero” safety implications. • Major, an incident that “could have had safety implications” had a human driver not intervened. • Critical, an incident in which the truck’s “self-driving actions” put itself or other motorists “in actual danger” and required a human driver to take full control. The document, which Otto submitted to the Colorado Department of Transportation prior to starting testing in that state, does not specify how many disengagements of each type its trucks have experienced.

Disengagements may have played a role in Uber’s earlier decision not to seek a permit for testing its autonomous Volvo XC90 SUVs in California. By avoiding a permit, the company would not be beholden to its requirements and could then, said many industry leaders, keep its self-driving software’s disengagements confidential. This is important, as disengagements are used as a general barometer to gauge the safety and progress of self-driving programs in development by various companies. That’s a notion firmly rejected by Anthony Levandowski, an Otto co-founder who now serves as the head of Uber’s autonomous-driving unit, a position in which he oversees both Uber and Otto autonomous operations. At the time, he said Uber did not apply for a permit out of principle, because he believed the regulations did not apply to the vehicles. With regard to the Otto trucks testing in California, he told Car and Driver in a statement, “We deliberately use only driver-assist tech in California. We keep the CA DMV abreast of our activities and have had discussions with them regarding our operations.” This time around, there may be a more pragmatic reason why Levandowski did not apply for a testing permit for Otto trucks: They’re likely ineligible to receive one. California’s regulations prohibit the testing of autonomous vehicles with a gross weight of 10,001 pounds or more.

Otto Is Pursuing a “True Level 4 System”

Although they are different products with different target customers, Levandowski described the company’s position on why it does not need a testing permit the same way he described the company’s position on the XC90s. Both programs, he reiterated, are testing advanced driver-assist features not subject to regulations. “This technology is the essential foundation upon which autonomous technologies are built,” he told C/D, “but is not autonomous in itself.” The line between advanced driver features and autonomous systems might be confusing for motorists, but, as mentioned, NHTSA has provided terms and definitions for the industry to follow. The agency’s classifications for levels of autonomy range from Level 0, in which no automation occurs and humans do all the driving, to Level 5, in which self-driving systems do all driving in all conditions. Driver-assist features such as the ones Otto says it is testing fall within Level 2 and Level 3, in which the car can conduct some driving tasks but human drivers must monitor vehicle performance and be available to retake control. By sharp contrast, NHTSA defines Level 4 systems as ones that require no human driver at all. “An automated system can conduct the driving task and monitor the driving environment, and the human need not take back control,” the agency’s Federal Automated Vehicles Policy states. Level 4 systems can operate in defined environments, such as highways, but not all environments. As Otto prepared to launch autonomous testing in Colorado in the fall of 2016, Ognen Stojanovski, who heads the company’s government-relations efforts, made it clear which type of technology the company was pursuing. In a September letter to officials with the Colorado Department of Transportation, he wrote, “We are building true Level 4 systems for highways.” Levandowski told us it’s no secret that Otto aims to build this Level 4 system, but he said, “We’re not there yet.”

How Passengers Treat Self-Driving Mode

In preparation for that self-driving Colorado test, which culminated in a highly orchestrated delivery of Budweiser in late October, Otto used California’s roads as its proving ground. Levandowski said the trucks cannot drive themselves in California in the same manner in which they did in Colorado and that software features created for the Interstate 25 run were unique for that particular road. But Ryvola wrote that Otto conducted “stress testing” of its system in California in advance of the Colorado testing, which was necessary in part because the company had installed a revamped computer, new lidar sensors, and a new wiring harness. His document confirms that it tested these both on California public roads and at a private testing facility in the state. In a typical California public-road testing scenario, Ryvola wrote, there are two “passengers” inside the cabin. One is a commercial driver responsible for the truck’s overall operation and the “active physical monitoring” of the self-driving system. The second person holds an equally vital safety role, “monitoring the output of the self-driving system and alerting the driver of any anomalies, as well as instructing when the driver should ‘disengage’ from self-driving mode.” Without the same oversight of state regulators that companies that hold permits adhere to, Otto deploys its trucks at all hours of the day in a variety of traffic and weather conditions, including morning rush-hour traffic and night driving. The goal of this testing is to capture every driving scenario possible and then utilize advanced machine-learning techniques to learn from these scenarios and improve development of what Ryvola, again, terms a self-driving vehicle. Improvement “is achieved through advanced machine-learning models that our perception system employs,” he wrote. “Although not every component of a self-driving vehicle can employ a machine-learning model, we use problem scenarios encountered on the road as a baseline of what our system must be able to handle.”

Reminiscent of Previous Confrontation

This would not be Otto’s first clash with a state DMV. In May 2016, a similar argument over testing licenses occurred in Nevada. In order to film a video for the company’s promotional materials, Otto began preparations for an autonomous test drive near Reno. But a top official with Nevada’s Department of Motor Vehicles warned Levandowski that such a test would violate the state’s autonomous-vehicle regulations because Otto had not yet been issued a testing license, nor specialty red license plates to mark the truck as an automated vehicle. Otto ignored the warning and proceeded with the test. A detailed report on Backchannel published in November 2016 revealed that Jude Hurin, an administrator with the Nevada DMV, called the drive “illegal” in emails that evaluated the aftermath. He considered shuttering the state’s entire autonomous-vehicle testing program to prevent further rogue testing.

Despite that strained relationship, Otto was later issued the first license to run an Autonomous Technology Certification Facility (ATCF) in Nevada. Self-driving vehicles cannot be sold or registered in the state without a certificate from such a facility. The certificate shows it complies with state laws and safety requirements. By operating its own certification facility, Otto may be empowered to give its own vehicles a stamp of approval without further regulatory oversight. These issues arise at a time when state transportation and motor-vehicle officials are grappling with questions of how to best regulate the testing of fledgling autonomous technologies. In the future, most agree that self-driving vehicles hold the promise of drastically reducing the number of traffic deaths and injuries on public roads. But in the present, these agencies are trying to foster innovation while not risking the safety of today’s human motorists.

Nevada and California were the first two states to pass legislation regarding the testing of self-driving vehicles. What they’ve learned, perhaps too late, is that they didn’t include significant penalties for companies that don’t follow the rules. One California lawmaker is now proposing such penalties in the wake of Uber’s testing, but for now, there’s no deterrent for companies that want to push the boundaries of regulation. “Silicon Valley follows the rule that it’s better to seek forgiveness than to ask permission, but with Uber, it has gotten tiresome,” Simpson said. “This time, they should have the book thrown at them.”