GM’s Cruise robotaxi unit, hot on the heels of receiving its permit to charge for rides all day in San Francisco, did not have a good weekend. Two major stall-out incidents took place, related to the “Outside Lands” music festival which ran all weekend on the west side of town.
The biggest problem did not take place at the festival, but rather on the other side of town, where as many as 10 Cruise vehicles stopped dead and blocked traffic in the busy North Beach area.
A city council member got into the fray and was highly critical. Cruise’s only response was to state that “”A large festival posed wireless bandwidth constraints causing delayed connectivity to our vehicles. We are actively investigating and working on solutions to prevent this from happening again. We apologize to those who were impacted.”
Cruise vehicles use cellular data networks to talk back to their HQ to receive commands and report their status. Most companies use at least 2 cellular radios to assure they can handle the loss of one, and they may use more. It appears Cruise lost all of them, not because of problems with their local cell towers, but because of network overload. Normally when Cruise cars get stuck, they ask for help from HQ, and operators there give the vehicles advice or escape routes. Sometimes that fails or they can’t resolve a problem, and they send a human driver to rescue the vehicle. According to data released by Cruise last week, that happens about an average of once/day though they claim it has been getting better.
In this case, the vehicles did not need rescue, and after about 15 minutes they were able to direct the vehicles on their way and they stopped blocking traffic on the small streets. Those blocked were of course frustrated. There were also rumors that Cruise’s operations center might have also been overloaded, as other things were happening at the music festival with other stalled vehicles needing help.
It should be noted that while these vehicles are being developed, it is not at all unexpected that they will block traffic from time to time. This is why they are out there, to learn what goes wrong and fix it. That doesn’t make it less annoying to those who get delayed, and Cruise definitely should work to fix these problems.
These vehicles don’t need network connections to drive, but they do need them to resolve unusual problems, and to get new destinations. As such, they need to make special efforts to make sure they don’t lose contact for extended periods, or that they are able to resolve problems on their own – even if it’s just finding somewhere to pull over — when the network is down. Right now, the vehicles are being extra conservative and are unwilling to take any risks resolving problems on their own, and this is disrupting traffic.
There are a number of steps Cruise could take, and perhaps should have taken, to assure these sorts of problems can’t happen. Cruise declined to comment on whether they do any of these.
More and better data connections
Having 2 or even 3 cell radios and SIMs is usually effective, but not always. The new 5G cell towers have facilities to let the cell companies guarantee higher levels of service to customers that pay more. It’s unclear why this wasn’t the case here, but robocars should definitely get service level agreements with their mobile carriers so they can demand bandwidth (at the expense of people who paid less) when it’s needed to avoid trouble on the road. One extreme step would be for the company to be come a virtual mobile network operator (MVNO) which acts like a cell company but uses the hardware of the big companies, installing little or none of its own equipment, but having contracts to assure service. Alphabet already operates an MVNO called Google Fi which is the company I use on my own phone, and Waymo may use its contracts.
There are also other data networks in many cities. Usually their hardware is not too expensive, but you want to get contracts that let you make use of them as a backup, paying (a lot) when you need to invoke the backup, but little when you don’t. These can include wifi networks and other bands.
There’s an irony that back over 15 years ago, Google tried to install a municipal wifi network in San Francisco, backed by now-governor Newsom. City council politics scuttled that project. It would have been a good potential backup channel, in particular obviously for Waymo.
SpaceX’s Starlink dishes can now provide a connection when they can see the sky. There are obviously locations in the city where the sky is blocked, but most can see some of it. The terminals are costly and power hungry, but if you can get a contract to use them as backup, they could do the job, though their upstream bandwidth is more limited and the cars want a lot of upstream. This is one area where Tesla, if it follows through on its announced plan to do robotaxi service, could get an edge. The CEO of Tesla may be able to wrangle a special favor from the CEO of SpaceX.
Some of these plans might make sense only during the early phases of deployment, which is what we’re in now, and when the extra cost is not an issue. Later, the vehicles should be able to handle almost all outages on their own.
Use broadcast channels
It would make great sense to allocate a spare digital TV channel for use by cars. These channels are 19 megabits, and they penetrate even into buildings. The electronics to receive them are found in any TV and they are quite cheap. Such a channel would be one way, but there can be a lot of value in at least being able to send commands and data to vehicles when you know something they don’t that can help them resolve their problem. You don’t need your own channel — all the existing channels allow you to add side-channels if you just want a few hundred kilobits. There are also many other broadcast media which could sell data capacity which would reach cars even far out in the country. (Rural areas have cellular dead zones of course, but the robotaxis don’t go there at this time.)
Sirius/XM satellite radio is another possible provider — the Chevy Bolt comes standard with this already — and works over most of the planet.
Make a mesh
Robocars have an extraordinary ability almost no every technology has — they move where you want them to. Cruise’s cars could form mesh networks and relay signals to their siblings. As long as one car has a connection — be in cellular, or wifi or anything else — it could get data to others, and they can daisy chain.
It would be possible that a car could even deliberately drive to somewhere near to a car that has lost contact, in order to relay data to it. You could even have more than one car do that, bringing a connection almost anywhere. A combination of wifi points around the city and relay cars could do the job. In the future, when it’s allowed, drones could also do this — flying in seconds to be above a car that needs a connection and relaying the signal to a tower. (This is possible today if you have a pilot who can get visual line of sight on the drone, at least in certain types of airspace.)
For security reasons, you want this mesh to only involve your sibling cars, not random strangers, but that’s OK if you have a good sized fleet. The other cars would not be doing anything complex like relaying sensor data or helping decide commands — but they would let HQ send commands after humans figure things out.
Get better at it
While these tricks can do the job, they can’t work everywhere, including certain tunnels and rural areas. Vehicles need to get better and resolving problems even with no help from HQ. They don’t need to be perfect, but 10 vehicles should never get stuck. Each should be able to at least stop blocking the road if it’s possible, even by moving very slowly. The one at the outside of any mess would go first. Getting multiple vehicles stuck is just something that should be on the “can’t happen” list, but that isn’t true yet for Cruise.
It is not confirmed that Cruise ran short of staff because of the heavy problems that evening. If so, they might want to draw a lesson from the companies that turn executives into operations center workers if the operations center gets overloaded. In theory, if there is a problem of this sort, then anybody in the company, up to and including the CEO, who is able to do the remote assistance job, should be on call and get alerted to go to a terminal and do what they can. Ideally it never happens — until it does.
Update: Before going to wide deployment, vendors need to be sure they can handle complete cell network failures, such as happened in the 1989 earthquake and during Katrina. While these events are very rare, vehicles should handle them well before they go into large deployment. (There can be more debate about this during small-fleet test projects.) In fact, ideally the vehicles should be ready to be a boon, and certainly not a problem, for public needs like evacuation. Unlike human driven vehicles, robotaxis can take people out of a danger zone and then go back in to get more people without risk to a driver. While they obviously should not block streets in these situations, ideally they can do many positive things, but they must do them even during communications failures.
More getting stuck
At the same time, over near the concert, vehicles were also getting stuck. This video shows a stalled Cruise in the traffic trying to get people away from the Festival. A Waymo was among the cars that drove around the stalled Cruise, to add insult to injury. This concert is notorious for causing traffic problems. Uber and Lyft rides had huge surge surcharges at this time and buses were overloaded as well.
I’m sure Cruise doesn’t need to be told this, but they need to improve their actions in these stall-outs. They are trying to be cautious, but as a result they are doing things that are not at all like human problems. The general goal is that robotaxis will drive better than humans, first of all on a safety basis, but also in other ways. Human driven vehicles just don’t often get stuck like this for long periods. If they have mechanical failures, usually they are pushed off the road by willing passers-by within a minute or two, and that’s very difficult to imagine happening with a robotaxi. Cruise recently implemented a system so that emergency workers can get into the cars and drive them away — Waymo has had this for some time — but they are not ready to trust random strangers to guide or drive them.
The public should be more tolerant of these mistakes, but the public is not likely to feel that it should. In the big picture, a few mistakes with early small fleets speed up the arrival of big fleets of vehicles that make fewer mistakes than humans, for a big net win — a few mistakes with a small fleet in exchange for reducing problems at large scale. We only can see what’s right in front of us, rather than what is to come. At least on social media.
Read the full article here
