Inside Tesla’s Autopilot labeling facilities

Business Insider spoke with 17 current and former employees on Tesla’s data entry team to understand what it’s like to power the company’s Autopilot and Full Self-Driving software. The team is split across three different Tesla facilities in Buffalo, New York, Palo Alto, California and Draper, Utah.

Projects in teams can last anywhere from months to days and can vary from workflows that require staff to tag short clips to track static images or satellite data overlays.

“It can get monotonous at times,” said one former worker. “You could spend eight hours a day for months just tagging lane lines and boundaries across thousands of videos.”

An ‘intimate look into someone’s life’

The clips can provide a unique window into the daily lives of Tesla executives. At one point, there was a project that required workers to tag data obtained from inside several owners’ garages through Tesla’s Sentry Mode feature, five workers said.

Another project, called “Selfie,” asked several annotators to label data captured by Tesla’s in-cabin cameras, according to two workers who saw employees working on the project. Four other workers said they were aware of the program. The Selfie program is designed to teach Tesla’s system how to identify when a driver was not paying attention to the road while Autopilot was in use, they said.

Tesla has said the vehicle’s in-cabin camera “shares short video clips of the in-cab camera with Tesla to help us develop future safety improvements and continuously improve the intelligence of features that rely on the in-cab camera,” according to the manuals its owner. Tesla owners must first opt-in to sharing their data in order for Tesla taggers to be able to access the video, the company says.

In other cases, workers found themselves tagging routes associated with YouTubers and even Elon Musk himself, BI previously reported.

“There’s something very strange about having this very intimate look into someone’s life,” said one current employee. “It feels weird to see someone’s daily movement, but it’s also an important part of correcting and refining the program.”

The videos were taken from all over the US, as well as some regions in Europe and South America, 15 workers said. Two workers recalled tagging videos they believed were taken from customers’ cars in Ukraine during the Russian invasion.

Business Insider reached out to Tesla, Musk and his legal team for comment, but did not receive a response before publication.

Workers can encounter data from any number of countries in a single workflow, which means they must constantly be aware of the different rules of the road for each region. At times, Tesla seemed to take a more relaxed stance toward those rules, seven former and current employees said. For example, some workers said they were told to ignore “No Turn on Red” or “No U-Turn” signs, meaning they would not train the system to obey those signs.

“It’s a driver-first mentality,” said one former employee. “I think the idea is that we want to train him to drive like a human, not a robot that just follows the rules.”

Sometimes the role requires workers to tag videos of accidents and near-misses. Seven workers recalled tagging videos involving Tesla accidents or those involving nearby vehicles. At one point a worker even shared a video among employees of an incident involving a young man on a bicycle being hit by a Tesla, four workers said. It was one of many videos and memes the workers shared, they said.

Last year, Reuters was the first to report on the bicycle clip and potential privacy issues on the note pages. Soon after the article was published, Tesla began restricting access to clips outside of worker-designated projects and added watermarks to some of the videos and images so they could easily track which employees shared images, nine workers told BI .

Tesla employee monitoring systems

Tesla has pretty strict employee monitoring systems at its Buffalo site. The location has a number of surveillance cameras in place that overlook the workspace, 11 workers told Business Insider.

Employees are also closely monitored using two different software systems.

One system, called HuMans, measures how long they should spend on each clip, four workers said. Annotators who consistently take more than their allotted time are likely to receive poor performance reviews or be placed under a performance improvement plan, or PIP, they said. The software was originally created to assist pilots in the US Air Force and also has the ability to track employees’ eye movements and take audio recordings, according to its website. But it’s unclear whether Tesla uses the software to track staff eye movement.

The company also uses a measure called “Flide Time” to track markers’ active time in the tagging software, 17 workers said. It can track keystrokes and how much time workers spend with the tagging software open, but it won’t track the time workers spend using other tools on their computer, they said. Depending on their level, workers can be expected to log anywhere from five to seven and a half hours of Flide Time, meaning they must be active in the software for at least that amount of time.

If workers are even five minutes short of their scheduled flight time, they could face disciplinary action, six workers said. If they miss Flide Time three times in the span of six months, they can be terminated, the workers added.

Some Tesla workers have tried to push the company’s staffing measures with little success.

In February 2023, some workers at Tesla’s Buffalo facility tried to unionize. Union organizers at the Buffalo facility told Bloomberg the company was tracking their keystrokes and said some workers were tired of being “treated like robots.”

That same month, Tesla laid off dozens of workers at the Buffalo plant. At the time, the National Labor Relations Board (NLRB) filed a complaint alleging that Tesla illegally fired several staff “in retaliation for union activity and to discourage union activity.” But Tesla denied the allegations, stating that the employees were fired due to poor performance. The NLRB did not respond to a request for the current status of the complaint.

When Tesla began building its driver assistance program in 2016, the company outsourced data tagging to a California-based company that had offices in Kenya, but Tesla brought the program in-house in 2019, Reuters reported.

Most recently, Tesla’s Autopilot team was hit with company-wide layoffs in April. Tesla has laid off about 300 of its staff in Buffalo, according to a WARN notice.

Tesla has said that its neural network will one day be able to train itself, but for now, it relies on its own human power.

The work is vital to Musk’s vision for the car company.

Over the years, Tesla’s CEO has repeatedly emphasized the importance of Tesla’s efforts toward achieving autonomous driving. In 2022, Musk said that Tesla’s self-driving technology is “the difference between Tesla being worth a lot of money or being worth basically nothing.”

Tesla plans to unveil its autonomous Robotaxi service later this year, which is expected to be built on the same self-driving software — and, of course, the tedious, clip-by-clip analysis of its tags.

Do you work for Tesla or have a tip? Contact the reporter via a non-work email and device at gkay@businessinsider.com or via the secure messaging app Signal at 248-894-6012.