Forums

Reporting Year 2017, Tesla did not test any vehicles on public roads in California in autonomous mode,Waymo&GM did.

Reporting Year 2017, Tesla did not test any vehicles on public roads in California in autonomous mode,Waymo&GM did.

GM/Cruise

Month #Disengagements''' Autonomous mi. public roads in SanFran''''''''''''''''''''''''''Miles/disengagement
12/01/16 '''''''''''''''''''''''''''008 ''''''''''''''''''''''''''''''''''''''''''''''02,948 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,369
01/01/17 '''''''''''''''''''''''''''012 ''''''''''''''''''''''''''''''''''''''''''''''04,729 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,394
02/01/17 '''''''''''''''''''''''''''010 ''''''''''''''''''''''''''''''''''''''''''''''04,160 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,416
03/01/17 '''''''''''''''''''''''''''008 ''''''''''''''''''''''''''''''''''''''''''''''05,596 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,700
04/01/17 '''''''''''''''''''''''''''019 ''''''''''''''''''''''''''''''''''''''''''''''06,908 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,364
05/01/17 '''''''''''''''''''''''''''007 ''''''''''''''''''''''''''''''''''''''''''''''08,838 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''1,263
06/01/17 '''''''''''''''''''''''''''009 ''''''''''''''''''''''''''''''''''''''''''''''10,487 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''1,165
07/01/17 '''''''''''''''''''''''''''012 ''''''''''''''''''''''''''''''''''''''''''''''10,442 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,870
08/01/17 '''''''''''''''''''''''''''008 ''''''''''''''''''''''''''''''''''''''''''''''14,877 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''1,860
09/01/17 '''''''''''''''''''''''''''005 ''''''''''''''''''''''''''''''''''''''''''''''16,594 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''3,319
10/01/17 '''''''''''''''''''''''''''003 ''''''''''''''''''''''''''''''''''''''''''''''27,576 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''9,192
11/01/17 '''''''''''''''''''''''''''004 ''''''''''''''''''''''''''''''''''''''''''''''18,517 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''4,629
Total '''''''''''''''''''''''''''105 ''''''''''''''''''''''''''''''''''''''''''''''131,672

WayMo

Month #Disengagements Autonomous miles on public roads'''''''''''''''''''''''''''''''''''''Miles/disengagement
12/01/16 '''''''''''''''''''''''''''011 ''''''''''''''''''''''''''''''''''''''''''''''57,616 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''5,238
01/01/17 '''''''''''''''''''''''''''007 ''''''''''''''''''''''''''''''''''''''''''''''45,392 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''6,485
02/01/17 '''''''''''''''''''''''''''004 ''''''''''''''''''''''''''''''''''''''''''''''35,459 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''8,865
03/01/17 '''''''''''''''''''''''''''004 ''''''''''''''''''''''''''''''''''''''''''''''35,873 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''8,968
04/01/17 '''''''''''''''''''''''''''010 ''''''''''''''''''''''''''''''''''''''''''''''27,236 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''2,724
05/01/17 '''''''''''''''''''''''''''005 ''''''''''''''''''''''''''''''''''''''''''''''16,617 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''3,323
06/01/17 '''''''''''''''''''''''''''006 ''''''''''''''''''''''''''''''''''''''''''''''13,917 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''2,320
07/01/17 '''''''''''''''''''''''''''003 ''''''''''''''''''''''''''''''''''''''''''''''19,182 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''6,394
08/01/17 '''''''''''''''''''''''''''003 ''''''''''''''''''''''''''''''''''''''''''''''20,450 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''6,817
09/01/17 '''''''''''''''''''''''''''006 ''''''''''''''''''''''''''''''''''''''''''''''22,967 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''3,828
10/01/17 '''''''''''''''''''''''''''003 ''''''''''''''''''''''''''''''''''''''''''''''27,308 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''9,103
11/01/17 '''''''''''''''''''''''''''001 ''''''''''''''''''''''''''''''''''''''''''''''30,516 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''30,516
Total '''''''''''''''''''''''''''063 ''''''''''''''''''''''''''''''''''''''''''''''352,544

TESLA
Please note:

1)"For Reporting Year 2017, Tesla did not test any vehicles on public roads in California in autonomous mode"
The above are Tesla's words not mine and they come from Tesla's reporting report.

2)In 2016 Tesla did test in autonomous mode and did report. (see links and data below).

3)Hard to meet year end autonomous drive across country when you have not yet started Real World Testing in autonomous mode in the State where
your engineering team is based.

https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115-a72a-97f6f2...

jkilch | February 2, 2018

Well I was mostly able to keep the board from scrambling the data. GM/Cruise columns

Month''''''''Disengagements'''''''''Autonomous mi. public roads in SanFran'''''Miles/Disengagement

TaoJones | February 2, 2018

Well, ain't that a kick in the nether regions.

SbMD | February 2, 2018

From Tesla:

“Tesla conducts testing to develop autonomous vehicles via simulation, in laboratories, on test tracks, and on public roads in various locations around the world. Additionally, because Tesla is the only participant in the program that has a fleet of hundreds of thousands of customer-owned vehicles that test autonomous technology in “shadow-mode” during their normal operation (these are not autonomous vehicles nor have they been driven in autonomous mode as defined by California law), Tesla is able to use billions of miles of real-world driving data to develop its autonomous technology.”

jkilch | February 3, 2018

DTAP

hoffmannjames | February 3, 2018

We've talked about this in another thread. Tesla does a lot to test FSD and does test FSD in California, just not in the way that requires reporting it. Basically, Tesla and Waymo are using 2 different approaches that make it impossible to measure them against each other. So we simply don't know one way or the other. But the lack of reporting is not bad news.

If Musk is to be believed, and I have no reason not to, when he says that Tesla's neural net is solid then Tesla will at some point reach that "critical mass" where their FSD will work on its own. I suspect that in the next couple of years, Tesla will suddenly announce that FSD is operational and it will be amazing.

But the numbers for Waymo seem really good and seem to indicate that Waymo has successfully developed a functional FSD. However, as discussed many times, Waymo is simply looking to create a ride-sharing service in select cities whereas Tesla is aiming to mass produce FSD capable cars that the public can own. So their goals are very different. As soon as Tesla does finish their FSD system, they will be able to send it to every Tesla AP2 car on the road via an over the air update. When they do, they will jump way ahead of Waymo or Cruise.

carlk | February 3, 2018

Every hardware equipped Tesla, which is every Tesla sold in the last couple years, are sending data to Tesla and doing "shadow mode" testing everyday. Tesla does not need to do what others are doing with a few test cars in some particular areas.

Bighorn | February 3, 2018

Jkilch really loves his Tesla, but on Friday nights after a few appletinis...

He’ll rue the day when the car has true autonomy and leaves him for good. #metoo

jkilch | February 3, 2018

@Bighorn | February 3, 2018
"He’ll rue the day when the car has true autonomy"

You could not be more wrong! I will actually buy another Tesla for my Dad that has Macular Degeneration.

Very hard to meet year end autonomous drive across country when you have not yet started Real World Testing in the State where your engineering team is based.

Shadow mode is EAP fallback is to the driver. Autonomous testing is for FSD

hoffmannjames | February 3, 2018

jkilch wrote: "Very hard to meet year end autonomous drive across country when you have not yet started Real World Testing in the State where your engineering team is based."
-------

Tesla is doing real world testing of FSD, just not in a way that requires reporting under California law. Sure, I guess Tesla could have 4-5 cars equipped with the latest FSD software, self-drive around the State of California, just to make some folks happy. but that would actually not be that useful to Tesla. The data that Tesla would get from this type of testing would be tiny compared to the data that they are already getting from shadow-mode. So it wouldn't really help Tesla all that much. The only real benefit would be to give the public one metric of how well the FSD is doing. Remember that companies like Waymo and Cruise have to do it because that is literally the only way for them to get data since they don't have tens of thousands of cars on the road providing them with data. Tesla does not need to do it because they are getting so much data from their existing cars.

jkilch | February 3, 2018

Tesla actually states in the letter that they are in the Development stage.
"Tesla is able to use billions of miles of real-world driving data to develop its autonomous technology.”

DTAP One has to Develop a system before testing the system. Waymo and Cruise have both Developed systems and gone to the next stage of Testing the system. Tesla by there own admission is still in the Development stage.

The next stage would be an Acceptance test either certified by the manufacturer Government or both.

SbMD | February 3, 2018

@jkilch - Tesla is gathering data and using this in simulation settings. This is a scientifically proven method of skill acquisition, and being applied in machine learning.

Frankly, no one here can qualify nor quantify the progress Tesla is making versus others. The DMV report, contrary to your belief, is not a proxy for progress.

Ergo, no miles logged doesn’t mean progress isn’t being made. By their own words, their method of testing does not fall under the rubric defined by CA for reporting. That’s all.

Uncle Paul | February 3, 2018

Believe that Tesla production vehicles are still the best in the marketplace.

Lots of companies are bragging that they have super systems in development, but none on the market yet.

Believe that Tesla is still far ahead of others, and also has lots of improvemts and capabilities in their own development.

jkilch | February 4, 2018

This is interesting in 2016 Tesla acknowledged the law applied in their following report.

"Pursuant to the requirements of 13 CCR § 227.46 Reporting Disengagement of Autonomous Mode, Tesla
Motors Inc. is submitting the following Disengagement Report from its Autonomous Vehicle Test
ve hicle fleet for the Calendar Year 2016
This report summarizes the Tesla Autonomous Vehicle Program disengagement events as specified by
DMV Article 3.7, Section 227.46 "

https://www.dmv.ca.gov/portal/wcm/connect/f1873c87-4f21-4beb-b665-050cad...

So in 2016 Tesla Reported they DID Test in Autonomous mode and 2017 they say they did NOT Test in autonomous mode. Very Interesting.

TESLA
Month #Disengagements''' Autonomous mi. public roads in SanFran''''''''''''''''''''''''''Miles/disengagement
01/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
02/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
03/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
04/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
05/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
06/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
07/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
08/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
09/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
10/01/16 '''''''''''''''''''''''''''182 ''''''''''''''''''''''''''''''''''''''''''''''00,401 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,002
11/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,020 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,020
12/01/16 '''''''''''''''''''''''''''000 ''''''''''''''''''''''''''''''''''''''''''''''00,000 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''0,000
Total '''''''''''''''''''''''''''124 ''''''''''''''''''''''''''''''''''''''''''''''00,421

jkilch | February 4, 2018

Autonomous mi. public roads in SanFran = Autonomous mi. public roads in California.
It is Cruise that is testing in SanFran.

SbMD | February 4, 2018

@jkilch, that’s not the way to interpret that report and current events. See my prior post which has the statement from Tesla as well as the posts of others here in this thread. You should look at the thread @Hoffmanjames is referring to, as well.

carlk | February 4, 2018

Don't think @jkilch understands how machine learning works. No need to argue with him unless he really wanted to learn.

jkilch | February 4, 2018

@SbMd I am not interpreting the data just reporting the facts. The only "interpretation" I made was that the data was "interesting".

jkilch | February 4, 2018

At some point they will have to Test the Autonomous system on California Highways before doing a coast to coast autonomous trip that was promised by the end of last year.

SbMD | February 4, 2018

@jkilch - "@SbMd I am not interpreting the data just reporting the facts. The only "interpretation" I made was that the data was "interesting"."

No, you are providing interpretations, and based on assumptions from incomplete information. Please see your statements below for some examples:

1) "Very hard to meet year end autonomous drive across country when you have not yet started Real World Testing in the State where your engineering team is based."

You are interpreting the data to say that it is "very hard" to have a goal of autonomous driving since your interpretation is that they "have not yet started Real World Testing". This is a clear example of interpretation, and not stating "facts".

This is actually a pretty big interpretation on your part, and with incomplete information. Similarly, it would be bad practice to say the converse, that Tesla has leapfrogged the competition in this space without knowing what they have done. Right now, no one can comment outside of Tesla about their progress.

2) "Waymo and Cruise have both Developed systems and gone to the next stage of Testing the system. Tesla by there own admission is still in the Development stage."

They are all in the "Development stage", @jkilch. Don't take the autonomous miles and disengagements as being proof of functionality. As Bryan Reimer from MIT stated in a past interview (and I am paraphrasing) that this is an interesting metric, but it is just a data point and not to be taken as a proxy for success in implementation.

3) "So in 2016 Tesla Reported they DID Test in Autonomous mode and 2017 they say they did NOT Test in autonomous mode. Very Interesting."

The testing, by Tesla's own report, did not meet criteria for reporting but have ben doing autonomous testing. The tests are being done, but they do not need to disclose based on the letter of CA law. There are reasons why one would do this, or not. Again, you interpret and interpret incorrectly.

I'll say it again: none of us -- including you -- have a true understanding of the progress of Tesla or any of the systems being tested. We should refrain from making any definitive statements as you have above, which are actually interpretive and based on incomplete information and in some cases are incorrect conclusions.

SbMD | February 4, 2018

@carlk - hopefully putting more balanced and informed comments, including proper qualification to acknowledge information gaps, helps all people reading and learning. Ideally that includes @jkilch.

jkilch | February 4, 2018

@SbMd You have twisted the timeline. Your comments are nonsense, and and not worth rebutting.

I repeat At some point Tesla will have to Test the Autonomous system on California Highways before doing a coast to coast autonomous trip that was promised by the end of last year.

My Dad has macular degeneration and had hopes Tesla would do the Coast to Coast last year and provide him a solution and extended independence.

I will not be responding to accusations and twisted timelines.

hoffmannjames | February 4, 2018

jkilch wrote: "So in 2016 Tesla Reported they DID Test in Autonomous mode and 2017 they say they did NOT Test in autonomous mode. Very Interesting."
------

I think you are misinterpreting the data. Reporting 0 miles and 0 disengagements does not mean 0 testing. Tesla actually states in their report that they DID test autonomous driving in 2017, just in a way that did not require reporting under California law. Saying that Tesla did not do any testing of autonomous driving in 2017 is just completely false.

jkilch | February 4, 2018

Links to 2015,2016 and 2017 California disengagement reports at the bottom of the page.

Research-Development-Testing-Acceptance-Production.

Again here is the master link to all reports and data-Disengagemet report Links for all years at the bottom

jkilch | February 4, 2018

https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/bkgd

Key Definitions
California Vehicle Code Section 38750 established several definitions that will guide the department's development of regulations.

"Autonomous technology" means technology that has the capability to drive a vehicle without the active physical control or monitoring by a human operator.

SbMD | February 4, 2018

@hoffmanjames + 1

@jkilch - "@SbMd You have twisted the timeline. Your comments are nonsense, and and not worth rebutting."
Well, not at all, but this is an open Forum so you can certainly post your opinion. Your follow-on posts are attempts at rebuttal, though...

bsteinesq | February 4, 2018

The section of the California Code requiring this report is highly specific about what is to be reported:

"13 CCR § 227.46
§ 227.46. Reporting Disengagement of Autonomous Mode.
(a) Upon receipt of a Manufacturer's Testing Permit, a manufacturer shall commence retaining data related to the disengagement of the autonomous mode. For the purposes of this section, “disengagement” means a deactivation of the autonomous mode when a failure of the autonomous technology is detected or when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle.
(b) Every manufacturer authorized under this article to test autonomous vehicles on public roads shall prepare and submit to the department an annual report summarizing the information compiled pursuant to subdivision (a) by January 1st, of each year.
(1) The first report shall cover the period from the date of issuance of the Manufacturer's Testing Permit to November 30th of the following year.
(2) After the first report, subsequent annual reports shall cover the period December 1st of the current year to November 30th of the following year.
(3) The annual report shall summarize disengagements for each month as follows:
(A) The total number of autonomous mode disengagements and the circumstances or testing conditions at the time of the disengagements including:
(i) The location: interstate, freeway, highway, rural road, street, or parking facility.
(ii) A description of the facts causing the disengagements, including: weather conditions, road surface conditions, construction, emergencies, accidents or collisions, and whether the disengagement was the result of a planned test of the autonomous technology.
(B) The total number of miles each autonomous vehicle tested in autonomous mode on public roads each month.
(C) The period of time elapsed from when the autonomous vehicle test driver was alerted of the technology failure and the driver assumed manual control of the vehicle. "

As Tesla explains in their 2017 report, non of the testing they did met the specific criteria required by California law, thus, there was nothing to report.

This does not mean there was no testing, only that the testing did not meet the specific requirements of the California Disengagement report. So, if 10,000 California Tesla's were testing the new software in "shadow mode" it would not be reported in this report. Similarly, if there were dozens of Tesla's actually live testing the new software in state other than California, that would not be included in this report. So, yeah, nothing to report on this report does not mean "zero testing."

Bighorn | February 4, 2018

The coast to coast trip will have a person in the driver's seat, so there are no regulatory hurdles to clear.

jkilch | February 4, 2018

The Title says "2017 California Autonomous Driving Summary--Max Miles/Month/disengagement" Not Zero testing anywhere. However I will further clarify the title so no one is confused

hoffmannjames | February 4, 2018

@jkilch I think your title is still wrong because Tesla did do testing in California.

jkilch | February 4, 2018

@hoffmannjames for clarification Do you agree with this statement? (I did not change the title yet)

"For Reporting Year 2017, Tesla did not test any vehicles on public roads in California in autonomous mode"

hoffmannjames | February 4, 2018

@jkilch Yes, I think I would agree with that statement.

bsteinesq | February 4, 2018

@jkilch: IMO, still a bit over broad. I would suggest something like "In California, for Reporting Year 2017, Tesla did not have any reports for the State required "Disengagement of Autonomous Mode" report."

SbMD | February 4, 2018

@jkilch, if you want, keep it simple: "In 2017, Tesla tests off of public roads"
Add to your OP the statement from Tesla which says this explicitly.
Keep the other info about the other automakers who reported, IMO.

RedJ | February 4, 2018

It’s possible that Tesla is playing legal semantics in order to avoid reporting. They could be testing FSD as an enhanced version of EAP with greater capabilities but keeping the “hand on the wheel” warning and shut down feature. That way they could say that technically the vehicle did not have the capability of driving without human monitoring. Just another possibility/wild guess.

jkilch | February 5, 2018

@RedJ I agree with your point about avoiding reporting. But then I would ask the question why avoid reporting if the results are good?

jkilch | February 5, 2018

The title has been modified and comes directly from the 1st sentence in the 3rd paragraph of the Tesla's cover letter to California. Waymo and GM did test in autonomous mode, that too is an accurate fact.

RedJ | February 5, 2018

@jkilch, I agree that they probably have a pretty high rate of disengagement’s for 2017 since this is the first year they actually started working on FSD (I wouldn’t count the 2016 marketing video).

The bottom line is I’m guessing they’re working fairly aggressively on it even if they don’t have anything that requires reporting under CA regulations (or at least their interpretation of it).

bill | February 6, 2018

Well it has been almost 1.5 years since any real advance in Auto Pilot, given that AP2 is still catching up to AP1. And there has been 0 functionality for Autonomous Driving.

So while a lot of people seem to still have faith, as defined by a belief with no proof, I see no reason other then faith that Tesla has done anything but loose ground in the FSD race.

bill | February 6, 2018

But they can put a car in outer space but since I am not going there it has no value to me.

Stiction | February 10, 2018

..."Telsa's neural net" ...