Tesla Flash Memory

[Wombat]

Wrong application?

https://www.drive.com.au/news/tesla...to-last-5-6-years--125093.html?trackLink=SMH3

 

[Doodski]

Wow. Just wow. So Tesla expects the owner to not only run the risk of a catastrophic file system failure but to purchase flash memory and a display once or maybe twice in a vehicles lifetime. That's wrong.

 

[Blumlein 88]

Not much you can do about flash drive wear.

I do recall a while ago a kerfuffle about the displays. Tesla didn't build them to an automotive standard set of conditions. They went with a lower level of environmental guidelines apparently just from lack of experience as an auto maker. So they by now knew these early S screens were going to fail early. Heat range and vibration standards they accepted for the screens were below what is known to be needed inside an automobile.

https://www.thedrive.com/tech/27989/teslas-screen-saga-shows-why-automotive-grade-matters
This is from nearly 2 years ago about what "automotive standard grade" meant for these. Tesla uses a lower industrial grade display.

 

[peanuts]

iv used ssds and ipads for a decade, not once have they failed. the flash storage in tesla must be rubbish.

 

[Blumlein 88]

iv used ssds and ipads for a decade, not once have they failed. the flash storage in tesla must be rubbish.

You might under-estimate the rates at which it is being accessed.

 

[Wombat]

Are SSDs any better?

 

[Soniclife]

Not much you can do about flash drive wear.

There is, you can put more in so it wears slower, if the driver is good, and you can write software that does not do writes as often, as it's the writes that kill it. This feels like typical start up thinking, where the concept of what you made still being used in 5 years is a problem for someone else.

The SSD in my main windows machine is 7 years old, it should be seeing a much higher rate of access than a car does.

 

[RayDunzl]

iv used ssds and ipads for a decade, not once have they failed. the flash storage in tesla must be rubbish.

Do you have them permanently mounted in your car on the upper side of your dash?

 

[Doodski]

I checked my 500 GB Samsung EVO SSD and in the past ~32 months it has had 22.4 TB written to the drive. Samsung guarantees the drive for up to 2,400 TBW, or is backed by a 5-year limited warranty. I use the PC for ~6-12 hours per day. I have a fan on the SSD drive for cooling. At the present rate of use the estimated life would be a very long timeeee. Apparently the memory used in the Tesla vehicles is pretty cheesy.

 

[mansr]

I checked my 500 GB Samsung EVO SSD and in the past ~32 months it has had 22.4 TB written to the drive. Samsung guarantees the drive for up to 2,400 TBW, or is backed by a 5-year limited warranty. I use the PC for ~6-12 hours per day. I have a fan on the SSD drive for cooling. At the present rate of use the estimated life would be a very long timeeee. Apparently the memory used in the Tesla vehicles is pretty cheesy.

There are many types and grades of flash memory. If you don't care about longevity, you can save some money by using a cheap variant with low endurance rating.

Flash memory can be tricky to manage correctly. Each block has a limited number of erase cycles which can be as low as 1,000 for cheap MLC chips. A typical endurance rating for decent SLC flash is 100,000 erase cycles. Regardless of the rating, measures must be taken to limit the number of writes to any single block. To this end, some wear levelling scheme is used. This means that block numbers as presented to higher level software do not map to fixed locations in the flash device. Instead, each time a block is written, a new physical location is chosen and stored in a mapping table. The old location is marked as unused and added to the list of free blocks to be used by future writes. This spreads the writes (more or less) evenly across the device, so even if the software is writing repeatedly to (what it sees as) the same location, it won't cause excessive wear of any physical block.

Wear levelling is the reason why SSD endurance is quoted in number of bytes written. This figure is simply the block size times erase endurance times number of blocks, minus some safety margin. Thus, the lifetime of the device as a whole can, as mentioned above, be increased simply by adding more flash memory than the bare minimum required.

Even with wear levelling, it is important to minimise the number of writes. This requires some level of discipline from the software developers, something which is often sorely lacking. For an application such as a car dashboard, there is in fact little reason to ever write anything at all. Little things like the last tuned radio station and other user settings are better to store in a separate EEPROM.

Another problem that has to be dealt with is gradual corruption of stored data. Random bit-flips are an expected occurrence, and each block includes around 10% error-correction data so as to cope with this. Whenever a block is read and the number of errors exceeds some threshold, a fresh copy is written. To ensure corruption is detected in time, while it is still correctable, background scrubbing may be necessary. This gets extra complicated with MLC/TLC/QLC flash where even the act of reading a block can cause corruption. Too vigorous scrubbing can thus generate additional writes, increasing wear.

In an SSD, all this complexity is handled by the controller chip, presenting to the host computer what appears to be a simple, reliable storage device. Some controllers are better at this than others, which is (in part) why performance varies between makes and models. Embedded devices often use plain flash chips, doing all the above-mentioned management in the OS. This can offer more flexibility in system design, provided the designers know what they are doing. If not, disaster awaits.

 

[Tks]

There are many types and grades of flash memory. If you don't care about longevity, you can save some money by using a cheap variant with low endurance rating.

Flash memory can be tricky to manage correctly. Each block has a limited number of erase cycles which can be as low as 1,000 for cheap MLC chips. A typical endurance rating for decent SLC flash is 100,000 erase cycles. Regardless of the rating, measures must be taken to limit the number of writes to any single block. To this end, some wear levelling scheme is used. This means that block numbers as presented to higher level software do not map to fixed locations in the flash device. Instead, each time a block is written, a new physical location is chosen and stored in a mapping table. The old location is marked as unused and added to the list of free blocks to be used by future writes. This spreads the writes (more or less) evenly across the device, so even if the software is writing repeatedly to (what it sees as) the same location, it won't cause excessive wear of any physical block.

Wear levelling is the reason why SSD endurance is quoted in number of bytes written. This figure is simply the block size times erase endurance times number of blocks, minus some safety margin. Thus, the lifetime of the device as a whole can, as mentioned above, be increased simply by adding more flash memory than the bare minimum required.

Even with wear levelling, it is important to minimise the number of writes. This requires some level of discipline from the software developers, something which is often sorely lacking. For an application such as a car dashboard, there is in fact little reason to ever write anything at all. Little things like the last tuned radio station and other user settings are better to store in a separate EEPROM.

Another problem that has to be dealt with is gradual corruption of stored data. Random bit-flips are an expected occurrence, and each block includes around 10% error-correction data so as to cope with this. Whenever a block is read and the number of errors exceeds some threshold, a fresh copy is written. To ensure corruption is detected in time, while it is still correctable, background scrubbing may be necessary. This gets extra complicated with MLC/TLC/QLC flash where even the act of reading a block can cause corruption. Too vigorous scrubbing can thus generate additional writes, increasing wear.

In an SSD, all this complexity is handled by the controller chip, presenting to the host computer what appears to be a simple, reliable storage device. Some controllers are better at this than others, which is (in part) why performance varies between makes and models. Embedded devices often use plain flash chips, doing all the above-mentioned management in the OS. This can offer more flexibility in system design, provided the designers know what they are doing. If not, disaster awaits.

I think the fellow you spoke to has nothing to worry about considering people have Flash Cards (SD etc...) that have been going on for years of constant video writing and deleting from daily camera use.

Especially since he's using Samsung SSD's which are known to be virtually the best in all regards aside from price (debatable, seeing as how they technically perform the best with vertically integrated design giving way to their current reign with the NAND and memory controllers as well).

Also, the bit flips issues can be mitigated in critical use cases with ECC as you say, which should only get better as time goes on and more active "background" diagnostic and management software.

 

[mansr]

I think the fellow you spoke to has nothing to worry about considering people have Flash Cards (SD etc...) that have been going on for years of constant video writing and deleting from daily camera use.

Memory cards wear out. Those of the micro-SD variety seem to be especially short-lived.

Especially since he's using Samsung SSD's which are known to be virtually the best in all regards aside from price (debatable, seeing as how they technically perform the best with vertically integrated design giving way to their current reign with the NAND and memory controllers as well).

Yes, Samsung SSDs are among the top choices for both reliability and performance. They know what they're doing.

Also, the bit flips issues can be mitigated in critical use cases with ECC as you say, which should only get better as time goes on and more active "background" diagnostic and management software.

Flash memory is useless without ECC. The only choice you have is the strength. Stronger ECC can correct a larger number of bit-flips at the expense of reduced capacity and increased computational cost.

 

[TheWalkman]

Last I checked, the average car on the road in the US is 12 years, so a five or six year expected life of the flash memory seems a bit cheap, although comparing the life cycle of an EV to an ICE auto may not be a truly fair comparison.

 

[somebodyelse]

It's far from clear whether the write rate has always been a problem, or whether it originally had a lower write rate that would have led to, say, a 20 year projected lifetime, and a subsequent firmware update changed it for the worse. In either case Tesla should have taken responsibility rather than being forced into a recall.

 

[Zoomer]

Hackadays take on Tesla's attempt to wriggle out of this by labeling the Flash a "wear item"

 

[restorer-john]

Hackadays take on Tesla's attempt to wriggle out of this by labeling the Flash a "wear item"

Flash is a wear item. Maybe if Tesla installed the flash as a M.2, esily accessible and replaceable, they wouldn't have all this trouble with recalls. A little flap cover, one screw and you're good to go for another 5 years- all for $50-$100.

 

[LTig]

No, the usual way is to make it as complicated as possible so that the customer is forced to let his dealer make the change for a nice charge (nice for the dealer). Just look how difficult it is to change the bulbs in the front lights of many cars.

 

[mansr]

Flash is a wear item. Maybe if Tesla installed the flash as a M.2, esily accessible and replaceable, they wouldn't have all this trouble with recalls. A little flap cover, one screw and you're good to go for another 5 years- all for $50-$100.

When the requirements for capacity and performance are moderate, as I'm sure they is the case here, and the full product is (literally) the price of a car, it is perfectly reasonable to design a storage solution with an expected lifetime at least as long as that of other critical components. Sooner or later, it will of course fail, but so will e.g. the wheel bearings. Nobody complains about needing to replace mechanical parts after 15 years, and a circuit board would/should be no different. The mistake here seems to have been some combination of underestimating the wear and using a non-automotive-qualified part, thus reducing its lifetime below spec.

A $100 M.2 SSD will definitely not be automotive-qualified.

 

[Berwhale]

Not much you can do about flash drive wear.

You can over-provision flash cells to increase performance and extend the useable lifetime of the drive.

3 years ago, I gave up 10% of my 250GB Samsung EVO 960 NVMe SSD for overprovisioning to 'optimize performance and lifespan'...

I learned a fair bit about configuring flash storage when I was responsible for specifing Violin Memory all flash arrays for a large database cluster at the bank I used to work for. 10TB of flash cost around $5m at the time and I had to do the IO calcs to work out how much of this 10TB would actually be availble for our valuable data. It turned out that we needed to over provision around 20% of the cells to avoid 'falling off the write cliff' (IO performance would drop from 250K to the low 20K IOPS whilst the array tried to erase cells before they could be re-written to). The meeting where I explained to the CTO why we needed to bench $2m worth of flash was quite memorable (The cluster had 2x 10TB arrays attached).

 

[restorer-john]

A $100 M.2 SSD will definitely not be automotive-qualified.

Probably true, but the point remains. Large capacity flash memory in a vehicle's infotainment display system should be replaceable in a straighforward manner, without needing a recall, and a whole lot (~135,000) of scrapped displays/boards etc. Personally, I think 5-7 years isn't too bad, IF the flash ram was easy to replace.

You have no doubt inspected the internals of ECUs and other vehicle systems. The components used are often absolutely bog standard silicon with the only real differences being the quality of the board, soldering, casework and prevention of environmental ingress. I've seen car audio built better than vehicle bodysystem electronics.