hckrnws

Buyer be warned i think this is extremely brand-dependent.

While i've had generally solid experience with sandisk for almost 20 years and had a few old drives (which i hear are slc-based so its not surprising) hold files for over 5 years no issue, i recently almost lost over 4 years of photos.

I had purchased some lexar drives from costco since they were dual interface (usb A / usb C) about 2 years ago, and it was usefull to just get some pictures off my phone. I usually don't rely on such a setup for long term but as with all things I was delayed tending to it. I figured there were 2 per box so i just copied them twice, and diffed them several times to make sure they were exact copies.

After 24 months, one of the drives had a %95 loss, almost every picture was lost cut-off bottom half or so. The other drive surprisingly seemed fine, though it had been plugged in every 6-9 months I recall, as I wanted to browse it a few times, it seems that this action saved the volume. Upon further inspection the good drive still lost 10 pictures in about 5 thousand, so it wasn't perfect.

Lexar.

https://www.ebay.com/itm/176810492981?chn=ps&_trkparms=ispr%...

> After 24 months, one of the drives had a %95 loss, almost every picture was lost cut-off bottom half or so.

If these are JPEGs with a grey or green lower half, it's likely only a few 16x16 macroblocks are corrupted and you can recover the rest.

This cannot be done programmatically because you have to guess what the average colour of the block was, but it can be worth it for precious pictures.

Really who knows if you're getting a legit drive any longer...

https://www.tomshardware.com/pc-components/ssds/fake-samsung...

Slightly related: I have a tool that writes random (incompressible) data to a disk and lets you verify it back without storing a copy (by using a csprng seed), initially developed for benchmarking SSDs that used to cheat to get better performance numbers but that can also be used for this purpose or to overwrite (“shred”) a disk: https://github.com/mqudsi/hddrand

I haven't used badblocks https://en.wikipedia.org/wiki/Badblocks in about 10 years, but I was annoyed that this exact feature wasn't available for testing accidental swapping of block locations. badblocks only writes the same data to each block and thus they are all indistinguishable.

TIL `badblocks -t random` repeats the same random block over and over :(

You can however set the block size to something quite large, which means you write the same random pattern spread out over multiple blocks repeatedly. If you pick an "odd" block size (like say, your native block size multiplied by 47), it's highly unlikely your disk under test will be swapping around "groups of 47 blocks." (I usually just do a nice multiple, like 4K16, but if you're super paranoid a weird multiple should be pretty much good enough). You won't get reporting of which exact* blocks on the drive are failing, but these days, that isn't really useful information - if any blocks are failing, warranty or ditch the drive.

I bought about 20 flash drives in 2019 at work to parcel out when needed, once or twice a quarter for users.

I needed one last week, and had to throw most of them away, they had all died from presumably dormancy, even new in the package.

I like the fact he's not just verifying all of them each year. AFAICR, reading the flash causes the row to be rewritten with the values just read.

I remember years ago working on the Wii, and there was a restriction on how often you could read the flash to avoid premature wearing. Not sure if that was just the specific type of storage, as googling suggests that NAND is subject to this and NOR isn't. I think pretty much all USB drives now use NOR flash, so maybe this isn't actually an issue any more.

reading the flash causes the row to be rewritten with the values just read

DRAM works that way but flash doesn't. Read disturb is a different issue.

pretty much all USB drives now use NOR flash

Nope, NOR flash is much more expensive than NAND so NOR is only used for firmware and everything else is NAND.

But the firmware might have the logic to rewrite the block when it reads it in case it hasn't been written in a while.

SSDs should definitely rewrite static data if it has too many ECC errors. Unfortunately we don't know much about what's going on in SSDs. Some could have much better data integrity than others.

> reading the flash causes the row to be rewritten

This only happens very rarely, though more frequently as NAND flash goes QLC and beyond.

Besides, other experiments have shown that data remanence is way more of an issue with drives that are almost completely worn out (way beyond their specified TBW) and about to croak. Even then you only get rare bitrot that can be checked for and compensated quite cheaply in most cases.

If you take fresh media, write it just once or a few times at most, use substantial overprovisioning to keep the drive in its fast pseudo-SLC mode, and reread the media periodically, NAND can be a good enough storage system for most casual needs.

What is the best consumer friendly long-term storage medium? Are we still better off with high capacity dvd/Blu ray discs?

Recordable blu-ray discs have a reported lifespan of hundreds of years if left untouched, but the high-capacity ones (128GB) are not especially cheap right now and I assume the writing process is slow. The drives themselves may not be easy to come by in future decades. But they are your best bet for "I want my data to outlive my grandchildren."

For the rest of us, a USB spinning rust hard drive formatted as exFAT is going to be hard to beat. You'll be able to plug this into virtually any computer made in the next few decades (modulo a USB adapter or two) and just read it. They are cheap (even allowing for the rising cost of storage), fast, and most importantly, they are easy. The data is stored magnetically, so is not susceptible to degradation just from sitting like SSDs or flash drives are.

Of course, you should not store any important data on only ONE drive. The 3-2-1 backup rule applies to archives as well: 3 copies, 2 different media, 1 off-site.

I decided to go with NTFS for the filesystem as it has journaling. Works fine on Linux, and obviously Windows. For macOS there are various add-ons that support NTFS, but my use case there is read-only.

Probably depends on what “consumer-friendly” entails, how it’s stored, and the quantity of data.

If we’re talking the average tech-illiterate to literate-but-cost-and-space-constrained person, probably Blu-Ray. A burner+reader combo with a stack of dual-layer discs is probably cost-effective. High-capacity HDDs would probably be equally effective if you can guarantee that they’re stored away from accidents and mishandling, but if it requires a SATA-to-USB adapter with assembly then it might possibly be out of reach for some consumers, and any risk of damage from movement could rule it out entirely.

If we’re talking tech-savvy consumers who don’t have the IT budget of a corporation, maybe LTO-5 or LTO-6 tapes could work. Tapes themselves are very affordable and have a good shelf lifespan. Used libraries can be had for under $600. The primary issues would be finding one with an interface that works with your existing equipment and software to support tape read and write.

Consumer? Apple or Google Photos or 'drive' functionality of either. The only real risk then is losing your account and Apple Photos has an option to keep them all locally on disk.

To be pedantic, the post you responded to asked about "storage medium", not storage services, which leads to the question of what storage medium they use and how long the services will be around.

I've been a big fan of M-Disc BD-R.

Honestly: multiple copies of encrypted cloud storage. (Encryption just for privacy.) You need decentralized backups anyway. Alternatively, two NAS systems with some RAID variation in different locations that back up each other can be more cost-effective for large capacities.

You're talking about backups which you wouldn't normally need to keep for decades and will be powered on regularly anyway. If it's archival, such as family photos for your kids when they grow up, cloud storage can lose them if you die or go to prison or for whatever reason don't keep paying the bill.

What's long-term? I have some dvd-rs that push 20-25 years and despite the plastic getting brittle they still work. I also have some ide drives that still work without problems after 40 years. I would rather aim for 20 years and upgrade the storage device if I still need to retain the data.

That's a thought I hadn't had. The plastic of the disk getting so brittle it shatters in the drive due to age. I wonder what's the embrittlement profile of polycarbonate stored in reasonable condition.

I've personally never noticed brittleness in old optical discs (unlike the polystyrene jewel cases, which often turn brittle). I don't think shattering is likely, but if it's a concern some optical drives allow limiting the maximum spin speed. If the drive supports it you can temporarily set it with the -x option of the "eject" command from util-linux.

On a related subject, physical media, like a song album. I started by wondering if there were ever any solid state distribution options (One Company tried SD cards) and then started digging into the underlying storage tech to see if I could find a write once long term stable process.

First the elephant in the room. Why solid state? because the drives to read the media are often the weak link. When the drives are no longer being manufactured how hard is it to make one? reading solid state drives is a relatively low precision electrical process compared to the high precision mechanical process needed for most media.

First on the chopping block was bulk storage. It tends to be delicate and hard to read and short lifespans. But if I limited myself to small storage there are some interesting options. fusible proms were promising but top out at a few megabytes. Mask roms? does anyone offer a mask rom service anymore?

Put a mask rom into a sd card... no, sd cards are too physically small. For a song album we want something bigger to put album art on. A thing the size of the original gameboy cartridge with a usb interface and a mask rom?

My conclusion, for that specific goal, indefinite future storage of a song album. Vinyl records. low tech enough that it is easy to make a player for them.

Interesting related thread that includes SSD firmware engineer fairfeather discussing refresh mechanism implementation: https://news.ycombinator.com/item?id=46038099

Rewriting the data each year hides the actual issue here. Have had plenty of "nice" flash drives rot to hell in 18+ months of dormancy

Does rewriting data help prevent bit rot? Does it mean powered drives can take advantage of it by periodically rewriting the same data over?

It depends on the type of flash being used and the controller managing it. That he did not even identify the chips should inform you of the extent that these results can be trusted.

All I can say for sure is that you should not trust any flash for long term storage, thumb drive or otherwise. In serious enough, high usage, high heat enviornments where everything working without problems or delay is part of what they are paying us to be responsible for, it is standard practice to clone fresh images to nvmes every time, with multiple spares that can be swapped out in minutes when they inevitably fail anyways.

It depends on how the flash modules are maintained and their quality, but yes having freshly written data will imply better data consistency on flash media.

Flash media relies on recharging, which may or many not happen often enough.

Did you miss that there are 10 different drives and so they have 10 different years of tests where they are testing a completely untouched drive?

I don't think you're reading the results properly.

I think they are reading it correctly. Year 1, they touched one drive and left 9 untouched. Year 2, they read one additional drive and left 8 untouched. Etc.

Yes, it's also confirmed on the OP's blog linked in the post.

Those drives aren't being read

What do you think I am I reading incorrectly? The post seems pretty clear:

"I filled 10 32-GB Kingston flash drives with pseudo-random data."

"The years where I'll first touch a new drive (assuming no errors) are: 1, 2, 3, 4, 6, 8, 11, 15, 20, 27"

And from the blog: "Q: You know you powered the drive by reading it, right? A: Yes, that’s why I wrote 10 drives to begin with. We want to see how something works if left unpowered for 1 year, 2 years, etc."

I could google it, but I would rather ask HN: what are the best pens (or pen(cil)/paper-combination) for keeping written text as long as possible? I had some Stabilo pen which was very nice ergonomically, but the blue ink faded within a couple of years (laying on my window sil in the sun, but still).

My guess is: regular graphite pencil on porous paper is best. Any ideas about further things I have to take into account?

I'd think thin copper sheet on something soft would also work. That indentation will probably outlast any sort of ink

I don't think there's a simple answer. For example, someone recommended black ink on white paper, but it really depends on the composition of that ink. Inorganic pigments last forever, but the ink used in black sharpies actually fades pretty quickly.

Pencil definitely lasts if the paper is undisturbed. I have some paperwork that's 100+ years old and with legible pencil text. On the flip side, if the paper is handled a lot, the writing will gradually fade because graphite particles just sit on the surface and can flake off.

On some level, the medium is your main problem. Low-grade paper, especially if stored in suboptimal conditions (hot attic, moist crawlspace, etc), may start falling apart in 20 years or less. Thick, acid-free stock stored under controlled conditions can survive hundreds of years.

Thanks for the insight.

Acid-free paper sounds like the way to go. Do you have experience with this? Or is it common knowledge? Just curious!

I also read letters from my grandparents, stored by my parents in a simple shoe box. No special conditions, just light-free and inside the home for decades. They were still very much readable. I did not pay enough attention, but I guess it was blue ink from back in the day that they used.

> Do you have experience with this? Or is it common knowledge? Just curious!

I collect vintage stuff that sometimes comes with paperwork, usually after spending a decade or two stashed away in the attic.

I vote for graphite on paper. Ink will run if the paper gets wet. Of all the damage that has occurred to my papers, water is the most common. I keep a copy of important phone numbers written inside my wallet in case I ever lose my phone. Between an unexpected rainstorm, to an unchecked pocket before putting pants in a washing machine, to a spilled drink, I have gotten my wallet wet several times. Every time I used ink, I had to rewrite the list, but now with graphite, it isn't a problem.

Thanks! I appreciate the input.

Do you just use regular graphite pencils, like with the HB scale or something?

I just use the same BIC mechanical pencils with #2 lead that I picked up in college. No reason to get complicated.

In the UK “registrar’s ink” is used for marriage certificates, I believe it is supposed to be good for many hundreds of years.

Black ink on white paper, stored in a cool dark place, will last many decades. If may fade but will remain readable. Want centuries? Use skin parchment. Millenia? An engraving pen on glass. Going for longer? Take a grinder to a block of granite, but the real problem there is the lack of geologically-stable storage on this planet.

An engraving pen on glass? Why not get some sheet glass and a stick of color. Write it directly onto the glass _in glass_

Granite is heavy and brittle. Instead, take a plate made of platinum or iridium, and engrave information on it. It offers excellent mechanical, chemical, and thermal durability. It can sink in volcanic lava and then hammered back out from the resulting rock, intact. (Expensive though.)

A couple of millennia might suffice. ;) Thanks for the input.

The engraving pen on glass is a good one. Any experience with it?

Beware that flash data lifetime is sensitive to temperature in the normal range people store stuff at. Store them in the roof space of your house that can exceed 40C each day and they might not last one year.

Definitely not a medium to passively store anything long term without power! Use Hard drives or Blu-ray instead.

The linked post currently has demonstrated that 6 years in some reasonable-ish condition is perfectly fine.

> they might not last one year.

> Definitely not a medium to passively store anything long term without power!

Do you have any evidence to back up this claim? I'm much more interested in data than fear mongering.

He stored them on a shelf which is probably 25C max. and that has an order of magnitude longer life than at 40C. [1]

[1] https://www.ni.com/en/support/documentation/supplemental/12/...

What's the simplest way to rewrite the data without actually copying the data? Like in place rewrite - you write what you read.

I've seen "dd if=/dev/removable of=/dev/removable" suggested. I don't know if it actually works or if the OS optimizes it to a no-op.

Certainly the OS can't optimize it to a no-op, since `dd` makes separate read and write syscalls.

I suppose your `dd` implementation itself could do so, but I don't know why it would.

the risk of catastrophic data loss from misuse of `dd` makes my hackles rise just looking at this.

I will never forget when I mixed up `if` and `of` during a routine backup.

`cat /dev/sda > /mnt/myDisk2` is so much safer, explicit, and in unix norms. It's also faster because you don't have to tune block size parameters.

Plus you can also do `pv /dev/sda > /mnt/myDisk2` to get transfer speed details.

Friends don't let friends use `dd` where `cat` can do the same job.

I stopped getting scared of `if` and `of` about a decade ago when I started explicitly saying (in my head) "input file" and "output file" rather than "if" and "of." You still can mess up the order, but imo no more easily than you can swap `cat in > out` for `cat out > in`.

> Friends don't let friends use `dd` where `cat` can do the same job.

Technically yes... but I like being able to explicitly set block sizes and force sync writes.

Wouldn't a ZFS Scrub get the job done?

Powered all the time on or powered off?

OP says powered off.

That's good. I want to keep some institutional knowledge and photos in "cold storage" and cloud subscriptions with a credit card and password are completely inviable.

I'll probably get a spinner and a flash drive and hope one of them survives the years.

If privacy is your primary problem with cloud storage, I would suggest veracrypt containers. And if you aren't storing too much data, I would also suggest DVD/BluRay optical media with DVDisaster and PAR2 archives. I keep a DVD spindle in a safe deposit box that gets updated each year.

Unless the data is huge, you're probably going to be better off with M-Disc Blurays or DVDs, as they're explicitly designed for what you're trying to do.

[dead]