Back in 1999 or so, I was on the team that helped Minnesota Public Radio choose a long-term storage format for their (extensive and valuable!!) audio archives, which they were about to digitize.
The conclusion we came to was that there was •no• storage medium that met their long-term preservation goals — true then, and almost certainly still true today. The best approach is perpetual and redundant forward copying of the data, from one physical storage device to another.
An archive isn’t a physical thing; it's a process.
Gizmodo asks “Will We Ever Solve the Problem of Losing Data?”I love this answer: “I know of one data storage form that can be permanent: a library. Or rather, a world full of libraries. Libraries aren’t perfect storage devices (nothing is), but they are fiercely committed to self-repair. A good library is a collection of people, technologies, and practices that pass an obligation—to remember the things we must remember—from the last generation to the next.”
gizmodo.com/will-we-ever-solve-the-problem-of-losing-data-2000792386 #librariesWill We Ever Solve the Problem of Losing Data?
When it comes to recording, tracking, and storing data, the concept of "permanence" can be a lot trickier than you'd expect.Gayoung Lee (Gizmodo)
Carlos Solís likes this.
reshared this
Nicole Parsons, trending_bot, your auntifa liza 🇵🇷 🦛 🦦, don Elías (como los buses) 🥨, Oblomov, Amandine y Glyn Moody reshared this.
Paul_IPv6
in reply to Paul Cantrell • • •Richard Johnson
in reply to Paul_IPv6 • • •@paul_ipv6
Standard practice among tape lib operators where I've worked: Multiple copies of each file/object. Min 3 matching reads from recent tapes + older tapes to create a new copy. Tapes regularly refreshed w/ new (usually higher capacity) cartridges.
Then along comes Orrible, "We are cancelling your tape orders. Move your data to our cloud where we'll keep one copy at much higher opex spend for you!"
This is not called doing business. Rather, it's doing business to you.
Carlos Solís likes this.
Paul_IPv6
in reply to Richard Johnson • • •there was a period where tapes were slow/expensive enough and disk was cheap where lots of folks bought into RAID as good enough for one copy. problem was, many didn't do their homework about what the differences and risks were of the various types of RAID. also doesn't solve the "what happens if your primary site is destroyed".
Climate Jenny 2.1
in reply to Paul Cantrell • • •Enola Knezevic
in reply to Climate Jenny 2.1 • • •Seiðr
in reply to Paul Cantrell • • •Oblomov reshared this.
Bruno Philipe
in reply to Paul Cantrell • • •Elizabeth 🏳️⚧️
in reply to Paul Cantrell • • •the thing is that this process is inherently brittle and myriad things can go wrong during copying, especially if the media is marginal. It’s inevitable that at least some loss is going to occur until we have a durable means to store data, and for older data rarely copied, it’s probable some could be lost forever. This is already happening with floppies.
This also presumes that the intersection of “someone or some people interested in preserving this data” and “someone or some people who are able to replace the old media” are the same people, which is also not guaranteed.
Elizabeth 🏳️⚧️
in reply to Elizabeth 🏳️⚧️ • • •an example of how brittle this process is:
Most of Google Video is essentially gone forever. So is most of MySpace and countless irreplaceable songs. The main reason being that those interested in preserving it and those with the ability to preserve it just… did not intersect, which is the fate that usually happens to most data. Usenet’s archives are also very incomplete.
Alex@rtnVFRmedia Suffolk UK
in reply to Elizabeth 🏳️⚧️ • • •@Elizafox
a large amount of British TV aimed at teens/young adults, especially that which contained the slightest bit of "pop music" from *1990s/00s* is likely permanently gone.
Not for lack of archiving technology (it was around in 1990s, including digital video storage by late 90s, but because there was so much arguing over over copyright that archives have been allowed to deteriorate or even deliberately bulk erased to prevent them falling into "the wrong hands" ( TV companies would rather destroy it than risk a lawsuit because hoarded master tapes ended up on YouTube)
Elizabeth 🏳️⚧️
in reply to Alex@rtnVFRmedia Suffolk UK • • •legraLeGra
in reply to Paul Cantrell • • •I just finished up working on the Artemis 2 archiving (it’s under review now for the public October data release) — this is how nasa handles digital data for long long term archiving:
pds.nasa.gov/home/about/
And how long is long term archiving. Recall the Silurian hypothesis.
PDS: About the PDS
pds.nasa.govGeorge B
in reply to Paul Cantrell • • •My current solution on my desktop is a btrfs partition with a bunch of random drives and a nightly job that rebalances the data across them if it needs to be (time boxed to 1 hour per night, restarts the night after it ends)
Its not quite zero touch but it does mean that if I replace a drive then it will slowly get redundant copies of data over the next few nights. The big failure mode is that it is not resistant to multiple drives failing before the first one is rebalanced
Lee Hauser
in reply to George B • • •George B
in reply to Lee Hauser • • •Yes that is true but that's the same failure mode of every single site storage system, including libraries as in the original quoted post. Having additional redundancy in multiple locations is important for some information but each location needs to have that kind of maintenance happening for properly long term storage
Joel VanderWerf
in reply to Paul Cantrell • • •"Keep circulating the tapes."
#mst3k
David McMullin
in reply to Paul Cantrell • • •MidgePhoto
in reply to Paul Cantrell • • •Binary in dots printed into Titanium might come moderately close, perhaps?
The Eternity Service as described by Ross Anderson is another element, protecting against loss of a physical copy archive, and against some other attacks.
#Scifi The trouble in #VernorVinge 's #AFireAcrossTheDeep SF novel, one of the two set in the #ZonesOfMind begins with Stromli Realm's team opening an ancient archive they discovered in the Low Beyond...
Justin Derrick
in reply to Paul Cantrell • • •Cecilia | MeraOrd
in reply to Justin Derrick • • •U+002C
in reply to Paul Cantrell • • •Jef
in reply to Paul Cantrell • • •The question is what happens when we won't be able to afford renewing the medium. Who gets to choose what to keep and what to loose?
MacCruiskeen
in reply to Paul Cantrell • • •Richard Rathe
in reply to Paul Cantrell • • •Quote from #SciAm article on this topic back in the 1990s/early 2000s...
"Digital information lasts forever--or five years, whichever comes first." - Jeff Rothenberg
Amandine reshared this.
𝔸𝕟𝕔𝕚𝕖𝕟𝕥 𝕊𝕠𝕦𝕟𝕕𝕤 🔉
in reply to Paul Cantrell • • •Glyph
in reply to Paul Cantrell • • •Guitarsophist
in reply to Paul Cantrell • • •MegatronicThronBanks
in reply to Paul Cantrell • • •3D-printing using ECCs onto some kind of ceramic alloy that's super hard and super temperature resistant.
Mans R
in reply to MegatronicThronBanks • • •MegatronicThronBanks
in reply to Mans R • • •Mans R
in reply to MegatronicThronBanks • • •MegatronicThronBanks
in reply to Mans R • • •@mansr Why would you need to? You only need to describe the format.
It's not a DVD or DLT, that requires precise physics to decode.
Mans R
in reply to MegatronicThronBanks • • •MegatronicThronBanks
in reply to Mans R • • •@mansr In a few human languages, on pages scattered throughout to protect them. With diagrams.
We did it with a gold record, and that was for aliens.
Oggie
in reply to Paul Cantrell • • •There were some real research systems looking into this, and it makes me incredibly sad that they basically have all stopped because it's not immediately profitable.
Living archives are always the functional solution, but it should still be something people work on!
The concept of immutable records should at least be appealing....
UnlikelyLass
in reply to Paul Cantrell • • •Jonathan Hendry
in reply to Paul Cantrell • • •@overholt
Buddhist non-attachment to old files is a pretty good strategy I find.
Louis Aslett
in reply to Paul Cantrell • • •Project Silica
Richard Black (Microsoft Research)cratermoon
in reply to Paul Cantrell • • •OddOpinions5
in reply to Paul Cantrell • • •* [ see note ]
yes, I figured this out in 1990, which is why every 5 years or so I buy a bigger hard drive, which costs less, and just copy everything to the new hard drive
works for me
not elegant or very user friendly, but it is just me
*getting shoulder strain patting myself on the back
ARGVMI~1.PIF
in reply to Paul Cantrell • • •Clark Shishido
in reply to Paul Cantrell • • •LOCKSS (Lots of Copies Keep Stuff Safe) tries to mitigate the problem of integrity when copying files forward. TL:DR - min 4 copies with continuous file integrity checks.
lockss.org
(edit: it's mentioned in the Gizmodo article)
LOCKSS Program
LOCKSS ProgramEric Lawton
in reply to Paul Cantrell • • •I did the same for the Alberta Provincial Archives.
Same conclusion.
Most of their material was on microfilm, which was beginning to perish. Early 1980s.
Clark Breyman (he/him)
in reply to Paul Cantrell • • •Carlos Solís
in reply to Paul Cantrell • •Jernej Simončič � likes this.
Marty Fouts
in reply to Paul Cantrell • • •Chuck
in reply to Paul Cantrell • • •LOCKSS Program
LOCKSS Programliquidlamp
in reply to Paul Cantrell • • •Oblomov reshared this.
Rob
in reply to Paul Cantrell • • •Jernej Simončič �
in reply to Rob • • •Svavar the Neurospicy
in reply to Jernej Simončič � • • •@jernej__s @Eurospoofer
This is already a problem with proprietary systems where the vendor goes out of business but the data has to be retained and usable for at least 7 years for legal reasons.
I worked at a bank in this situation and it was a giant pain because the system couldn’t run on any modern / approved operating system so it couldn’t be hosted in any location without massive security issues and mitigations.
So the first step is open data formats that can be maintained
Jonas
in reply to Paul Cantrell • • •Svavar the Neurospicy
in reply to Paul Cantrell • • •I managed the computers for my dad’s video store in the late twentieth and early 21st centuries (1995 - 2007).
The store’s rental software was based around a MS SQL server in RAID 1 and I replaced both drives annually just to be safe. Even then, I ended up having to do an emergency recovery because I moved to another country and let the replacement schedule slip.
So the only way to store data long-term is to re-establish the Knights Templar and task them with guarding it.
Brian M Knoblock 🇺🇸🇳🇴 🦊
in reply to Paul Cantrell • • •For me: nine external HDDs that I’ve collected one a year for the past ten before prices went up. Rotate around which one is updated, using SD cards as the source for the copy, each card concentrating on a specific directory or directories.
It certainly is a process that has to be done on a schedule or it will never consistently be done.
Anna
in reply to Paul Cantrell • • •DeterioratedStucco
in reply to Anna • • •Same.
billy joe bowers 🗽
in reply to Paul Cantrell • • •This was the concern in music recording early in my career when digital was becoming more of a thing. Basically I feel like everyone kind of just gave up on it.
Not really, but all media was always time limited and fragile, so it's nothing new. People try, but I feel like everyone knows you can only do so much.
Mr.Mark
in reply to billy joe bowers 🗽 • • •I listed to a Perez Prado LP this morning while making breakfast. Recorded about 75 years ago, well-kept, nearly perfect condition.
I’m sure that if it is stored properly and taken care of those grooves will still sound good 100 or more years from now.
When it comes to audio, still nothing beats vinyl.
(so, if you ask me, we should save all the rest of the oil that’s in the ground for records and stop burning it for fuel!)
billy joe bowers 🗽
in reply to Mr.Mark • • •Pretty sure that's how some have decided that's the best way to preserve music. Smithsonian?
OddOpinions5
in reply to Paul Cantrell • • •miki
in reply to Paul Cantrell • • •ohir
in reply to Paul Cantrell • • •> no storage medium that met their long-term preservation goals
May I suggest a clay disc? ;)
Charlie Stross
in reply to Paul Cantrell • • •Guitarsophist
in reply to Charlie Stross • • •@cstross Well, considering the current state of humanity, 1,000 years is probably more than enough. What we need is a printer with baked clay output and huge storage crypts in Iraq or someplace. Such crypts would have a big advantage over data centers in that they would need no electricity or water at all.
Of course in quantum cosmological terms, information is never destroyed. So there's that.
@inthehands @blotosmetek @overholt
zbrown
in reply to Paul Cantrell • • •Todd Knarr
in reply to Paul Cantrell • • •Any media intended for long-term storage needs to use physically-readable markings. One constant is that we'll have some way of taking a picture of something and turning it into an image file. At that point, software can decode the markings. That doesn't solve all the problems, but it should make them solvable.
The alternative, as you say, is continuous copy-forward.