Built from twenty years of running a large video archive.

The work is informed by two decades of operating a multi-petabyte institutional video archive, mostly irreplaceable material, migrated across successive LTO generations. The design grew from what went wrong in that time and what the archive kept needing: damaged media, dust, unreadable regions of tape, hardware and software going out of support, dependence on a catalog, migration after migration, checksum verification, and restoring one portion of a very large video file rather than the whole thing.

The work is published under a project name because authorship is not meant to be the basis of trust. The specifications, the software, and the evidence should stand or fall on independent examination.

An archive should not depend on the company that wrote its software. Many institutional archive products combine recurring license costs with proprietary on-tape formats and vendor-controlled workflows. The cost matters, particularly for mission-driven institutions on constrained budgets. The deeper risk is losing access to the archive if the vendor, the product, the license server, or the support arrangement disappears. Remanence exists to remove that dependency at the level of the format, where it does the most damage.

Existing tools also tended to solve one side of the problem. Preservation systems prioritize durable storage; production systems prioritize fast access. We needed one design that treats tape as the sequential medium it is, stays readable without any one application, verifies restored files, rebuilds a cartridge's contents without the catalog, repairs damaged regions from parity, and retrieves a segment of a very large file. No available format combined those properties, so the work began.

Until recently, building something like this took years and specialist knowledge. Working with tape at this level means low-level SCSI and the particulars of LTO drives and libraries, ground obscure enough that only a few large suppliers took it on, and priced accordingly. That has changed. The standards are openly available, suitable hardware is cheap on the secondary market, and current AI models can carry much of the implementation work.

The models cannot supply the judgment of what an archive actually needs and where it must not cut corners. That judgment came from twenty years of running the archive. With it in hand, the specifications and the reference implementation were developed and tested against a virtual tape library in about six weeks.

The implication reaches beyond this project. Institutions no longer need to depend on vendors charging tens of thousands of dollars a year for this class of software. The speed has a cost. It cannot buy the one thing an archival format most needs, which is decades of use in many hands, and so it calls for more outside inspection and testing rather than less.

Remanence has not accumulated decades of deployment across many institutions, hardware vendors, drive generations, and independent implementations. That limitation is real, and the response is to publish everything needed for others to examine the design. What that evidence consists of is listed on the overview; the licenses and persistent identifiers are listed with the specifications. The project has been tested seriously and still needs broader independent testing. Questions, corrections, implementation results, and independent test results are welcome.

Read the specifications → ·  View the source → ·  Contact ArchiveTech →

Magnetic remanence is the magnetization a material keeps after the external field is removed, which is roughly what tape preserves after the write head has passed. The formats take the REM- prefix from it. REM-OBJECT defines the preserved package, REM-PARITY protects and maps packages on tape, and REM-ENCRYPT optionally seals copies stored off-site. Remanence is also the name of the Rust reference implementation that reads and writes all three.