Built from twenty years of operating a large video archive.

The work is informed by two decades of operating a multi-petabyte institutional video archive — predominantly irreplaceable material, migrated across successive LTO generations.

The design grew from real failures and constraints: damaged media, dust exposure, unreadable tape regions, hardware and software obsolescence, catalog dependence, migration requirements, checksum verification, and the need to restore portions of very large video files.

The work is published under a project name because authorship is not intended to be the basis of trust. The specifications, software, and evidence should stand or fall through independent examination.

The archive should not be dependent on the company that wrote it. Many institutional archive products combine recurring licence costs with proprietary on-tape formats and vendor-controlled workflows.

Cost matters, particularly for mission-driven institutions on constrained budgets. The deeper risk is losing independent access to the archive if a vendor, product, licence server, or support arrangement disappears. Remanence was created to reduce that dependency at the format level.

Existing tools tended to solve only one side of the problem. Preservation systems often prioritize durable storage; production systems prioritize fast access. The required architecture needed to do both:

• Treat tape as a sequential medium rather than pretending it is a disk.
• Remain readable without one proprietary application.
• Verify restored files independently.
• Rebuild cartridge information without the central catalog.
• Recover damaged regions using separate parity data.
• Retrieve a needed segment from a very large file.

No available format combined these properties in the required way, so the work began.

Until recently, building something like this took years of development and specialist knowledge. Working with tape at this level means low-level SCSI and the particulars of LTO drives and libraries — ground arcane enough that only a few large suppliers took it on, and priced accordingly.

That ground has shifted. The standards and specifications are openly available, suitable hardware is inexpensive on the secondary market, and current AI models can carry much of the implementation work. What they cannot supply is the judgment of what an archive actually needs and where it must not cut corners: which failures matter, which dependencies are unacceptable, how recovery must work. That judgment came from twenty years of operating the archive. With it in hand, the specifications and the reference implementation were developed and tested against a virtual tape library in about six weeks.

The implication reaches beyond this project: institutions no longer need to be dependent on vendors charging tens of thousands of dollars a year for this class of software. But the honesty has to cut both ways. Speed cannot buy the one thing an archival format most needs — decades of use in many hands. That is the trade-off, stated plainly, and it is why the speed is a reason for additional inspection and testing rather than a substitute for it.

Remanence has not yet accumulated decades of deployment across many institutions, hardware vendors, drive generations, and independent implementations. That limitation is real. The response is to publish the evidence needed for others to examine the design:

• Open specifications.
• Pinned test vectors.
• A second reader written independently from the specification.
• Physical and virtual tape testing.
• Explicit malformed-input behavior.
• Source code and issue tracking.

The project has been tested seriously and still needs broader independent testing.

Magnetic remanence is the magnetization a material retains after the external magnetic field is removed — roughly what tape preserves after the write head has passed. The work takes its name from that property, and the formats carry its REM- prefix.

REM-OBJECT defines the preserved object. REM-PARITY protects and maps those objects on tape. REM-ENCRYPT optionally seals sensitive copies stored off-site. Remanence is also the name of the Rust reference implementation that reads and writes all three.

Specifications: Creative Commons Attribution 4.0. Test vectors: Creative Commons Zero. Reference implementation: Apache License 2.0. Versioned releases carry persistent DOI identifiers.

Questions, corrections, implementation results, and independent testing are welcome.

Read the specifications → ·  View the source → ·  Contact ArchiveTech →