An archive in daily use, built to outlive its software.
Remanence is a set of open specifications, and a reference implementation, for writing large collections to LTO tape and getting them back. It came out of an archive that has to be two things at once: a production store that people pull material from every day, and a preservation store that must still be readable in thirty years. The design aims at both: retrieval fast enough for daily work, and tapes that stay understandable and recoverable after the catalog, the application, the vendor, and the people who set it up are gone.
The reference implementation has run on one virtual tape library and one physical library with one drive generation. It is not in production anywhere, including in the archive it was built for. Do not make it the only copy of anything you cannot afford to lose.
The three format specifications are review drafts, open for comment until 30 April 2027 and frozen on 31 July 2027. Tapes written before the freeze carry no compatibility guarantee. The tape layout described on this site is the revised one written by the development branch; the tagged release still writes the earlier layout, as explained under the specifications. The full position on maturity is on the Test it page.
At a glance
- Fixity
- A SHA-256 checksum for every file, stored inside the archive
- Damage recovery
- Reed–Solomon parity, kept apart from the data. The default costs 3.1% of capacity and survives 512 MiB of contiguous damage
- Container
- POSIX pax tar. A standard
tarextracts the files - Partial restore
- Any byte range of any file, located by arithmetic
- Catalog
- Not needed for recovery. A finalized tape describes itself
- Encryption
- Optional, per object, post-quantum hybrid
- Licenses
- CC-BY-4.0 specifications, CC0 test vectors, Apache-2.0 software
Who this is for
Tape keeps bits for a long time, and archives are seldom lost because the tape faded. The usual causes are more ordinary. The application that wrote the tapes is discontinued, and nothing else can read its format. The catalog that knew which file sat on which cartridge dies with the server it lived on. One latent bug in the writing software corrupts every copy in the same way. The one person who understood the system leaves. Remanence is designed for those conditions from the beginning.
It is for people responsible for digital material that has to stay recoverable for decades: video, audio, photographs, research data, institutional records. It will matter most if you also need material back the same day, one clip out of a large master rather than a whole cartridge, and if you want to be able to check for yourself that what came back is what went in.
Nothing here has to be taken on trust. The specifications, the software, the test vectors, and the recovery procedures are all published, so every claim on this site can be examined directly.
Three formats, three jobs
The design is split into layers so that the archive package, the tape layout, and encryption are separate concerns. Each can be replaced or revised without the others. None of it is locked inside one application.
REM-OBJECT is the package. It is a tar archive with extra rules: a
fixed order, a checksum for every file, and a manifest at the end that records
each file's path, size, checksum, and position. Because every file starts on a
fixed grid, a reader can work out where any byte range lives and fetch just
that. A standard tar program extracts the files with no special
software.
REM-PARITY is the arrangement on the cartridge. Packages are written one after another, with Reed–Solomon recovery data stored apart from what it protects, so a damaged stretch of tape can be rebuilt from what survives. A small block at the start identifies the tape, and three separated copies of the complete index at the end let a bare cartridge be mapped and verified with no catalog at all.
REM-ENCRYPT is optional and sits on top. It seals a package for a copy that leaves your control, using a hybrid of conventional and post-quantum cryptography, and it leaves the durable package underneath unchanged. Most of an archive should not be encrypted: a lost key turns an intact archive into an unreadable one. It belongs on the off-site copy, not on every copy.
A component, not a product
Remanence does one job: it writes archives to tape and reads them back, verifiably, through a complete API and command line. It is not a hierarchical storage manager. The catalog people search, the policy that decides how many copies go where, the scheduling, and the disk and cloud copies of a real archive belong to a larger system above it.
Archive software is usually bought as one application that does everything, which means it also owns everything, including the on-tape format. A component avoids that dependency. It has an open format below it and open interfaces above it, so it can be wrapped in a larger system, or replaced, without touching the tapes. In our own evaluation environment Remanence sits under an orchestration layer that holds our institution's rules. That layer is ours and is not published; every institution's rules are its own.
What has been tested, and what has not
The practice these formats formalize, plain tar with a full catalog appended to each tape, has run in our production archive for years with no format-related corruption. The REM formats add deterministic encoding, parity, and catalog-free recovery to that practice. The implementation has been exercised on the equipment described on the Test it page and on nothing else.
One library family and one drive generation is a narrow basis for a format meant to last decades. What the project publishes so that others can widen it:
- The specifications, and a plain-language companion to them.
- The Rust reference implementation.
- A second reader in Python, written from the specifications rather than from the Rust code.
- Test vectors with pinned checksums, including deliberately malformed objects and the errors a correct reader must produce for them.
- Machine-checked proofs, in Lean, of nine small pieces of the format logic, with a public inventory of what is and is not proved.
Early users should keep an independent copy of anything irreplaceable and restore real material from physical cartridges regularly. Why the project was built, and how, is on the About page.