How the archive stays readable, verifiable, and recoverable.

A REM-OBJECT is a tar archive, in the POSIX pax form, with additional rules. The base was chosen because tar is old, published, and available everywhere; the LTFS page gives the reasoning. A standard tar program extracts the files from a REM-OBJECT with no knowledge of Remanence. The extra rules are what make it an archival format.

The first rule is that the same input always produces the same bytes. Order, padding, and metadata representation are fixed, so two independent implementations given the same files should produce identical archives. A changed byte therefore means damage or an implementation error.

The second rule is that every file's data begins on a fixed grid, and the package ends with a manifest recording each file's path, size, SHA-256 checksum, and position. Where a file sits, or where a byte range inside it sits, follows by arithmetic from the manifest. A reader can go straight to that region and read only what it needs. For a large video master this means one clip can be restored without pulling the whole file off tape first.

Integrity is checked at three levels. Each file has its own checksum, the manifest has one, and the whole object has one. The first two live inside the object, and the tape's index repeats the manifest checksum, so a bare cartridge can be verified file by file with no catalog. The checksum over the whole object cannot sit inside the bytes it covers, so it lives in the catalog, where it lets a stored copy be checked wholesale without reading into its contents. Because the manifest travels with the object, a reader can list the contents of any object and rebuild a lost catalog from the objects themselves.

Two further choices follow from the grid. The package is never compressed as a whole, because compressing the stream would make every position depend on decompressing everything before it. Material that benefits from compression should be compressed before archiving and kept in that form. And the package records only what belongs to a file rather than to the machine it came from: modification times and extended attributes are kept, and the executable bit; ownership and the other permission bits are not, because they describe a system that will not exist when the archive is read. The extended-attributes reference has the full rules, including how attributes are handled on restore.

Tape drives are built to stream. A drive that has to start and stop for every small file spends its time repositioning instead of writing, throughput collapses, and the mechanism wears. The catalog has the same problem in a different form: millions of records for tiny files that nobody will ever ask for individually. Real collections contain a great deal of such material: sidecar files, proxies, and sometimes the contents of a whole workstation archived alongside the masters.

The format therefore lets one REM-OBJECT hold many members. In such a bundle every member is still named and checksummed in the manifest, and every member's data still starts on the chunk grid, so one member, or a byte range inside it, can be restored without reading the rest. With the catalog available, a reader goes straight to the member's blocks. Without it, the tape index records where the bundle's manifest sits; the reader fetches the manifest, which is small, and then seeks to the member. The tape index and the catalog spend one row on the bundle rather than one per member.

Alignment has a cost, and it sets the lower limit of what a bundle is for. Each non-empty member occupies at least one chunk, 256 KiB to 1 MiB depending on the block size chosen for the tape. A bundle of files that are themselves a few hundred kilobytes or more wastes little. A bundle of a million one-kilobyte files would waste almost all of its space. For that case, and for files the format cannot represent natively, the reference implementation packs a subtree into one ordinary member using plain tar, with a small index member beside it. The index lists every inner file with its offset, length, and its own SHA-256 checksum. A single inner file is restored by reading the index, then reading only that file's byte range from the packed member, then checking its checksum, and this works on encrypted objects too. What is given up is that inner files are not members of the archive in their own right, and byte-range restore inside an inner file is not offered. The convention is described in an informative appendix of the REM-OBJECT specification.

Which arrangement a file gets is decided above Remanence. A video master gets an object of its own. A folder of sidecar files becomes a bundle. A tree of a million tiny files becomes a blob. The reference command line does not make that choice for you: rem put writes one object per file, and a bundle is built first and then stored as one object. In our own environment the orchestration layer applies the policy.

An LTO drive already writes with very high accuracy. It verifies data as it writes, and its own error correction makes undetected bit errors extremely rare. Parity in REM-PARITY is not there to make up for the medium. It is there for failures that arrive from outside the drive's control: dust, a scratch, rough handling, a stretch of tape that has become unreadable as a region.

As objects are written, Remanence computes Reed–Solomon recovery data for them and writes it to parity files of its own, placed apart from the data they protect, so that one localized failure is unlikely to take both. Every object is covered whatever its size. With the default settings, parity costs 3.1% of capacity and can rebuild up to 512 MiB of contiguous damage.

Each object is written as a separate tape file, meaning a run of blocks closed by a filemark. A drive can position to any block number, and that is how a reader reaches a member or a byte range. The filemarks give each object a boundary the drive itself understands, so an object can be read, bounded, and repaired as a unit. Parity is never mixed into an object, and an object read back is exactly the object that was written.

The bootstrap at the start is a small fixed-size block with a recognizable signature. It records the cartridge's identity, its block size, and its parity scheme, and deliberately nothing else: no list of objects, because that list would have to be rewritten as the tape grew. When the cartridge is finalized, the complete inventory is written three times at the end, with a gap of about a gigabyte between copies so that one damaged region cannot reach all three. A reader starts from the end of the tape, checks copy C, and falls back to B and then A. Any one intact copy is enough to map the tape, locate every object, verify it, and find the parity available for repair.

Until a cartridge is finalized, the record of what has been committed to it is a journal on the writing host, described in the next section. If all three index copies were ever lost, the bootstrap still allows a reader to walk the tape from the beginning and find every object, though matching each one to its identity may need the host's records.

Remanence differs from plain tar in one respect an operator should understand before relying on it: it keeps state on the host. There are four kinds, all ordinary files under directories the operator chooses.

• Journals. One per tape, with a checkpoint journal beside it. These are the authority for a tape that is still open for writing: what has been committed to it and where its physical end is. They are small and only ever appended to.
• The audit log. An append-only record of every state-changing operation, in daily segments.
• The catalog. One SQLite file listing the tapes, objects, and files the site knows about and where each one is. It is a projection of the journals and the audit log, and the command rem rebuild-catalog-from-journals regenerates it from them. It exists to make everyday searching fast and can be treated as disposable as long as the journals and audit log survive.
• Caches. Per-tape catalog caches and a read-order calibration store, both regenerable.

In practice: back up the journal and audit directories, because they are what the catalog is rebuilt from; the SQLite file itself need not be. A finalized cartridge carries everything needed to identify it, list its contents, verify them, and restore its objects with the published tools, so a site left with only cartridges can recover its data. What current software cannot yet do is rebuild a working catalog from the tapes alone, since the rebuild reads journals rather than tapes; that is listed as future work. A tape that was still open for writing when its journals were lost can be read and scanned from the beginning, but not appended to.

Encryption is specified separately from the durable format, and it is optional. A plain REM-OBJECT is unchanged when the cryptography is revised, so the preserved package is never tied to one generation of cipher, and an archive that needs no confidentiality loses nothing by leaving encryption out. We recommend leaving it out for every copy except the one that leaves your control. The reasons are on the encryption page.

Each encrypted object gets a fresh key. That key is sealed to one or more recipients (up to eight) using HPKE, the standard Hybrid Public Key Encryption scheme of RFC 9180, with the X-Wing key encapsulation. X-Wing combines the conventional X25519 exchange with the post-quantum ML-KEM-768 standardized in FIPS 203. Remanence pins X-Wing at revision 10 of its March 2026 draft. That standalone draft has since expired, and the same construction is now carried by the IETF's Crypto Forum Research Group under the name MLKEM768-X25519 and registered for HPKE, under identifier 0x647a, in a standards-track draft. A break in either half on its own does not open the object. Sealing to more than one recipient is recommended, so that one lost key does not lose the object.

The sealed key travels in the object's header, so the writing machine keeps no lasting secret, and the header records the cryptographic choices so that they cannot be silently downgraded. The grid is preserved under encryption, which means a byte range can be restored from the ciphertext directly. Decrypted, the result is an ordinary REM-OBJECT.

A new implementation can be tested against known answers. The published test-vector archive contains valid objects with pinned checksums, and deliberately malformed ones with the exact error a correct reader must produce for each. Anyone writing a reader should start there: make the valid cases pass, then make every malformed case fail in the required way. Vectors expose ambiguous wording in the specification, incompatible implementations, and readers that accept damaged structures they should reject.

77be73e780e9ff2c265c8357b6ba684b4c69800213820ae1331850f742b1d83d  remanence-test-vectors.tar

A second reader, written in Python from the specifications without calling the Rust code, re-derives the vectors independently. That is independence inside the project, not review by an outside institution, and the open items list says which outside review would help most.

Nine small pieces of the format logic go a step further than testing. A test checks particular inputs. A proof covers every possible input. The checksum loop, the parity arithmetic, the layout calculations, and the manifest bookkeeping have been translated into a form the Lean proof assistant can reason about, and their correctness has been proved and machine-checked. Each proof covers a small, pure piece of logic and states exactly where its guarantee ends. This is not a claim that the whole system is proved. Guards in the test suite tie each proved piece to the production code so the two cannot drift apart, and the inventory of what is and is not proved is public.

• REM-OBJECT Core Format, 1.0.0-draft.3. The archive package and its integrity.
• REM-PARITY, 1.0.0-draft.2. The published tape layout (generation 1, see below).
• REM-PARITY, 1.0.0-draft.5, in preparation. The generation-2 layout shown in Fig. 2, which the current development software writes.
• REM-ENCRYPT, 1.0.0-draft.3. Optional object encryption.
• The formats, explained. A longer plain-language companion to all three.
• Software release DOI. The reference implementation, with a snapshot of the documents. Each specification has a reserved concept DOI of its own; the first document deposits are still pending.

Specifications: Creative Commons Attribution 4.0. Test vectors: Creative Commons Zero. Reference implementation: Apache License 2.0.

They are review drafts. Comments are open until 30 April 2027, and the documents freeze on 31 July 2027, one year after publication. The three months in between exist so that changes made in response to review are published, and visible, before the text is fixed. Until the freeze, breaking changes are allowed, and a tape written from a pre-freeze draft carries no guarantee that later software will read it. Keep the source material for every evaluation tape until then. After the freeze, an on-tape structure that a frozen format validates will never be invalidated; incompatible successors will live beside it under a new major version.

REM-PARITY is between two layouts. The published text describes an earlier design, generation 1, in which the tape's self-description block was repeated at intervals down the tape and recovered by rescanning those copies. Development replaced it with a simpler arrangement: one bootstrap at the start and three separated copies of the index at the end, which is what Fig. 2 shows and what this page describes. That generation-2 design is implemented on the development branch and written up in the draft in preparation, but its text is not yet the published one, and its fuzz-testing and supervised physical-tape gates are still open. The tagged v0.1.0 release writes generation 1; the development branch writes and reads only generation 2. The published test vectors describe generation 1. An evaluator should expect this and should read the open items before writing tapes.

No standards body has reviewed these documents. There is no ISO number and no RFC. They were written by the people who wrote the implementation, so the stability guarantee is the project's own undertaking rather than anyone else's approval. It is worth something because it is narrow enough to check, and because the rules for how the documents may change are written down in full: how the formats change explains them, and the versioning register lists every versioned component. Command-line interfaces and operational workflows are not covered by the guarantee while the implementation is young. A disagreement between prose, vectors, and implementation is a finding the project wants: check the known open items, then open an issue.

BagIt and OCFL solve an adjacent problem. Both describe how preserved objects live on a filesystem: a bag of files with checksum manifests, or a versioned object tree. Neither says anything about tape. They do not cover how bytes are laid out on a cartridge, how damage is survived, how a range inside a large file is reached without reading everything before it, or how an archive is recovered when no filesystem and no catalog survive.

The REM formats cover that tape layer, and the two compose without friction. A BagIt bag or an OCFL object can be archived as the contents of a REM-OBJECT without either standard noticing the other. An institution with existing BagIt tooling loses nothing by keeping it.

Fixity

Evidence that a file's bytes are unchanged since it was archived. Here, a SHA-256 checksum of every file, stored inside the package and checked on restore.

Parity

Extra data computed from the archive, here by a Reed–Solomon code, from which a damaged or unreadable region can be reconstructed. Stored apart from the data it protects.

Catalog

The index an archive uses to find things: which file is on which tape, at which position. In Remanence it is one SQLite file, rebuilt from the journals and audit log rather than trusted on its own.

Journal

A small append-only file per tape on the writing host, recording what has been committed to that tape. The authority for a tape that is still open for writing.

Catalog-free recovery

Mapping, verifying, and restoring a cartridge using only what is written on the tape itself.

Self-describing tape

A cartridge that carries its own inventory: what it holds, where each file sits, and how to check it.

Byte-range restore

Retrieving part of a file, for example one clip of a large video master, without reading the whole file first.

Manifest

The final member of a REM-OBJECT: a table of every file's path, size, checksum, and position within the object.

Tape file

A run of blocks closed by a filemark on a cartridge. A drive can position to any block number or to a filemark; Remanence writes each object as one tape file so the object has a boundary the drive understands.

Finalization

Closing a cartridge for writing. Remanence then writes three separated copies of the complete index at the end of the tape.

Review draft

A specification published for public comment that may still change incompatibly. The REM specifications are review drafts until 31 July 2027.