Three open specifications divide the archive into separate responsibilities. REM-OBJECT defines the archive package. REM-PARITY defines how it is protected and located on tape. REM-ENCRYPT optionally protects sensitive copies stored outside physical control. This page explains the governing ideas in plain language; the published specification documents remain the normative technical sources.
REM-OBJECT starts with a format that future systems are likely to understand. A
REM-OBJECT is a deliberately constrained POSIX pax tar archive: a standard pax-aware
tar program can extract its files without Remanence or another
proprietary application. The archival value comes from the additional rules governing
order, metadata, checksums, addressing, and the final manifest.
The same input produces the same archive bytes. REM-OBJECT fixes ordering, padding, and metadata representation, so two independent implementations given the same input should create the same output. This makes differences meaningful: a changed byte indicates damage or an implementation error rather than harmless formatting variation.
Part of a large file can be retrieved without reading the whole object. Every file's payload begins on a predictable chunk boundary, and the manifest records enough information to calculate where a file — or a byte range inside it — resides. A reader can therefore seek directly to the required region. For a large video master, this allows one required clip to be restored without retrieving the entire file first.
Compression is kept outside the archive container. Compressing the entire archive stream would make the position of later bytes depend on decompressing everything before them, which would defeat direct byte-range addressing. Material that benefits from compression should be compressed before archiving and then preserved in that form.
The object carries the information needed to understand and verify itself. The final member is a deterministic CBOR manifest containing each file's path, size, SHA-256 checksum, and block address. Integrity is checked at three defined levels: individual files, the manifest, and the complete object. The file and manifest digests travel inside the object itself. The whole-object digest necessarily travels beside it — a digest cannot sit inside the very bytes it covers — recorded in the catalog for every copy, echoed with the manifest digest in the tape's bootstrap records, and carried inside the authenticated encryption envelope for an encrypted copy. Because the description travels with the object, a reader can list its contents and rebuild a lost catalog from the archived objects themselves.
Tape drives are built to stream. A drive made to start and stop for every small file spends its time repositioning instead of writing — throughput collapses, and the constant back-and-forth adds mechanical wear. The catalog suffers its own version of the same problem: millions of database records for tiny files that nobody will ever ask for individually. Real collections contain such material more often than anyone plans for; sooner or later, someone archives an entire system drive alongside the master material.
Remanence therefore bundles, configurably, before anything reaches tape. Small files are packed into a single archive object — which is itself plain tar, so nothing about recoverability changes — and written in one continuous stream. The object's manifest still lists every member with its path, size, checksum, and position, so per-file fixity and partial restore are preserved. What changes is where the fine-grained record lives: the catalog keeps one row for the bundle, and the per-file detail travels inside the self-describing object.
Which files bundle and which keep individual identity is policy, not accident. A video master deserves an object of its own; a folder of ten thousand small sidecar files does not. The working rule from our production experience: files that will be addressed individually keep individual identity — the rest are carried in bulk, deliberately.
Separate recovery data can rebuild unreadable regions. As archive data is written, REM-PARITY calculates Reed–Solomon recovery information and stores it in parity tape files of its own. Data and the recovery information protecting it are placed apart, so one localized media failure is less likely to destroy both. Every object receives protection regardless of its size. The technical detail: parity is computed over GF(2⁸) with a Cauchy generator.
The placement discipline is strict. Each archive object remains a distinct tape file, separated by standard tape filemarks, and parity is never interleaved inside an object — an object read back is exactly the object that was written. Positioning, inspection, and recovery therefore all operate at the tape-file level, using the mechanisms every tape drive already provides.
The cartridge carries its own map. A bootstrap block is a single fixed-size block with a recognizable signature, so a reader can find one by scanning even a tape it knows nothing about. Inside it are the cartridge's identity, the block size, the parity scheme in use, and a verified pointer to the tape's directory — the list of every tape file on the cartridge, which of them are objects and which are parity, and where each one sits. Copies are written at the beginning of the tape, at checkpoints along it, and at the end, so damage to one region does not take the map with it. From any surviving copy a reader can map the tape, locate objects, verify their integrity, and identify the parity available for repair. The operational database makes normal searching faster, but it is not the only surviving description of the cartridge.
Encryption remains optional and separate from the durable archive format. The underlying REM-OBJECT does not change when cryptographic methods are revised, which prevents the preserved object format from becoming permanently tied to one generation of encryption.
Each encrypted object receives a new key. That key is sealed using HPKE and the X-Wing hybrid mechanism, combining the classical X25519 with the post-quantum ML-KEM-768, so a break in either component on its own does not break the encapsulation.
The wrapped key travels in the object header, addressed to one or more recipients — wrapping to more than one is recommended, so a single lost key does not lose the object. Because the key travels with the object, the writing machine keeps no lasting secret, and the header binds the cryptographic choices so they cannot be silently downgraded. The chunk grid is preserved under encryption, so partial restore runs directly on the ciphertext. Decrypted, the object is a standard REM-OBJECT.
A new implementation can be tested against known answers. The specification release includes a fixed collection of valid and malformed archive examples with a published SHA-256 checksum. An independent implementation should reproduce the valid examples exactly and reject malformed examples with the specified errors.
This helps reveal ambiguous wording, incompatible implementations, or software that accepts damaged structures incorrectly. The independent Python reader, written only against the specification text, must reproduce every vector — if the specification were ambiguous, or the reference implementation wrong, this is where the fact would show. Anyone writing their own implementation should begin with the vectors: make the valid cases pass, then make every malformed case fail with the required error.
b9be8760fd4a85a922e5fa8eaf86840eec0719a5407030b9f6a35f0606ea79bd remanence-test-vectors.tar
Selected kernels of the format logic are additionally machine-checked. Nine areas — among them the CRC-64 loop, the parity capacity and mapping arithmetic, the sidecar layout bounds, and the manifest folds — are extracted to Lean and proved against written specification theorems, with the Lean type checker as the trust anchor and no local placeholders permitted. Each proof states its boundary precisely, and drift guards pin the extractions to the production code. This is a deliberately narrow effort — proofs of pure kernels, not a verified system — and one we intend to deepen. The reviewer-facing inventory of what is proved, and what is not, is public.
The specification documents are under Creative Commons Attribution (CC-BY-4.0), the test vectors under Creative Commons Zero (CC0), and the reference implementation under the Apache License 2.0.