Design comparison

Hardware tape encryption compared with encryption in the object.

An archive does the same few things to its copies for as long as it holds them. It reads them back to confirm they are still intact, repairs the damage it finds, makes a further copy when one is lost, and moves everything onto new media every few years as drives go out of support. Whatever protects the contents has to survive all of that work without getting in its way.

LTO-4 and later drives encrypt in hardware, at rated speed, and for many archives that is the right answer. Remanence encrypts earlier, in the object format, before anything reaches a drive. The difference shows up in that repeated work: under drive encryption each of those operations needs a key released to a drive, and under encryption in the object, storing, copying, repairing from parity and checking the stored bytes need no key at all.

What the drive already does

Every LTO drive since generation 4 holds an AES-256-GCM engine on the data path. Data is compressed, then encrypted, then written, at rated speed. The key arrives over the ordinary SCSI interface and is not kept in the drive, and the drive never writes it to the cartridge.

What the format does not do is manage keys. The LTO Program is explicit about it:

"Key management is not part of the LTO Ultrium drive specification."

So the keys come from somewhere else: from the writing application, from an external key manager over KMIP, or from a store belonging to the library itself — on HPE's MSL libraries, a USB token in the chassis. That choice is made by a product rather than by the format, and what it decides is not how strong the encryption is but what has to survive alongside the key in order to use it: a running service, a vendor's product line, or nothing but the key.

The drive's encryption engine, the key-management arrangement around it, and the product that holds the keys are three different things. Most of what follows concerns the second and third; the engine itself is not what we are arguing with. The detailed comparison covers each arrangement, and the cartridge-level mechanics, in full.

Where the difference lies

Three differences, in the order they have mattered to us.

The key stays in the safe. An object is written once; everything after that recurs for as long as the collection is held — reading copies back, scrubbing for blocks that have begun to fail, rebuilding from parity, making further copies, migrating to newer media. Under drive encryption all of that runs through the decryption path, so a key has to be released, to unattended jobs, for years, for work whose only purpose is to confirm that stored bytes have not changed. With encryption in the object the ciphertext is what gets stored, checksummed, replicated and repaired, so a copy can be checked without being opened and the key never leaves custody. Every occasion a key is handled is an occasion it can leak, and this removes nearly all of them. It also separates custody from access: another institution can hold a copy, confirm it is intact and repair it from parity, without being able to see what it contains. Authenticated verification of an object's contents, and any restore, still need a key. Upkeep does not.

Any drive can write any copy. Where encryption is applied by library policy — the usual arrangement — the policy binds to a partition, and the drives serving it encrypt everything they write. An estate holding both encrypted and unencrypted material then divides into encrypting and non-encrypting drives. That is a real cost in a small library: the drives cannot be pooled, so a given throughput needs more of them; wear falls unevenly on the two groups; and losing one drive does not reduce capacity so much as remove the ability to write one of the two kinds of copy until it is replaced. Nothing in the hardware requires the split — encryption parameters are set per mount — but it is how the policy is normally deployed. With encryption in the object no drive is configured differently from another, and the estate is a single pool.

One ciphertext travels to every medium. Drive encryption protects the copies that are on tape and nothing else, so a disk or off-site copy needs a second mechanism with its own key custody, and rebuilding one copy from another passes through plaintext on the host. Sealed in the object, the material is encrypted once and that byte string is fanned out unchanged to every medium.

We put that last deliberately, because we do not encrypt every copy and would not advise it. Encryption is a barrier to preservation: it adds a way for readable data to become unreadable, and the barrier is permanent while the threat is usually circumstantial. It earns its place on the copy that leaves your control, and rarely everywhere. But an archive that does want several encrypted copies on several media should not need several schemes to get them, and with this design it does not.

ENCRYPTION IN THE DRIVE ingest staging host + cable tape disk cloud ciphertext not covered not covered ENCRYPTION IN THE OBJECT ingest staging host + cable tape disk cloud one ciphertext, copied unchanged
Fig. 1 — The dashed line is where plaintext ends. Under drive encryption it sits at the tape head, so plaintext crosses the host on every operation and the copies that are not on tape are not covered at all.

The two designs side by side

PropertyDrive encryptionREM-ENCRYPT
Ciphertext beginsAt the tape headAt ingest, before any storage is touched
What is protectedThe cartridgeThe object, on every medium it is copied to
Upkeep without a keyNoStoring, replicating, repairing from parity, scrubbing
Which drives can write which copiesPolicy usually binds encryption to a partition, so the estate divides by modeAny drive writes any copy
Migrate to new mediaDecrypt, cross the host in the clear, re-encryptCopy the bytes; no key is loaded
What the medium carriesA key identifier; some key managers also write a wrapped key to the cartridgeThe whole envelope: suites, wrapping, recipients, wrapped key
Needed to read a copy backA compatible drive, the key, and a host that can send the SCSI security commandsBoth published specifications, a conforming implementation, and one matching recipient key
Audit and revocationFrom the key managerNothing in the format; whatever your custody component provides
Hardware compressionRetainedForfeited
Certified moduleSpecific drive models and firmware have held FIPS 140 validationsNone

Row by row, with sources and the qualifications each one needs, in the detailed comparison.

What it costs

The framing is our own work, and it has not been reviewed outside the project. The primitives are standardised, but the envelope around them is ours, and REM-ENCRYPT is a review draft. What exists is the published specification, pinned test vectors and a separately written verifier.

Hardware compression is given up. Ciphertext does not compress, and REM-OBJECT has no compression of its own, so material that benefits from it has to be compressed before archiving. For an archive of compressible material this is the strongest argument for the drive.

There is no certified module, and no audit trail in the format. Where a FIPS certificate is a procurement requirement, the drive route exists and ours does not. A key manager also gives logging, revocation and dual control that a key in a safe does not — though a custody component can supply all three.

An encrypted copy is not silent. Its identifier, size, chunk size and recipient labels are visible without a key. Member names, sizes, count and contents are not.

Key custody is yours to provide

REM-ENCRYPT specifies no key registry and no custody protocol, deliberately. Key custody is a component an operator brings, in the same way the catalog and the copy policy are — Remanence is a component, not a product. Putting custody inside it would be self-defeating anyway: the part designed to run without keys would become the part that holds them.

What Remanence needs is narrow. It is handed a key at the moment an object is opened, it performs the unwrap, and it stores nothing. What surrounds that is yours to design, and it is smaller than a key server: an X-Wing private key's stored form is a 32-byte seed, which can go on paper, into metal, or through Shamir's scheme so that no one person and no one safe can open the archive alone. An envelope takes up to eight labelled recipients and any one of them opens it, so custody becomes a few named roles — a cold key held split, a warm key for approved restores, optionally one held by a second institution.

None of it sits on the archive's availability path. If the custody arrangement is unreachable the archive still verifies, repairs, replicates and migrates; only opening an encrypted copy waits.

The roles, the restore procedure, the twice-yearly drill and the limits on rotation are set out in key custody. It describes the shape we think that component should have, not software Remanence ships.

Choosing between them

Drive encryption is the right choice in several common situations.

  • Where the format is not yours to change. Most archives are written by software they did not write, into a container they do not control. The drive is the only control available, and it is an effective one.
  • Where the concern is a cartridge leaving the building — theft, loss in transit, disposal of retired media. That is what the feature was designed for, and it costs nothing in speed or capacity.
  • Where a validated module, an audit trail or revocation is required.
  • Where tape is the only medium. Application-managed encryption with the raw key escrowed by you keeps the compression, the option of a validated drive module, and a recovery path needing only a drive and a host. What it does not give is keyless upkeep, or one ciphertext across media.

Where an archive spans several media and has to be verified, repaired, replicated and migrated for decades, often by people and processes that should not be entitled to read it, we found the boundary belongs earlier, at the object. Whichever is chosen, the key custody plan decides the outcome, and it is the part most often left until last. An archive that outlives its keys is lost as thoroughly as if the tapes had burned, and for that reason the encrypted representation is designed around key custody before anything else.

Encryption in the REM formats is optional and separate from the durable object. An archive that does not need confidentiality writes plain REM-OBJECT and loses none of the fixity, parity or catalog-free recovery. Keeping the two apart means cryptography can be revised on its own schedule without the preserved format being revised with it.

The full argument, with the drive-side mechanics and the sources behind each claim, is in encryption explained. What we encrypt with, and why a post-quantum construction, is on the security page; the envelope is specified in REM-ENCRYPT.