When I worked at a company called Smarsh, I wrote an encrypted file format for them for use in cloud agnostic storage utilizing envelope encryption. I read dozens of state of the art documents, various RFCs, and researched even proprietary formats for prior art references.
We also used Go, which I consider the best possible language to use with respect to working on encryption. Kudos to Filippo Valsorda for his work on Go, and age. Hands down the best work in the space at the moment. It doesn't get better. I've done my homework.
But in my opinion, this is still an unsolved problem. age simply makes a dent in an otherwise immature space, with simple questions:
1. What cyphertext, salt, iv structure layout should the encrypted payload follow? There are de facto binary structure layouts that prior art has established a practice for, but it does vary in order, and no papers talk about embedding specific details you need for the decryption process such as required lengths that otherwise need agreed upon static values.
2. What headers should be provided, if any, to provide mechanisms for versioning and reading encryption strategy? Should part of this metadata be in the file extension instead, such as `.aes256-gcm' and `.aes256-gcm.key'?
To my surprise, despite the longevity of the industry, there was no simple file format that I could turn to that didn't involve being tightly coupled to email or a concept of a recipient.
Unfortunately, age was:
1. Too young to consider for enterprise rollout.
2. Failed to differentiate itself from previous standards that were tightly coupled to email or recipients.
3. It fails to meet encryption standards that are more aligned with compliance than the state of the art.
Regarding point 3, I appreciate its desire to stay small and utilize ChaCha20-Poly1305, but that's insufficient for deployments with specific requirements.
I think someone needs to come out with an alternative to age that allows you to specify the cipher, and supported ciphers in the standard need to have specific binary layouts that can be agreed upon.
I would personally prefer some metadata in the file extension, but I think this is naive, because re-encrypting a file would lose this information, or dumping binaries to pathnames with arbitrary file names would destroy this information, so it must be in a file header.
The file header needs a specific magic number that can be read by utilities, and those utilities can further probe the file for metadata.
age does some of these things, but it doesn't have answers for others.
But gpg isn't a solution, either. Nor are numerous other encrypted payload formats for various reasons.
I think there are no standards for the things you mention because there shouldn't be any. If they existed, they would be necessarily complex and bloated like JWT or XML, introducing unneeded complexity in anything that adopted them. age tries to be as much complexity needed for the use case and no more. Other designs can make their different choices for their different use cases. That's good!
No contest on (1). On (2) I want to mention that "recipient" might have been an unfortunate choice of word, but it has nothing to do with email, it just means "a thing that can be encrypted to" like a public key, and any envelope encryption scheme will have something like that even if it doesn't give it this or any name. For (3) I have no idea what your requirements are, but I would claim age is pretty much state of the art (partially because it's not hard, age is trivial by cryptography research standards). Wireguard is being merrily deployed everywhere and uses ChaCha20-Poly1305 as well, FWIW.
(age does have a magic number in the header, and I see you came to realize why metadata in files is a bad idea. It's a bad idea also because it would be attacker-controlled.)
A part of the issue is that we tried to avoid utility deployment in parallel with the micro services we were already building.
So we emphasized trying to focus on the data itself first, and as a result what standards we could rely on, anticipating that we would use some implementation in Go.
openssl-enc doesn’t output to a file standard. It has known binary layouts but they’re not for use other than within the utility. `openssl enc’ was not something we were going to utilize for clients’ billions of files nor was replicating the binary layout for compatibility.
At the time we deployed age had been released for about two weeks, IIRC. Not something I as a principal can take to management and say we’re going to use for some of the largest clients in the world. Ironically, we could take on the risk internally with our own research.
But we needed cipher compatibility with the KMSs we were working with. age doesn’t provide that when everyone else is speaking aes-256-gcm.
That makes perfect sense, two weeks is way too early to deploy something in production.
Also, good call on not using openssl(1) in production. Last time I checked that CLI was primarily meant for testing, and anyway is full of sharp edges.
Not sure what AES-256-GCM vs ChaCha20Poly1305 has to do with KMSs though? I ask because age is specifically designed to support pluggable key wrapping mechanisms to support KMSs. You can write a plugin that talks to your KMS to wrap the file key, and use age for everything else. Surely you're not sending the whole file payload to the KMS.
We also used Go, which I consider the best possible language to use with respect to working on encryption. Kudos to Filippo Valsorda for his work on Go, and age. Hands down the best work in the space at the moment. It doesn't get better. I've done my homework.
But in my opinion, this is still an unsolved problem. age simply makes a dent in an otherwise immature space, with simple questions:
1. What cyphertext, salt, iv structure layout should the encrypted payload follow? There are de facto binary structure layouts that prior art has established a practice for, but it does vary in order, and no papers talk about embedding specific details you need for the decryption process such as required lengths that otherwise need agreed upon static values.
2. What headers should be provided, if any, to provide mechanisms for versioning and reading encryption strategy? Should part of this metadata be in the file extension instead, such as `.aes256-gcm' and `.aes256-gcm.key'?
To my surprise, despite the longevity of the industry, there was no simple file format that I could turn to that didn't involve being tightly coupled to email or a concept of a recipient.
Unfortunately, age was:
1. Too young to consider for enterprise rollout.
2. Failed to differentiate itself from previous standards that were tightly coupled to email or recipients.
3. It fails to meet encryption standards that are more aligned with compliance than the state of the art.
Regarding point 3, I appreciate its desire to stay small and utilize ChaCha20-Poly1305, but that's insufficient for deployments with specific requirements.
I think someone needs to come out with an alternative to age that allows you to specify the cipher, and supported ciphers in the standard need to have specific binary layouts that can be agreed upon.
I would personally prefer some metadata in the file extension, but I think this is naive, because re-encrypting a file would lose this information, or dumping binaries to pathnames with arbitrary file names would destroy this information, so it must be in a file header.
The file header needs a specific magic number that can be read by utilities, and those utilities can further probe the file for metadata.
age does some of these things, but it doesn't have answers for others.
But gpg isn't a solution, either. Nor are numerous other encrypted payload formats for various reasons.