Nonces: I'm not going to go into why they're important for your encryption, just what they mean for the filesystem. A nonce is some additional input to your encryption function - 96 bits is typical. A nonce doesn't have to be secret - an incrementing counter is generally fine - but the important thing is that you must never, ever reuse the same nonce twice.
Using the physical offset on disk as a nonce is the typical solution when you can't store a real nonce somewhere - if an attacker only gets to see your disk image once, they won't see any blocks that have been encrypted with the same nonce. But if they can look more than once and see blocks that have been overwritten with new data, then they can see multiple messages (blocks) that have been encrypted with the same key:nonce - which is really bad. Or worse, if you're using an SSD when you overwrite data you aren't really overwriting data - the old data is still there until the SSD's internal garbage collection reclaims it, and a sophisticated enough attacker could certainly find it. So implementing nonces correctly is emphatically worth it, if we can.
Anyways, it's the filesystem's responsibility to generate these nonces, and store them somewhere so we have them when we go to decrypt. Conveniently, keys in bcache have a version number field - I'm going to expand that to 96 bits (which is a good thing to do for other reasons anyways), and then we just have to make sure that when encryption is enabled we're generating unique version numbers.
There's a corner case with partially overwritten extents and copygc: when we rewrite a partially overwritten extent, we can't generate a new version number - that would break other uses of version numbers. So that means when we go to rewrite and reencrypt the data, we're actually encrypting different data (we start encrypting from a different position than before) with the same version number - but we can solve that issue by also incorporating the offset field into our nonce.
So normal data extents are easy. We also need to encrypt metadata too, though - btree nodes and journal entries. They already have fields we can use for nonces, fortunately.
Algorithms: I'm primarily interested in supporting ChaCha20 and Poly1305. ChaCha20 beats AES in software (by a lot), and even if you have hardware AES ChaCha20 is quite fast. AES will almost definitely be supported, as well as other MACs.
Using a cryptographic MAC is also generally considered to be quite important when you're using encryption - there are real world attacks they protect against. I don't know if any of the attacks they guard against practically apply to a filesystem (vs. networked applications), but I don't particularly care to test that - anyways, if you're paying the cost in CPU for encryption you can afford a MAC. Checksumming with just crc32c will probably be allowed, but discouraged.
edit: crc32c will not be allowed with encryption, poly1305 will be used. There's some nasty attacks on stream cyphers you really need a HMAC to deal with.Also no longer planning on adding AES support - it's too vulnerable to side channel attacks when implemented in software. Fortunately, chacha20 is plenty fast - I measure roughly 4 GB/second on my Haswell machine.
Also, getting nonces right is turning out to be ever so much fun.