New feature: specify a device's durability
 
Just pushed a new feature (only lightly tested so far): when formatting, you can specify a "durability" for each device: the effect of this is that data on that device will be counted as being replicated that many times.

So if you've got a filesystem with two SSDs and a big hardware RAID array: you probably want all your data to be replicated - you don't want to lose data if one of the SSDs dies - but you don't want bcachefs replicating it if it's on the hardware RAID array. With this feature, you can format with --replicas=2 and --durability=2 for the hardware RAID and it'll do what you want: if data is on the SSDs bcachefs will make sure there's another copy on another device, but not for data on the hardware RAID.

You can also set durability to 0 for a device: this, combined with the other options for foreground/background/promote targets, will tell bcachefs to use a device for writethrough caching - "don't write anything to this device that we can't lose", so even with replicas=1 bcachefs will always ensure any data on that device will also have another copy on another device.

For the moment, durability can only be set at format time - hopefully I'll be able to make it settable at runtime like most other options, but with the way the allocation code works that looks very tricky so I haven't done it just yet.