Panasas Storage System: redundancy and usable space
We have 5 shelves installed, each with 10 OSDs (Object Storage Devices, or storage blades). Each OSD has two 3TB drives on it, with a formatted capacity of 5.2TB, giving a net writable 262TB. 10.5TB of that is held aside as hot-spare capacity, yielding 251TB of usable storage. Each shelf (1/5 of that) gives you 50TB of usable storage.
There are several factors that decrease the usable amount of storage space.
- The most obvious is hot-spare storage blades which are held in reserve for reconstruction when another storage blade fails in the blade set. This is analogous to a hot spare in a RAID5 disk array. These are not actually physical blades that are unused, but a logical space reservation spread across all blades, so that you can utilize the maximum number of spindles possible for active I/O traffic.
- The second is what Panasas refers to as “horizontal” parity. This is essentially a RAID implementation over an arbitrary number of storage blades, so as your data blocks (objects) are written in parallel to multiple blades, one of the blades receives a client-calculated parity block. This protects against the failure of a storage blade.
- The third, and perhaps most obscure factor, is what Panasas calls “vertical” parity. This is in effect a software RAID implementation on a single drive (technically a pair of drives on a storage blade), and is used to protect against sub-full-disk errors (sector errors) on a disk. If a blade fails in the system and, during recovery, another blade with parity or reconstruction data encounters a read error, that data will typically not be recoverable. Vertical parity provides an extra layer of robustness in a rebuild or recovery scenario. In addition, it is used to ensure integrity of read data. With modern commodity disk drives, unrecoverable read bit errors are estimated to occur on average ever 10^14-10^15 bits, which is approximately every 10-100TB. Vertical parity ensures the integrity of the read data.
All of these taken together, along with the Panasas client’s parity computation, compose a very robust system with the resiliency to survive multiple types of media errors and failures, and provides very high assurances as to the integrity of the data you are reading and writing. Obviously they do make the usable capacity of the storage array somewhat less than the raw disk capacity.