What is Ceph, and when do you actually need it?

· INTECH

At some point a growing system outgrows “just add a bigger disk.” You need storage that scales past a single box, survives hardware dying, and doesn’t tie you to a specific vendor’s SAN. That’s the problem Ceph was built for.

This is the plain-language intro — what Ceph is and when it’s worth it. The deeper, hands-on pieces come after.

What Ceph is

Ceph is an open-source, software-defined, distributed storage system. Two ideas make it interesting:

  1. One cluster, three kinds of storage. From the same pool of disks, Ceph serves object storage (S3-compatible), block storage (virtual disks), and a POSIX file system. Most products need one storage silo per use case; Ceph unifies them.
  2. Commodity hardware, no single point of failure. It runs on ordinary servers and disks, spreads data across them, and keeps working — and healing — when any of them fail.

No proprietary appliance, no forklift upgrade, no per-terabyte licensing.

How it works, briefly

Under everything sits RADOS — a reliable, self-managing object store. A few kinds of daemon make it go:

  • OSDs (Object Storage Daemons) — typically one per disk. They store the actual data and handle replication, recovery, and rebalancing.
  • Monitors (MON) — keep the authoritative map of the cluster and reach consensus on its state. You run an odd number (3 or 5) so they can hold quorum.
  • Manager (MGR) — runs alongside the monitors for metrics, the dashboard, and management modules.
  • Metadata Servers (MDS) — only needed if you use the file system (CephFS).

The clever part is CRUSH, the algorithm that decides where each piece of data lives. Instead of asking a central index “where is this object?”, clients compute the location. No central lookup means no bottleneck and no single point of failure — and it’s what lets Ceph scale out cleanly.

Durability comes from either replication (keep N copies) or erasure coding (more space-efficient, more CPU) — you choose per pool, per workload.

The three faces

  • RBD (block) — virtual disks for VMs and containers. This is what backs OpenStack volumes and Kubernetes persistent volumes (via Rook).
  • CephFS (file) — a shared POSIX file system for many clients at once (needs the MDS).
  • RGW (object) — an S3/Swift-compatible gateway, so applications that speak S3 just work.

One cluster, and you can use all three at the same time.

Why teams choose it

  • Scales from a handful of nodes to petabytes, by adding servers.
  • No single point of failure — data is replicated and the cluster self-heals when a disk or node dies.
  • No vendor lock-in — open source, on hardware you choose.
  • S3-compatible object storage without paying a cloud provider per request.
  • Integrates with OpenStack and Kubernetes as the storage layer.

The honest catch

Ceph is powerful, and that power has a cost: operational complexity. It is not a weekend NAS. It expects enough nodes to make redundancy meaningful (realistically three or more), and it rewards someone who understands CRUSH, placement groups, and how it behaves when things go wrong.

So the rule of thumb:

  • Small, simple, single-box storage needs? Ceph is overkill — use something simpler.
  • Scale, high availability, S3/block/file from one platform, no lock-in? This is exactly where Ceph earns its keep.

Get the design and the day-2 operations right and it runs quietly for years. Get them wrong and you have a distributed system failing in distributed ways.

Running Ceph, planning a cluster, or staring at a HEALTH_WARN you can’t explain? Linux administration & Ceph is part of what we do — and more technical Ceph write-ups are on the way.