Ceph is a powerful, scalable, and highly resilient distributed storage platform that can provide object, block, and file storage under a unified system. It consists of various daemons such as Monitors (MONs), Object Storage Daemons (OSDs), and Metadata Servers (MDS) for CephFS. Ceph's primary interface for interacting with the system is the ceph command-line tool, which provides various subcommands to manage, monitor, and troubleshoot the Ceph cluster.
Below is a detailed guide on the most commonly used ceph subcommands and their purposes.
General Structure of ceph Command
ceph [options] <subcommand>[arguments]
[options]: Flags such as --admin-daemon, --cluster, -s for status, etc.
<subcommand>: The specific operation or management action you wish to perform (e.g., status, health, osd, monitor).
[arguments]: Additional arguments required by the subcommand.
Common ceph Subcommands
1. Cluster Information and Status
ceph status (or ceph -s)
Displays the overall health and status of the Ceph cluster. This command is frequently used for monitoring purposes.
cephstatus
Example Output:
cluster:
id: e4a89f8c-4a64-4df5-bcdf-e246b6f8120e
health: HEALTH_OK
services:
mon: 3 daemons, quorum mon1,mon2,mon3
mgr: mgr1(active), mgr2(standby)
osd: 20 osds: 20 up, 20 in
mds: cephfs-1/1/1 up {0=mds1=up:active}, 1 up:standby
ceph health
Returns a concise health report for the cluster. This can be combined with additional options to filter warnings or errors.
Example Output:
Options:
ceph health detail – Provides a more detailed health report with issues or warnings.
2. Monitor Commands
ceph mon dump
Displays the list of monitors, their ranks, and their addresses.
ceph quorum_status
Provides information about the quorum of the monitor nodes.
ceph mon stat
Shows the status of the monitor nodes.
3. OSD Commands
ceph osd status
Provides a list of the Object Storage Daemons (OSDs) in the cluster and their statuses (e.g., up, down, in, out).
ceph osd tree
Displays the OSDs organized in a hierarchical tree, often including information about the CRUSH hierarchy.
ceph osd df
Displays a detailed breakdown of disk usage and available space across OSDs in the cluster.
ceph osd pool create
Creates a new pool within the Ceph cluster. A pool is a logical group of objects that can have its own replication settings.
Example:
ceph osd pool delete
Deletes a pool from the cluster. Be cautious with this command, as it permanently deletes data.
ceph osd crush rule list
Lists the CRUSH (Controlled Replication Under Scalable Hashing) rules used for data placement in the cluster.
ceph osd reweight
Adjusts the weight of an OSD to rebalance data across the cluster.
Example:
ceph osd out
Marks an OSD as "out" of the cluster, which stops data from being written to that OSD.
ceph osd in
Marks an OSD as "in" the cluster, allowing it to receive data again.
4. Pool Commands
ceph osd pool ls
Lists all pools in the cluster.
ceph osd pool set
Sets or modifies parameters of a given pool.
Example (adjusting replication size):
ceph osd pool get
Gets the value of a particular option for a given pool.
Example (getting the replication size):
5. MDS and CephFS Commands
ceph fs status
Displays the status of CephFS (Ceph File System).
ceph mds stat
Shows the status of all metadata servers (MDS) in the cluster.
ceph fs ls
Lists all CephFS file systems in the cluster.
ceph fs new
Creates a new CephFS file system.
6. Client and Admin Commands
ceph config dump
Displays the current configuration for all Ceph components.
ceph log
Displays logs for the Ceph cluster.
Example (viewing logs of the monitor service):
7. RBD (RADOS Block Device) Commands
rbd ls
Lists all RADOS block devices in the cluster.
rbd create
Creates a new RADOS block device.
Example:
rbd rm
Removes an existing RBD image.
rbd map
Maps a RBD device to a local kernel module, making it available to be mounted as a block device on the system.
rbd unmap
Unmaps a previously mapped RBD device.
8. Administrative Commands
ceph auth list
Lists all authorization keys and capabilities in the cluster.
ceph auth add
Adds a new authorization key with specified capabilities.
Example:
ceph mgr module enable
Enables a manager module in Ceph. Ceph Manager (Mgr) modules offer additional services to the cluster.
Example:
Conclusion
The ceph command-line tool is highly versatile and allows administrators to control every aspect of a Ceph cluster. Whether it's managing OSDs, monitoring cluster health, configuring pools, or setting up RBD devices, ceph provides robust commands to handle these tasks efficiently.
For daily monitoring and administration, commands like ceph status, ceph osd tree, and ceph osd df provide essential insights into cluster performance, health, and resource utilization.