crushtool

crushtool is a utility in Ceph used to interact with CRUSH (Controlled Replication Under Scalable Hashing) maps. CRUSH is the algorithm Ceph uses to determine how data is distributed and stored across the cluster. The crushtool provides various commands to create, decompile, compile, and analyze CRUSH maps, which control how Ceph places data onto OSDs based on failure domains, weights, and replication rules.

Common crushtool Commands and Subcommands

1. decompile

Decompiles a binary CRUSH map into a human-readable text format.

crushtool --decompile <binary-crush-map> -o <output-text-file>

• Example:

  crushtool --decompile /etc/ceph/crushmap -o crushmap.txt

This command takes the binary CRUSH map /etc/ceph/crushmap and decompiles it into the human-readable file crushmap.txt.

2. compile

Compiles a text-format CRUSH map into a binary format that Ceph uses.

crushtool --compile <input-text-file> -o <output-binary-crush-map>

• Example:

  crushtool --compile crushmap.txt -o /etc/ceph/crushmap

This command compiles the human-readable crushmap.txt into the binary CRUSH map /etc/ceph/crushmap.

3. show

Displays the contents of a compiled CRUSH map in a more readable format directly in the terminal without decompiling it into a separate file.

crushtool --show <binary-crush-map>

• Example:

  crushtool --show /etc/ceph/crushmap

This command prints the contents of the binary CRUSH map in a readable format.

4. analyze

Analyzes how data will be distributed according to a given CRUSH map. You can use it to simulate data placement and failure domains.

crushtool --analyze <binary-crush-map> --rule <rule-id> --type <type-name> --num-rep <num-replicas>

• Example:

  crushtool --analyze /etc/ceph/crushmap --rule 0 --type osd --num-rep 3

This command simulates data placement using the CRUSH map's rule 0, distributing replicas across OSDs with a replication factor of 3.

5. test

Tests how a specific CRUSH map distributes data for a particular bucket or item, allowing administrators to verify placement behavior.

crushtool --test --rule <rule-id> --num-rep <num-replicas> --show-mappings --show-utilization --output-csv <output-file> <binary-crush-map>

• Example:

  crushtool --test --rule 0 --num-rep 3 --show-mappings /etc/ceph/crushmap

This command tests the distribution of data according to rule 0 with 3 replicas and displays the placement mappings for the current CRUSH map.

6. reweight

Reweights the CRUSH map’s OSDs or buckets based on a new weight, adjusting the distribution of data.

crushtool --reweight-item <item-name> --weight <new-weight> -i <input-crush-map> -o <output-crush-map>

• Example:

  crushtool --reweight-item osd.0 --weight 0.8 -i /etc/ceph/crushmap -o new-crushmap

This command reweights osd.0 to 0.8 in the CRUSH map.

7. add-bucket

Adds a new bucket to the CRUSH map. Buckets define failure domains such as racks, hosts, and OSDs.

crushtool --add-bucket <bucket-name> <bucket-type> -i <input-crush-map> -o <output-crush-map>

• Example:

  crushtool --add-bucket rack0 rack -i /etc/ceph/crushmap -o new-crushmap

This command adds a new bucket named rack0 of type rack to the CRUSH map.

8. set-item-class

Assigns a class to an item (such as an OSD) in the CRUSH map. Ceph uses classes to differentiate between storage types (e.g., SSD, HDD).

crushtool --set-item-class <class> <item-name> -i <input-crush-map> -o <output-crush-map>

• Example:

  crushtool --set-item-class ssd osd.1 -i /etc/ceph/crushmap -o new-crushmap

This command sets osd.1 to belong to the ssd class.

9. rm-bucket

Removes a bucket from the CRUSH map.

crushtool --rm-bucket <bucket-name> -i <input-crush-map> -o <output-crush-map>

• Example:

  crushtool --rm-bucket rack0 -i /etc/ceph/crushmap -o new-crushmap

This command removes the bucket rack0 from the CRUSH map.

10. rm-item

Removes an item (such as an OSD) from the CRUSH map.

crushtool --rm-item <item-name> -i <input-crush-map> -o <output-crush-map>

• Example:

  crushtool --rm-item osd.0 -i /etc/ceph/crushmap -o new-crushmap

This command removes osd.0 from the CRUSH map.

11. move

Moves a bucket to a different location in the hierarchy of the CRUSH map.

crushtool --move <bucket-name> <new-location> -i <input-crush-map> -o <output-crush-map>

• Example:

  crushtool --move host1 rack1 -i /etc/ceph/crushmap -o new-crushmap

This command moves the bucket host1 to rack1.

12. tree

Prints the CRUSH map hierarchy in a tree-like format.

crushtool --tree <binary-crush-map>

• Example:

  crushtool --tree /etc/ceph/crushmap

This command displays the hierarchy of the CRUSH map, showing how OSDs, hosts, racks, etc., are organized.

Typical Use Cases:

1. Customizing Data Placement: Ceph admins use crushtool to modify CRUSH maps for fine-tuned control over how data is replicated and distributed across failure domains (e.g., racks, hosts, OSDs).

2. Simulating Data Distribution: With commands like analyze and test, admins can simulate how CRUSH maps will distribute data across the cluster, ensuring even utilization and high availability.

3. Reweighting for Performance: Using reweight, admins can adjust the weights of OSDs or other CRUSH components to ensure even distribution of data, optimizing performance and capacity usage.

4. Failure Domain Management: By adding and removing buckets, Ceph administrators can redefine failure domains like hosts and racks, ensuring the cluster's resilience against hardware failures.

5. Storage Tiering with Classes: Assigning items to classes (e.g., SSDs or HDDs) allows for storage tiering, where Ceph can manage fast storage devices (SSDs) differently from slower ones (HDDs).

Conclusion

The crushtool is an essential utility for managing and customizing CRUSH maps in a Ceph cluster. It offers various commands for creating, compiling, testing, and analyzing CRUSH maps, giving administrators the ability to define and optimize how Ceph distributes and replicates data across the cluster. Through crushtool, failure domains, replication rules, and storage tiers can be efficiently controlled, providing resilience and performance to the Ceph cluster.