Proxmox Ceph OSD Partition Created With Only 10GB

Server Fault Asked on January 5, 2022

How do you define the Ceph OSD Disk Partition Size?

It always creates with only 10 GB usable space.

  • Disk size = 3.9 TB
  • Partition size = 3.7 TB
  • Using ceph-disk prepare and ceph-disk activate (See below)
  • OSD created but only with 10 GB, not 3.7 TB


Commands Used

root@proxmox:~# ceph-disk prepare --cluster ceph --cluster-uuid fea02667-f17d-44fd-a4c2-a8e19d05ed51 --fs-type xfs /dev/sda4

meta-data=/dev/sda4              isize=2048   agcount=4, agsize=249036799 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0, reflink=0
data     =                       bsize=4096   blocks=996147194, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=486399, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


root@proxmox:~# ceph-disk activate /dev/sda4

creating /var/lib/ceph/tmp/mnt.jyqJTM/keyring
added entity osd.0 auth auth(auid = 18446744073709551615 key=AQAohgpdjwb3NRAAIrINUiXDWQ5iMWp4Ueah3Q== with 0 caps)
got monmap epoch 3
2019-06-19 19:59:54.006226 7f966e628e00 -1 bluestore(/var/lib/ceph/tmp/mnt.jyqJTM/block) _read_bdev_label failed to open /var/lib/ceph/tmp/mnt.jyqJTM/block: (2) No such file or directory
2019-06-19 19:59:54.006285 7f966e628e00 -1 bluestore(/var/lib/ceph/tmp/mnt.jyqJTM/block) _read_bdev_label failed to open /var/lib/ceph/tmp/mnt.jyqJTM/block: (2) No such file or directory
2019-06-19 19:59:55.668619 7f966e628e00 -1 created object store /var/lib/ceph/tmp/mnt.jyqJTM for osd.0 fsid fea02667-f17d-44fd-a4c2-a8e19d05ed51
Created symlink /run/systemd/system/[email protected] → /lib/systemd/system/[email protected].

# Don't worry about my keys/IDs, its just a dev environment.


Disk Layout

root@proxmox:~# fdisk -l
Disk /dev/sda: 3.9 TiB, 4294967296000 bytes, 8388608000 sectors

Device         Start        End    Sectors   Size Type
/dev/sda1         34       2047       2014  1007K BIOS boot
/dev/sda2       2048    1050623    1048576   512M EFI System
/dev/sda3    1050624  419430400  418379777 199.5G Linux LVM
/dev/sda4  419430408 8388607966 7969177559   3.7T Ceph OSD


Ceph OSD Disk Size Incorrect (10 GB not 1.7 TB)

root@proxmox:~# ceph status
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0B
    usage:   1.00GiB used, 9.00GiB / 10GiB avail



Full Install Details

If you want details on Proxmox install and creating Ceph OSD with partitions, read on…


  • Disk Size: 2TB NVMe (/dev/sda)
  • Operating system (Proxmox) installed with 200 GB and rest of disk is empty (1800 GB).
  • Once booted and in the web interface, create a cluster and join two hosts to ensure a green quorum status
  • Now do script below

Config Script

# Install Ceph
pveceph install

# Configure Network (Just run on Primary Proxmox Server, your LAN network)
pveceph init --network

# Create Monitor
pveceph createmon

# View Disks Before
sgdisk --print /dev/sda

sgdisk --largest-new=4 --change-name="4:CephOSD" 
--typecode=4:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sda
# View Disks After (Compare)
sgdisk --print /dev/sda

# Reboot for changes to be in affect

# Note your cluster ID (fsid) at this point from the web interface 
Datacenter > Server > Ceph

# Prepare the Ceph OSD Disk, replace cluster-uuid with above fsid
ceph-disk prepare --cluster ceph --cluster-uuid fea02667-f17d-44fd-a4c2-a8e19d05ed51 --fs-type xfs /dev/sda4

# Activate the Ceph OSD Disk 
ceph-disk activate /dev/sda4

# Check Ceph OSD Disk Size
ceph status

I have read posts highly recommending to use disks instead of partitions because of performance issues, I understand the warnings, but In my case I’m using NVMe SSD storage and accept any risks.

One Answer

We ran into a similar issue recently where our OSD was limited to only 10GB despite a much larger actual disk space and found that it's related to the bluestore backend in Ceph. When bluestore is backed by a file, if that file does not already exist, it will create it:

bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory
bluestore(/var/lib/ceph/osd/ceph-0) mkfs generated fsid db688b70-0f63-4086-ba4a-ef36dc4a3147

and then resize it based on the bluestore_block_size configuration:

bluestore(/var/lib/ceph/osd/ceph-0) _setup_block_symlink_or_file resized block file to 10 GiB

The default value for bluestore_block_size in ceph version 14.2.2, for example, is 10GB [1]:

    Option("bluestore_block_size", Option::TYPE_SIZE, Option::LEVEL_DEV)
    .set_description("Size of file to create for backing bluestore"),

This can be adjusted by setting the following in your ceph.conf:

bluestore_block_size = <block file size in bytes>


Answered by melwitt on January 5, 2022

Add your own answers!

Ask a Question

Get help from others!

© 2024 All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP