| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 16:42:41 | sean-k-mooney | e.g. that does not look like it was created orginally for ceph | |
| 16:43:26 | melwitt | hm, I found this https://forum.proxmox.com/threads/proxmox-ceph-osd-partition-created-with-only-10gb.55291/ | |
| 16:43:31 | sean-k-mooney | oh its for swift orginially | |
| 16:46:20 | sean-k-mooney | i think the "sudo ceph-osd -c ${CEPH_CONF_FILE} -i ${OSD_ID} --mkfs" is the one we would need to modify | |
| 16:49:32 | sean-k-mooney | melwitt: that does seam like the same issue more or less | |
| 16:49:48 | melwitt | yeah, I'm having trouble understanding it | |
| 16:50:50 | melwitt | the last comment links to another post https://forum.proxmox.com/threads/where-can-i-tune-journal-size-of-ceph-bluestore.44000/ where they're talking about tuning journal size and bluestore_block_db_size and bluestore_block_wal_size | |
| 16:51:02 | melwitt | and I don't know what any of that is or means | |
| 16:51:22 | melwitt | (in ceph.conf) | |
| 17:01:31 | sean-k-mooney | those are not realated to the data storage size of the osd | |
| 17:01:55 | sean-k-mooney | blustore has an embeed database that track where the logic block are located on disk | |
| 17:02:27 | sean-k-mooney | wal i think it the write ahead log or somethingl like that | |
| 17:02:39 | sean-k-mooney | its part of how it does write journalling | |
| 17:03:02 | sean-k-mooney | in both cases they are turning parmatner for how bluestore can save its metadata | |
| 17:03:56 | sean-k-mooney | unlike file sotre it can save it inline in the blockdevice it is managening or it can save it oh external devices and they support tuneing of the sizing of them independelty | |
| 17:04:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Handle Neutron errors in _post_live_migration() https://review.opendev.org/729763 | |
| 17:09:57 | melwitt | sean-k-mooney: found a new thing https://bugzilla.redhat.com/show_bug.cgi?id=1597048 | |
| 17:09:57 | openstack | bugzilla.redhat.com bug 1597048 in RADOS "ceph osd df not showing correct disk size and causing cluster to go to full state" [High,Closed: notabug] - Assigned to bhubbard | |
| 17:10:59 | dansmith | imagine that :) | |
| 17:11:24 | melwitt | what | |
| 17:11:27 | sean-k-mooney | it should be 3.7TB but is 10G | |
| 17:11:41 | dansmith | melwitt: "not showing correct disk size" | |
| 17:11:57 | melwitt | yeah? | |
| 17:12:14 | melwitt | I'm still googling for why bluestore is maxed out at 10G | |
| 17:12:17 | dansmith | melwitt: just saying, I think we've stumbled into a realization that our df reporting on ceph in libvirt es no bueno right? | |
| 17:12:23 | melwitt | no | |
| 17:13:08 | dansmith | oh did I miss something? I thought those RHN articles were indicating that we're reporting the wrong thing still | |
| 17:13:51 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=1597048#c8 | |
| 17:13:51 | openstack | bugzilla.redhat.com bug 1597048 in RADOS "ceph osd df not showing correct disk size and causing cluster to go to full state" [High,Closed: notabug] - Assigned to bhubbard | |
| 17:14:08 | dansmith | like, if we're reporting the total size of the osd, but that's shared by vms and images, we'll be telling placement it can allocate all that space for instances but it can't | |
| 17:14:20 | sean-k-mooney | so it look like they hever actully got to the root cause of why the bluestore file was a 10G file | |
| 17:14:37 | sean-k-mooney | they just redeployed with file store an ignored it | |
| 17:15:01 | melwitt | yeah.. but then what does this mean? "The BlueStore block device was a file named with a block, not a symlink to block device partition of this disk and that file size was 10G hence it was showing the size of the OSD as 10G." | |
| 17:15:24 | sean-k-mooney | i think they ment | |
| 17:15:45 | sean-k-mooney | that in stead of it being a symlink to /dev/sdX | |
| 17:15:55 | sean-k-mooney | it was a file named block | |
| 17:15:58 | sean-k-mooney | that was 10G | |
| 17:16:13 | sean-k-mooney | hence -rw-r--r--. 1 ceph ceph 10737418240 Jul 2 16:51 block | |
| 17:16:32 | melwitt | right.. so you think it's correct that it's pointing at a file named block? and that the problem is that the file is not larger than 10G? | |
| 17:16:35 | sean-k-mooney | they were expecting it to be a symlink to the actual hdd | |
| 17:16:49 | sean-k-mooney | well in that case yes | |
| 17:16:54 | sean-k-mooney | and also likely in our case | |
| 17:17:16 | melwitt | ok. from reading that I thought maybe it was pointing wrongly at a file | |
| 17:17:34 | sean-k-mooney | /var/lib/ceph/osd/ceph-0/block is liekly a 10G file | |
| 17:17:38 | dansmith | I think that bug is that they deployed on file instead of having the bluestore osd use the disk they wanted | |
| 17:17:52 | sean-k-mooney | dansmith: yes | |
| 17:17:56 | dansmith | we want file, they wanted disk, right? | |
| 17:18:02 | sean-k-mooney | yes | |
| 17:18:17 | melwitt | oh | |
| 17:18:47 | melwitt | ok so why is /var/lib/ceph/osd/ceph-0/block only 10G ... who creates it ... | |
| 17:19:04 | dansmith | right, that I think we still don't know.. where the 10G comes from and how we change it | |
| 17:19:18 | dansmith | because the osd itself (the driver) seems to create that as a flat 10G file if it's not there | |
| 17:20:09 | melwitt | yeah, at least before now I did not know that the 10G comes from the size of the file named "block" so now I'm gonna see if I can find where that file is created | |
| 17:28:53 | melwitt | hm https://github.com/ceph/ceph/blob/master/src/common/legacy_config_opts.h#L940 | |
| 17:29:37 | dansmith | lol | |
| 17:29:53 | melwitt | https://github.com/ceph/ceph/blob/8c1a077e560248760ac441f315b84304aa693e72/src/common/options.cc#L4122 | |
| 17:30:15 | dansmith | so maybe we're supposed to create that block file to be what we want it to be | |
| 17:30:17 | melwitt | looks like they changed the default to 100G at some point | |
| 17:30:18 | dansmith | definitely obscure though | |
| 17:30:41 | sean-k-mooney | dansmith: yes we are ment to create the file/partion first normaly when deploying ceph | |
| 17:30:49 | melwitt | well I think you can set bluestore_block_size in ceph.conf no? | |
| 17:30:57 | melwitt | oh | |
| 17:31:19 | sean-k-mooney | am likely | |
| 17:31:29 | sean-k-mooney | but ceph does not expect to have to create this normally | |
| 17:31:31 | melwitt | https://github.com/ceph/ceph/commit/57890fce7064811780823e298b31e7fced2fa0e3 | |
| 17:31:45 | sean-k-mooney | if you use the tooling they provide tehy create the partions ahead of time | |
| 17:32:12 | melwitt | that's more recent, change from 1 TB -> 100G default. but in older versions the default was 10G, trying to see when that was so we can compare with what version we're running | |
| 17:33:13 | sean-k-mooney | this is the funtion that actully creates teh file https://github.com/ceph/ceph/blob/nautilus/src/os/bluestore/BlueStore.cc#L5934 | |
| 17:33:21 | melwitt | v15.1.0 is Octopus | |
| 17:34:02 | sean-k-mooney | if the block file is not present it creates it https://github.com/ceph/ceph/blob/nautilus/src/os/bluestore/BlueStore.cc#L5931-L5934 | |
| 17:34:36 | melwitt | and somehow 'size' is passed in from the config option I assume | |
| 17:34:49 | sean-k-mooney | that is what im currently trying to find yes | |
| 17:36:36 | sean-k-mooney | this maybe https://github.com/ceph/ceph/blob/nautilus/src/os/bluestore/BlueStore.cc#L5943-L5944 | |
| 17:38:33 | sean-k-mooney | ah no its here | |
| 17:38:36 | sean-k-mooney | https://github.com/ceph/ceph/blob/nautilus/src/os/bluestore/BlueStore.cc#L6050-L6052 | |
| 17:38:41 | sean-k-mooney | so in the mkfs call | |
| 17:39:25 | sean-k-mooney | so when we do this | |
| 17:39:27 | sean-k-mooney | https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L482-L483 | |
| 17:39:29 | melwitt | ah yup, and it's pulling the conf option | |
| 17:39:43 | sean-k-mooney | it cause the mkfs function to be invoked on the backend store | |
| 17:39:59 | sean-k-mooney | whic for bluestore uses that config option to create the 10G file | |
| 17:40:14 | melwitt | the interesting thing is, I wonder why it uses the legacy option and not the new one. I don't understand how that works in their code. cause in nautilus they have both the 10G and 100G default in the legacy conf vs the non | |
| 17:40:24 | sean-k-mooney | if var/lib/ceph/osd/ceph-0/block is not a symplink to a device | |
| 17:41:07 | sean-k-mooney | melwitt: its legacy on master | |
| 17:41:15 | sean-k-mooney | it might not be on nautalius | |
| 17:41:37 | melwitt | oh, I thought you mentioned earlier that CI is using nautilus | |
| 17:41:57 | sean-k-mooney | actully its alos here on master https://github.com/ceph/ceph/blob/master/src/common/options.cc#L4127-L4131 | |
| 17:42:05 | sean-k-mooney | melwitt: yes it is | |
| 17:42:25 | sean-k-mooney | actully just above that | |
| 17:42:27 | sean-k-mooney | https://github.com/ceph/ceph/blob/master/src/common/options.cc#L4122-L4125 | |
| 17:42:49 | melwitt | yeah I'm saying it's weird that it's not defaulting to 100G like that is showing | |
| 17:42:57 | melwitt | the old default was 10G | |
| 17:43:02 | sean-k-mooney | yep | |
| 17:43:57 | sean-k-mooney | we are pulling 14.2.2 https://github.com/ceph/ceph/blob/v14.2.2/src/common/options.cc#L4339 | |
| 17:44:02 | sean-k-mooney | which is 10 | |
| 17:44:32 | sean-k-mooney | they backported the 100G change to nautilus | |
| 17:44:44 | sean-k-mooney | but its not in the tag we are pulling | |
| 17:45:23 | sean-k-mooney | i think legacy_config_opts.h is just an old way to define config options | |
| 17:45:27 | melwitt | ohhh | |
| 17:45:37 | melwitt | good find. ok at least everything makes sense now | |