| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 16:34:10 | dansmith | sean-k-mooney: yeah I thought we were assuming that | |
| 16:34:16 | dansmith | because it's clearly just placement | |
| 16:34:31 | dansmith | my job might be slower (or faster) causing us to hit it more than we were or something | |
| 16:35:05 | sean-k-mooney | so we just either a.) swap back to file store to get the old behavior or b.) mount our loopback file in such a way that the bluestore block device uses our 24G loopback device instead of creating its own | |
| 16:35:29 | dansmith | yeah so I figured going back to xfs would be ideal for compatibility with everything | |
| 16:35:38 | dansmith | my system clearly gets xfs | |
| 16:35:51 | dansmith | I assume the workers are getting blue because they're newer ubuntu or something | |
| 16:36:00 | dansmith | I'm still on bionic | |
| 16:36:01 | sean-k-mooney | dansmith: sure but eventurally we will have to move since i think filestore is deprected in ceph | |
| 16:36:05 | dansmith | sure | |
| 16:36:16 | dansmith | pain now or pain later | |
| 16:36:21 | dansmith | pain later might be someone else's pain :P | |
| 16:36:52 | sean-k-mooney | so i guess what we are looking for is a ceph config option to select filestore for the osd backend | |
| 16:37:43 | sean-k-mooney | that or we set it on the osd create command | |
| 16:38:02 | dansmith | yeah | |
| 16:38:37 | sean-k-mooney | so this code https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L475-L484 | |
| 16:39:03 | sean-k-mooney | that inital sudo ceph -c ${CEPH_CONF_FILE} osd create | |
| 16:39:06 | dansmith | well, the other option is to figure out how to make blue use 20ish G instead of 10, | |
| 16:39:20 | dansmith | which would be less impactful than retooling the mount stuff in the ceph plugin | |
| 16:40:00 | sean-k-mooney | well we are mounting it on /var/lib/ceph | |
| 16:40:08 | sean-k-mooney | so i guess this is already plugin specific | |
| 16:40:24 | sean-k-mooney | we are likely resuing the same function that is used for cinder and just passing the mount path | |
| 16:40:57 | sean-k-mooney | ya we are just calling create_disk https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L387 | |
| 16:41:01 | dansmith | cinder just wants a loop not a mounted fs though right/ | |
| 16:41:52 | sean-k-mooney | maybe this is the fucntion in devstack https://github.com/openstack/devstack/blob/eee60c76719c02c08dba7b7fb703798a056b22b9/functions#L758-L789 | |
| 16:42:19 | sean-k-mooney | that kind of looks like a hack | |
| 16:42:41 | sean-k-mooney | e.g. that does not look like it was created orginally for ceph | |
| 16:43:26 | melwitt | hm, I found this https://forum.proxmox.com/threads/proxmox-ceph-osd-partition-created-with-only-10gb.55291/ | |
| 16:43:31 | sean-k-mooney | oh its for swift orginially | |
| 16:46:20 | sean-k-mooney | i think the "sudo ceph-osd -c ${CEPH_CONF_FILE} -i ${OSD_ID} --mkfs" is the one we would need to modify | |
| 16:49:32 | sean-k-mooney | melwitt: that does seam like the same issue more or less | |
| 16:49:48 | melwitt | yeah, I'm having trouble understanding it | |
| 16:50:50 | melwitt | the last comment links to another post https://forum.proxmox.com/threads/where-can-i-tune-journal-size-of-ceph-bluestore.44000/ where they're talking about tuning journal size and bluestore_block_db_size and bluestore_block_wal_size | |
| 16:51:02 | melwitt | and I don't know what any of that is or means | |
| 16:51:22 | melwitt | (in ceph.conf) | |
| 17:01:31 | sean-k-mooney | those are not realated to the data storage size of the osd | |
| 17:01:55 | sean-k-mooney | blustore has an embeed database that track where the logic block are located on disk | |
| 17:02:27 | sean-k-mooney | wal i think it the write ahead log or somethingl like that | |
| 17:02:39 | sean-k-mooney | its part of how it does write journalling | |
| 17:03:02 | sean-k-mooney | in both cases they are turning parmatner for how bluestore can save its metadata | |
| 17:03:56 | sean-k-mooney | unlike file sotre it can save it inline in the blockdevice it is managening or it can save it oh external devices and they support tuneing of the sizing of them independelty | |
| 17:04:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Handle Neutron errors in _post_live_migration() https://review.opendev.org/729763 | |
| 17:09:57 | melwitt | sean-k-mooney: found a new thing https://bugzilla.redhat.com/show_bug.cgi?id=1597048 | |
| 17:09:57 | openstack | bugzilla.redhat.com bug 1597048 in RADOS "ceph osd df not showing correct disk size and causing cluster to go to full state" [High,Closed: notabug] - Assigned to bhubbard | |
| 17:10:59 | dansmith | imagine that :) | |
| 17:11:24 | melwitt | what | |
| 17:11:27 | sean-k-mooney | it should be 3.7TB but is 10G | |
| 17:11:41 | dansmith | melwitt: "not showing correct disk size" | |
| 17:11:57 | melwitt | yeah? | |
| 17:12:14 | melwitt | I'm still googling for why bluestore is maxed out at 10G | |
| 17:12:17 | dansmith | melwitt: just saying, I think we've stumbled into a realization that our df reporting on ceph in libvirt es no bueno right? | |
| 17:12:23 | melwitt | no | |
| 17:13:08 | dansmith | oh did I miss something? I thought those RHN articles were indicating that we're reporting the wrong thing still | |
| 17:13:51 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=1597048#c8 | |
| 17:13:51 | openstack | bugzilla.redhat.com bug 1597048 in RADOS "ceph osd df not showing correct disk size and causing cluster to go to full state" [High,Closed: notabug] - Assigned to bhubbard | |
| 17:14:08 | dansmith | like, if we're reporting the total size of the osd, but that's shared by vms and images, we'll be telling placement it can allocate all that space for instances but it can't | |
| 17:14:20 | sean-k-mooney | so it look like they hever actully got to the root cause of why the bluestore file was a 10G file | |
| 17:14:37 | sean-k-mooney | they just redeployed with file store an ignored it | |
| 17:15:01 | melwitt | yeah.. but then what does this mean? "The BlueStore block device was a file named with a block, not a symlink to block device partition of this disk and that file size was 10G hence it was showing the size of the OSD as 10G." | |
| 17:15:24 | sean-k-mooney | i think they ment | |
| 17:15:45 | sean-k-mooney | that in stead of it being a symlink to /dev/sdX | |
| 17:15:55 | sean-k-mooney | it was a file named block | |
| 17:15:58 | sean-k-mooney | that was 10G | |
| 17:16:13 | sean-k-mooney | hence -rw-r--r--. 1 ceph ceph 10737418240 Jul 2 16:51 block | |
| 17:16:32 | melwitt | right.. so you think it's correct that it's pointing at a file named block? and that the problem is that the file is not larger than 10G? | |
| 17:16:35 | sean-k-mooney | they were expecting it to be a symlink to the actual hdd | |
| 17:16:49 | sean-k-mooney | well in that case yes | |
| 17:16:54 | sean-k-mooney | and also likely in our case | |
| 17:17:16 | melwitt | ok. from reading that I thought maybe it was pointing wrongly at a file | |
| 17:17:34 | sean-k-mooney | /var/lib/ceph/osd/ceph-0/block is liekly a 10G file | |
| 17:17:38 | dansmith | I think that bug is that they deployed on file instead of having the bluestore osd use the disk they wanted | |
| 17:17:52 | sean-k-mooney | dansmith: yes | |
| 17:17:56 | dansmith | we want file, they wanted disk, right? | |
| 17:18:02 | sean-k-mooney | yes | |
| 17:18:17 | melwitt | oh | |
| 17:18:47 | melwitt | ok so why is /var/lib/ceph/osd/ceph-0/block only 10G ... who creates it ... | |
| 17:19:04 | dansmith | right, that I think we still don't know.. where the 10G comes from and how we change it | |
| 17:19:18 | dansmith | because the osd itself (the driver) seems to create that as a flat 10G file if it's not there | |
| 17:20:09 | melwitt | yeah, at least before now I did not know that the 10G comes from the size of the file named "block" so now I'm gonna see if I can find where that file is created | |
| 17:28:53 | melwitt | hm https://github.com/ceph/ceph/blob/master/src/common/legacy_config_opts.h#L940 | |
| 17:29:37 | dansmith | lol | |
| 17:29:53 | melwitt | https://github.com/ceph/ceph/blob/8c1a077e560248760ac441f315b84304aa693e72/src/common/options.cc#L4122 | |
| 17:30:15 | dansmith | so maybe we're supposed to create that block file to be what we want it to be | |
| 17:30:17 | melwitt | looks like they changed the default to 100G at some point | |
| 17:30:18 | dansmith | definitely obscure though | |
| 17:30:41 | sean-k-mooney | dansmith: yes we are ment to create the file/partion first normaly when deploying ceph | |
| 17:30:49 | melwitt | well I think you can set bluestore_block_size in ceph.conf no? | |
| 17:30:57 | melwitt | oh | |
| 17:31:19 | sean-k-mooney | am likely | |
| 17:31:29 | sean-k-mooney | but ceph does not expect to have to create this normally | |
| 17:31:31 | melwitt | https://github.com/ceph/ceph/commit/57890fce7064811780823e298b31e7fced2fa0e3 | |
| 17:31:45 | sean-k-mooney | if you use the tooling they provide tehy create the partions ahead of time | |
| 17:32:12 | melwitt | that's more recent, change from 1 TB -> 100G default. but in older versions the default was 10G, trying to see when that was so we can compare with what version we're running | |
| 17:33:13 | sean-k-mooney | this is the funtion that actully creates teh file https://github.com/ceph/ceph/blob/nautilus/src/os/bluestore/BlueStore.cc#L5934 | |
| 17:33:21 | melwitt | v15.1.0 is Octopus | |
| 17:34:02 | sean-k-mooney | if the block file is not present it creates it https://github.com/ceph/ceph/blob/nautilus/src/os/bluestore/BlueStore.cc#L5931-L5934 | |
| 17:34:36 | melwitt | and somehow 'size' is passed in from the config option I assume | |
| 17:34:49 | sean-k-mooney | that is what im currently trying to find yes | |
| 17:36:36 | sean-k-mooney | this maybe https://github.com/ceph/ceph/blob/nautilus/src/os/bluestore/BlueStore.cc#L5943-L5944 | |
| 17:38:33 | sean-k-mooney | ah no its here | |