| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 16:21:54 | dansmith | https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/ | |
| 16:21:58 | sean-k-mooney | if there is a config it would be the ceph config file | |
| 16:22:03 | melwitt | oh, sorry, just saying that's a command that is setting 10G deliberately | |
| 16:22:28 | dansmith | melwitt: yep I think that's understood now | |
| 16:22:28 | melwitt | from what sean-k-mooney was saying, I thought no one saw a deliberate setting of it yet | |
| 16:22:41 | melwitt | that it was happening "automatically" | |
| 16:22:58 | melwitt | I am caught up now | |
| 16:23:16 | dansmith | all I'm saying is, I imagine that bluestore can have more than 10G of backing store, and if it's not basing that on actual disk free space, it's probably a config somewhere or something :) | |
| 16:23:51 | melwitt | yeah, I understand now | |
| 16:24:09 | melwitt | I misunderstood what sean was saying earlier | |
| 16:25:22 | dansmith | the bluestore config actually seems to be mostly focused on using physical devices | |
| 16:27:12 | sean-k-mooney | melwitt: it happens frequently that people missunderstad me. well not that often if i type what i ment to type but i am bad at not doing that | |
| 16:28:56 | melwitt | sean-k-mooney: eh, I often have trouble understanding people so by our powers combined... ! | |
| 16:30:46 | dansmith | I HAVE NO IDEA WHAT YOU PEOPLE ARE SAYING | |
| 16:31:09 | melwitt | I AM GOOD AT DEALING WITH PEOPLE | |
| 16:31:47 | sean-k-mooney | stephenfin: talking about storage does this to people | |
| 16:32:16 | sean-k-mooney | just taking a step back | |
| 16:32:30 | sean-k-mooney | we are happy we know where the 10G size is comming from now | |
| 16:32:36 | melwitt | StOrAGE | |
| 16:33:10 | sean-k-mooney | and that we are proably just hitting a real no valid host error because we are actully providing 10G to cpeh instead of 24 | |
| 16:33:28 | sean-k-mooney | so the ceph jobs failrues are not related to dansmith's recent changes to the job | |
| 16:33:35 | sean-k-mooney | yes? | |
| 16:34:10 | dansmith | sean-k-mooney: yeah I thought we were assuming that | |
| 16:34:16 | dansmith | because it's clearly just placement | |
| 16:34:31 | dansmith | my job might be slower (or faster) causing us to hit it more than we were or something | |
| 16:35:05 | sean-k-mooney | so we just either a.) swap back to file store to get the old behavior or b.) mount our loopback file in such a way that the bluestore block device uses our 24G loopback device instead of creating its own | |
| 16:35:29 | dansmith | yeah so I figured going back to xfs would be ideal for compatibility with everything | |
| 16:35:38 | dansmith | my system clearly gets xfs | |
| 16:35:51 | dansmith | I assume the workers are getting blue because they're newer ubuntu or something | |
| 16:36:00 | dansmith | I'm still on bionic | |
| 16:36:01 | sean-k-mooney | dansmith: sure but eventurally we will have to move since i think filestore is deprected in ceph | |
| 16:36:05 | dansmith | sure | |
| 16:36:16 | dansmith | pain now or pain later | |
| 16:36:21 | dansmith | pain later might be someone else's pain :P | |
| 16:36:52 | sean-k-mooney | so i guess what we are looking for is a ceph config option to select filestore for the osd backend | |
| 16:37:43 | sean-k-mooney | that or we set it on the osd create command | |
| 16:38:02 | dansmith | yeah | |
| 16:38:37 | sean-k-mooney | so this code https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L475-L484 | |
| 16:39:03 | sean-k-mooney | that inital sudo ceph -c ${CEPH_CONF_FILE} osd create | |
| 16:39:06 | dansmith | well, the other option is to figure out how to make blue use 20ish G instead of 10, | |
| 16:39:20 | dansmith | which would be less impactful than retooling the mount stuff in the ceph plugin | |
| 16:40:00 | sean-k-mooney | well we are mounting it on /var/lib/ceph | |
| 16:40:08 | sean-k-mooney | so i guess this is already plugin specific | |
| 16:40:24 | sean-k-mooney | we are likely resuing the same function that is used for cinder and just passing the mount path | |
| 16:40:57 | sean-k-mooney | ya we are just calling create_disk https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L387 | |
| 16:41:01 | dansmith | cinder just wants a loop not a mounted fs though right/ | |
| 16:41:52 | sean-k-mooney | maybe this is the fucntion in devstack https://github.com/openstack/devstack/blob/eee60c76719c02c08dba7b7fb703798a056b22b9/functions#L758-L789 | |
| 16:42:19 | sean-k-mooney | that kind of looks like a hack | |
| 16:42:41 | sean-k-mooney | e.g. that does not look like it was created orginally for ceph | |
| 16:43:26 | melwitt | hm, I found this https://forum.proxmox.com/threads/proxmox-ceph-osd-partition-created-with-only-10gb.55291/ | |
| 16:43:31 | sean-k-mooney | oh its for swift orginially | |
| 16:46:20 | sean-k-mooney | i think the "sudo ceph-osd -c ${CEPH_CONF_FILE} -i ${OSD_ID} --mkfs" is the one we would need to modify | |
| 16:49:32 | sean-k-mooney | melwitt: that does seam like the same issue more or less | |
| 16:49:48 | melwitt | yeah, I'm having trouble understanding it | |
| 16:50:50 | melwitt | the last comment links to another post https://forum.proxmox.com/threads/where-can-i-tune-journal-size-of-ceph-bluestore.44000/ where they're talking about tuning journal size and bluestore_block_db_size and bluestore_block_wal_size | |
| 16:51:02 | melwitt | and I don't know what any of that is or means | |
| 16:51:22 | melwitt | (in ceph.conf) | |
| 17:01:31 | sean-k-mooney | those are not realated to the data storage size of the osd | |
| 17:01:55 | sean-k-mooney | blustore has an embeed database that track where the logic block are located on disk | |
| 17:02:27 | sean-k-mooney | wal i think it the write ahead log or somethingl like that | |
| 17:02:39 | sean-k-mooney | its part of how it does write journalling | |
| 17:03:02 | sean-k-mooney | in both cases they are turning parmatner for how bluestore can save its metadata | |
| 17:03:56 | sean-k-mooney | unlike file sotre it can save it inline in the blockdevice it is managening or it can save it oh external devices and they support tuneing of the sizing of them independelty | |
| 17:04:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Handle Neutron errors in _post_live_migration() https://review.opendev.org/729763 | |
| 17:09:57 | melwitt | sean-k-mooney: found a new thing https://bugzilla.redhat.com/show_bug.cgi?id=1597048 | |
| 17:09:57 | openstack | bugzilla.redhat.com bug 1597048 in RADOS "ceph osd df not showing correct disk size and causing cluster to go to full state" [High,Closed: notabug] - Assigned to bhubbard | |
| 17:10:59 | dansmith | imagine that :) | |
| 17:11:24 | melwitt | what | |
| 17:11:27 | sean-k-mooney | it should be 3.7TB but is 10G | |
| 17:11:41 | dansmith | melwitt: "not showing correct disk size" | |
| 17:11:57 | melwitt | yeah? | |
| 17:12:14 | melwitt | I'm still googling for why bluestore is maxed out at 10G | |
| 17:12:17 | dansmith | melwitt: just saying, I think we've stumbled into a realization that our df reporting on ceph in libvirt es no bueno right? | |
| 17:12:23 | melwitt | no | |
| 17:13:08 | dansmith | oh did I miss something? I thought those RHN articles were indicating that we're reporting the wrong thing still | |
| 17:13:51 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=1597048#c8 | |
| 17:13:51 | openstack | bugzilla.redhat.com bug 1597048 in RADOS "ceph osd df not showing correct disk size and causing cluster to go to full state" [High,Closed: notabug] - Assigned to bhubbard | |
| 17:14:08 | dansmith | like, if we're reporting the total size of the osd, but that's shared by vms and images, we'll be telling placement it can allocate all that space for instances but it can't | |
| 17:14:20 | sean-k-mooney | so it look like they hever actully got to the root cause of why the bluestore file was a 10G file | |
| 17:14:37 | sean-k-mooney | they just redeployed with file store an ignored it | |
| 17:15:01 | melwitt | yeah.. but then what does this mean? "The BlueStore block device was a file named with a block, not a symlink to block device partition of this disk and that file size was 10G hence it was showing the size of the OSD as 10G." | |
| 17:15:24 | sean-k-mooney | i think they ment | |
| 17:15:45 | sean-k-mooney | that in stead of it being a symlink to /dev/sdX | |
| 17:15:55 | sean-k-mooney | it was a file named block | |
| 17:15:58 | sean-k-mooney | that was 10G | |
| 17:16:13 | sean-k-mooney | hence -rw-r--r--. 1 ceph ceph 10737418240 Jul 2 16:51 block | |
| 17:16:32 | melwitt | right.. so you think it's correct that it's pointing at a file named block? and that the problem is that the file is not larger than 10G? | |
| 17:16:35 | sean-k-mooney | they were expecting it to be a symlink to the actual hdd | |
| 17:16:49 | sean-k-mooney | well in that case yes | |
| 17:16:54 | sean-k-mooney | and also likely in our case | |
| 17:17:16 | melwitt | ok. from reading that I thought maybe it was pointing wrongly at a file | |
| 17:17:34 | sean-k-mooney | /var/lib/ceph/osd/ceph-0/block is liekly a 10G file | |
| 17:17:38 | dansmith | I think that bug is that they deployed on file instead of having the bluestore osd use the disk they wanted | |
| 17:17:52 | sean-k-mooney | dansmith: yes | |
| 17:17:56 | dansmith | we want file, they wanted disk, right? | |
| 17:18:02 | sean-k-mooney | yes | |
| 17:18:17 | melwitt | oh | |
| 17:18:47 | melwitt | ok so why is /var/lib/ceph/osd/ceph-0/block only 10G ... who creates it ... | |
| 17:19:04 | dansmith | right, that I think we still don't know.. where the 10G comes from and how we change it | |
| 17:19:18 | dansmith | because the osd itself (the driver) seems to create that as a flat 10G file if it's not there | |