| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 16:12:08 | dansmith | fwiw, I don't have a block file (yet) and mine is reporting 24G | |
| 16:12:11 | dansmith | so i dunno why it's different | |
| 16:13:00 | dansmith | ah, my osd0.log: | |
| 16:13:01 | dansmith | xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf | |
| 16:13:16 | dansmith | so that's different than bluestore I guess you're saying? | |
| 16:13:35 | sean-k-mooney | i think if we just left /var/lib/ceph mounted under / as part of the root filestem and moved wehere we mount the 24G loopback device file to /var/lib/ceph/osd/ceph-0/block ceph would have all 24G | |
| 16:13:47 | sean-k-mooney | dansmith: yes that is teh filestore backend | |
| 16:13:51 | sean-k-mooney | that use need a folder to use | |
| 16:13:55 | dansmith | ah, CI is using the nautilus version of ceph, I'm on luminous | |
| 16:14:11 | sean-k-mooney | luminous is the defualt in the devstack plugin ya | |
| 16:14:17 | dansmith | but CI is using nautilus | |
| 16:14:18 | sean-k-mooney | but ci i guess is overriding it | |
| 16:14:29 | dansmith | ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable), process ceph-osd, pid 8952 | |
| 16:14:38 | sean-k-mooney | ya | |
| 16:14:44 | dansmith | can we set the backing store driver? | |
| 16:14:48 | dansmith | back to xfs? | |
| 16:15:07 | sean-k-mooney | we could but i think the better solution is to change how we do the mounting | |
| 16:15:14 | sean-k-mooney | bluestore is not the default | |
| 16:15:17 | sean-k-mooney | upstream | |
| 16:15:27 | sean-k-mooney | in ceph and downstream as of osp 16 | |
| 16:15:38 | sean-k-mooney | so its nice to test with bluestore | |
| 16:15:54 | dansmith | unless you see where we're setting it to bluestore, it would seem maybe the default changed in nautilus? | |
| 16:16:22 | sean-k-mooney | well after lumious in any case but yes i dont think we currently set it directly | |
| 16:16:54 | dansmith | you said "bluestore is not the default" above | |
| 16:17:03 | dansmith | so I'm confused about what you're proposing | |
| 16:17:42 | sean-k-mooney | oh i ment is | |
| 16:17:49 | sean-k-mooney | it is now the default in ceph | |
| 16:17:58 | sean-k-mooney | i filestore used to be the default before | |
| 16:18:14 | dansmith | okay that's what I was saying | |
| 16:18:36 | dansmith | I still don't get where the 10G comes from, other than that something is clearly different with blue vs xfs stores | |
| 16:18:46 | sean-k-mooney | bluestore has been the default for a few releases now. filestore is deprecated upstream and downstream in ops | |
| 16:19:06 | sean-k-mooney | dansmith: i think that is the default size that the ceph tool uses | |
| 16:19:19 | sean-k-mooney | when its creating a backing file | |
| 16:19:36 | dansmith | okay I don't see that anywhere | |
| 16:20:19 | sean-k-mooney | its being created by the ceph osd itself here https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph-osd.0_log.txt#4 | |
| 16:20:26 | dansmith | I imagine that keeping the loopback mount for var lib ceph is ideal for the plugin as long as we have stable branches that use that | |
| 16:20:38 | dansmith | sean-k-mooney: yeah I get that :) | |
| 16:20:50 | dansmith | sean-k-mooney: I'm saying I don't know where 10G is set or assumed or whatever ;) | |
| 16:21:00 | sean-k-mooney | yes we can proably change that in the job? | |
| 16:21:07 | melwitt | dansmith: is it not here? https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph-osd.0_log.txt#18 | |
| 16:21:24 | dansmith | lol | |
| 16:21:28 | dansmith | yes, I understand 10G is being used | |
| 16:21:34 | sean-k-mooney | or if the destack pluging is branched we can change it only on the branchs that use nautilus | |
| 16:21:35 | dansmith | I'm saying I don't see a config for that | |
| 16:21:54 | dansmith | https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/ | |
| 16:21:58 | sean-k-mooney | if there is a config it would be the ceph config file | |
| 16:22:03 | melwitt | oh, sorry, just saying that's a command that is setting 10G deliberately | |
| 16:22:28 | dansmith | melwitt: yep I think that's understood now | |
| 16:22:28 | melwitt | from what sean-k-mooney was saying, I thought no one saw a deliberate setting of it yet | |
| 16:22:41 | melwitt | that it was happening "automatically" | |
| 16:22:58 | melwitt | I am caught up now | |
| 16:23:16 | dansmith | all I'm saying is, I imagine that bluestore can have more than 10G of backing store, and if it's not basing that on actual disk free space, it's probably a config somewhere or something :) | |
| 16:23:51 | melwitt | yeah, I understand now | |
| 16:24:09 | melwitt | I misunderstood what sean was saying earlier | |
| 16:25:22 | dansmith | the bluestore config actually seems to be mostly focused on using physical devices | |
| 16:27:12 | sean-k-mooney | melwitt: it happens frequently that people missunderstad me. well not that often if i type what i ment to type but i am bad at not doing that | |
| 16:28:56 | melwitt | sean-k-mooney: eh, I often have trouble understanding people so by our powers combined... ! | |
| 16:30:46 | dansmith | I HAVE NO IDEA WHAT YOU PEOPLE ARE SAYING | |
| 16:31:09 | melwitt | I AM GOOD AT DEALING WITH PEOPLE | |
| 16:31:47 | sean-k-mooney | stephenfin: talking about storage does this to people | |
| 16:32:16 | sean-k-mooney | just taking a step back | |
| 16:32:30 | sean-k-mooney | we are happy we know where the 10G size is comming from now | |
| 16:32:36 | melwitt | StOrAGE | |
| 16:33:10 | sean-k-mooney | and that we are proably just hitting a real no valid host error because we are actully providing 10G to cpeh instead of 24 | |
| 16:33:28 | sean-k-mooney | so the ceph jobs failrues are not related to dansmith's recent changes to the job | |
| 16:33:35 | sean-k-mooney | yes? | |
| 16:34:10 | dansmith | sean-k-mooney: yeah I thought we were assuming that | |
| 16:34:16 | dansmith | because it's clearly just placement | |
| 16:34:31 | dansmith | my job might be slower (or faster) causing us to hit it more than we were or something | |
| 16:35:05 | sean-k-mooney | so we just either a.) swap back to file store to get the old behavior or b.) mount our loopback file in such a way that the bluestore block device uses our 24G loopback device instead of creating its own | |
| 16:35:29 | dansmith | yeah so I figured going back to xfs would be ideal for compatibility with everything | |
| 16:35:38 | dansmith | my system clearly gets xfs | |
| 16:35:51 | dansmith | I assume the workers are getting blue because they're newer ubuntu or something | |
| 16:36:00 | dansmith | I'm still on bionic | |
| 16:36:01 | sean-k-mooney | dansmith: sure but eventurally we will have to move since i think filestore is deprected in ceph | |
| 16:36:05 | dansmith | sure | |
| 16:36:16 | dansmith | pain now or pain later | |
| 16:36:21 | dansmith | pain later might be someone else's pain :P | |
| 16:36:52 | sean-k-mooney | so i guess what we are looking for is a ceph config option to select filestore for the osd backend | |
| 16:37:43 | sean-k-mooney | that or we set it on the osd create command | |
| 16:38:02 | dansmith | yeah | |
| 16:38:37 | sean-k-mooney | so this code https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L475-L484 | |
| 16:39:03 | sean-k-mooney | that inital sudo ceph -c ${CEPH_CONF_FILE} osd create | |
| 16:39:06 | dansmith | well, the other option is to figure out how to make blue use 20ish G instead of 10, | |
| 16:39:20 | dansmith | which would be less impactful than retooling the mount stuff in the ceph plugin | |
| 16:40:00 | sean-k-mooney | well we are mounting it on /var/lib/ceph | |
| 16:40:08 | sean-k-mooney | so i guess this is already plugin specific | |
| 16:40:24 | sean-k-mooney | we are likely resuing the same function that is used for cinder and just passing the mount path | |
| 16:40:57 | sean-k-mooney | ya we are just calling create_disk https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L387 | |
| 16:41:01 | dansmith | cinder just wants a loop not a mounted fs though right/ | |
| 16:41:52 | sean-k-mooney | maybe this is the fucntion in devstack https://github.com/openstack/devstack/blob/eee60c76719c02c08dba7b7fb703798a056b22b9/functions#L758-L789 | |
| 16:42:19 | sean-k-mooney | that kind of looks like a hack | |
| 16:42:41 | sean-k-mooney | e.g. that does not look like it was created orginally for ceph | |
| 16:43:26 | melwitt | hm, I found this https://forum.proxmox.com/threads/proxmox-ceph-osd-partition-created-with-only-10gb.55291/ | |
| 16:43:31 | sean-k-mooney | oh its for swift orginially | |
| 16:46:20 | sean-k-mooney | i think the "sudo ceph-osd -c ${CEPH_CONF_FILE} -i ${OSD_ID} --mkfs" is the one we would need to modify | |
| 16:49:32 | sean-k-mooney | melwitt: that does seam like the same issue more or less | |
| 16:49:48 | melwitt | yeah, I'm having trouble understanding it | |
| 16:50:50 | melwitt | the last comment links to another post https://forum.proxmox.com/threads/where-can-i-tune-journal-size-of-ceph-bluestore.44000/ where they're talking about tuning journal size and bluestore_block_db_size and bluestore_block_wal_size | |
| 16:51:02 | melwitt | and I don't know what any of that is or means | |