| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 16:07:51 | sean-k-mooney | its creating a ceph-0 file inside it it think | |
| 16:07:58 | dansmith | sure, that's inside the mount | |
| 16:08:04 | dansmith | it's creating a 10G flat file right? | |
| 16:08:11 | sean-k-mooney | i think so | |
| 16:08:19 | sean-k-mooney | that is then being used for the osd | |
| 16:08:41 | dansmith | right | |
| 16:09:02 | sean-k-mooney | so we are creatinga 24G flatifile and attaching it as a loopback device then mounting it on /mnt/ceph | |
| 16:09:30 | sean-k-mooney | sorry | |
| 16:09:33 | sean-k-mooney | /var/lib/ceph | |
| 16:09:43 | dansmith | right | |
| 16:09:52 | sean-k-mooney | then inside that they are creating another flatfile | |
| 16:09:53 | dansmith | and then it's creating a file called block inside there as the actual thing the osd uses | |
| 16:09:57 | sean-k-mooney | and using that for the osd | |
| 16:10:05 | sean-k-mooney | yep | |
| 16:10:11 | sean-k-mooney | so this is wrong | |
| 16:10:11 | dansmith | and that thing is 10G | |
| 16:10:28 | sean-k-mooney | i think we are expecting them to use the /var/lib/ceph mound directly for the osd | |
| 16:10:46 | sean-k-mooney | i suspect this behavior changed when we changed form the filestore to bluestore backend | |
| 16:11:23 | sean-k-mooney | we should mount the loopback device at /var/lib/ceph/osd/ceph-0/block | |
| 16:11:30 | sean-k-mooney | instead that way it would have teh full 24G | |
| 16:11:31 | dansmith | I dunno what "directly" means.. they still have to store their data in their special format right? | |
| 16:11:49 | dansmith | and it's normally a raw disk they want, so if we give them a filesystem they need to create a flat file to emulate the block device on no? | |
| 16:12:08 | dansmith | fwiw, I don't have a block file (yet) and mine is reporting 24G | |
| 16:12:11 | dansmith | so i dunno why it's different | |
| 16:13:00 | dansmith | ah, my osd0.log: | |
| 16:13:01 | dansmith | xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf | |
| 16:13:16 | dansmith | so that's different than bluestore I guess you're saying? | |
| 16:13:35 | sean-k-mooney | i think if we just left /var/lib/ceph mounted under / as part of the root filestem and moved wehere we mount the 24G loopback device file to /var/lib/ceph/osd/ceph-0/block ceph would have all 24G | |
| 16:13:47 | sean-k-mooney | dansmith: yes that is teh filestore backend | |
| 16:13:51 | sean-k-mooney | that use need a folder to use | |
| 16:13:55 | dansmith | ah, CI is using the nautilus version of ceph, I'm on luminous | |
| 16:14:11 | sean-k-mooney | luminous is the defualt in the devstack plugin ya | |
| 16:14:17 | dansmith | but CI is using nautilus | |
| 16:14:18 | sean-k-mooney | but ci i guess is overriding it | |
| 16:14:29 | dansmith | ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable), process ceph-osd, pid 8952 | |
| 16:14:38 | sean-k-mooney | ya | |
| 16:14:44 | dansmith | can we set the backing store driver? | |
| 16:14:48 | dansmith | back to xfs? | |
| 16:15:07 | sean-k-mooney | we could but i think the better solution is to change how we do the mounting | |
| 16:15:14 | sean-k-mooney | bluestore is not the default | |
| 16:15:17 | sean-k-mooney | upstream | |
| 16:15:27 | sean-k-mooney | in ceph and downstream as of osp 16 | |
| 16:15:38 | sean-k-mooney | so its nice to test with bluestore | |
| 16:15:54 | dansmith | unless you see where we're setting it to bluestore, it would seem maybe the default changed in nautilus? | |
| 16:16:22 | sean-k-mooney | well after lumious in any case but yes i dont think we currently set it directly | |
| 16:16:54 | dansmith | you said "bluestore is not the default" above | |
| 16:17:03 | dansmith | so I'm confused about what you're proposing | |
| 16:17:42 | sean-k-mooney | oh i ment is | |
| 16:17:49 | sean-k-mooney | it is now the default in ceph | |
| 16:17:58 | sean-k-mooney | i filestore used to be the default before | |
| 16:18:14 | dansmith | okay that's what I was saying | |
| 16:18:36 | dansmith | I still don't get where the 10G comes from, other than that something is clearly different with blue vs xfs stores | |
| 16:18:46 | sean-k-mooney | bluestore has been the default for a few releases now. filestore is deprecated upstream and downstream in ops | |
| 16:19:06 | sean-k-mooney | dansmith: i think that is the default size that the ceph tool uses | |
| 16:19:19 | sean-k-mooney | when its creating a backing file | |
| 16:19:36 | dansmith | okay I don't see that anywhere | |
| 16:20:19 | sean-k-mooney | its being created by the ceph osd itself here https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph-osd.0_log.txt#4 | |
| 16:20:26 | dansmith | I imagine that keeping the loopback mount for var lib ceph is ideal for the plugin as long as we have stable branches that use that | |
| 16:20:38 | dansmith | sean-k-mooney: yeah I get that :) | |
| 16:20:50 | dansmith | sean-k-mooney: I'm saying I don't know where 10G is set or assumed or whatever ;) | |
| 16:21:00 | sean-k-mooney | yes we can proably change that in the job? | |
| 16:21:07 | melwitt | dansmith: is it not here? https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph-osd.0_log.txt#18 | |
| 16:21:24 | dansmith | lol | |
| 16:21:28 | dansmith | yes, I understand 10G is being used | |
| 16:21:34 | sean-k-mooney | or if the destack pluging is branched we can change it only on the branchs that use nautilus | |
| 16:21:35 | dansmith | I'm saying I don't see a config for that | |
| 16:21:54 | dansmith | https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/ | |
| 16:21:58 | sean-k-mooney | if there is a config it would be the ceph config file | |
| 16:22:03 | melwitt | oh, sorry, just saying that's a command that is setting 10G deliberately | |
| 16:22:28 | dansmith | melwitt: yep I think that's understood now | |
| 16:22:28 | melwitt | from what sean-k-mooney was saying, I thought no one saw a deliberate setting of it yet | |
| 16:22:41 | melwitt | that it was happening "automatically" | |
| 16:22:58 | melwitt | I am caught up now | |
| 16:23:16 | dansmith | all I'm saying is, I imagine that bluestore can have more than 10G of backing store, and if it's not basing that on actual disk free space, it's probably a config somewhere or something :) | |
| 16:23:51 | melwitt | yeah, I understand now | |
| 16:24:09 | melwitt | I misunderstood what sean was saying earlier | |
| 16:25:22 | dansmith | the bluestore config actually seems to be mostly focused on using physical devices | |
| 16:27:12 | sean-k-mooney | melwitt: it happens frequently that people missunderstad me. well not that often if i type what i ment to type but i am bad at not doing that | |
| 16:28:56 | melwitt | sean-k-mooney: eh, I often have trouble understanding people so by our powers combined... ! | |
| 16:30:46 | dansmith | I HAVE NO IDEA WHAT YOU PEOPLE ARE SAYING | |
| 16:31:09 | melwitt | I AM GOOD AT DEALING WITH PEOPLE | |
| 16:31:47 | sean-k-mooney | stephenfin: talking about storage does this to people | |
| 16:32:16 | sean-k-mooney | just taking a step back | |
| 16:32:30 | sean-k-mooney | we are happy we know where the 10G size is comming from now | |
| 16:32:36 | melwitt | StOrAGE | |
| 16:33:10 | sean-k-mooney | and that we are proably just hitting a real no valid host error because we are actully providing 10G to cpeh instead of 24 | |
| 16:33:28 | sean-k-mooney | so the ceph jobs failrues are not related to dansmith's recent changes to the job | |
| 16:33:35 | sean-k-mooney | yes? | |
| 16:34:10 | dansmith | sean-k-mooney: yeah I thought we were assuming that | |
| 16:34:16 | dansmith | because it's clearly just placement | |
| 16:34:31 | dansmith | my job might be slower (or faster) causing us to hit it more than we were or something | |
| 16:35:05 | sean-k-mooney | so we just either a.) swap back to file store to get the old behavior or b.) mount our loopback file in such a way that the bluestore block device uses our 24G loopback device instead of creating its own | |
| 16:35:29 | dansmith | yeah so I figured going back to xfs would be ideal for compatibility with everything | |
| 16:35:38 | dansmith | my system clearly gets xfs | |
| 16:35:51 | dansmith | I assume the workers are getting blue because they're newer ubuntu or something | |
| 16:36:00 | dansmith | I'm still on bionic | |
| 16:36:01 | sean-k-mooney | dansmith: sure but eventurally we will have to move since i think filestore is deprected in ceph | |
| 16:36:05 | dansmith | sure | |
| 16:36:16 | dansmith | pain now or pain later | |
| 16:36:21 | dansmith | pain later might be someone else's pain :P | |