| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 16:01:02 | sean-k-mooney | that was orginically for cinder | |
| 16:03:08 | sean-k-mooney | oh | |
| 16:03:09 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph_log.txt | |
| 16:03:19 | sean-k-mooney | pgmap v5: 0 pgs: ; 0 B data, 704 KiB used, 9.0 GiB / 10 GiB avail | |
| 16:03:38 | sean-k-mooney | so ceph does think it has only 10G | |
| 16:03:53 | dansmith | oh nice, but from where? | |
| 16:04:08 | sean-k-mooney | there is a ceph follder at the root of the contoler logs | |
| 16:04:22 | dansmith | from my devstack: 2020-07-24 08:25:52.400945 mgr.x client.14099 192.168.201.41:0/3299763660 2 : cluster [DBG] pgmap v5: 0 pgs: ; 0B data, 188MiB used, 23.8GiB / 24.0GiB avail | |
| 16:04:30 | dansmith | no I mean where is it getting the 10G | |
| 16:04:57 | sean-k-mooney | im wondering if we are using the filestore backend and didnt resize the filesystem or something? | |
| 16:05:09 | sean-k-mooney | although DF on the host shose 24G right | |
| 16:05:15 | dansmith | it does, | |
| 16:05:19 | sean-k-mooney | is that the block device size or filesystem | |
| 16:05:22 | dansmith | and my local devstack shows the 24G | |
| 16:05:27 | dansmith | filesystem | |
| 16:06:16 | dansmith | doesn't look like we grab the ceph configs | |
| 16:06:19 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph-osd.0_log.txt#10 | |
| 16:06:33 | sean-k-mooney | so its using filestore | |
| 16:06:42 | sean-k-mooney | but wy is that 10G | |
| 16:06:56 | sean-k-mooney | oh its using bluestore not file store | |
| 16:06:59 | sean-k-mooney | but same question | |
| 16:07:01 | dansmith | you mean files in /var/lib/ceph right? | |
| 16:07:17 | sean-k-mooney | bluestore(/var/lib/ceph/osd/ceph-0) _setup_block_symlink_or_file resized block file to 10 GiB | |
| 16:07:33 | sean-k-mooney | its not using the mount | |
| 16:07:50 | dansmith | not using the mount for what? | |
| 16:07:51 | sean-k-mooney | its creating a ceph-0 file inside it it think | |
| 16:07:58 | dansmith | sure, that's inside the mount | |
| 16:08:04 | dansmith | it's creating a 10G flat file right? | |
| 16:08:11 | sean-k-mooney | i think so | |
| 16:08:19 | sean-k-mooney | that is then being used for the osd | |
| 16:08:41 | dansmith | right | |
| 16:09:02 | sean-k-mooney | so we are creatinga 24G flatifile and attaching it as a loopback device then mounting it on /mnt/ceph | |
| 16:09:30 | sean-k-mooney | sorry | |
| 16:09:33 | sean-k-mooney | /var/lib/ceph | |
| 16:09:43 | dansmith | right | |
| 16:09:52 | sean-k-mooney | then inside that they are creating another flatfile | |
| 16:09:53 | dansmith | and then it's creating a file called block inside there as the actual thing the osd uses | |
| 16:09:57 | sean-k-mooney | and using that for the osd | |
| 16:10:05 | sean-k-mooney | yep | |
| 16:10:11 | sean-k-mooney | so this is wrong | |
| 16:10:11 | dansmith | and that thing is 10G | |
| 16:10:28 | sean-k-mooney | i think we are expecting them to use the /var/lib/ceph mound directly for the osd | |
| 16:10:46 | sean-k-mooney | i suspect this behavior changed when we changed form the filestore to bluestore backend | |
| 16:11:23 | sean-k-mooney | we should mount the loopback device at /var/lib/ceph/osd/ceph-0/block | |
| 16:11:30 | sean-k-mooney | instead that way it would have teh full 24G | |
| 16:11:31 | dansmith | I dunno what "directly" means.. they still have to store their data in their special format right? | |
| 16:11:49 | dansmith | and it's normally a raw disk they want, so if we give them a filesystem they need to create a flat file to emulate the block device on no? | |
| 16:12:08 | dansmith | fwiw, I don't have a block file (yet) and mine is reporting 24G | |
| 16:12:11 | dansmith | so i dunno why it's different | |
| 16:13:00 | dansmith | ah, my osd0.log: | |
| 16:13:01 | dansmith | xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf | |
| 16:13:16 | dansmith | so that's different than bluestore I guess you're saying? | |
| 16:13:35 | sean-k-mooney | i think if we just left /var/lib/ceph mounted under / as part of the root filestem and moved wehere we mount the 24G loopback device file to /var/lib/ceph/osd/ceph-0/block ceph would have all 24G | |
| 16:13:47 | sean-k-mooney | dansmith: yes that is teh filestore backend | |
| 16:13:51 | sean-k-mooney | that use need a folder to use | |
| 16:13:55 | dansmith | ah, CI is using the nautilus version of ceph, I'm on luminous | |
| 16:14:11 | sean-k-mooney | luminous is the defualt in the devstack plugin ya | |
| 16:14:17 | dansmith | but CI is using nautilus | |
| 16:14:18 | sean-k-mooney | but ci i guess is overriding it | |
| 16:14:29 | dansmith | ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable), process ceph-osd, pid 8952 | |
| 16:14:38 | sean-k-mooney | ya | |
| 16:14:44 | dansmith | can we set the backing store driver? | |
| 16:14:48 | dansmith | back to xfs? | |
| 16:15:07 | sean-k-mooney | we could but i think the better solution is to change how we do the mounting | |
| 16:15:14 | sean-k-mooney | bluestore is not the default | |
| 16:15:17 | sean-k-mooney | upstream | |
| 16:15:27 | sean-k-mooney | in ceph and downstream as of osp 16 | |
| 16:15:38 | sean-k-mooney | so its nice to test with bluestore | |
| 16:15:54 | dansmith | unless you see where we're setting it to bluestore, it would seem maybe the default changed in nautilus? | |
| 16:16:22 | sean-k-mooney | well after lumious in any case but yes i dont think we currently set it directly | |
| 16:16:54 | dansmith | you said "bluestore is not the default" above | |
| 16:17:03 | dansmith | so I'm confused about what you're proposing | |
| 16:17:42 | sean-k-mooney | oh i ment is | |
| 16:17:49 | sean-k-mooney | it is now the default in ceph | |
| 16:17:58 | sean-k-mooney | i filestore used to be the default before | |
| 16:18:14 | dansmith | okay that's what I was saying | |
| 16:18:36 | dansmith | I still don't get where the 10G comes from, other than that something is clearly different with blue vs xfs stores | |
| 16:18:46 | sean-k-mooney | bluestore has been the default for a few releases now. filestore is deprecated upstream and downstream in ops | |
| 16:19:06 | sean-k-mooney | dansmith: i think that is the default size that the ceph tool uses | |
| 16:19:19 | sean-k-mooney | when its creating a backing file | |
| 16:19:36 | dansmith | okay I don't see that anywhere | |
| 16:20:19 | sean-k-mooney | its being created by the ceph osd itself here https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph-osd.0_log.txt#4 | |
| 16:20:26 | dansmith | I imagine that keeping the loopback mount for var lib ceph is ideal for the plugin as long as we have stable branches that use that | |
| 16:20:38 | dansmith | sean-k-mooney: yeah I get that :) | |
| 16:20:50 | dansmith | sean-k-mooney: I'm saying I don't know where 10G is set or assumed or whatever ;) | |
| 16:21:00 | sean-k-mooney | yes we can proably change that in the job? | |
| 16:21:07 | melwitt | dansmith: is it not here? https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph-osd.0_log.txt#18 | |
| 16:21:24 | dansmith | lol | |
| 16:21:28 | dansmith | yes, I understand 10G is being used | |
| 16:21:34 | sean-k-mooney | or if the destack pluging is branched we can change it only on the branchs that use nautilus | |
| 16:21:35 | dansmith | I'm saying I don't see a config for that | |
| 16:21:54 | dansmith | https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/ | |
| 16:21:58 | sean-k-mooney | if there is a config it would be the ceph config file | |
| 16:22:03 | melwitt | oh, sorry, just saying that's a command that is setting 10G deliberately | |
| 16:22:28 | dansmith | melwitt: yep I think that's understood now | |
| 16:22:28 | melwitt | from what sean-k-mooney was saying, I thought no one saw a deliberate setting of it yet | |
| 16:22:41 | melwitt | that it was happening "automatically" | |
| 16:22:58 | melwitt | I am caught up now | |
| 16:23:16 | dansmith | all I'm saying is, I imagine that bluestore can have more than 10G of backing store, and if it's not basing that on actual disk free space, it's probably a config somewhere or something :) | |
| 16:23:51 | melwitt | yeah, I understand now | |