| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 15:48:50 | dansmith | so really we need to sum up all the pools on the same store, and then set reserved= for any space they use I guess, but then we race with those other uses in our reporting | |
| 15:48:55 | dansmith | and could go negative | |
| 15:49:19 | melwitt | yeah, I'm trying to remember, I could have sworn this get_pool_info was only used to report free space, not total space, but I could be totally making that up | |
| 15:49:45 | melwitt | or that that's what it's used for ultimately in higher layers | |
| 15:50:23 | melwitt | let me look up what "total" used to be, maybe it meant "total available" | |
| 15:51:48 | melwitt | no, looks like it was total. had total, total used, and total available | |
| 15:52:47 | sean-k-mooney | dansmith: one thing that i just tought of | |
| 15:53:02 | sean-k-mooney | by default replicate pools have a replciation factor of 3 | |
| 15:53:17 | sean-k-mooney | so if we have 24G of space we would only have 8 useable | |
| 15:53:31 | melwitt | but looking at the clip again https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L374-L382 I did parse out total_bytes to go with 'total', max_avail to go with 'free', and bytes_used to go with 'used'. so this should be fine.... | |
| 15:53:32 | dansmith | I'm confused about whether we're replicating or not | |
| 15:53:36 | dansmith | sean-k-mooney: ^ | |
| 15:53:51 | sean-k-mooney | that is the default unless we create a erasure encoded pool | |
| 15:53:54 | dansmith | and even still, 24/3==10 only for very small values of 3 :P | |
| 15:54:13 | dansmith | hmm, okay what is CEPH_REPLICAS then? | |
| 15:54:31 | melwitt | that's the number of replicas for when it creates the pools | |
| 15:54:41 | sean-k-mooney | well we have 24G for ceph but we have multiple pools right? | |
| 15:54:51 | sean-k-mooney | the images pool will also be using that | |
| 15:55:01 | dansmith | right, vms and images | |
| 15:56:55 | sean-k-mooney | https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L109 | |
| 15:56:57 | sean-k-mooney | its 1 | |
| 15:57:05 | sean-k-mooney | CEPH_REPLICAS | |
| 15:57:22 | sean-k-mooney | wich for ci makes sense | |
| 15:57:23 | dansmith | right, I think we established that earlier :) | |
| 15:57:31 | melwitt | yeah, I was saying earlier I've never seen CI use anything other than the default of 1 | |
| 15:58:09 | sean-k-mooney | well we dont need to test anything else in our ci since we are not really testing ceph | |
| 15:58:18 | dansmith | https://pastebin.com/6gcGhTHQ | |
| 15:58:21 | sean-k-mooney | just ceph integration with other thngs | |
| 15:58:26 | dansmith | this is what my ceph df shows on a clean devstack | |
| 15:58:38 | dansmith | interestingly I didn't update my backing size from 8 to 24, but still got 24 | |
| 15:58:50 | gibi_pto | so I'm going away for a week. I will be back on 3rd of Aug | |
| 15:59:01 | dansmith | gibi_pto: p/ | |
| 15:59:06 | gibi_pto | o/ | |
| 15:59:10 | lyarwood | \o | |
| 15:59:46 | sean-k-mooney | dansmith: i think VOLUME_BACKING_FILE_SIZE is a devstack setting | |
| 16:00:05 | dansmith | oh, I see, and ceph plugin uses that, gotcha | |
| 16:00:11 | sean-k-mooney | yes https://github.com/openstack/devstack/blob/e0d06adffcf4c8da1aefebc66f2de9a440badbf6/stackrc#L766 | |
| 16:00:21 | sean-k-mooney | and devstack defaults it to 24 | |
| 16:00:39 | sean-k-mooney | so that is where that is comming form | |
| 16:01:02 | sean-k-mooney | that was orginically for cinder | |
| 16:03:08 | sean-k-mooney | oh | |
| 16:03:09 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph_log.txt | |
| 16:03:19 | sean-k-mooney | pgmap v5: 0 pgs: ; 0 B data, 704 KiB used, 9.0 GiB / 10 GiB avail | |
| 16:03:38 | sean-k-mooney | so ceph does think it has only 10G | |
| 16:03:53 | dansmith | oh nice, but from where? | |
| 16:04:08 | sean-k-mooney | there is a ceph follder at the root of the contoler logs | |
| 16:04:22 | dansmith | from my devstack: 2020-07-24 08:25:52.400945 mgr.x client.14099 192.168.201.41:0/3299763660 2 : cluster [DBG] pgmap v5: 0 pgs: ; 0B data, 188MiB used, 23.8GiB / 24.0GiB avail | |
| 16:04:30 | dansmith | no I mean where is it getting the 10G | |
| 16:04:57 | sean-k-mooney | im wondering if we are using the filestore backend and didnt resize the filesystem or something? | |
| 16:05:09 | sean-k-mooney | although DF on the host shose 24G right | |
| 16:05:15 | dansmith | it does, | |
| 16:05:19 | sean-k-mooney | is that the block device size or filesystem | |
| 16:05:22 | dansmith | and my local devstack shows the 24G | |
| 16:05:27 | dansmith | filesystem | |
| 16:06:16 | dansmith | doesn't look like we grab the ceph configs | |
| 16:06:19 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/ceph/ceph-osd.0_log.txt#10 | |
| 16:06:33 | sean-k-mooney | so its using filestore | |
| 16:06:42 | sean-k-mooney | but wy is that 10G | |
| 16:06:56 | sean-k-mooney | oh its using bluestore not file store | |
| 16:06:59 | sean-k-mooney | but same question | |
| 16:07:01 | dansmith | you mean files in /var/lib/ceph right? | |
| 16:07:17 | sean-k-mooney | bluestore(/var/lib/ceph/osd/ceph-0) _setup_block_symlink_or_file resized block file to 10 GiB | |
| 16:07:33 | sean-k-mooney | its not using the mount | |
| 16:07:50 | dansmith | not using the mount for what? | |
| 16:07:51 | sean-k-mooney | its creating a ceph-0 file inside it it think | |
| 16:07:58 | dansmith | sure, that's inside the mount | |
| 16:08:04 | dansmith | it's creating a 10G flat file right? | |
| 16:08:11 | sean-k-mooney | i think so | |
| 16:08:19 | sean-k-mooney | that is then being used for the osd | |
| 16:08:41 | dansmith | right | |
| 16:09:02 | sean-k-mooney | so we are creatinga 24G flatifile and attaching it as a loopback device then mounting it on /mnt/ceph | |
| 16:09:30 | sean-k-mooney | sorry | |
| 16:09:33 | sean-k-mooney | /var/lib/ceph | |
| 16:09:43 | dansmith | right | |
| 16:09:52 | sean-k-mooney | then inside that they are creating another flatfile | |
| 16:09:53 | dansmith | and then it's creating a file called block inside there as the actual thing the osd uses | |
| 16:09:57 | sean-k-mooney | and using that for the osd | |
| 16:10:05 | sean-k-mooney | yep | |
| 16:10:11 | sean-k-mooney | so this is wrong | |
| 16:10:11 | dansmith | and that thing is 10G | |
| 16:10:28 | sean-k-mooney | i think we are expecting them to use the /var/lib/ceph mound directly for the osd | |
| 16:10:46 | sean-k-mooney | i suspect this behavior changed when we changed form the filestore to bluestore backend | |
| 16:11:23 | sean-k-mooney | we should mount the loopback device at /var/lib/ceph/osd/ceph-0/block | |
| 16:11:30 | sean-k-mooney | instead that way it would have teh full 24G | |
| 16:11:31 | dansmith | I dunno what "directly" means.. they still have to store their data in their special format right? | |
| 16:11:49 | dansmith | and it's normally a raw disk they want, so if we give them a filesystem they need to create a flat file to emulate the block device on no? | |
| 16:12:08 | dansmith | fwiw, I don't have a block file (yet) and mine is reporting 24G | |
| 16:12:11 | dansmith | so i dunno why it's different | |
| 16:13:00 | dansmith | ah, my osd0.log: | |
| 16:13:01 | dansmith | xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf | |
| 16:13:16 | dansmith | so that's different than bluestore I guess you're saying? | |
| 16:13:35 | sean-k-mooney | i think if we just left /var/lib/ceph mounted under / as part of the root filestem and moved wehere we mount the 24G loopback device file to /var/lib/ceph/osd/ceph-0/block ceph would have all 24G | |
| 16:13:47 | sean-k-mooney | dansmith: yes that is teh filestore backend | |
| 16:13:51 | sean-k-mooney | that use need a folder to use | |
| 16:13:55 | dansmith | ah, CI is using the nautilus version of ceph, I'm on luminous | |
| 16:14:11 | sean-k-mooney | luminous is the defualt in the devstack plugin ya | |
| 16:14:17 | dansmith | but CI is using nautilus | |
| 16:14:18 | sean-k-mooney | but ci i guess is overriding it | |
| 16:14:29 | dansmith | ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable), process ceph-osd, pid 8952 | |
| 16:14:38 | sean-k-mooney | ya | |