| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 15:26:57 | bauzas | tl;dr: ssh: connect to host review.openstack.org port 29418: Network is unreachable | |
| 15:27:11 | bauzas | have I missed a memo ? | |
| 15:27:25 | melwitt | there's a openstackstatus above ^ said there will be a short outage | |
| 15:29:00 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: WIP: Offline Reshape tool spec https://review.opendev.org/742908 | |
| 15:29:05 | bauzas | yay, it worked | |
| 15:29:09 | bauzas | melwitt: thanks | |
| 15:29:46 | bauzas | calling it a day | |
| 15:32:15 | melwitt | dansmith: MAX_AVAIL should be total actually, just taking number of replicas into account. if you only have 1 replica (default NUM_REPLICAS=1) then MAX_AVAIL should match whatever total says in 'ceph df' | |
| 15:32:49 | dansmith | melwitt: you're reporting free as max_avail though in that thing aren't you? | |
| 15:32:59 | dansmith | or does MAX_AVAIL != max_avail ? | |
| 15:32:59 | melwitt | but if you've set NUM_REPLICAS=2 when you deployed a devstack, then since the devstack ceph plugin creates 2 OSDs on the same HDD in that case, it would be 2x the real disk | |
| 15:33:07 | melwitt | no MAX_AVAIL is a ceph thing | |
| 15:33:30 | melwitt | (if you're referring to what is written about ceph df in rbd_utils.py) | |
| 15:33:32 | dansmith | you mean half the disk I assume | |
| 15:33:34 | dansmith | yeah | |
| 15:33:57 | melwitt | no like the old behavior used to report 20G if you had a 10G disk, of you had created 2 OSDs that point at the same HDD | |
| 15:34:02 | dansmith | so maybe (24 - overhead) / 2 == 10 or something | |
| 15:34:22 | melwitt | you're using NUM_REPLICAS=1 right? you didn't set it in the job | |
| 15:34:31 | melwitt | if so, there shouldn't be a difference | |
| 15:34:38 | dansmith | I'm not setting it, but let me look if it's getting set | |
| 15:35:06 | melwitt | I doubt it, I've never seen it set in CI before. I had to set it locally to do the testing for that MAX_AVAIL change | |
| 15:35:26 | dansmith | yeah I don't even see that variable anywhere | |
| 15:35:41 | dansmith | is that a devstack-plugin-ceph thing? | |
| 15:35:45 | melwitt | yeah sec | |
| 15:36:02 | lyarwood | dansmith: https://docs.ceph.com/docs/jewel/rados/operations/pools/#create-a-pool ; sudo ceph -c /etc/ceph/ceph.conf osd pool create vms 8 8 ; that doesn't mean create a 8GB pool | |
| 15:36:09 | melwitt | bah sorry it's CEPH_REPLICAS https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L109 | |
| 15:36:15 | dansmith | lyarwood: yeah we established that :) | |
| 15:36:22 | lyarwood | ah sorry wasn't watching irc | |
| 15:36:30 | dansmith | lyarwood: somewhere in the plugin I saw a comment that made it sound like that was size | |
| 15:37:16 | melwitt | 10G honestly I would have thought is just the cloud image's disk size, no? | |
| 15:37:20 | dansmith | melwitt: yeah 1 | |
| 15:37:23 | melwitt | or do we probably use something larger in CI | |
| 15:37:26 | dansmith | melwitt: no, said above, it's 24G | |
| 15:37:37 | dansmith | melwitt: https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/df.txt | |
| 15:37:46 | dansmith | and it's overridden to 24G in the devstack log | |
| 15:37:50 | lyarwood | that's total for the three different pools | |
| 15:37:59 | lyarwood | vms images and volumes? | |
| 15:38:04 | melwitt | oh I see | |
| 15:38:14 | dansmith | lyarwood: and are the pools set to something specific for size? that's what we're trying to find and can't :) | |
| 15:38:38 | dansmith | lyarwood: the way it looks now I'd assume it just reports that they're all 24G in size, with various amounts free like zfs does for filesystems on a pool, | |
| 15:38:42 | dansmith | but I'm just guessing | |
| 15:38:52 | dansmith | I'm stacking a ceph devstack so I can poke but right now all I have is logs | |
| 15:39:28 | dansmith | if total decreases as we use space, then we're not really reporting the right thing to placement | |
| 15:39:32 | dansmith | which could be part of the problem of coruse | |
| 15:40:02 | lyarwood | dansmith: yup true and that's also going to bounce around alot during a tempest run | |
| 15:40:07 | dansmith | yep | |
| 15:40:34 | dansmith | I'm pretty sure this is not a consequence of my job, by the way, I think mine is just a little slower because we have some glance features turned on, so we probably have a little more of a logjam than normal | |
| 15:41:06 | dansmith | oh jeez, you know what I just realized? | |
| 15:41:25 | dansmith | we might be snapshotting to the file store and not the ceph store in some cases, actually | |
| 15:41:27 | dansmith | hmm | |
| 15:41:55 | dansmith | nova does the snapshots itself so maybe not, but if we ever do a raw image upload.. the default store is the file store | |
| 15:42:04 | dansmith | not that that would cause this, but it might be changing the timing characteristics | |
| 15:42:32 | dansmith | I'll have to think on that a bit | |
| 15:42:33 | melwitt | well, this doesn't look promising for MAX_AVAIL, it sounds like it would decrease with use and is not a total https://access.redhat.com/solutions/3537961 | |
| 15:43:08 | dansmith | ah yeah | |
| 15:43:41 | dansmith | melwitt: did you read this? https://access.redhat.com/solutions/2273951 | |
| 15:44:03 | dansmith | we're not replicated I guess so maybe that doesn't affect us in CI, but probably has some impact for real users of this | |
| 15:44:12 | melwitt | no | |
| 15:45:47 | melwitt | so there are multiple reasons MAX_AVAIL shouldn't be used :( | |
| 15:46:31 | dansmith | not it! | |
| 15:47:42 | dansmith | the other problem I'm guessing, | |
| 15:47:44 | melwitt | yeah... I'm thinking whether to revert that or tweak it to take total and divide by pool size, the latter would do what was actually desired and report total with replication considered | |
| 15:48:11 | dansmith | is that if we report the real actual total (even minus replication overhead), but other pools can consume space from the same store, | |
| 15:48:17 | dansmith | we will tell placement we have more room than it can allocate | |
| 15:48:50 | dansmith | so really we need to sum up all the pools on the same store, and then set reserved= for any space they use I guess, but then we race with those other uses in our reporting | |
| 15:48:55 | dansmith | and could go negative | |
| 15:49:19 | melwitt | yeah, I'm trying to remember, I could have sworn this get_pool_info was only used to report free space, not total space, but I could be totally making that up | |
| 15:49:45 | melwitt | or that that's what it's used for ultimately in higher layers | |
| 15:50:23 | melwitt | let me look up what "total" used to be, maybe it meant "total available" | |
| 15:51:48 | melwitt | no, looks like it was total. had total, total used, and total available | |
| 15:52:47 | sean-k-mooney | dansmith: one thing that i just tought of | |
| 15:53:02 | sean-k-mooney | by default replicate pools have a replciation factor of 3 | |
| 15:53:17 | sean-k-mooney | so if we have 24G of space we would only have 8 useable | |
| 15:53:31 | melwitt | but looking at the clip again https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L374-L382 I did parse out total_bytes to go with 'total', max_avail to go with 'free', and bytes_used to go with 'used'. so this should be fine.... | |
| 15:53:32 | dansmith | I'm confused about whether we're replicating or not | |
| 15:53:36 | dansmith | sean-k-mooney: ^ | |
| 15:53:51 | sean-k-mooney | that is the default unless we create a erasure encoded pool | |
| 15:53:54 | dansmith | and even still, 24/3==10 only for very small values of 3 :P | |
| 15:54:13 | dansmith | hmm, okay what is CEPH_REPLICAS then? | |
| 15:54:31 | melwitt | that's the number of replicas for when it creates the pools | |
| 15:54:41 | sean-k-mooney | well we have 24G for ceph but we have multiple pools right? | |
| 15:54:51 | sean-k-mooney | the images pool will also be using that | |
| 15:55:01 | dansmith | right, vms and images | |
| 15:56:55 | sean-k-mooney | https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L109 | |
| 15:56:57 | sean-k-mooney | its 1 | |
| 15:57:05 | sean-k-mooney | CEPH_REPLICAS | |
| 15:57:22 | sean-k-mooney | wich for ci makes sense | |
| 15:57:23 | dansmith | right, I think we established that earlier :) | |
| 15:57:31 | melwitt | yeah, I was saying earlier I've never seen CI use anything other than the default of 1 | |
| 15:58:09 | sean-k-mooney | well we dont need to test anything else in our ci since we are not really testing ceph | |
| 15:58:18 | dansmith | https://pastebin.com/6gcGhTHQ | |
| 15:58:21 | sean-k-mooney | just ceph integration with other thngs | |
| 15:58:26 | dansmith | this is what my ceph df shows on a clean devstack | |
| 15:58:38 | dansmith | interestingly I didn't update my backing size from 8 to 24, but still got 24 | |
| 15:58:50 | gibi_pto | so I'm going away for a week. I will be back on 3rd of Aug | |
| 15:59:01 | dansmith | gibi_pto: p/ | |
| 15:59:06 | gibi_pto | o/ | |
| 15:59:10 | lyarwood | \o | |
| 15:59:46 | sean-k-mooney | dansmith: i think VOLUME_BACKING_FILE_SIZE is a devstack setting | |
| 16:00:05 | dansmith | oh, I see, and ceph plugin uses that, gotcha | |