| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-11 | |||
| 14:17:19 | sean-k-mooney | and then we will use ceph when doing normal snapshots right | |
| 14:17:36 | dansmith | right, nova doesn't specify one when it does the snapshot.. however, since it's rbd-backed, I think snapshot does the ceph stuff under the covers, so it may work just by virtue of the fact that it doesn't actually upload, but just tells glance about it | |
| 14:17:53 | dansmith | the cirros upload will go to file currently, yeah | |
| 14:18:24 | sean-k-mooney | is this job using the rbd image backend | |
| 14:18:34 | dansmith | yeah, that's the whole point :) | |
| 14:18:37 | sean-k-mooney | or are we only using ceph for bfv | |
| 14:18:47 | sean-k-mooney | ya that is what i assuemed just checking | |
| 14:18:48 | dansmith | no, this is rbd on the image backend | |
| 14:19:44 | sean-k-mooney | ya so im wondering if we want to either add a post playbook to test the file snapshot or if we want to add multi store to another job for that case | |
| 14:20:09 | sean-k-mooney | for example we have a multi cell job right | |
| 14:20:27 | sean-k-mooney | we could configure multisore on that too and check it uploads to the cell local one | |
| 14:21:40 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184 | |
| 14:26:00 | sean-k-mooney | dansmith: by the way before aarents upload a new version were you expecting to reuse this config option or add a new one https://review.opendev.org/#/c/734776/2/nova/conf/compute.py@912 | |
| 14:26:54 | dansmith | sean-k-mooney: tbh, I didn't know we had that one | |
| 14:27:45 | dansmith | I'm not really sure why we have that, as image downloads would be gated by the build limit | |
| 14:27:47 | sean-k-mooney | so when i asked about it in the ptg i tought we siad it was only for the filter schdulier btu i guess there is a seperate one for compute | |
| 14:28:18 | dansmith | that config you linked isn't for the filter/weigher, but for compute presumably yeah? | |
| 14:28:31 | sean-k-mooney | correct its in compute | |
| 14:29:03 | sean-k-mooney | but do we use it anywhere https://github.com/openstack/nova/search?q=max_concurrent_disk_ops&unscoped_q=max_concurrent_disk_ops | |
| 14:29:30 | sean-k-mooney | it was added by https://github.com/openstack/nova/commit/728f20e8f4ac2e3d4b893b7169b81d20471d0be9 | |
| 14:29:31 | dansmith | in libvirt | |
| 14:30:32 | sean-k-mooney | ah we do with compute_utils.disk_ops_semaphore | |
| 14:31:25 | dansmith | I mean... I was expecting there to be one for snapshotting as a whole, so we didn't even start a high-level snapshot operation until we could get to it, | |
| 14:31:38 | dansmith | this would presumably lock much further down the chain, so you're already paused and waiting for room | |
| 14:32:02 | dansmith | which seems less than ideal to me | |
| 14:32:20 | sean-k-mooney | ya it looks like its only locking on the actull io operation i.e. invoking qemu-image or glance | |
| 14:34:28 | dansmith | right | |
| 14:34:32 | dansmith | I commented | |
| 14:36:40 | sean-k-mooney | dansmith: that current semaphore is used in snapshot but only at the end https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L2465 | |
| 14:37:24 | dansmith | ah for the actual upload I guess? | |
| 14:37:48 | dansmith | seems like that semaphore was added kinda willy-nilly in a bunch of places without a systemic view of "never call this code without the semaphore held" | |
| 14:37:57 | stephenfin | sean-k-mooney: In brief, what would happen if you configured disk_allocation_ratio too high? Would you see outright corruption of existing images or simply failure to e.g. download images for a new instance? | |
| 14:37:57 | dansmith | because I assume if we grab it too high we'll double acquire | |
| 14:38:26 | stephenfin | I'm not actually sure what happens with sparse image if you run out of disk space | |
| 14:38:33 | stephenfin | *a | |
| 14:38:38 | dansmith | stephenfin: you'll get disk errors in the guest when you run out of space | |
| 14:38:49 | dansmith | stephenfin: like your disk was failing if it was a real disk | |
| 14:39:02 | stephenfin | and if I tried to schedule a new guest to that host? | |
| 14:39:09 | sean-k-mooney | dansmith: ya probably | |
| 14:39:33 | dansmith | which generally means "you asked me to write this to disk, disk said no" which if it's in the journal flush could mean whole fs corruption, but likely a remount-readonly at minimum and application data loss | |
| 14:40:01 | sean-k-mooney | you can allso get into cases where you cant log in or exectue commands | |
| 14:40:09 | sean-k-mooney | in the guest | |
| 14:40:10 | dansmith | for sure | |
| 14:40:21 | stephenfin | dansmith++ sweet, thanks | |
| 14:40:37 | sean-k-mooney | i guess on the host too if the nova instnace directoy is in / | |
| 14:40:39 | stephenfin | I'm writing an overcommit doc atm and trying to tease out implications of these things | |
| 14:40:45 | sean-k-mooney | or under it on the same partion | |
| 14:41:04 | dansmith | sean-k-mooney: yes, it could be quite bad | |
| 14:41:17 | sean-k-mooney | stephenfin: honestly disk allocation ratio shoudl have a max of 1 | |
| 14:41:20 | dansmith | sean-k-mooney: presumably the host fs has some reserved-for-root set and qemu can't exhaust that though | |
| 14:41:49 | dansmith | sean-k-mooney: agree, overcommit of disk is just asking for problems | |
| 14:41:49 | sean-k-mooney | dansmith: you would hope. | |
| 14:42:11 | sean-k-mooney | peopel get a way with it with lvm thin providiouning by carfully monitoring it | |
| 14:42:24 | sean-k-mooney | but it makes me uncomfortable if i care about the data | |
| 14:42:42 | dansmith | it's the worst thing you can overcommit | |
| 14:43:07 | sean-k-mooney | dansmith: the host_reserved_disk_mb helps but if you allow suspend you can exceed that as we need to snapshot the guest ram and save it to disk | |
| 14:44:12 | sean-k-mooney | when kolla was young it did nto have logrotate and i made the mistake of running our internal intel dev cluserter with debug logging on | |
| 14:44:24 | sean-k-mooney | fixing out of disk was fun... | |
| 14:44:40 | sean-k-mooney | because it caused db curruption | |
| 14:44:55 | dansmith | I think stephenfin gets the picture.. "it's bad, yo" | |
| 14:45:35 | sean-k-mooney | stephenfin: are you going to write doc to cover the interaction betweeen file backed memory and reserved disk by the way | |
| 14:46:05 | sean-k-mooney | if not i can proably try to write something | |
| 14:46:17 | stephenfin | you mean above what I put in the config opt docs here https://review.opendev.org/#/c/734658/ ? | |
| 14:46:43 | stephenfin | oh, reserved _disk_ | |
| 14:46:47 | sean-k-mooney | yes | |
| 14:46:49 | sean-k-mooney | disk | |
| 14:47:12 | sean-k-mooney | and i guess we shoudl menetion the reserved hugepages option too | |
| 14:47:16 | stephenfin | hmm, I hadn't thought of that. Only seeing your comments now | |
| 14:47:32 | stephenfin | Sure, I can do that early next week | |
| 14:48:06 | sean-k-mooney | ok if you dont get to it remind me and i can try to do it | |
| 14:48:36 | sean-k-mooney | basically if its the same file system they shoudl add the value of the file backed memory option to reserved disk | |
| 15:15:36 | aarents | stephenfin: if host disk is full qemu may PAUSE himself (by default I think) see man qemu werror=action,rerror=action) | |
| 15:16:34 | sean-k-mooney | dansmith: looks like glance started listening on http https://zuul.opendev.org/t/openstack/build/8afb850f7844450389823d7f84b90c61/log/controller/logs/screen-g-api.txt#345 but devstack seams to be checking https https://zuul.opendev.org/t/openstack/build/8afb850f7844450389823d7f84b90c61/log/controller/logs/devstacklog.txt#19995 | |
| 15:16:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add a guide on overcommit https://review.opendev.org/735175 | |
| 15:16:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tox: Stop linting as part of docs target https://review.opendev.org/735176 | |
| 15:17:04 | dansmith | that wsgi thing works for me locally | |
| 15:17:11 | stephenfin | sean-k-mooney, dansmith, aarents: that's the doc anyway | |
| 15:17:33 | sean-k-mooney | dansmith: i can deploy with the local.conf that was generated and see what happens | |
| 15:17:39 | dansmith | sean-k-mooney: it's getting a 503, which means it's hitting apache or something which should be proxying | |
| 15:17:56 | sean-k-mooney | im wondering if it enabling the tls code | |
| 15:18:10 | sean-k-mooney | e.g. the tls proxy | |
| 15:18:20 | sean-k-mooney | but not actully configuring it for some reason | |
| 15:18:33 | dansmith | idk | |
| 15:18:51 | sean-k-mooney | the tls proxy is enabled | |
| 15:18:55 | sean-k-mooney | https://94b38a054383d32ffbe4-eecb7ff79b5c6e121d6c20fabad8af79.ssl.cf1.rackcdn.com/734184/10/check/nova-ceph-multistore/8afb850/controller/logs/local_conf.txt | |
| 15:19:03 | dansmith | unless something changed recently | |
| 15:19:29 | dansmith | maybe the ceph plugin job I'm inheriting from does that? | |
| 15:19:42 | dansmith | because at least a couple weeks ago, devstack didn't with just that wsgi switch in the localrc | |
| 15:19:49 | sean-k-mooney | dansmith: i think its enabeld by default | |
| 15:20:09 | dansmith | in d-g? because it wasn't for me, unless it's changed very recently | |
| 15:20:11 | sean-k-mooney | in the devstack or tempest job | |
| 15:20:36 | sean-k-mooney | not d-g but the base devstack zuulv3 job i think | |
| 15:20:42 | dansmith | yeah, that's what I mean | |
| 15:20:45 | dansmith | something other than devstack defaults | |
| 15:21:00 | dansmith | well, I'd not think this would change it, unless the base devstack job breaks with wsgi | |
| 15:21:12 | dansmith | maybe glance has a mod_wsgi job I can look at | |
| 15:21:28 | dansmith | doesn't look like it | |
| 15:22:22 | sean-k-mooney | which raiase the question how do they test this? | |
| 15:22:45 | aarents | sean-k-mooney: dansmith Should I need a reno for a new config parameter ? | |
| 15:22:49 | sean-k-mooney | looking at there zuul jobs they only use the normal tepest jobs | |