| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-11 | |||
| 12:48:39 | stephenfin | end of the month | |
| 12:49:33 | stephenfin | so all good | |
| 12:49:59 | stephenfin | Anyone tried deploying devstack on an RPi? | |
| 12:51:35 | artom | stephenfin, ah, sorry, in my head they were back to back | |
| 13:53:01 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184 | |
| 13:55:59 | dansmith | sean-k-mooney: yeah, default is uwsgi and apparently none of the magic glance stuff works without mod_wsgi.. the image import, image conversion, etc | |
| 13:56:11 | sean-k-mooney | ah i see | |
| 13:56:19 | sean-k-mooney | that is likely a glance bug... | |
| 13:56:36 | dansmith | I needed it for the copy-to-store, but forgot to copy that from my local config, but hadn't gotten far enough to notice and it's required for the conversion too | |
| 13:56:46 | dansmith | I dunno, don't we have problems with uwsgi and background tasks? | |
| 13:56:52 | dansmith | it's because those things happen in threads | |
| 13:57:07 | sean-k-mooney | oh so its the same issue as we have with the heartbeats | |
| 13:57:12 | dansmith | right | |
| 13:57:54 | sean-k-mooney | i guess you could argure that they are relying on behavior of the implemeation that is not requried | |
| 13:57:58 | sean-k-mooney | but ya fair point | |
| 13:58:29 | sean-k-mooney | e.g. the lifetime smeantics of mod_wsgi or the python one | |
| 13:58:34 | dansmith | well, it has been fine for multiple things (pre-wsgi, mod_wsgi) until uwsgi, but yeah they could use a conductor like service | |
| 13:58:36 | dansmith | but that kinda sucks | |
| 13:58:49 | dansmith | and needs to be on the same machine as the api I think because of the file uploads it needs to access | |
| 13:59:18 | sean-k-mooney | ya fair enough | |
| 13:59:45 | dansmith | anyway, it'll be a dependency for using this stuff | |
| 14:13:06 | dansmith | sean-k-mooney: so, I'm tweaking my spec to remove the "functional tests should be easy" section, which I've realized is actually going to be super difficult | |
| 14:13:21 | dansmith | because we stub out ALL of the image backend stuff in functional tests for libvirt, | |
| 14:13:40 | dansmith | and would require me to write at least 10x the actual code to just have functional support | |
| 14:13:40 | sean-k-mooney | i assume we are just going to keep your new multistore ceph job ya | |
| 14:13:41 | dansmith | instead, | |
| 14:13:46 | sean-k-mooney | if so i think that is more then enough | |
| 14:14:20 | dansmith | I'm thinking we just change the existing job to do what I have here, which means the first tempest test kicks a copy-to-store from the file backend, and then the rest of the tests just have it set as it would | |
| 14:14:30 | dansmith | yeah, cool | |
| 14:15:02 | dansmith | my only concern is what happens if/when we test snapshot with the file backend as the default, | |
| 14:15:05 | sean-k-mooney | if we have a regression in the future i think we can corss the function test bridge then | |
| 14:15:24 | dansmith | but if that does something stupid I might need to do some devstack surgery to make the rbd still the default, but upload the cirros image to the file backend | |
| 14:15:57 | sean-k-mooney | well i think we can specify the store on upload | |
| 14:16:04 | dansmith | yeah, we can, | |
| 14:16:10 | dansmith | just requires devstack surgery | |
| 14:16:17 | dansmith | but I want to see if it just does the smarter thing | |
| 14:16:33 | sean-k-mooney | you are setting the default to cheap | |
| 14:16:38 | sean-k-mooney | whcih is the file backend right | |
| 14:16:43 | sean-k-mooney | but novas to robust | |
| 14:16:53 | sean-k-mooney | so the glacce upload shoudl be to file | |
| 14:17:19 | sean-k-mooney | and then we will use ceph when doing normal snapshots right | |
| 14:17:36 | dansmith | right, nova doesn't specify one when it does the snapshot.. however, since it's rbd-backed, I think snapshot does the ceph stuff under the covers, so it may work just by virtue of the fact that it doesn't actually upload, but just tells glance about it | |
| 14:17:53 | dansmith | the cirros upload will go to file currently, yeah | |
| 14:18:24 | sean-k-mooney | is this job using the rbd image backend | |
| 14:18:34 | dansmith | yeah, that's the whole point :) | |
| 14:18:37 | sean-k-mooney | or are we only using ceph for bfv | |
| 14:18:47 | sean-k-mooney | ya that is what i assuemed just checking | |
| 14:18:48 | dansmith | no, this is rbd on the image backend | |
| 14:19:44 | sean-k-mooney | ya so im wondering if we want to either add a post playbook to test the file snapshot or if we want to add multi store to another job for that case | |
| 14:20:09 | sean-k-mooney | for example we have a multi cell job right | |
| 14:20:27 | sean-k-mooney | we could configure multisore on that too and check it uploads to the cell local one | |
| 14:21:40 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184 | |
| 14:26:00 | sean-k-mooney | dansmith: by the way before aarents upload a new version were you expecting to reuse this config option or add a new one https://review.opendev.org/#/c/734776/2/nova/conf/compute.py@912 | |
| 14:26:54 | dansmith | sean-k-mooney: tbh, I didn't know we had that one | |
| 14:27:45 | dansmith | I'm not really sure why we have that, as image downloads would be gated by the build limit | |
| 14:27:47 | sean-k-mooney | so when i asked about it in the ptg i tought we siad it was only for the filter schdulier btu i guess there is a seperate one for compute | |
| 14:28:18 | dansmith | that config you linked isn't for the filter/weigher, but for compute presumably yeah? | |
| 14:28:31 | sean-k-mooney | correct its in compute | |
| 14:29:03 | sean-k-mooney | but do we use it anywhere https://github.com/openstack/nova/search?q=max_concurrent_disk_ops&unscoped_q=max_concurrent_disk_ops | |
| 14:29:30 | sean-k-mooney | it was added by https://github.com/openstack/nova/commit/728f20e8f4ac2e3d4b893b7169b81d20471d0be9 | |
| 14:29:31 | dansmith | in libvirt | |
| 14:30:32 | sean-k-mooney | ah we do with compute_utils.disk_ops_semaphore | |
| 14:31:25 | dansmith | I mean... I was expecting there to be one for snapshotting as a whole, so we didn't even start a high-level snapshot operation until we could get to it, | |
| 14:31:38 | dansmith | this would presumably lock much further down the chain, so you're already paused and waiting for room | |
| 14:32:02 | dansmith | which seems less than ideal to me | |
| 14:32:20 | sean-k-mooney | ya it looks like its only locking on the actull io operation i.e. invoking qemu-image or glance | |
| 14:34:28 | dansmith | right | |
| 14:34:32 | dansmith | I commented | |
| 14:36:40 | sean-k-mooney | dansmith: that current semaphore is used in snapshot but only at the end https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L2465 | |
| 14:37:24 | dansmith | ah for the actual upload I guess? | |
| 14:37:48 | dansmith | seems like that semaphore was added kinda willy-nilly in a bunch of places without a systemic view of "never call this code without the semaphore held" | |
| 14:37:57 | stephenfin | sean-k-mooney: In brief, what would happen if you configured disk_allocation_ratio too high? Would you see outright corruption of existing images or simply failure to e.g. download images for a new instance? | |
| 14:37:57 | dansmith | because I assume if we grab it too high we'll double acquire | |
| 14:38:26 | stephenfin | I'm not actually sure what happens with sparse image if you run out of disk space | |
| 14:38:33 | stephenfin | *a | |
| 14:38:38 | dansmith | stephenfin: you'll get disk errors in the guest when you run out of space | |
| 14:38:49 | dansmith | stephenfin: like your disk was failing if it was a real disk | |
| 14:39:02 | stephenfin | and if I tried to schedule a new guest to that host? | |
| 14:39:09 | sean-k-mooney | dansmith: ya probably | |
| 14:39:33 | dansmith | which generally means "you asked me to write this to disk, disk said no" which if it's in the journal flush could mean whole fs corruption, but likely a remount-readonly at minimum and application data loss | |
| 14:40:01 | sean-k-mooney | you can allso get into cases where you cant log in or exectue commands | |
| 14:40:09 | sean-k-mooney | in the guest | |
| 14:40:10 | dansmith | for sure | |
| 14:40:21 | stephenfin | dansmith++ sweet, thanks | |
| 14:40:37 | sean-k-mooney | i guess on the host too if the nova instnace directoy is in / | |
| 14:40:39 | stephenfin | I'm writing an overcommit doc atm and trying to tease out implications of these things | |
| 14:40:45 | sean-k-mooney | or under it on the same partion | |
| 14:41:04 | dansmith | sean-k-mooney: yes, it could be quite bad | |
| 14:41:17 | sean-k-mooney | stephenfin: honestly disk allocation ratio shoudl have a max of 1 | |
| 14:41:20 | dansmith | sean-k-mooney: presumably the host fs has some reserved-for-root set and qemu can't exhaust that though | |
| 14:41:49 | dansmith | sean-k-mooney: agree, overcommit of disk is just asking for problems | |
| 14:41:49 | sean-k-mooney | dansmith: you would hope. | |
| 14:42:11 | sean-k-mooney | peopel get a way with it with lvm thin providiouning by carfully monitoring it | |
| 14:42:24 | sean-k-mooney | but it makes me uncomfortable if i care about the data | |
| 14:42:42 | dansmith | it's the worst thing you can overcommit | |
| 14:43:07 | sean-k-mooney | dansmith: the host_reserved_disk_mb helps but if you allow suspend you can exceed that as we need to snapshot the guest ram and save it to disk | |
| 14:44:12 | sean-k-mooney | when kolla was young it did nto have logrotate and i made the mistake of running our internal intel dev cluserter with debug logging on | |
| 14:44:24 | sean-k-mooney | fixing out of disk was fun... | |
| 14:44:40 | sean-k-mooney | because it caused db curruption | |
| 14:44:55 | dansmith | I think stephenfin gets the picture.. "it's bad, yo" | |