| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 18:09:58 | efried | IIUC the default multiplier is really big so that by default you won't effectively-disable a compute until it's seen a really lot of build failures. | |
| 18:10:12 | dansmith | efried: no | |
| 18:10:20 | dansmith | efried: read the reno and commit for reasoning about the multiplier | |
| 18:10:39 | dansmith | efried: it has to compete against disk weigher, which is like scored by mb free or something | |
| 18:11:12 | dansmith | efried: a big multiplier makes it more likely to have an effect, not les | |
| 18:11:14 | dansmith | *less | |
| 18:11:18 | efried | ah | |
| 18:11:30 | dansmith | and the threshold is now just "report failures if nonzero", so it's not a threshold | |
| 18:11:57 | dansmith | if you want it off, set the "threshold" to zero and computes won't even report the number | |
| 18:12:13 | efried | ack | |
| 18:12:14 | dansmith | and if you do, then you tune the multiplier based on whatever else you have configured | |
| 18:15:11 | efried | got it. Thanks dansmith. (And no, mriedem, I'm helping donnyd figure out why his CI hosts get effectively-disabled-without-actually-being-disabled when they spend some time trying to boot with not-yet-ready images) | |
| 18:16:07 | mriedem | :P | |
| 18:16:14 | mriedem | o-) | |
| 18:16:16 | efried | dansmith: save me tracing the code, does it still have the "reset to zero" behavior as soon as we get one success? | |
| 18:16:30 | dansmith | IIRC yes | |
| 18:16:47 | dansmith | efried: https://review.opendev.org/#/c/572195/6/nova/compute/manager.py | |
| 18:16:54 | donnyd | thats pending it ever actually gets rescheduled | |
| 18:17:15 | donnyd | I have enough space that essentially it doesn't | |
| 18:17:19 | efried | yeah | |
| 18:17:23 | dansmith | https://review.opendev.org/#/c/572195/6/nova/compute/stats.py | |
| 18:17:30 | efried | dansmith: is this one of those things that you should theoretically be able to reset via SIGHUP? | |
| 18:17:34 | dansmith | see my FIXME in there | |
| 18:17:40 | dansmith | efried: for sure | |
| 18:18:22 | efried | k. btw, not sure if you saw the update, but between bnemec and me, SIGHUP is (soon to be) fixed. | |
| 18:18:38 | efried | soon as we can get this code merged & released | |
| 18:21:53 | donnyd | What i really need is a faster way to download glance images so this isn't an issue | |
| 18:24:13 | efried | just walk a flash drive across the room | |
| 18:24:28 | donnyd | LOL efried | |
| 18:25:31 | donnyd | Well the glance image store is on an nvme drive that will move at the speed of the rest of the network... however glance doesn't feel that same way, and limits my download speeds to what one core can do... which is about 100M/s | |
| 18:25:59 | donnyd | I think it has something to do with requests if I am not mistaken | |
| 18:33:02 | sean-k-mooney | donnyd: are you useing a 1G network for your managment network | |
| 18:33:09 | donnyd | 10 | |
| 18:33:19 | donnyd | and the controllers are all 40 | |
| 18:33:48 | donnyd | So on the control plane for the compute side 10, and controller side 40 | |
| 18:33:55 | mriedem | donnyd: are you using ceph? | |
| 18:33:58 | donnyd | no | |
| 18:34:15 | donnyd | filestore is the fastest i have measured thus far | |
| 18:34:18 | melwitt | mriedem: just fyi if you didn't see, I updated the multi-cell archive patches yesterday night | |
| 18:34:41 | mriedem | melwitt: i didn't, and was going to look about 30 minutes ago, but was distracted, but i'll look in a bit, thanks | |
| 18:34:50 | donnyd | i tried cinder and swift backends, and they only made it slower | |
| 18:35:15 | melwitt | mriedem: coolness, thanks | |
| 18:35:19 | donnyd | if I was on ceph, there would be no wait time at all | |
| 18:35:19 | mriedem | donnyd: have you tried pre-caching the images on the computes when you have a new image? | |
| 18:35:25 | melwitt | and sorry for the delay | |
| 18:35:59 | mriedem | donnyd: the only thing i'd have to check is if there is some config to keep the image cache manager periodic from deleting the images if there are no guests on the host using them | |
| 18:36:02 | donnyd | mriedem: I didn't know that was an option... but I am also pretty sure nodepool is all over it whenever there is a new image | |
| 18:36:48 | mriedem | the image cache stuff in nova isn't documented at all outside of config (i don't think anyway), so i wouldn't be surprised | |
| 18:36:49 | donnyd | mostly the issue is when new images are loaded, they can't be downloaded fast enough by compute | |
| 18:36:54 | mriedem | i don't know it all that well either | |
| 18:37:10 | mriedem | right, by pre-caching you wouldn't need compute to download them, the images would be there | |
| 18:37:26 | mriedem | but that's something yo'ud have to orchestrate outside of nova | |
| 18:37:48 | donnyd | well nodepool asks for them nearly the instant that they are active in glance | |
| 18:37:56 | donnyd | at least from what i can see | |
| 18:38:14 | mriedem | ok yeah so maybe wouldn't help | |
| 18:38:43 | donnyd | when you say pre-caching, i am thinking you mean have nova launch something based on the new image and then kill it immediately after its active. | |
| 18:39:00 | mriedem | that's one way | |
| 18:39:05 | mriedem | probably the easiest | |
| 18:39:16 | donnyd | I don't know how to setup the http store for glance and have it download direct from say an apache server | |
| 18:39:21 | mriedem | or you could have something push the image here https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.image_cache_subdirectory_name | |
| 18:39:58 | mriedem | jroll: doesn't verizon do extensive image pre-caching? | |
| 18:40:08 | melwitt | pre-caching (warming the cache by booting instance with new image 1 per compute host) is the only other way I know to deal with avoiding download speed issues, other than using ceph. I know at least in the past they did the cache warming thing at yahoo | |
| 18:40:24 | mriedem | ok yeah that ^ | |
| 18:40:25 | jroll | if we do it idk about it | |
| 18:40:26 | melwitt | mriedem: haha, my slow message composition strikes again | |
| 18:40:28 | mriedem | was going to ask penick but he's not here | |
| 18:40:31 | jroll | rax did some of that too | |
| 18:40:34 | donnyd | I thought about setting up a share from my block image server that would share _base across hypervisors | |
| 18:40:44 | donnyd | making it only have to download once | |
| 18:40:48 | donnyd | which is acceptable | |
| 18:41:05 | donnyd | but i am not sure of the other performance implications of doing such a thing | |
| 18:41:11 | jroll | mriedem: is there something I can help with around that? | |
| 18:41:13 | mriedem | yeah, people do that https://bugs.launchpad.net/nova/+bug/1804262 | |
| 18:41:15 | openstack | Launchpad bug 1804262 in OpenStack Compute (nova) "ComputeManager._run_image_cache_manager_pass times out when running on NFS" [Medium,In progress] - Assigned to Matthew Booth (mbooth-9) | |
| 18:41:27 | mriedem | jroll: not really | |
| 18:41:33 | jroll | k :) | |
| 18:41:43 | mriedem | i thought i saw something (blog/talk?) where penick was talking about doing this | |
| 18:41:54 | jroll | possibly | |
| 18:42:08 | mriedem | i was probably surprised because it was some non-baremetal thing | |
| 18:42:11 | donnyd | I was only going to mount _base so the first hypervisor to download the image would speed it up for the rest | |
| 18:42:15 | jroll | just not sure if there's a question to be answered or if you're just curious | |
| 18:42:39 | jroll | rax did bare metal image caching and it was awesomesauce | |
| 18:42:50 | mriedem | donnyd: yeah i'm pretty sure i was talking with someone in here awhile back that had the same idea | |
| 18:42:53 | melwitt | yeah sharing _base/ makes sense | |
| 18:43:36 | donnyd | jroll: Well when new images are loaded from nodepool, it takes a while for compute to download from glance X # of hypervisors | |
| 18:43:55 | donnyd | so some pre-caching or sharing of _base would likely speed things up | |
| 18:44:27 | donnyd | Glance is seemingly limited to one core's worth of power when downloading | |
| 18:45:18 | donnyd | which for my controllers is unfortunately not fast | |
| 18:46:24 | donnyd | melwitt: my only concern in mounting _base on a shared drive is that what happens when multiple hypervisors are all trying to download the same image at the same time | |
| 18:47:05 | mriedem | there might be a lock in the code, but would need to verify | |
| 18:47:08 | melwitt | hm, yeah | |
| 18:47:20 | mriedem | in fact, i think starlingx people added a configurable lock there | |
| 18:47:32 | jroll | donnyd: gotcha, thanks for context | |
| 18:47:37 | donnyd | would they all download it anyways, or as mriedem just said would they see another hypervisor already has that in-flight | |
| 18:48:25 | artom | sean-k-mooney, do you recall if the SRIOV live migration code ever updated instance.pci_requests? | |
| 18:48:38 | artom | I don't see anything quickly glancing through your patches | |
| 18:48:56 | mriedem | donnyd: this is what i'm thinking of https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/imagebackend.py#L257 | |
| 18:49:18 | sean-k-mooney | we do update them when we allocate the claimed pci deivce i belive | |
| 18:50:14 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#compute.max_concurrent_disk_ops was the thing starlingx added | |
| 18:51:18 | donnyd | well I should give it a spin.. probably cut the "new image loaded so i will fail a bunch" down to almost nothing | |