| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 17:57:50 | dansmith | looks like regular live migration also looks at them to determine if they're all neutron-related | |
| 17:58:07 | dansmith | just trying to confirm that blowing them away is not something we can just ignore :) | |
| 18:01:02 | efried | dansmith: I'm looking at consecutive_build_service_disable_threshold and not seeing how it could be working | |
| 18:01:24 | dansmith | efried: we removed the compute side of that, if that's what you're looking at | |
| 18:01:40 | efried | "the compute side" | |
| 18:01:53 | efried | meaning... the part that makes it behave in any way other than a bool? | |
| 18:01:54 | dansmith | the part where the compute node self-disables | |
| 18:02:21 | dansmith | there should be renos about this | |
| 18:02:33 | efried | ack | |
| 18:02:46 | dansmith | change-consecutive-boot-failure-counter-to-weigher-428de7da0ed2033a.yaml | |
| 18:03:25 | efried | dansmith: okay | |
| 18:03:26 | efried | so | |
| 18:04:00 | efried | this is now up to deployers installing their own weigher? | |
| 18:04:12 | dansmith | eh? | |
| 18:04:19 | dansmith | did you read the reno? | |
| 18:04:56 | dansmith | efried: https://review.opendev.org/#/c/572195/ | |
| 18:05:01 | efried | ack | |
| 18:05:18 | dansmith | added a weigher, changed the meaning of the threshold for compatibility reasons | |
| 18:05:25 | mriedem | https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights | |
| 18:05:28 | mriedem | BuildFailureWeigher | |
| 18:07:24 | dansmith | efried: you hip to that jive? | |
| 18:07:58 | efried | Still trying to grok the implications. | |
| 18:08:31 | efried | So in a biggish cloud, if I have a bunch of failures in a row on a particular host, even if I don't hit the default multiplier, it's still going to make scheduling to that node way less likely | |
| 18:08:36 | dansmith | mriedem: beat you to the youtube ref: https://www.youtube.com/watch?v=sgW3RxKdN0Q | |
| 18:09:04 | dansmith | efried: hit the multiplier meaning "change" it? | |
| 18:09:27 | efried | sorry, no, I just mean... | |
| 18:09:58 | efried | IIUC the default multiplier is really big so that by default you won't effectively-disable a compute until it's seen a really lot of build failures. | |
| 18:10:12 | dansmith | efried: no | |
| 18:10:20 | dansmith | efried: read the reno and commit for reasoning about the multiplier | |
| 18:10:39 | dansmith | efried: it has to compete against disk weigher, which is like scored by mb free or something | |
| 18:11:12 | dansmith | efried: a big multiplier makes it more likely to have an effect, not les | |
| 18:11:14 | dansmith | *less | |
| 18:11:18 | efried | ah | |
| 18:11:30 | dansmith | and the threshold is now just "report failures if nonzero", so it's not a threshold | |
| 18:11:57 | dansmith | if you want it off, set the "threshold" to zero and computes won't even report the number | |
| 18:12:13 | efried | ack | |
| 18:12:14 | dansmith | and if you do, then you tune the multiplier based on whatever else you have configured | |
| 18:15:11 | efried | got it. Thanks dansmith. (And no, mriedem, I'm helping donnyd figure out why his CI hosts get effectively-disabled-without-actually-being-disabled when they spend some time trying to boot with not-yet-ready images) | |
| 18:16:07 | mriedem | :P | |
| 18:16:14 | mriedem | o-) | |
| 18:16:16 | efried | dansmith: save me tracing the code, does it still have the "reset to zero" behavior as soon as we get one success? | |
| 18:16:30 | dansmith | IIRC yes | |
| 18:16:47 | dansmith | efried: https://review.opendev.org/#/c/572195/6/nova/compute/manager.py | |
| 18:16:54 | donnyd | thats pending it ever actually gets rescheduled | |
| 18:17:15 | donnyd | I have enough space that essentially it doesn't | |
| 18:17:19 | efried | yeah | |
| 18:17:23 | dansmith | https://review.opendev.org/#/c/572195/6/nova/compute/stats.py | |
| 18:17:30 | efried | dansmith: is this one of those things that you should theoretically be able to reset via SIGHUP? | |
| 18:17:34 | dansmith | see my FIXME in there | |
| 18:17:40 | dansmith | efried: for sure | |
| 18:18:22 | efried | k. btw, not sure if you saw the update, but between bnemec and me, SIGHUP is (soon to be) fixed. | |
| 18:18:38 | efried | soon as we can get this code merged & released | |
| 18:21:53 | donnyd | What i really need is a faster way to download glance images so this isn't an issue | |
| 18:24:13 | efried | just walk a flash drive across the room | |
| 18:24:28 | donnyd | LOL efried | |
| 18:25:31 | donnyd | Well the glance image store is on an nvme drive that will move at the speed of the rest of the network... however glance doesn't feel that same way, and limits my download speeds to what one core can do... which is about 100M/s | |
| 18:25:59 | donnyd | I think it has something to do with requests if I am not mistaken | |
| 18:33:02 | sean-k-mooney | donnyd: are you useing a 1G network for your managment network | |
| 18:33:09 | donnyd | 10 | |
| 18:33:19 | donnyd | and the controllers are all 40 | |
| 18:33:48 | donnyd | So on the control plane for the compute side 10, and controller side 40 | |
| 18:33:55 | mriedem | donnyd: are you using ceph? | |
| 18:33:58 | donnyd | no | |
| 18:34:15 | donnyd | filestore is the fastest i have measured thus far | |
| 18:34:18 | melwitt | mriedem: just fyi if you didn't see, I updated the multi-cell archive patches yesterday night | |
| 18:34:41 | mriedem | melwitt: i didn't, and was going to look about 30 minutes ago, but was distracted, but i'll look in a bit, thanks | |
| 18:34:50 | donnyd | i tried cinder and swift backends, and they only made it slower | |
| 18:35:15 | melwitt | mriedem: coolness, thanks | |
| 18:35:19 | mriedem | donnyd: have you tried pre-caching the images on the computes when you have a new image? | |
| 18:35:19 | donnyd | if I was on ceph, there would be no wait time at all | |
| 18:35:25 | melwitt | and sorry for the delay | |
| 18:35:59 | mriedem | donnyd: the only thing i'd have to check is if there is some config to keep the image cache manager periodic from deleting the images if there are no guests on the host using them | |
| 18:36:02 | donnyd | mriedem: I didn't know that was an option... but I am also pretty sure nodepool is all over it whenever there is a new image | |
| 18:36:48 | mriedem | the image cache stuff in nova isn't documented at all outside of config (i don't think anyway), so i wouldn't be surprised | |
| 18:36:49 | donnyd | mostly the issue is when new images are loaded, they can't be downloaded fast enough by compute | |
| 18:36:54 | mriedem | i don't know it all that well either | |
| 18:37:10 | mriedem | right, by pre-caching you wouldn't need compute to download them, the images would be there | |
| 18:37:26 | mriedem | but that's something yo'ud have to orchestrate outside of nova | |
| 18:37:48 | donnyd | well nodepool asks for them nearly the instant that they are active in glance | |
| 18:37:56 | donnyd | at least from what i can see | |
| 18:38:14 | mriedem | ok yeah so maybe wouldn't help | |
| 18:38:43 | donnyd | when you say pre-caching, i am thinking you mean have nova launch something based on the new image and then kill it immediately after its active. | |
| 18:39:00 | mriedem | that's one way | |
| 18:39:05 | mriedem | probably the easiest | |
| 18:39:16 | donnyd | I don't know how to setup the http store for glance and have it download direct from say an apache server | |
| 18:39:21 | mriedem | or you could have something push the image here https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.image_cache_subdirectory_name | |
| 18:39:58 | mriedem | jroll: doesn't verizon do extensive image pre-caching? | |
| 18:40:08 | melwitt | pre-caching (warming the cache by booting instance with new image 1 per compute host) is the only other way I know to deal with avoiding download speed issues, other than using ceph. I know at least in the past they did the cache warming thing at yahoo | |
| 18:40:24 | mriedem | ok yeah that ^ | |
| 18:40:25 | jroll | if we do it idk about it | |
| 18:40:26 | melwitt | mriedem: haha, my slow message composition strikes again | |
| 18:40:28 | mriedem | was going to ask penick but he's not here | |
| 18:40:31 | jroll | rax did some of that too | |
| 18:40:34 | donnyd | I thought about setting up a share from my block image server that would share _base across hypervisors | |
| 18:40:44 | donnyd | making it only have to download once | |
| 18:40:48 | donnyd | which is acceptable | |
| 18:41:05 | donnyd | but i am not sure of the other performance implications of doing such a thing | |
| 18:41:11 | jroll | mriedem: is there something I can help with around that? | |
| 18:41:13 | mriedem | yeah, people do that https://bugs.launchpad.net/nova/+bug/1804262 | |
| 18:41:15 | openstack | Launchpad bug 1804262 in OpenStack Compute (nova) "ComputeManager._run_image_cache_manager_pass times out when running on NFS" [Medium,In progress] - Assigned to Matthew Booth (mbooth-9) | |