| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 18:02:21 | dansmith | there should be renos about this | |
| 18:02:33 | efried | ack | |
| 18:02:46 | dansmith | change-consecutive-boot-failure-counter-to-weigher-428de7da0ed2033a.yaml | |
| 18:03:25 | efried | dansmith: okay | |
| 18:03:26 | efried | so | |
| 18:04:00 | efried | this is now up to deployers installing their own weigher? | |
| 18:04:12 | dansmith | eh? | |
| 18:04:19 | dansmith | did you read the reno? | |
| 18:04:56 | dansmith | efried: https://review.opendev.org/#/c/572195/ | |
| 18:05:01 | efried | ack | |
| 18:05:18 | dansmith | added a weigher, changed the meaning of the threshold for compatibility reasons | |
| 18:05:25 | mriedem | https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights | |
| 18:05:28 | mriedem | BuildFailureWeigher | |
| 18:07:24 | dansmith | efried: you hip to that jive? | |
| 18:07:58 | efried | Still trying to grok the implications. | |
| 18:08:31 | efried | So in a biggish cloud, if I have a bunch of failures in a row on a particular host, even if I don't hit the default multiplier, it's still going to make scheduling to that node way less likely | |
| 18:08:36 | dansmith | mriedem: beat you to the youtube ref: https://www.youtube.com/watch?v=sgW3RxKdN0Q | |
| 18:09:04 | dansmith | efried: hit the multiplier meaning "change" it? | |
| 18:09:27 | efried | sorry, no, I just mean... | |
| 18:09:58 | efried | IIUC the default multiplier is really big so that by default you won't effectively-disable a compute until it's seen a really lot of build failures. | |
| 18:10:12 | dansmith | efried: no | |
| 18:10:20 | dansmith | efried: read the reno and commit for reasoning about the multiplier | |
| 18:10:39 | dansmith | efried: it has to compete against disk weigher, which is like scored by mb free or something | |
| 18:11:12 | dansmith | efried: a big multiplier makes it more likely to have an effect, not les | |
| 18:11:14 | dansmith | *less | |
| 18:11:18 | efried | ah | |
| 18:11:30 | dansmith | and the threshold is now just "report failures if nonzero", so it's not a threshold | |
| 18:11:57 | dansmith | if you want it off, set the "threshold" to zero and computes won't even report the number | |
| 18:12:13 | efried | ack | |
| 18:12:14 | dansmith | and if you do, then you tune the multiplier based on whatever else you have configured | |
| 18:15:11 | efried | got it. Thanks dansmith. (And no, mriedem, I'm helping donnyd figure out why his CI hosts get effectively-disabled-without-actually-being-disabled when they spend some time trying to boot with not-yet-ready images) | |
| 18:16:07 | mriedem | :P | |
| 18:16:14 | mriedem | o-) | |
| 18:16:16 | efried | dansmith: save me tracing the code, does it still have the "reset to zero" behavior as soon as we get one success? | |
| 18:16:30 | dansmith | IIRC yes | |
| 18:16:47 | dansmith | efried: https://review.opendev.org/#/c/572195/6/nova/compute/manager.py | |
| 18:16:54 | donnyd | thats pending it ever actually gets rescheduled | |
| 18:17:15 | donnyd | I have enough space that essentially it doesn't | |
| 18:17:19 | efried | yeah | |
| 18:17:23 | dansmith | https://review.opendev.org/#/c/572195/6/nova/compute/stats.py | |
| 18:17:30 | efried | dansmith: is this one of those things that you should theoretically be able to reset via SIGHUP? | |
| 18:17:34 | dansmith | see my FIXME in there | |
| 18:17:40 | dansmith | efried: for sure | |
| 18:18:22 | efried | k. btw, not sure if you saw the update, but between bnemec and me, SIGHUP is (soon to be) fixed. | |
| 18:18:38 | efried | soon as we can get this code merged & released | |
| 18:21:53 | donnyd | What i really need is a faster way to download glance images so this isn't an issue | |
| 18:24:13 | efried | just walk a flash drive across the room | |
| 18:24:28 | donnyd | LOL efried | |
| 18:25:31 | donnyd | Well the glance image store is on an nvme drive that will move at the speed of the rest of the network... however glance doesn't feel that same way, and limits my download speeds to what one core can do... which is about 100M/s | |
| 18:25:59 | donnyd | I think it has something to do with requests if I am not mistaken | |
| 18:33:02 | sean-k-mooney | donnyd: are you useing a 1G network for your managment network | |
| 18:33:09 | donnyd | 10 | |
| 18:33:19 | donnyd | and the controllers are all 40 | |
| 18:33:48 | donnyd | So on the control plane for the compute side 10, and controller side 40 | |
| 18:33:55 | mriedem | donnyd: are you using ceph? | |
| 18:33:58 | donnyd | no | |
| 18:34:15 | donnyd | filestore is the fastest i have measured thus far | |
| 18:34:18 | melwitt | mriedem: just fyi if you didn't see, I updated the multi-cell archive patches yesterday night | |
| 18:34:41 | mriedem | melwitt: i didn't, and was going to look about 30 minutes ago, but was distracted, but i'll look in a bit, thanks | |
| 18:34:50 | donnyd | i tried cinder and swift backends, and they only made it slower | |
| 18:35:15 | melwitt | mriedem: coolness, thanks | |
| 18:35:19 | donnyd | if I was on ceph, there would be no wait time at all | |
| 18:35:19 | mriedem | donnyd: have you tried pre-caching the images on the computes when you have a new image? | |
| 18:35:25 | melwitt | and sorry for the delay | |
| 18:35:59 | mriedem | donnyd: the only thing i'd have to check is if there is some config to keep the image cache manager periodic from deleting the images if there are no guests on the host using them | |
| 18:36:02 | donnyd | mriedem: I didn't know that was an option... but I am also pretty sure nodepool is all over it whenever there is a new image | |
| 18:36:48 | mriedem | the image cache stuff in nova isn't documented at all outside of config (i don't think anyway), so i wouldn't be surprised | |
| 18:36:49 | donnyd | mostly the issue is when new images are loaded, they can't be downloaded fast enough by compute | |
| 18:36:54 | mriedem | i don't know it all that well either | |
| 18:37:10 | mriedem | right, by pre-caching you wouldn't need compute to download them, the images would be there | |
| 18:37:26 | mriedem | but that's something yo'ud have to orchestrate outside of nova | |
| 18:37:48 | donnyd | well nodepool asks for them nearly the instant that they are active in glance | |
| 18:37:56 | donnyd | at least from what i can see | |
| 18:38:14 | mriedem | ok yeah so maybe wouldn't help | |
| 18:38:43 | donnyd | when you say pre-caching, i am thinking you mean have nova launch something based on the new image and then kill it immediately after its active. | |
| 18:39:00 | mriedem | that's one way | |
| 18:39:05 | mriedem | probably the easiest | |
| 18:39:16 | donnyd | I don't know how to setup the http store for glance and have it download direct from say an apache server | |
| 18:39:21 | mriedem | or you could have something push the image here https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.image_cache_subdirectory_name | |
| 18:39:58 | mriedem | jroll: doesn't verizon do extensive image pre-caching? | |
| 18:40:08 | melwitt | pre-caching (warming the cache by booting instance with new image 1 per compute host) is the only other way I know to deal with avoiding download speed issues, other than using ceph. I know at least in the past they did the cache warming thing at yahoo | |
| 18:40:24 | mriedem | ok yeah that ^ | |
| 18:40:25 | jroll | if we do it idk about it | |
| 18:40:26 | melwitt | mriedem: haha, my slow message composition strikes again | |
| 18:40:28 | mriedem | was going to ask penick but he's not here | |
| 18:40:31 | jroll | rax did some of that too | |
| 18:40:34 | donnyd | I thought about setting up a share from my block image server that would share _base across hypervisors | |
| 18:40:44 | donnyd | making it only have to download once | |
| 18:40:48 | donnyd | which is acceptable | |
| 18:41:05 | donnyd | but i am not sure of the other performance implications of doing such a thing | |
| 18:41:11 | jroll | mriedem: is there something I can help with around that? | |
| 18:41:13 | mriedem | yeah, people do that https://bugs.launchpad.net/nova/+bug/1804262 | |
| 18:41:15 | openstack | Launchpad bug 1804262 in OpenStack Compute (nova) "ComputeManager._run_image_cache_manager_pass times out when running on NFS" [Medium,In progress] - Assigned to Matthew Booth (mbooth-9) | |
| 18:41:27 | mriedem | jroll: not really | |
| 18:41:33 | jroll | k :) | |
| 18:41:43 | mriedem | i thought i saw something (blog/talk?) where penick was talking about doing this | |
| 18:41:54 | jroll | possibly | |
| 18:42:08 | mriedem | i was probably surprised because it was some non-baremetal thing | |
| 18:42:11 | donnyd | I was only going to mount _base so the first hypervisor to download the image would speed it up for the rest | |
| 18:42:15 | jroll | just not sure if there's a question to be answered or if you're just curious | |