| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 17:53:35 | artom | Ah, no, it does the _decrease_pool_count() thing, which "claims" the resource | |
| 17:54:20 | sean-k-mooney | how does it interact with this? | |
| 17:54:33 | sean-k-mooney | the claims have 3 states | |
| 17:54:41 | sean-k-mooney | free,cliamed and allocated | |
| 17:54:50 | sean-k-mooney | something like that | |
| 17:55:07 | sean-k-mooney | one of them meens its reserved for an instace the the otehr is its in use | |
| 17:55:10 | dansmith | I'm saying I would expect we use pci_requests if we're doing a rebuild on the instance | |
| 17:55:23 | dansmith | or other operations | |
| 17:55:43 | dansmith | point is just that corrupting our own accounting because sriov is tracked in neutron is not okay I don't think | |
| 17:55:53 | sean-k-mooney | so we are claiming the ot reserve them and later we update the instace with them weh we move it to the allocated state i think. its beed a whild and i dont really rememerb the details | |
| 17:56:08 | sean-k-mooney | this is the import cahgne https://review.opendev.org/#/c/620115/35/nova/compute/manager.py | |
| 17:56:12 | mriedem | we do'nt claim for rebuild | |
| 17:56:16 | mriedem | it's a noop claim | |
| 17:56:28 | dansmith | mriedem: sure, not related to claim | |
| 17:56:59 | dansmith | cross-cell-migrate looks at pci_requests | |
| 17:57:50 | dansmith | looks like regular live migration also looks at them to determine if they're all neutron-related | |
| 17:58:07 | dansmith | just trying to confirm that blowing them away is not something we can just ignore :) | |
| 18:01:02 | efried | dansmith: I'm looking at consecutive_build_service_disable_threshold and not seeing how it could be working | |
| 18:01:24 | dansmith | efried: we removed the compute side of that, if that's what you're looking at | |
| 18:01:40 | efried | "the compute side" | |
| 18:01:53 | efried | meaning... the part that makes it behave in any way other than a bool? | |
| 18:01:54 | dansmith | the part where the compute node self-disables | |
| 18:02:21 | dansmith | there should be renos about this | |
| 18:02:33 | efried | ack | |
| 18:02:46 | dansmith | change-consecutive-boot-failure-counter-to-weigher-428de7da0ed2033a.yaml | |
| 18:03:25 | efried | dansmith: okay | |
| 18:03:26 | efried | so | |
| 18:04:00 | efried | this is now up to deployers installing their own weigher? | |
| 18:04:12 | dansmith | eh? | |
| 18:04:19 | dansmith | did you read the reno? | |
| 18:04:56 | dansmith | efried: https://review.opendev.org/#/c/572195/ | |
| 18:05:01 | efried | ack | |
| 18:05:18 | dansmith | added a weigher, changed the meaning of the threshold for compatibility reasons | |
| 18:05:25 | mriedem | https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights | |
| 18:05:28 | mriedem | BuildFailureWeigher | |
| 18:07:24 | dansmith | efried: you hip to that jive? | |
| 18:07:58 | efried | Still trying to grok the implications. | |
| 18:08:31 | efried | So in a biggish cloud, if I have a bunch of failures in a row on a particular host, even if I don't hit the default multiplier, it's still going to make scheduling to that node way less likely | |
| 18:08:36 | dansmith | mriedem: beat you to the youtube ref: https://www.youtube.com/watch?v=sgW3RxKdN0Q | |
| 18:09:04 | dansmith | efried: hit the multiplier meaning "change" it? | |
| 18:09:27 | efried | sorry, no, I just mean... | |
| 18:09:58 | efried | IIUC the default multiplier is really big so that by default you won't effectively-disable a compute until it's seen a really lot of build failures. | |
| 18:10:12 | dansmith | efried: no | |
| 18:10:20 | dansmith | efried: read the reno and commit for reasoning about the multiplier | |
| 18:10:39 | dansmith | efried: it has to compete against disk weigher, which is like scored by mb free or something | |
| 18:11:12 | dansmith | efried: a big multiplier makes it more likely to have an effect, not les | |
| 18:11:14 | dansmith | *less | |
| 18:11:18 | efried | ah | |
| 18:11:30 | dansmith | and the threshold is now just "report failures if nonzero", so it's not a threshold | |
| 18:11:57 | dansmith | if you want it off, set the "threshold" to zero and computes won't even report the number | |
| 18:12:13 | efried | ack | |
| 18:12:14 | dansmith | and if you do, then you tune the multiplier based on whatever else you have configured | |
| 18:15:11 | efried | got it. Thanks dansmith. (And no, mriedem, I'm helping donnyd figure out why his CI hosts get effectively-disabled-without-actually-being-disabled when they spend some time trying to boot with not-yet-ready images) | |
| 18:16:07 | mriedem | :P | |
| 18:16:14 | mriedem | o-) | |
| 18:16:16 | efried | dansmith: save me tracing the code, does it still have the "reset to zero" behavior as soon as we get one success? | |
| 18:16:30 | dansmith | IIRC yes | |
| 18:16:47 | dansmith | efried: https://review.opendev.org/#/c/572195/6/nova/compute/manager.py | |
| 18:16:54 | donnyd | thats pending it ever actually gets rescheduled | |
| 18:17:15 | donnyd | I have enough space that essentially it doesn't | |
| 18:17:19 | efried | yeah | |
| 18:17:23 | dansmith | https://review.opendev.org/#/c/572195/6/nova/compute/stats.py | |
| 18:17:30 | efried | dansmith: is this one of those things that you should theoretically be able to reset via SIGHUP? | |
| 18:17:34 | dansmith | see my FIXME in there | |
| 18:17:40 | dansmith | efried: for sure | |
| 18:18:22 | efried | k. btw, not sure if you saw the update, but between bnemec and me, SIGHUP is (soon to be) fixed. | |
| 18:18:38 | efried | soon as we can get this code merged & released | |
| 18:21:53 | donnyd | What i really need is a faster way to download glance images so this isn't an issue | |
| 18:24:13 | efried | just walk a flash drive across the room | |
| 18:24:28 | donnyd | LOL efried | |
| 18:25:31 | donnyd | Well the glance image store is on an nvme drive that will move at the speed of the rest of the network... however glance doesn't feel that same way, and limits my download speeds to what one core can do... which is about 100M/s | |
| 18:25:59 | donnyd | I think it has something to do with requests if I am not mistaken | |
| 18:33:02 | sean-k-mooney | donnyd: are you useing a 1G network for your managment network | |
| 18:33:09 | donnyd | 10 | |
| 18:33:19 | donnyd | and the controllers are all 40 | |
| 18:33:48 | donnyd | So on the control plane for the compute side 10, and controller side 40 | |
| 18:33:55 | mriedem | donnyd: are you using ceph? | |
| 18:33:58 | donnyd | no | |
| 18:34:15 | donnyd | filestore is the fastest i have measured thus far | |
| 18:34:18 | melwitt | mriedem: just fyi if you didn't see, I updated the multi-cell archive patches yesterday night | |
| 18:34:41 | mriedem | melwitt: i didn't, and was going to look about 30 minutes ago, but was distracted, but i'll look in a bit, thanks | |
| 18:34:50 | donnyd | i tried cinder and swift backends, and they only made it slower | |
| 18:35:15 | melwitt | mriedem: coolness, thanks | |
| 18:35:19 | mriedem | donnyd: have you tried pre-caching the images on the computes when you have a new image? | |
| 18:35:19 | donnyd | if I was on ceph, there would be no wait time at all | |
| 18:35:25 | melwitt | and sorry for the delay | |
| 18:35:59 | mriedem | donnyd: the only thing i'd have to check is if there is some config to keep the image cache manager periodic from deleting the images if there are no guests on the host using them | |
| 18:36:02 | donnyd | mriedem: I didn't know that was an option... but I am also pretty sure nodepool is all over it whenever there is a new image | |
| 18:36:48 | mriedem | the image cache stuff in nova isn't documented at all outside of config (i don't think anyway), so i wouldn't be surprised | |
| 18:36:49 | donnyd | mostly the issue is when new images are loaded, they can't be downloaded fast enough by compute | |
| 18:36:54 | mriedem | i don't know it all that well either | |
| 18:37:10 | mriedem | right, by pre-caching you wouldn't need compute to download them, the images would be there | |
| 18:37:26 | mriedem | but that's something yo'ud have to orchestrate outside of nova | |
| 18:37:48 | donnyd | well nodepool asks for them nearly the instant that they are active in glance | |
| 18:37:56 | donnyd | at least from what i can see | |
| 18:38:14 | mriedem | ok yeah so maybe wouldn't help | |
| 18:38:43 | donnyd | when you say pre-caching, i am thinking you mean have nova launch something based on the new image and then kill it immediately after its active. | |
| 18:39:00 | mriedem | that's one way | |
| 18:39:05 | mriedem | probably the easiest | |
| 18:39:16 | donnyd | I don't know how to setup the http store for glance and have it download direct from say an apache server | |