| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 17:46:05 | sean-k-mooney | right after we call check_can_live_migrate_source in check_can_live_migrate_destination | |
| 17:46:17 | dansmith | artom: so it's very likely that you're blowing those away with this yeah? | |
| 17:46:22 | artom | Although sean-k-mooney's saying apparently all recent-ish NICs can do SRIOV, so maybe I *do* have the hardware? | |
| 17:46:52 | sean-k-mooney | artom: i have hardware and i set up port forwading so you can ssh in. | |
| 17:47:03 | sean-k-mooney | but im going to check both again after dinner | |
| 17:49:20 | sean-k-mooney | im going to grab dinner but ill kick of a devstack run before i go and setup the test enviromint | |
| 17:49:57 | artom | dansmith, seems likely, yeah | |
| 17:51:11 | sean-k-mooney | we dont use the pci request spec object from the instance sriov migration by the way | |
| 17:51:27 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6428-L8993 | |
| 17:51:46 | dansmith | sean-k-mooney: because why? we store details in neutron about it? | |
| 17:52:13 | artom | Hrmm | |
| 17:52:15 | artom | So actually | |
| 17:52:16 | sean-k-mooney | the only info we need to pass back as the pci address of the new device | |
| 17:52:18 | artom | AFAICT | |
| 17:52:31 | artom | The PCI stuff in the claim doesn't actually *claim* any resources | |
| 17:52:34 | sean-k-mooney | so we store that in the vif port profile which is wher ewe pass it to neutron | |
| 17:52:36 | artom | Just tests that it's supported | |
| 17:53:00 | dansmith | sean-k-mooney: okay, but nova's own structure would still have them in the pci_requests right? | |
| 17:53:07 | dansmith | so what about like rebuild or something? | |
| 17:53:07 | sean-k-mooney | yes | |
| 17:53:35 | artom | Ah, no, it does the _decrease_pool_count() thing, which "claims" the resource | |
| 17:54:20 | sean-k-mooney | how does it interact with this? | |
| 17:54:33 | sean-k-mooney | the claims have 3 states | |
| 17:54:41 | sean-k-mooney | free,cliamed and allocated | |
| 17:54:50 | sean-k-mooney | something like that | |
| 17:55:07 | sean-k-mooney | one of them meens its reserved for an instace the the otehr is its in use | |
| 17:55:10 | dansmith | I'm saying I would expect we use pci_requests if we're doing a rebuild on the instance | |
| 17:55:23 | dansmith | or other operations | |
| 17:55:43 | dansmith | point is just that corrupting our own accounting because sriov is tracked in neutron is not okay I don't think | |
| 17:55:53 | sean-k-mooney | so we are claiming the ot reserve them and later we update the instace with them weh we move it to the allocated state i think. its beed a whild and i dont really rememerb the details | |
| 17:56:08 | sean-k-mooney | this is the import cahgne https://review.opendev.org/#/c/620115/35/nova/compute/manager.py | |
| 17:56:12 | mriedem | we do'nt claim for rebuild | |
| 17:56:16 | mriedem | it's a noop claim | |
| 17:56:28 | dansmith | mriedem: sure, not related to claim | |
| 17:56:59 | dansmith | cross-cell-migrate looks at pci_requests | |
| 17:57:50 | dansmith | looks like regular live migration also looks at them to determine if they're all neutron-related | |
| 17:58:07 | dansmith | just trying to confirm that blowing them away is not something we can just ignore :) | |
| 18:01:02 | efried | dansmith: I'm looking at consecutive_build_service_disable_threshold and not seeing how it could be working | |
| 18:01:24 | dansmith | efried: we removed the compute side of that, if that's what you're looking at | |
| 18:01:40 | efried | "the compute side" | |
| 18:01:53 | efried | meaning... the part that makes it behave in any way other than a bool? | |
| 18:01:54 | dansmith | the part where the compute node self-disables | |
| 18:02:21 | dansmith | there should be renos about this | |
| 18:02:33 | efried | ack | |
| 18:02:46 | dansmith | change-consecutive-boot-failure-counter-to-weigher-428de7da0ed2033a.yaml | |
| 18:03:25 | efried | dansmith: okay | |
| 18:03:26 | efried | so | |
| 18:04:00 | efried | this is now up to deployers installing their own weigher? | |
| 18:04:12 | dansmith | eh? | |
| 18:04:19 | dansmith | did you read the reno? | |
| 18:04:56 | dansmith | efried: https://review.opendev.org/#/c/572195/ | |
| 18:05:01 | efried | ack | |
| 18:05:18 | dansmith | added a weigher, changed the meaning of the threshold for compatibility reasons | |
| 18:05:25 | mriedem | https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights | |
| 18:05:28 | mriedem | BuildFailureWeigher | |
| 18:07:24 | dansmith | efried: you hip to that jive? | |
| 18:07:58 | efried | Still trying to grok the implications. | |
| 18:08:31 | efried | So in a biggish cloud, if I have a bunch of failures in a row on a particular host, even if I don't hit the default multiplier, it's still going to make scheduling to that node way less likely | |
| 18:08:36 | dansmith | mriedem: beat you to the youtube ref: https://www.youtube.com/watch?v=sgW3RxKdN0Q | |
| 18:09:04 | dansmith | efried: hit the multiplier meaning "change" it? | |
| 18:09:27 | efried | sorry, no, I just mean... | |
| 18:09:58 | efried | IIUC the default multiplier is really big so that by default you won't effectively-disable a compute until it's seen a really lot of build failures. | |
| 18:10:12 | dansmith | efried: no | |
| 18:10:20 | dansmith | efried: read the reno and commit for reasoning about the multiplier | |
| 18:10:39 | dansmith | efried: it has to compete against disk weigher, which is like scored by mb free or something | |
| 18:11:12 | dansmith | efried: a big multiplier makes it more likely to have an effect, not les | |
| 18:11:14 | dansmith | *less | |
| 18:11:18 | efried | ah | |
| 18:11:30 | dansmith | and the threshold is now just "report failures if nonzero", so it's not a threshold | |
| 18:11:57 | dansmith | if you want it off, set the "threshold" to zero and computes won't even report the number | |
| 18:12:13 | efried | ack | |
| 18:12:14 | dansmith | and if you do, then you tune the multiplier based on whatever else you have configured | |
| 18:15:11 | efried | got it. Thanks dansmith. (And no, mriedem, I'm helping donnyd figure out why his CI hosts get effectively-disabled-without-actually-being-disabled when they spend some time trying to boot with not-yet-ready images) | |
| 18:16:07 | mriedem | :P | |
| 18:16:14 | mriedem | o-) | |
| 18:16:16 | efried | dansmith: save me tracing the code, does it still have the "reset to zero" behavior as soon as we get one success? | |
| 18:16:30 | dansmith | IIRC yes | |
| 18:16:47 | dansmith | efried: https://review.opendev.org/#/c/572195/6/nova/compute/manager.py | |
| 18:16:54 | donnyd | thats pending it ever actually gets rescheduled | |
| 18:17:15 | donnyd | I have enough space that essentially it doesn't | |
| 18:17:19 | efried | yeah | |
| 18:17:23 | dansmith | https://review.opendev.org/#/c/572195/6/nova/compute/stats.py | |
| 18:17:30 | efried | dansmith: is this one of those things that you should theoretically be able to reset via SIGHUP? | |
| 18:17:34 | dansmith | see my FIXME in there | |
| 18:17:40 | dansmith | efried: for sure | |
| 18:18:22 | efried | k. btw, not sure if you saw the update, but between bnemec and me, SIGHUP is (soon to be) fixed. | |
| 18:18:38 | efried | soon as we can get this code merged & released | |
| 18:21:53 | donnyd | What i really need is a faster way to download glance images so this isn't an issue | |
| 18:24:13 | efried | just walk a flash drive across the room | |
| 18:24:28 | donnyd | LOL efried | |
| 18:25:31 | donnyd | Well the glance image store is on an nvme drive that will move at the speed of the rest of the network... however glance doesn't feel that same way, and limits my download speeds to what one core can do... which is about 100M/s | |
| 18:25:59 | donnyd | I think it has something to do with requests if I am not mistaken | |
| 18:33:02 | sean-k-mooney | donnyd: are you useing a 1G network for your managment network | |
| 18:33:09 | donnyd | 10 | |
| 18:33:19 | donnyd | and the controllers are all 40 | |
| 18:33:48 | donnyd | So on the control plane for the compute side 10, and controller side 40 | |
| 18:33:55 | mriedem | donnyd: are you using ceph? | |
| 18:33:58 | donnyd | no | |
| 18:34:15 | donnyd | filestore is the fastest i have measured thus far | |
| 18:34:18 | melwitt | mriedem: just fyi if you didn't see, I updated the multi-cell archive patches yesterday night | |