| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-25 | |||
| 19:14:56 | lyarwood | melwitt: I'll respin and/or update in the morning. | |
| 19:14:59 | mriedem | it's intentional because of superconductor mode | |
| 19:15:02 | mriedem | melwitt: ^ | |
| 19:15:13 | dansmith | yeah, what mriedem said | |
| 19:15:14 | mriedem | otherwise affinity tests will below up | |
| 19:15:28 | dansmith | so people either have to disable affinity or enable that workaround | |
| 19:15:29 | mriedem | there are a few things disabled by default like that in devstack | |
| 19:15:31 | melwitt | lyarwood: ok, np at all. just wanted to make sure in case you didn't see | |
| 19:15:33 | dansmith | because we still don't have affinity in placement | |
| 19:16:14 | mriedem | https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls | |
| 19:16:40 | mriedem | anything not marked complete in that list probably has some kind of flag to disable it in devstack | |
| 19:17:31 | mriedem | and we don't test cross_az_attach=false in the gate anywhere so...that just flies under the radar | |
| 19:17:35 | melwitt | mriedem, dansmith: thanks. yeah, I see that the logic is based on whether we have a superconductor going or not. just trying to work out whether we want or if there's way to set it False if we know everything's on the same MQ. context, we got a regression reported https://bugs.launchpad.net/nova/+bug/1863190 that's looking not like a regression since I saw [workarounds]disable_group_policy_check_upcall = True in the config | |
| 19:17:36 | openstack | Launchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New] | |
| 19:18:26 | mriedem | i think there is at least one known latent multi-cell bug with how (anti-)affinity works, but i'm fuzzy on the details | |
| 19:18:28 | melwitt | two parallel anti-affinity requests seeming to violate policy in the single MQ deployment | |
| 19:18:33 | mriedem | if you don't have multiple cells then i guess that doesn't apply | |
| 19:19:01 | mriedem | if you have single cell and are support anti-affinity then you need the late check enabled in the compute | |
| 19:19:02 | melwitt | yeah if you are multi MQ then you can't get anti-affinity if the requests hit at the same time | |
| 19:19:11 | melwitt | right | |
| 19:19:37 | mriedem | [workarounds]disable_group_policy_check_upcall = True means you're opting into the wildness | |
| 19:19:43 | melwitt | I was just thinking you'd think our default devstack with one MQ should set it False | |
| 19:19:50 | mriedem | it's false by default | |
| 19:20:12 | melwitt | it is, but something in the default devstack logic is setting it True | |
| 19:20:32 | melwitt | that is, I cloned the devstack repo and brought up a vanilla devstack and I'm getting it set to True | |
| 19:20:34 | mriedem | yeah, because superconductor is the default mode | |
| 19:20:51 | mriedem | superconductor is the ideal mode for deploying nova, so we test in the gate with that by default | |
| 19:21:11 | melwitt | yeah. and I'm thinking this sort of "bug" will keep being reported occasionally bc people not realizing what devstack is doing | |
| 19:21:32 | mriedem | it's a known limitation, link them to the docs | |
| 19:22:02 | mriedem | tempest has (anti)affinity tests as well but i don't think they make parallel requests | |
| 19:22:09 | mriedem | because of this | |
| 19:22:29 | melwitt | yeah. they have to be very parallel too. the first time I tried to repro I didn't get it but after a few tries I got it | |
| 19:24:41 | mriedem | right, i remember poking around in those tempest tests awhile back related to all of this | |
| 19:26:33 | mriedem | oh also i think tempest tests the affinity stuff with 2 servers in the same create request - same (anti)affinity group, and that works b/c the scheduler knows about the decisions within the same request | |
| 19:26:55 | melwitt | ah, yeah | |
| 19:27:06 | mriedem | https://github.com/openstack/tempest/blob/7a588ded216f74ddd0015c3065d4fae10de2161f/tempest/api/compute/admin/test_servers_on_multinodes.py | |
| 19:27:46 | mriedem | and https://github.com/openstack/tempest/blob/f419f4d36fd0f99a9c53fe3a984d172b02e828c5/tempest/api/compute/servers/test_server_group.py | |
| 19:28:24 | mriedem | but of course you get nfv mano systems in the wild that are robots just firing off rapid requests | |
| 19:28:50 | mriedem | but those are likely single cell and shouldn't be disabling that late affinity check upcall :) | |
| 19:29:12 | mriedem | starlingx had patches for a lot of this server group stuff | |
| 19:29:20 | melwitt | yeah, I see | |
| 19:29:57 | mriedem | to try and mitigate some of it, but it wasn't all perfect either, e.g. locks within conductor but that would only lock *that* conductor (or scheduler) worker, not across all - unless you use an external locking mechanism, like a db or etcd or something | |
| 19:30:22 | melwitt | right | |
| 19:30:22 | melwitt | yeah, I know of one where they run with a single scheduler and serialize affinity requests | |
| 19:30:26 | mriedem | trying to make scheduling requests serialized for group based scheduling | |
| 19:30:35 | mriedem | for starlingx with like 1 node and 1 worker then it's probably fine | |
| 19:32:08 | melwitt | ok. I'll finish repro'ing the situation with the workaround set to False to make sure the late affinity check triggers, and write up something for the bug. and will link the doc | |
| 19:44:45 | mriedem | melwitt: if you're so inclined and it's something that keeps coming up it might be worth writing something up in the troubleshooting docs, e.g. why are my servers that are in x policy landing on the same/different hosts when they shouldn't? | |
| 19:44:53 | mriedem | and explain the parallel issue | |
| 19:45:24 | mriedem | i found it nice to write something up once and then just point people to that | |
| 19:45:41 | mriedem | https://docs.openstack.org/nova/latest/admin/support-compute.html | |
| 19:51:01 | melwitt | mriedem: yup I think that's a good idea. I'll do that | |
| 19:51:09 | melwitt | thanks for suggesting | |
| 19:55:47 | mriedem | my first few weeks on the new job were me in slack being like "why x? why y? where is z documented?" and then taking the answers and trying to document them to feel like i was useful | |
| 19:58:22 | melwitt | that's a good investment. every time I don't do that, I regret it later | |
| 19:59:31 | melwitt | and I usually forget because I have the memory recall of a hamster | |
| 20:37:49 | melwitt | johnsom: hey, finally got a chance to dig into the bug report you opened awhile back about anti-affinity, tl;dr is I don't find that there's been a regression. pls see my latest comment explaining https://bugs.launchpad.net/nova/+bug/1863190 | |
| 20:37:50 | openstack | Launchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New] | |
| 20:38:46 | johnsom | melwitt Ok, thank you for having a look. I got some feedback that it changed around queens, but I didn't go back and confirm either way. | |
| 20:41:33 | johnsom | melwitt My money is on that setting being the variable. | |
| 20:42:23 | melwitt | johnsom: it looks like it was likely a timing difference bc the change that disabled the late affinity upcall was back in pike https://review.opendev.org/477556 | |
| 20:43:09 | johnsom | Lol, that is "around" in OpenStack time. | |
| 20:43:50 | melwitt | around for certain values of around | |
| 20:44:38 | johnsom | Yep | |
| 20:46:35 | johnsom | rm_work FYI: https://bugs.launchpad.net/nova/+bug/1863190 comment 7 | |
| 20:46:36 | openstack | Launchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New] | |
| 20:47:47 | rm_work | hmm k | |
| 20:48:02 | rm_work | we switched to hard-anti-affinity and made sure we have retries enabled | |
| 20:49:52 | melwitt | rm_work: you have to have your cell conductors and computes configured a certain way to be able to handle racing affinity requests. if you have one shared MQ the configs can be set to support it. if you have multiple MQs there's no enforcement of affinity for racing requests until affinity support is added to placement | |
| 20:50:10 | rm_work | k | |
| 20:50:19 | rm_work | our symptom was that a late check will catch it, but | |
| 20:50:27 | rm_work | for soft-anti-affinity, it doesn't actually BLOCK it | |
| 20:50:35 | rm_work | which means soft-anti-affinity is pretty useless | |
| 20:50:53 | rm_work | hard-anti-affinity it catches and sends for rescheduling, but our problem was we had reschedules disabled | |
| 20:50:58 | rm_work | we have since fixed that | |
| 20:51:16 | melwitt | yeah. I'm not sure what soft-anti-affinity is for ... give it a try and if not, meh I guess | |
| 20:51:25 | openstackgerrit | Merged openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834 | |
| 20:51:29 | rm_work | it's supposed to be "best attempt" | |
| 20:51:37 | melwitt | yeah | |
| 20:51:40 | rm_work | so if you are down to only one schedulable HV, it'll still *work* because that's better than nothing | |
| 20:51:57 | rm_work | but in our case, using pack scheduling, it basically never does anything unless it catches up front | |
| 20:52:05 | rm_work | the late-catch will do nothing as it's still "valid" | |
| 20:52:11 | rm_work | which makes it not so useful | |
| 20:52:16 | melwitt | right | |
| 20:52:43 | melwitt | so what did you do? enable some retries? | |
| 20:53:17 | rm_work | switched to hard-aa and set scheduling retries to 3 (which is the original default, i think -- we had set it specifically to 0) | |
| 20:58:22 | melwitt | rm_work: yeah, ok. we do have a gap regarding the default pack scheduling + server group requests as mriedem mentioned earlier, and a way we could deal with that is to do something similar to starlingx where we serialize server group request claims at the scheduler, but we'd need to use a distributed lock since we have multiple scheduler workers. not something we already have in nova so would take more effort to add. would be a spec | |
| 20:58:23 | melwitt | and all | |
| 21:20:57 | openstackgerrit | Merged openstack/nova stable/pike: Improve metadata server performance with large security groups https://review.opendev.org/697523 | |
| 21:24:29 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-hypervisors policies https://review.opendev.org/715071 | |
| 21:32:12 | mriedem | melwitt: tbc, i think the solution for the affinity problem during scheduling likely involves placement as has been discussed for years, not serializing things like starlingx did as a workaround | |
| 21:32:28 | mriedem | but how that would work in placement has always been difficult to model | |
| 21:33:08 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in os-hypervisors policy https://review.opendev.org/715074 | |
| 21:33:26 | mriedem | but then placement is your distributed lock :) | |
| 21:33:31 | melwitt | mriedem: yeah, sorry, I was missing that the pack pattern problem with anti-affinity would go away with placement affinity | |
| 21:33:48 | melwitt | this stuff confuses the hell out of me | |
| 21:35:31 | melwitt | so, nix what I said earlier rm_work ^ | |
| 21:40:39 | mriedem | , | |
| 21:40:39 | mriedem | "if you have multiple MQs there's no enforcement of affinity for racing requests until affinity support is added to placement" is not quite true, you just need the conductors configured to hit the API DB (compute -> cell conductor -> API DB); devstack doesn't configure the cell conductor with the API DB connection so again, devstack is doing the *ideal* separate setup but not what most (if any) nova deployments are probably d | |
| 21:41:09 | mriedem | pretty sure most nova deployments just have the api db connection configured everywhere | |