Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-25
19:19:11 melwitt right
19:19:37 mriedem [workarounds]disable_group_policy_check_upcall = True means you're opting into the wildness
19:19:43 melwitt I was just thinking you'd think our default devstack with one MQ should set it False
19:19:50 mriedem it's false by default
19:20:12 melwitt it is, but something in the default devstack logic is setting it True
19:20:32 melwitt that is, I cloned the devstack repo and brought up a vanilla devstack and I'm getting it set to True
19:20:34 mriedem yeah, because superconductor is the default mode
19:20:51 mriedem superconductor is the ideal mode for deploying nova, so we test in the gate with that by default
19:21:11 melwitt yeah. and I'm thinking this sort of "bug" will keep being reported occasionally bc people not realizing what devstack is doing
19:21:32 mriedem it's a known limitation, link them to the docs
19:22:02 mriedem tempest has (anti)affinity tests as well but i don't think they make parallel requests
19:22:09 mriedem because of this
19:22:29 melwitt yeah. they have to be very parallel too. the first time I tried to repro I didn't get it but after a few tries I got it
19:24:41 mriedem right, i remember poking around in those tempest tests awhile back related to all of this
19:26:33 mriedem oh also i think tempest tests the affinity stuff with 2 servers in the same create request - same (anti)affinity group, and that works b/c the scheduler knows about the decisions within the same request
19:26:55 melwitt ah, yeah
19:27:06 mriedem https://github.com/openstack/tempest/blob/7a588ded216f74ddd0015c3065d4fae10de2161f/tempest/api/compute/admin/test_servers_on_multinodes.py
19:27:46 mriedem and https://github.com/openstack/tempest/blob/f419f4d36fd0f99a9c53fe3a984d172b02e828c5/tempest/api/compute/servers/test_server_group.py
19:28:24 mriedem but of course you get nfv mano systems in the wild that are robots just firing off rapid requests
19:28:50 mriedem but those are likely single cell and shouldn't be disabling that late affinity check upcall :)
19:29:12 mriedem starlingx had patches for a lot of this server group stuff
19:29:20 melwitt yeah, I see
19:29:57 mriedem to try and mitigate some of it, but it wasn't all perfect either, e.g. locks within conductor but that would only lock *that* conductor (or scheduler) worker, not across all - unless you use an external locking mechanism, like a db or etcd or something
19:30:22 melwitt right
19:30:22 melwitt yeah, I know of one where they run with a single scheduler and serialize affinity requests
19:30:26 mriedem trying to make scheduling requests serialized for group based scheduling
19:30:35 mriedem for starlingx with like 1 node and 1 worker then it's probably fine
19:32:08 melwitt ok. I'll finish repro'ing the situation with the workaround set to False to make sure the late affinity check triggers, and write up something for the bug. and will link the doc
19:44:45 mriedem melwitt: if you're so inclined and it's something that keeps coming up it might be worth writing something up in the troubleshooting docs, e.g. why are my servers that are in x policy landing on the same/different hosts when they shouldn't?
19:44:53 mriedem and explain the parallel issue
19:45:24 mriedem i found it nice to write something up once and then just point people to that
19:45:41 mriedem https://docs.openstack.org/nova/latest/admin/support-compute.html
19:51:01 melwitt mriedem: yup I think that's a good idea. I'll do that
19:51:09 melwitt thanks for suggesting
19:55:47 mriedem my first few weeks on the new job were me in slack being like "why x? why y? where is z documented?" and then taking the answers and trying to document them to feel like i was useful
19:58:22 melwitt that's a good investment. every time I don't do that, I regret it later
19:59:31 melwitt and I usually forget because I have the memory recall of a hamster
20:37:49 melwitt johnsom: hey, finally got a chance to dig into the bug report you opened awhile back about anti-affinity, tl;dr is I don't find that there's been a regression. pls see my latest comment explaining https://bugs.launchpad.net/nova/+bug/1863190
20:37:50 openstack Launchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New]
20:38:46 johnsom melwitt Ok, thank you for having a look. I got some feedback that it changed around queens, but I didn't go back and confirm either way.
20:41:33 johnsom melwitt My money is on that setting being the variable.
20:42:23 melwitt johnsom: it looks like it was likely a timing difference bc the change that disabled the late affinity upcall was back in pike https://review.opendev.org/477556
20:43:09 johnsom Lol, that is "around" in OpenStack time.
20:43:50 melwitt around for certain values of around
20:44:38 johnsom Yep
20:46:35 johnsom rm_work FYI: https://bugs.launchpad.net/nova/+bug/1863190 comment 7
20:46:36 openstack Launchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New]
20:47:47 rm_work hmm k
20:48:02 rm_work we switched to hard-anti-affinity and made sure we have retries enabled
20:49:52 melwitt rm_work: you have to have your cell conductors and computes configured a certain way to be able to handle racing affinity requests. if you have one shared MQ the configs can be set to support it. if you have multiple MQs there's no enforcement of affinity for racing requests until affinity support is added to placement
20:50:10 rm_work k
20:50:19 rm_work our symptom was that a late check will catch it, but
20:50:27 rm_work for soft-anti-affinity, it doesn't actually BLOCK it
20:50:35 rm_work which means soft-anti-affinity is pretty useless
20:50:53 rm_work hard-anti-affinity it catches and sends for rescheduling, but our problem was we had reschedules disabled
20:50:58 rm_work we have since fixed that
20:51:16 melwitt yeah. I'm not sure what soft-anti-affinity is for ... give it a try and if not, meh I guess
20:51:25 openstackgerrit Merged openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834
20:51:29 rm_work it's supposed to be "best attempt"
20:51:37 melwitt yeah
20:51:40 rm_work so if you are down to only one schedulable HV, it'll still *work* because that's better than nothing
20:51:57 rm_work but in our case, using pack scheduling, it basically never does anything unless it catches up front
20:52:05 rm_work the late-catch will do nothing as it's still "valid"
20:52:11 rm_work which makes it not so useful
20:52:16 melwitt right
20:52:43 melwitt so what did you do? enable some retries?
20:53:17 rm_work switched to hard-aa and set scheduling retries to 3 (which is the original default, i think -- we had set it specifically to 0)
20:58:22 melwitt rm_work: yeah, ok. we do have a gap regarding the default pack scheduling + server group requests as mriedem mentioned earlier, and a way we could deal with that is to do something similar to starlingx where we serialize server group request claims at the scheduler, but we'd need to use a distributed lock since we have multiple scheduler workers. not something we already have in nova so would take more effort to add. would be a spec
20:58:23 melwitt and all
21:20:57 openstackgerrit Merged openstack/nova stable/pike: Improve metadata server performance with large security groups https://review.opendev.org/697523
21:24:29 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-hypervisors policies https://review.opendev.org/715071
21:32:12 mriedem melwitt: tbc, i think the solution for the affinity problem during scheduling likely involves placement as has been discussed for years, not serializing things like starlingx did as a workaround
21:32:28 mriedem but how that would work in placement has always been difficult to model
21:33:08 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in os-hypervisors policy https://review.opendev.org/715074
21:33:26 mriedem but then placement is your distributed lock :)
21:33:31 melwitt mriedem: yeah, sorry, I was missing that the pack pattern problem with anti-affinity would go away with placement affinity
21:33:48 melwitt this stuff confuses the hell out of me
21:35:31 melwitt so, nix what I said earlier rm_work ^
21:40:39 mriedem ,
21:40:39 mriedem "if you have multiple MQs there's no enforcement of affinity for racing requests until affinity support is added to placement" is not quite true, you just need the conductors configured to hit the API DB (compute -> cell conductor -> API DB); devstack doesn't configure the cell conductor with the API DB connection so again, devstack is doing the *ideal* separate setup but not what most (if any) nova deployments are probably d
21:41:09 mriedem pretty sure most nova deployments just have the api db connection configured everywhere
21:41:22 openstackgerrit Merged openstack/nova stable/stein: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713962
21:41:40 mriedem and yeah you have to have reschedules enabled to....reschedule :)
21:42:28 mriedem the only things that do the late affinity check in the computes are server create and evacuate, so you can still violate affinity policy for other moves (unshelve, cold and live migrate)
21:42:37 melwitt yeah ... I was realizing that slowly regarding the difference between database access vs MQ access
21:42:55 mriedem and the only flows that reschedule today from the compute are create and cold migrate/resize
21:43:10 mriedem i think evacuate just fails the operation if you fail the late affinity check
21:44:12 melwitt I can't remember why that "impossible to contact bc MQ" was ever a thing wrt to affinity
21:45:35 melwitt was it before alternate_hosts became a thing maybe?
21:46:35 melwitt sigh ... have to correct my comment yet again
21:54:22 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing instance usage log policies https://review.opendev.org/715080
21:55:05 mriedem alternate hosts just solved the problem of the cell conductor needing to go back to the scheduler and API DB
21:55:41 mriedem there is actually still an upcall bug there which i don't think got fixed,
21:56:11 mriedem during reschedule the conductor will update the instance.availability_zone for the alternate host and to do that it needs to hit the aggregates table which is in the API DB
21:57:04 melwitt yeah, I mean I had thought in the past it was said that the late affinity check would be impossible due to MQ isolation. but as you explained that's not true. so if that was ever said, I wondered why. it might have been before alternate hosts was added
21:57:29 mriedem err i guess i fixed that https://review.opendev.org/#/q/topic:bug/1781286+(status:open+OR+status:merged)+branch:master
21:58:46 mriedem anytime, it's time to social distance myself into the kitchen, o/
22:01:36 dansmith melwitt: I think what you're thinking of is that we don't currently have a way for the child cell to know about the top-level mq, and I don't think we should
22:02:04 dansmith we do have a separate api_db connection string, so as a hack, child conductors can use that as if they were a top-level conductor to still hit that database,
22:02:18 dansmith which at least reduces the scope of who at the lower level can talk up,

Earlier   Later