| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-30 | |||
| 11:17:08 | ralonsoh | we are calling this from a monkey patched process with the "fix" | |
| 11:17:09 | sean-k-mooney | not if its executed via privsep | |
| 11:17:17 | ralonsoh | oh my... | |
| 11:17:21 | ralonsoh | ok, got it!!! | |
| 11:17:53 | sean-k-mooney | noonedeadpunk: you enabled the placment prefilter for this in the nova.conf too right | |
| 11:19:16 | sean-k-mooney | noonedeadpunk: https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.enable_isolated_aggregate_filtering | |
| 11:19:25 | noonedeadpunk | sean-k-mooney: I think I just set `enable_isolated_aggregate_filtering` | |
| 11:19:30 | noonedeadpunk | yeah | |
| 11:19:56 | noonedeadpunk | and actually VMs are not created on host. the only one which is wrong placed for the last month is this one | |
| 11:20:02 | sean-k-mooney | noonedeadpunk: by the way you can also do tenant isolation with placement too | |
| 11:20:05 | noonedeadpunk | which was spawned there after resize | |
| 11:20:40 | sean-k-mooney | noonedeadpunk: did you way your using train | |
| 11:21:24 | noonedeadpunk | sorry didn't get the question:( | |
| 11:21:38 | sean-k-mooney | what release are you running train? | |
| 11:22:45 | sean-k-mooney | i start typeing "by the way are you using train" the swaped to "did you say your using train" in my head and type a mix of the two hehe | |
| 11:23:35 | sean-k-mooney | there is nothing here that is resize specific | |
| 11:23:37 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/train/nova/scheduler/request_filter.py#L31-L88 | |
| 11:23:46 | noonedeadpunk | it's a... 20.2.1.dev1, which to be exact 118ee682571a4bd41c8009dbe2e47fdd1f85a630 | |
| 11:24:26 | sean-k-mooney | noonedeadpunk: can you check for that "'Request filter %r took %.1f seconds'," debug message if you have debug logs | |
| 11:24:57 | sean-k-mooney | the simpelst way to check would be to boot a vm and set the schduler into debug mode then resize it | |
| 11:25:17 | sean-k-mooney | and confirm the isolate_aggregates prefilter is run | |
| 11:25:41 | sean-k-mooney | but they shoudl all be run https://github.com/openstack/nova/blob/stable/train/nova/scheduler/request_filter.py#L221-L237 | |
| 11:34:03 | noonedeadpunk | sean-k-mooney: I thiink I have only info level, not debug one... | |
| 11:34:44 | sean-k-mooney | thats ok. with a quick look i dont see any reason that it woudl be skipped | |
| 11:34:54 | noonedeadpunk | I see... | |
| 11:35:11 | sean-k-mooney | if it is being skipped for some reason then a functional test should be able to repoduces it | |
| 11:36:08 | sean-k-mooney | this logic should be driver independent | |
| 11:38:47 | noonedeadpunk | have the same ALL_REQUEST_FILTERS here | |
| 11:39:05 | noonedeadpunk | sean-k-mooney: ok, thanks for your help, will try to reproduce it and catch logs | |
| 11:40:01 | sean-k-mooney | if you do feel free to file a bug with the steps. we shoudl be able to trun it into a functional regression test and then fix it but im not aware of something that would cause it to break currently | |
| 11:40:22 | sean-k-mooney | that does not mean it cant just not seeing it quickly | |
| 11:46:22 | tobias-urdin | sean-k-mooney: sorry was afk for a while, should it be treated as a bug? | |
| 11:47:35 | sean-k-mooney | tobias-urdin: no we previusly had a bug for this for rbd then meged a skip and reverted it | |
| 11:47:58 | sean-k-mooney | tobias-urdin: the use of rbd does not put the instance state dir on shared storage | |
| 11:48:08 | sean-k-mooney | tobias-urdin: this could be a blueprint/spec | |
| 11:48:14 | sean-k-mooney | but its a new feature not a bug | |
| 11:48:41 | tobias-urdin | sean-k-mooney: ok, thanks :) | |
| 11:48:57 | sean-k-mooney | tobias-urdin: its one i would not mind seeing added by the way. i think it would be useful | |
| 11:49:27 | tobias-urdin | sean-k-mooney: i agree | |
| 11:55:32 | noonedeadpunk | sean-k-mooney: looking at code, I have one question - should I need another aggregate which would include the rest of the hosts? | |
| 11:56:00 | sean-k-mooney | noonedeadpunk: off the top of my head i dont think so | |
| 11:56:14 | noonedeadpunk | ok, just to double check, as I don't have one | |
| 11:56:29 | noonedeadpunk | I mean I have the only aggregate with required trait | |
| 11:56:52 | sean-k-mooney | all the rest will be in the implcit nova aggreate | |
| 11:59:20 | noonedeadpunk | was just thinking how this works. https://github.com/openstack/nova/blob/stable/train/nova/scheduler/request_filter.py#L74-L76 Like we get list of aggregates without the trait? | |
| 12:02:15 | sean-k-mooney | yes so we first get all the required traits from the instance | |
| 12:03:06 | sean-k-mooney | then we all the aggreates that have require traits that are not in the set we require for the instnace | |
| 12:03:20 | sean-k-mooney | we then add those as forbiden aggreates to the placment query | |
| 12:03:42 | noonedeadpunk | ah, ok, yeah | |
| 12:03:45 | sean-k-mooney | so since you have no required traits in the flaovr or image for that instance | |
| 12:03:55 | sean-k-mooney | it will get back all aggreates with a require trait | |
| 12:04:06 | sean-k-mooney | in this case the one aggreate you created | |
| 12:04:16 | sean-k-mooney | and then add those to the forbidden list | |
| 12:04:21 | noonedeadpunk | ok, thanks so much for explanation | |
| 12:04:42 | sean-k-mooney | that becmoes a "mem_of=!<aggreate uuid>" paramter to placemnt | |
| 12:05:08 | sean-k-mooney | ! being not | |
| 12:05:09 | openstack | sean-k-mooney: Error: "being" is not a valid command. | |
| 12:05:25 | noonedeadpunk | lol | |
| 12:05:26 | sean-k-mooney | hehe | |
| 12:05:49 | sean-k-mooney | sorry that hsoul be member_of | |
| 12:06:07 | sean-k-mooney | anyway you get the point we prefix the uuid with ! whenever we want to exclude it | |
| 12:07:36 | noonedeadpunk | yes, sure, thanks again) | |
| 13:33:16 | dansmith | lyarwood: I may have missed something about the live migration ceph job.. this is a grenade job, but this is the one I care most about: https://review.opendev.org/#/c/742056/6/.zuul.yaml@90 | |
| 13:33:31 | dansmith | which looks like it has already been removed and we're not running, right? | |
| 13:36:48 | lyarwood | dansmith: it's still there just hidden in the original nova-live-migration job | |
| 13:36:57 | lyarwood | dansmith: https://github.com/openstack/nova/blob/2beb18444fccac2ffe73292423ff801a9f327f49/.zuul.yaml#L59-L70 | |
| 13:37:01 | lyarwood | dansmith: https://github.com/openstack/nova/blob/2beb18444fccac2ffe73292423ff801a9f327f49/playbooks/legacy/nova-live-migration/run.yaml#L51-L54 | |
| 13:37:26 | lyarwood | dansmith: https://github.com/openstack/nova/blob/2beb18444fccac2ffe73292423ff801a9f327f49/gate/live_migration/hooks/run_tests.sh#L55-L65 | |
| 13:37:50 | dansmith | lyarwood: confusing, but okay.. so *that* is the thing I think is important not to lose, but grenade I guess I'm less worried about | |
| 13:38:54 | lyarwood | kk ack, I've -W'd my switch over change until I've covered that in a ceph specific job | |
| 13:38:55 | dansmith | I'm still worried about losing pressure to close the loop and these gaps sticking around for a while, but if the basic multinode live migration switcheroo bit is still there, then I don't want to block smaller process hacks like the grenade bit if that makes it easier | |
| 13:39:39 | lyarwood | np this shouldn't take long to work out and backport to stable/victoria so we can close the zuulv3 stuff out there tbh | |
| 13:39:53 | lyarwood | I've worked out how to copy the config and keys around already | |
| 13:40:08 | lyarwood | in here https://review.opendev.org/#/c/755270/ | |
| 13:40:22 | lyarwood | but I think the role needs to land in devstack itself | |
| 13:40:58 | dansmith | ah, wait wait, | |
| 13:41:12 | dansmith | gmann's patch *does* remove that chunk: https://review.opendev.org/#/c/752557/8/.zuul.yaml@a68 | |
| 13:41:22 | dansmith | you were saying that job isn't already gone, | |
| 13:41:34 | dansmith | but his patch *does* reduce that coverage... | |
| 13:41:45 | lyarwood | that's my change | |
| 13:41:48 | lyarwood | and is -W'd | |
| 13:42:15 | lyarwood | until we get the coverage sorted | |
| 13:42:17 | dansmith | okay, I guess I'm not sure which that is, | |
| 13:42:18 | lyarwood | I'll add a comment | |
| 13:43:53 | dansmith | ah, this guy: https://review.opendev.org/#/c/752557/ | |
| 13:44:25 | lyarwood | that's the same change as above | |
| 13:44:40 | dansmith | lol | |
| 13:45:25 | lyarwood | https://review.opendev.org/#/c/742056/ is the grenade change | |
| 13:45:26 | dansmith | I know, I just didn't realize that was yours, thought it was gmann's because he had the -W until a minute ago | |
| 13:45:38 | lyarwood | ah np | |
| 13:45:46 | dansmith | yeah, I got it now | |
| 13:46:12 | dansmith | btw, I'm super happy to see you guys also getting spanked by zuul on syntax stuff | |
| 13:46:33 | dansmith | fixing up glance's jobs, zuul was getting sick of me throwing up patches that didn't even parse :) | |
| 13:46:55 | lyarwood | haha yeah, all the damn time | |
| 13:47:05 | lyarwood | wish there was a way of running some of this stuff locally | |
| 13:47:08 | dansmith | there needs to be a separately-installable zuulint tool or something we can throw into a tox env | |
| 13:47:11 | dansmith | yeah | |
| 14:02:09 | sean-k-mooney | lyarwood: there is a docker compose file for running zull quickly | |
| 14:02:18 | sean-k-mooney | but ya gettign the gate jobs to work is non trivial | |
| 14:03:16 | sean-k-mooney | the devstack/tempest based jobs pull in a lot of repos so you need to donload a lot of repos to get them to work unless you take a few steps that mean your nolonger really mirrioring what happens in the gate | |