Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-05
15:49:11 sean-k-mooney ok i see what you have changed. hum ok
15:51:10 bauzas sean-k-mooney: I explained the issue in https://bugs.launchpad.net/nova/+bug/1902925/comments/3
15:51:10 openstack Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza)
15:51:33 sean-k-mooney ya i just wanted to read the repoducer and code fix
15:52:21 sean-k-mooney i guess we have not extended this api since we did the 5.0 rpc bump
15:52:44 sean-k-mooney none of the other fileds are optional
15:53:07 tobias-urdin any good (and somewhat "supported" way) to extend the nova metadata API to include some custom paths? IIRC some ways of extending nova has been deprecated/removed over the years
15:53:23 sean-k-mooney yes
15:53:58 sean-k-mooney tobias-urdin: https://docs.openstack.org/nova/latest/admin/vendordata.html
15:54:34 sean-k-mooney that or i guess you coudl use middleware
15:54:51 sean-k-mooney but in general nova is not extensible in this way intentionally
15:55:07 noonedeadpunk sean-k-mooney: I think I might figued out why isolated aggregates got instances from time to time. any reason not to pass rebuilds through scheduler? https://opendev.org/openstack/nova/src/branch/master/nova/scheduler/manager.py#L146
15:55:26 sean-k-mooney noonedeadpunk: rebuilds cant change host
15:55:32 sean-k-mooney noonedeadpunk: they are not move operations
15:55:46 sean-k-mooney and they do go to the schduler if the image changes
15:55:49 noonedeadpunk uh.... I see
15:56:14 noonedeadpunk well, I continue get 1 instance per month or smth like that on the isolated aggregate
15:56:16 sean-k-mooney to validate that the current host is still ok with the new image
15:56:22 noonedeadpunk and have no clue how that might happen...
15:57:03 sean-k-mooney are you using the placment way fo doing it
15:57:08 sean-k-mooney or the filter
15:58:06 sean-k-mooney the placement way that should not happen as the traits request should block it but the filter reuires all tenanats to be mapped to an aggreate
15:58:17 sean-k-mooney or unmpped tenants can go to any host
15:58:57 openstackgerrit Sylvain Bauza proposed openstack/nova master: Fix the compute RPC 5.12 issue https://review.opendev.org/761458
15:59:10 bauzas dansmith: gibi: stephenfin: last round, hopefully
15:59:28 bauzas and then I'll backport the changes
15:59:54 gibi thanks
16:11:23 noonedeadpunk sean-k-mooney: do placement traits, exactly like specified in https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html which you gave me one day
16:11:41 noonedeadpunk out of code I see no reason why this might happen
16:11:45 noonedeadpunk but it does
16:12:57 sean-k-mooney ya im not sure either
16:13:27 sean-k-mooney unless you have multiple schduler adn one of them has a differnt config
16:13:34 noonedeadpunk and if previously it was only during resizes or smth like that I found just new VM created a week ago...
16:13:39 sean-k-mooney e.g. on one of them you dont have the prefilter enabled
16:14:29 noonedeadpunk well, I think I've checked that.... in terms of prefilter you mean scheduler.enable_isolated_aggregate_filtering ?
16:15:30 sean-k-mooney bauzas: so that is what was breakign the grenade jobs? good find
16:16:09 bauzas only for the rebuild case
16:16:18 bauzas so, maybe...
16:16:19 bauzas idk
16:16:30 gibi grenade does not run evacuate or grenade tests as far as I know
16:16:33 gibi it runs live migration
16:16:57 sean-k-mooney it runs full tempest before and after
16:17:04 sean-k-mooney i think
16:17:22 gibi really? I only found a smoke result
16:17:26 sean-k-mooney im thinging that if the vm that landed on the unupgraded node was rebuilt it would fail
16:17:27 gibi + the live migration
16:17:34 sean-k-mooney maybe im wrong
16:20:21 sean-k-mooney https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-grenade-multinode/run.yaml#L40
16:20:37 sean-k-mooney its running the compute api tests and senario tests
16:21:09 sean-k-mooney oh just the smoke subset of those?
16:21:24 dansmith just smoke before, not sure about full after t hough
16:22:41 sean-k-mooney if its runnign rebuild after then if it booted on the upgraded node we would get teh type error
16:22:57 sean-k-mooney if it booted on the un upgraded node it would have rebuilt fine
16:23:21 sean-k-mooney which would have made the test failure intermitent
16:23:23 dansmith I dunno why you say that,
16:23:29 dansmith the control plane would be upgraded,
16:23:55 dansmith oh you mean because the pin is set to auto and the presence of an old compute would keep it pinned I guess?
16:24:02 sean-k-mooney yes
16:24:20 sean-k-mooney it would be pinned but the old nova code would not expect the parmater and the new code would
16:24:27 dansmith that only works for U->V jobs, since V supported it, it'll be using the new version
16:24:40 dansmith you need to be looking at U->V grenade multinode jobs I'd expect right?
16:25:06 dansmith also, as bad as the gate has been lately, it wouldn't surprise me if people have just been rechecking past that occasional fail
16:25:20 sean-k-mooney well v->master woudl work since they woudl both use 5.12+
16:25:29 dansmith that's my point
16:25:35 sean-k-mooney u->v would (posssible) be intermitent
16:25:42 sean-k-mooney so yes
16:25:54 dansmith right, very many fewer things running that configuration
16:26:44 sean-k-mooney i have just been seeign some intermitent grendade job failure before the ptg so was wondering if this was the issue or if there are others
16:27:06 sean-k-mooney most of the issue seam to be realted to volumes however rather then rebuild
16:27:06 dansmith could be.. so many CI fails lately, I expect people are doing a lot of recheck grinding
16:32:17 gibi this is a recent grenade multinode run from stable/victoria https://1cc2260295ba1f69c29d-8ad4cd99420b0d8b2b27089e00008c76.ssl.cf1.rackcdn.com/761424/1/check/nova-grenade-multinode/e3cf1bf/logs/index.html
16:32:27 gibi I see two test reports
16:32:28 gibi https://1cc2260295ba1f69c29d-8ad4cd99420b0d8b2b27089e00008c76.ssl.cf1.rackcdn.com/761424/1/check/nova-grenade-multinode/e3cf1bf/logs/old/testr_results.html
16:32:34 gibi and
16:32:35 gibi https://1cc2260295ba1f69c29d-8ad4cd99420b0d8b2b27089e00008c76.ssl.cf1.rackcdn.com/761424/1/check/nova-grenade-multinode/e3cf1bf/logs/testr_results.html
16:32:50 gibi is there a 3rd report somewhere in the tree?
16:33:17 sean-k-mooney nope
16:33:22 sean-k-mooney just those two
16:33:35 sean-k-mooney so we are not running rebuild in the grenade job
16:34:04 sean-k-mooney i tought we were but i guess not
16:35:19 sean-k-mooney the grenade failures i was seeing were likely something else so
16:36:16 sean-k-mooney its been like 2 weeks so all that is left in my brain on the topic is "i have seen more grenade failures lately then i normally do"
16:48:04 openstackgerrit Sylvain Bauza proposed openstack/nova stable/victoria: Add a regression test for 5.12 compute API issue https://review.opendev.org/761638
16:48:05 openstackgerrit Sylvain Bauza proposed openstack/nova stable/victoria: Fix the compute RPC 5.12 issue https://review.opendev.org/761639
16:50:27 bauzas elod: stable changes are up there ^
16:51:01 bauzas hopefully master changes will be merged tonight so we could move on tomorrow
16:51:09 bauzas and ideally release subsequently
16:51:20 bauzas (release stable/victoria)
16:51:58 elod bauzas: thx, looking :)
16:52:03 bauzas elod: don't
16:52:11 bauzas the master change isn't merged yet so I -2 it
16:52:28 elod don't worry I'll wait with the +2 until master is merged ;)
16:53:23 elod (if I don't find any mistake with the backport, ofc)
16:53:27 elod :]
17:00:12 openstackgerrit Merged openstack/nova master: Add a regression test for 5.12 compute API issue https://review.opendev.org/761457
17:03:40 bauzas elod: heh ^
17:11:28 elod bauzas: ok, so the regression test part is ready and looks OK. +2'd
17:11:37 bauzas <3
17:11:38 elod one more to go :)
17:16:31 elod the backport of the fix also looks good to me and the fix is on the gate in master, so we just have to wait.

Earlier   Later