| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-05 | |||
| 15:37:27 | gibi | bauzas: never said that we should not ask for a func test. I'm saying that we tend to forget about it as the current bug shows | |
| 15:37:28 | dansmith | yeah, so we could always pin the version to .0, | |
| 15:37:37 | dansmith | but coverage in tempest will be hard, | |
| 15:37:57 | dansmith | plus tempest needs to be graceful as some api calls will fail expectedly if the version doesn't support the new feature | |
| 15:38:00 | bauzas | technically, we need to set the pin to the previous release version | |
| 15:38:09 | sean-k-mooney | bauzas: same i want to stick on utc in my case | |
| 15:38:18 | dansmith | I'd prefer some test that ensures we've hit all the versions for each call in unit/func or something | |
| 15:38:40 | bauzas | dansmith: that's my thoughts | |
| 15:38:55 | gibi | dansmith: I thinked about that angle as that would be a good thing in my eys | |
| 15:38:57 | bauzas | checking it thru tempest is something I'd love, but I'm pragmativ | |
| 15:39:05 | bauzas | pragmatic | |
| 15:39:06 | dansmith | gibi: yeah | |
| 15:39:21 | bauzas | we honestly have the pattern to ask with my functest | |
| 15:39:28 | bauzas | it's just a simple request | |
| 15:39:39 | bauzas | and I guess (or I hope) none of the cores to miss this | |
| 15:39:52 | gibi | I have my doubt about my memory | |
| 15:40:01 | sean-k-mooney | testing rpc versions? trying to catch up on the converstation | |
| 15:40:03 | gibi | so I won't promise I will always remember | |
| 15:40:26 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 15:40:26 | gibi | sean-k-mooney: basically avoiding the bug https://bugs.launchpad.net/nova/+bug/1902925 | |
| 15:40:32 | bauzas | gibi: well, RPC and DB upgrades are possibly the hugest changes we could review, right? | |
| 15:40:40 | gibi | right | |
| 15:40:54 | bauzas | I could understand this for a simple method | |
| 15:41:21 | bauzas | but for all the manager services, the risk is present | |
| 15:41:37 | bauzas | but either way, the meeting is in 20 mins | |
| 15:41:45 | bauzas | probably the best is to discuss it there | |
| 15:41:49 | sean-k-mooney | that had functional test for what it was worth | |
| 15:41:59 | bauzas | sean-k-mooney: the cyborg patches ? nope | |
| 15:42:10 | sean-k-mooney | my orginial one did https://review.opendev.org/#/c/715326/ | |
| 15:42:27 | sean-k-mooney | https://review.opendev.org/#/c/715326/29/nova/tests/functional/test_servers.py | |
| 15:43:07 | dansmith | gibi: details of db, rpc, and general upgrade issues in patches have always required lots of human review to get right.. in the early days when we went from not-upgradeable to where we are now.. we added lots of tests where we could, and developer traps like the required db migration tests, | |
| 15:43:20 | dansmith | gibi: but automating all the things is hard and the issues are complex | |
| 15:43:45 | bauzas | sean-k-mooney: you won't catch this error then | |
| 15:43:46 | dansmith | gibi: so I'm all for trying to catch more stuff, especially in a case like this where we just lacked such a test, but ... human review is not replaceable, obviously | |
| 15:43:58 | bauzas | sean-k-mooney: see my regression test, it does capture the bug https://review.opendev.org/#/c/761457/1/nova/tests/functional/regressions/test_bug_1902925.py | |
| 15:44:15 | bauzas | actually https://review.opendev.org/#/c/761457/3/nova/tests/functional/regressions/test_bug_1902925.py | |
| 15:45:21 | bauzas | stephenfin: I don't get your -1 https://review.opendev.org/#/c/761458/3/releasenotes/notes/bug_1902925-351f563340a1e9a5.yaml@11 | |
| 15:45:36 | bauzas | stephenfin: the 'fixes' reno section is purposed to show the fixed bugs | |
| 15:45:51 | bauzas | so that's normal we won't show this note until we merge the patch | |
| 15:46:10 | stephenfin | I'm saying that the docs job won't pass until you do what gibi suggested | |
| 15:46:34 | sean-k-mooney | bauzas: becaue it need the version cap to trigger it | |
| 15:47:00 | openstack | bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] https://launchpad.net/bugs/1902925 - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 15:47:00 | stephenfin | you need to add a leading to underscore '.. bug 1902925:', i.e. '.. _bug 1902925:' | |
| 15:47:01 | bauzas | stephenfin: ah that, no worries I'll fix it | |
| 15:48:05 | gibi | dansmith: I agree that we need human review. All I want is to aid that huma review _if possible_ | |
| 15:48:18 | dansmith | gibi: for sure | |
| 15:49:11 | sean-k-mooney | ok i see what you have changed. hum ok | |
| 15:51:10 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 15:51:10 | bauzas | sean-k-mooney: I explained the issue in https://bugs.launchpad.net/nova/+bug/1902925/comments/3 | |
| 15:51:33 | sean-k-mooney | ya i just wanted to read the repoducer and code fix | |
| 15:52:21 | sean-k-mooney | i guess we have not extended this api since we did the 5.0 rpc bump | |
| 15:52:44 | sean-k-mooney | none of the other fileds are optional | |
| 15:53:07 | tobias-urdin | any good (and somewhat "supported" way) to extend the nova metadata API to include some custom paths? IIRC some ways of extending nova has been deprecated/removed over the years | |
| 15:53:23 | sean-k-mooney | yes | |
| 15:53:58 | sean-k-mooney | tobias-urdin: https://docs.openstack.org/nova/latest/admin/vendordata.html | |
| 15:54:34 | sean-k-mooney | that or i guess you coudl use middleware | |
| 15:54:51 | sean-k-mooney | but in general nova is not extensible in this way intentionally | |
| 15:55:07 | noonedeadpunk | sean-k-mooney: I think I might figued out why isolated aggregates got instances from time to time. any reason not to pass rebuilds through scheduler? https://opendev.org/openstack/nova/src/branch/master/nova/scheduler/manager.py#L146 | |
| 15:55:26 | sean-k-mooney | noonedeadpunk: rebuilds cant change host | |
| 15:55:32 | sean-k-mooney | noonedeadpunk: they are not move operations | |
| 15:55:46 | sean-k-mooney | and they do go to the schduler if the image changes | |
| 15:55:49 | noonedeadpunk | uh.... I see | |
| 15:56:14 | noonedeadpunk | well, I continue get 1 instance per month or smth like that on the isolated aggregate | |
| 15:56:16 | sean-k-mooney | to validate that the current host is still ok with the new image | |
| 15:56:22 | noonedeadpunk | and have no clue how that might happen... | |
| 15:57:03 | sean-k-mooney | are you using the placment way fo doing it | |
| 15:57:08 | sean-k-mooney | or the filter | |
| 15:58:06 | sean-k-mooney | the placement way that should not happen as the traits request should block it but the filter reuires all tenanats to be mapped to an aggreate | |
| 15:58:17 | sean-k-mooney | or unmpped tenants can go to any host | |
| 15:58:57 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Fix the compute RPC 5.12 issue https://review.opendev.org/761458 | |
| 15:59:10 | bauzas | dansmith: gibi: stephenfin: last round, hopefully | |
| 15:59:28 | bauzas | and then I'll backport the changes | |
| 15:59:54 | gibi | thanks | |
| 16:11:23 | noonedeadpunk | sean-k-mooney: do placement traits, exactly like specified in https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html which you gave me one day | |
| 16:11:41 | noonedeadpunk | out of code I see no reason why this might happen | |
| 16:11:45 | noonedeadpunk | but it does | |
| 16:12:57 | sean-k-mooney | ya im not sure either | |
| 16:13:27 | sean-k-mooney | unless you have multiple schduler adn one of them has a differnt config | |
| 16:13:34 | noonedeadpunk | and if previously it was only during resizes or smth like that I found just new VM created a week ago... | |
| 16:13:39 | sean-k-mooney | e.g. on one of them you dont have the prefilter enabled | |
| 16:14:29 | noonedeadpunk | well, I think I've checked that.... in terms of prefilter you mean scheduler.enable_isolated_aggregate_filtering ? | |
| 16:15:30 | sean-k-mooney | bauzas: so that is what was breakign the grenade jobs? good find | |
| 16:16:09 | bauzas | only for the rebuild case | |
| 16:16:18 | bauzas | so, maybe... | |
| 16:16:19 | bauzas | idk | |
| 16:16:30 | gibi | grenade does not run evacuate or grenade tests as far as I know | |
| 16:16:33 | gibi | it runs live migration | |
| 16:16:57 | sean-k-mooney | it runs full tempest before and after | |
| 16:17:04 | sean-k-mooney | i think | |
| 16:17:22 | gibi | really? I only found a smoke result | |
| 16:17:26 | sean-k-mooney | im thinging that if the vm that landed on the unupgraded node was rebuilt it would fail | |
| 16:17:27 | gibi | + the live migration | |
| 16:17:34 | sean-k-mooney | maybe im wrong | |
| 16:20:21 | sean-k-mooney | https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-grenade-multinode/run.yaml#L40 | |
| 16:20:37 | sean-k-mooney | its running the compute api tests and senario tests | |
| 16:21:09 | sean-k-mooney | oh just the smoke subset of those? | |
| 16:21:24 | dansmith | just smoke before, not sure about full after t hough | |
| 16:22:41 | sean-k-mooney | if its runnign rebuild after then if it booted on the upgraded node we would get teh type error | |
| 16:22:57 | sean-k-mooney | if it booted on the un upgraded node it would have rebuilt fine | |
| 16:23:21 | sean-k-mooney | which would have made the test failure intermitent | |