| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-05 | |||
| 15:18:46 | sean-k-mooney | its other that do | |
| 15:19:57 | sean-k-mooney | fortunetlly DLS will not be a thing in europe after 2021 | |
| 15:20:27 | dansmith | hopefully not on the west coast either, but it's not set yet | |
| 15:20:56 | sean-k-mooney | it was ment to happen this year but got delayed so this was ment to be the last switch | |
| 15:21:34 | sean-k-mooney | the current plan is contries adopting permenatn summer time will swap for the last time in the spring | |
| 15:21:51 | sean-k-mooney | and the rest will swap for the last time in the fall | |
| 15:32:15 | gibi | stephenfin, bauzas: spent some time thinking automating to catch bugs like https://bugs.launchpad.net/nova/+bug/1902925 . Besides code review (that fails some time like in this case) what we can do is to extend the grenade testing. | |
| 15:32:15 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 15:32:36 | gibi | As far as I understand it run livemigration between mixed computes | |
| 15:32:41 | bauzas | sean-k-mooney: I'm against summer time | |
| 15:33:01 | bauzas | gibi: you need then two compute services | |
| 15:33:09 | gibi | bauzas: we have multinode grenade | |
| 15:33:10 | bauzas | and a rolling upgrade scenario | |
| 15:33:33 | bauzas | because the rpc pins will automatically set the version to the oldest compute one | |
| 15:33:42 | bauzas | (if set to 'auto') | |
| 15:34:01 | gibi | I think nova-grenade-multinode does what we need | |
| 15:34:11 | bauzas | and again, tbh, I wonder whether it's just a code review usage | |
| 15:34:46 | gibi | as per https://github.com/openstack/nova/blob/d25bc07d26212408211b64953af7ef6047ca3d9d/playbooks/legacy/nova-grenade-multinode/run.yaml#L47-L50 | |
| 15:34:49 | bauzas | dansmith: your thoughts on it ? tl;dr: automatical uprade testing vs. asking for functional tests that would verify a RPC version minor bump | |
| 15:35:37 | bauzas | gibi: if we run two computes, then okay, we don't need them to be on separate nodes but the other services | |
| 15:35:43 | bauzas | ie. aio+compute | |
| 15:35:53 | bauzas | which is what grenade-multinode is doing AFAIR | |
| 15:35:58 | bauzas | so you're right | |
| 15:36:13 | bauzas | gibi: but then we need to test all the RPC calls in tempest | |
| 15:36:18 | bauzas | good luck with this | |
| 15:36:42 | bauzas | I just feel the simpliest is just to ask for functests | |
| 15:36:53 | bauzas | I wrote them yesterday night and it took me 20 mins | |
| 15:37:27 | gibi | bauzas: never said that we should not ask for a func test. I'm saying that we tend to forget about it as the current bug shows | |
| 15:37:28 | dansmith | yeah, so we could always pin the version to .0, | |
| 15:37:37 | dansmith | but coverage in tempest will be hard, | |
| 15:37:57 | dansmith | plus tempest needs to be graceful as some api calls will fail expectedly if the version doesn't support the new feature | |
| 15:38:00 | bauzas | technically, we need to set the pin to the previous release version | |
| 15:38:09 | sean-k-mooney | bauzas: same i want to stick on utc in my case | |
| 15:38:18 | dansmith | I'd prefer some test that ensures we've hit all the versions for each call in unit/func or something | |
| 15:38:40 | bauzas | dansmith: that's my thoughts | |
| 15:38:55 | gibi | dansmith: I thinked about that angle as that would be a good thing in my eys | |
| 15:38:57 | bauzas | checking it thru tempest is something I'd love, but I'm pragmativ | |
| 15:39:05 | bauzas | pragmatic | |
| 15:39:06 | dansmith | gibi: yeah | |
| 15:39:21 | bauzas | we honestly have the pattern to ask with my functest | |
| 15:39:28 | bauzas | it's just a simple request | |
| 15:39:39 | bauzas | and I guess (or I hope) none of the cores to miss this | |
| 15:39:52 | gibi | I have my doubt about my memory | |
| 15:40:01 | sean-k-mooney | testing rpc versions? trying to catch up on the converstation | |
| 15:40:03 | gibi | so I won't promise I will always remember | |
| 15:40:26 | gibi | sean-k-mooney: basically avoiding the bug https://bugs.launchpad.net/nova/+bug/1902925 | |
| 15:40:26 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 15:40:32 | bauzas | gibi: well, RPC and DB upgrades are possibly the hugest changes we could review, right? | |
| 15:40:40 | gibi | right | |
| 15:40:54 | bauzas | I could understand this for a simple method | |
| 15:41:21 | bauzas | but for all the manager services, the risk is present | |
| 15:41:37 | bauzas | but either way, the meeting is in 20 mins | |
| 15:41:45 | bauzas | probably the best is to discuss it there | |
| 15:41:49 | sean-k-mooney | that had functional test for what it was worth | |
| 15:41:59 | bauzas | sean-k-mooney: the cyborg patches ? nope | |
| 15:42:10 | sean-k-mooney | my orginial one did https://review.opendev.org/#/c/715326/ | |
| 15:42:27 | sean-k-mooney | https://review.opendev.org/#/c/715326/29/nova/tests/functional/test_servers.py | |
| 15:43:07 | dansmith | gibi: details of db, rpc, and general upgrade issues in patches have always required lots of human review to get right.. in the early days when we went from not-upgradeable to where we are now.. we added lots of tests where we could, and developer traps like the required db migration tests, | |
| 15:43:20 | dansmith | gibi: but automating all the things is hard and the issues are complex | |
| 15:43:45 | bauzas | sean-k-mooney: you won't catch this error then | |
| 15:43:46 | dansmith | gibi: so I'm all for trying to catch more stuff, especially in a case like this where we just lacked such a test, but ... human review is not replaceable, obviously | |
| 15:43:58 | bauzas | sean-k-mooney: see my regression test, it does capture the bug https://review.opendev.org/#/c/761457/1/nova/tests/functional/regressions/test_bug_1902925.py | |
| 15:44:15 | bauzas | actually https://review.opendev.org/#/c/761457/3/nova/tests/functional/regressions/test_bug_1902925.py | |
| 15:45:21 | bauzas | stephenfin: I don't get your -1 https://review.opendev.org/#/c/761458/3/releasenotes/notes/bug_1902925-351f563340a1e9a5.yaml@11 | |
| 15:45:36 | bauzas | stephenfin: the 'fixes' reno section is purposed to show the fixed bugs | |
| 15:45:51 | bauzas | so that's normal we won't show this note until we merge the patch | |
| 15:46:10 | stephenfin | I'm saying that the docs job won't pass until you do what gibi suggested | |
| 15:46:34 | sean-k-mooney | bauzas: becaue it need the version cap to trigger it | |
| 15:47:00 | stephenfin | you need to add a leading to underscore '.. bug 1902925:', i.e. '.. _bug 1902925:' | |
| 15:47:00 | openstack | bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] https://launchpad.net/bugs/1902925 - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 15:47:01 | bauzas | stephenfin: ah that, no worries I'll fix it | |
| 15:48:05 | gibi | dansmith: I agree that we need human review. All I want is to aid that huma review _if possible_ | |
| 15:48:18 | dansmith | gibi: for sure | |
| 15:49:11 | sean-k-mooney | ok i see what you have changed. hum ok | |
| 15:51:10 | bauzas | sean-k-mooney: I explained the issue in https://bugs.launchpad.net/nova/+bug/1902925/comments/3 | |
| 15:51:10 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 15:51:33 | sean-k-mooney | ya i just wanted to read the repoducer and code fix | |
| 15:52:21 | sean-k-mooney | i guess we have not extended this api since we did the 5.0 rpc bump | |
| 15:52:44 | sean-k-mooney | none of the other fileds are optional | |
| 15:53:07 | tobias-urdin | any good (and somewhat "supported" way) to extend the nova metadata API to include some custom paths? IIRC some ways of extending nova has been deprecated/removed over the years | |
| 15:53:23 | sean-k-mooney | yes | |
| 15:53:58 | sean-k-mooney | tobias-urdin: https://docs.openstack.org/nova/latest/admin/vendordata.html | |
| 15:54:34 | sean-k-mooney | that or i guess you coudl use middleware | |
| 15:54:51 | sean-k-mooney | but in general nova is not extensible in this way intentionally | |
| 15:55:07 | noonedeadpunk | sean-k-mooney: I think I might figued out why isolated aggregates got instances from time to time. any reason not to pass rebuilds through scheduler? https://opendev.org/openstack/nova/src/branch/master/nova/scheduler/manager.py#L146 | |
| 15:55:26 | sean-k-mooney | noonedeadpunk: rebuilds cant change host | |
| 15:55:32 | sean-k-mooney | noonedeadpunk: they are not move operations | |
| 15:55:46 | sean-k-mooney | and they do go to the schduler if the image changes | |
| 15:55:49 | noonedeadpunk | uh.... I see | |
| 15:56:14 | noonedeadpunk | well, I continue get 1 instance per month or smth like that on the isolated aggregate | |
| 15:56:16 | sean-k-mooney | to validate that the current host is still ok with the new image | |
| 15:56:22 | noonedeadpunk | and have no clue how that might happen... | |
| 15:57:03 | sean-k-mooney | are you using the placment way fo doing it | |
| 15:57:08 | sean-k-mooney | or the filter | |
| 15:58:06 | sean-k-mooney | the placement way that should not happen as the traits request should block it but the filter reuires all tenanats to be mapped to an aggreate | |
| 15:58:17 | sean-k-mooney | or unmpped tenants can go to any host | |
| 15:58:57 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Fix the compute RPC 5.12 issue https://review.opendev.org/761458 | |
| 15:59:10 | bauzas | dansmith: gibi: stephenfin: last round, hopefully | |
| 15:59:28 | bauzas | and then I'll backport the changes | |
| 15:59:54 | gibi | thanks | |