| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-05 | |||
| 10:20:13 | bauzas | but from what I've seen, nobody is really doing it | |
| 10:20:16 | gibi | still I want to automate it if possible :D | |
| 10:20:39 | lyarwood | shouldn't we cover mixed compute upgrades in the multinode grenade job? | |
| 10:20:46 | bauzas | gibi: well, we don't really set new versions a lot right? | |
| 10:20:50 | gibi | becuase all are code review rules are as good as the way we enforce them | |
| 10:21:00 | gibi | bauzas: we do it less and less, I agree | |
| 10:21:01 | bauzas | like, we only had one rpc minor bump per release since a while | |
| 10:21:16 | bauzas | gibi: well, we have a code review documentation | |
| 10:21:24 | bauzas | and I expect cores to know it at least | |
| 10:21:39 | bauzas | I mean, that's a breaking change to accept a RPC change | |
| 10:22:43 | bauzas | maybe we rushed over accepting some feature that was long overdue, but maybe considering to require a functest would ensure that we would put the burden on code owners | |
| 10:22:45 | gibi | bauzas: you are correct that we assume that core reviews catch these kind of problems, but they don't as you found | |
| 10:23:16 | gibi | if we do these thing less and less it means that we will easier to forget what to look at in these changes | |
| 10:24:07 | gibi | as we don't excersize this knowledge | |
| 10:26:26 | gibi | so I agree that one thing is to raise awerness for this issue as you did. | |
| 10:27:16 | gibi | but also I will think about some kind of automation as I cannot promise I won't forget this rule again 6 months from now when we bump the next | |
| 10:44:06 | bauzas | gibi: ahah lol, i had to rush off home because I forgot my kids at the school :whoops: | |
| 10:44:38 | bauzas | gibi: fwiw, the change we merged was a bit hairy, so I do understand that it was difficult to find the problem | |
| 10:45:02 | bauzas | gibi: that's why I said we should at least ask to provide a functional test, that's it | |
| 10:45:41 | gibi | yeah, I should not forget to ask a functional test pining to old RPC version when a new RPC version is proposed | |
| 12:05:33 | brinzhang0 | gibi: hi good morning | |
| 12:05:59 | brinzhang0 | gibi: Hope you can review | |
| 12:06:00 | brinzhang0 | Cyborg shelve/unshelve support patch https://review.opendev.org/#/c/729563/ :D | |
| 12:06:33 | gibi | brinzhang0: add to my queue | |
| 12:06:42 | gibi | added | |
| 12:06:51 | brinzhang0 | gibi: thanks | |
| 13:36:01 | bauzas | brinzhang0: gibi: hah, this time the new argument is nullable :p | |
| 13:36:23 | bauzas | but maybe it's time to ask for a functional testclass verifying the RPC API ? :) | |
| 14:11:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Bump the lowest eventlet version to 0.26.1 https://review.opendev.org/761427 | |
| 15:12:05 | iurygregory | Hi nova folks, a friend of mine using openstack queens asked me if it's possible to update the config drive of an instance? | |
| 15:12:31 | bauzas | gibi: the next nova meeting is in 45 mins, right? | |
| 15:12:36 | bauzas | tz change | |
| 15:14:35 | gibi | bauzas: yes, the meeting is at 16:00 UTC which is 17:00 CET | |
| 15:14:43 | bauzas | cool cool | |
| 15:14:48 | bauzas | nicer for us :) | |
| 15:15:23 | gibi | :) | |
| 15:18:40 | sean-k-mooney | openstack meetings are alwasy utc and never move | |
| 15:18:46 | sean-k-mooney | its other that do | |
| 15:19:57 | sean-k-mooney | fortunetlly DLS will not be a thing in europe after 2021 | |
| 15:20:27 | dansmith | hopefully not on the west coast either, but it's not set yet | |
| 15:20:56 | sean-k-mooney | it was ment to happen this year but got delayed so this was ment to be the last switch | |
| 15:21:34 | sean-k-mooney | the current plan is contries adopting permenatn summer time will swap for the last time in the spring | |
| 15:21:51 | sean-k-mooney | and the rest will swap for the last time in the fall | |
| 15:32:15 | gibi | stephenfin, bauzas: spent some time thinking automating to catch bugs like https://bugs.launchpad.net/nova/+bug/1902925 . Besides code review (that fails some time like in this case) what we can do is to extend the grenade testing. | |
| 15:32:15 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 15:32:36 | gibi | As far as I understand it run livemigration between mixed computes | |
| 15:32:41 | bauzas | sean-k-mooney: I'm against summer time | |
| 15:33:01 | bauzas | gibi: you need then two compute services | |
| 15:33:09 | gibi | bauzas: we have multinode grenade | |
| 15:33:10 | bauzas | and a rolling upgrade scenario | |
| 15:33:33 | bauzas | because the rpc pins will automatically set the version to the oldest compute one | |
| 15:33:42 | bauzas | (if set to 'auto') | |
| 15:34:01 | gibi | I think nova-grenade-multinode does what we need | |
| 15:34:11 | bauzas | and again, tbh, I wonder whether it's just a code review usage | |
| 15:34:46 | gibi | as per https://github.com/openstack/nova/blob/d25bc07d26212408211b64953af7ef6047ca3d9d/playbooks/legacy/nova-grenade-multinode/run.yaml#L47-L50 | |
| 15:34:49 | bauzas | dansmith: your thoughts on it ? tl;dr: automatical uprade testing vs. asking for functional tests that would verify a RPC version minor bump | |
| 15:35:37 | bauzas | gibi: if we run two computes, then okay, we don't need them to be on separate nodes but the other services | |
| 15:35:43 | bauzas | ie. aio+compute | |
| 15:35:53 | bauzas | which is what grenade-multinode is doing AFAIR | |
| 15:35:58 | bauzas | so you're right | |
| 15:36:13 | bauzas | gibi: but then we need to test all the RPC calls in tempest | |
| 15:36:18 | bauzas | good luck with this | |
| 15:36:42 | bauzas | I just feel the simpliest is just to ask for functests | |
| 15:36:53 | bauzas | I wrote them yesterday night and it took me 20 mins | |
| 15:37:27 | gibi | bauzas: never said that we should not ask for a func test. I'm saying that we tend to forget about it as the current bug shows | |
| 15:37:28 | dansmith | yeah, so we could always pin the version to .0, | |
| 15:37:37 | dansmith | but coverage in tempest will be hard, | |
| 15:37:57 | dansmith | plus tempest needs to be graceful as some api calls will fail expectedly if the version doesn't support the new feature | |
| 15:38:00 | bauzas | technically, we need to set the pin to the previous release version | |
| 15:38:09 | sean-k-mooney | bauzas: same i want to stick on utc in my case | |
| 15:38:18 | dansmith | I'd prefer some test that ensures we've hit all the versions for each call in unit/func or something | |
| 15:38:40 | bauzas | dansmith: that's my thoughts | |
| 15:38:55 | gibi | dansmith: I thinked about that angle as that would be a good thing in my eys | |
| 15:38:57 | bauzas | checking it thru tempest is something I'd love, but I'm pragmativ | |
| 15:39:05 | bauzas | pragmatic | |
| 15:39:06 | dansmith | gibi: yeah | |
| 15:39:21 | bauzas | we honestly have the pattern to ask with my functest | |
| 15:39:28 | bauzas | it's just a simple request | |
| 15:39:39 | bauzas | and I guess (or I hope) none of the cores to miss this | |
| 15:39:52 | gibi | I have my doubt about my memory | |
| 15:40:01 | sean-k-mooney | testing rpc versions? trying to catch up on the converstation | |
| 15:40:03 | gibi | so I won't promise I will always remember | |
| 15:40:26 | gibi | sean-k-mooney: basically avoiding the bug https://bugs.launchpad.net/nova/+bug/1902925 | |
| 15:40:26 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 15:40:32 | bauzas | gibi: well, RPC and DB upgrades are possibly the hugest changes we could review, right? | |
| 15:40:40 | gibi | right | |
| 15:40:54 | bauzas | I could understand this for a simple method | |
| 15:41:21 | bauzas | but for all the manager services, the risk is present | |
| 15:41:37 | bauzas | but either way, the meeting is in 20 mins | |
| 15:41:45 | bauzas | probably the best is to discuss it there | |
| 15:41:49 | sean-k-mooney | that had functional test for what it was worth | |
| 15:41:59 | bauzas | sean-k-mooney: the cyborg patches ? nope | |
| 15:42:10 | sean-k-mooney | my orginial one did https://review.opendev.org/#/c/715326/ | |
| 15:42:27 | sean-k-mooney | https://review.opendev.org/#/c/715326/29/nova/tests/functional/test_servers.py | |
| 15:43:07 | dansmith | gibi: details of db, rpc, and general upgrade issues in patches have always required lots of human review to get right.. in the early days when we went from not-upgradeable to where we are now.. we added lots of tests where we could, and developer traps like the required db migration tests, | |
| 15:43:20 | dansmith | gibi: but automating all the things is hard and the issues are complex | |
| 15:43:45 | bauzas | sean-k-mooney: you won't catch this error then | |
| 15:43:46 | dansmith | gibi: so I'm all for trying to catch more stuff, especially in a case like this where we just lacked such a test, but ... human review is not replaceable, obviously | |
| 15:43:58 | bauzas | sean-k-mooney: see my regression test, it does capture the bug https://review.opendev.org/#/c/761457/1/nova/tests/functional/regressions/test_bug_1902925.py | |
| 15:44:15 | bauzas | actually https://review.opendev.org/#/c/761457/3/nova/tests/functional/regressions/test_bug_1902925.py | |