| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-04 | |||
| 18:27:41 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) victoria "Upgrades to compute RPC API 5.12 are broken" [Critical,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 18:27:41 | bauzas | dansmith: triaged https://bugs.launchpad.net/nova/+bug/1902925 lemme say if I told something wrong | |
| 18:29:03 | dansmith | bauzas: yep, sounds good.. it makes it sound like the error was combined with the rpc client being right and the server being wrong, | |
| 18:29:13 | dansmith | but even if the client hadn't bumped the version, it would still be wrong | |
| 18:29:17 | dansmith | but yeah, what you said is all right | |
| 18:29:22 | bauzas | ack | |
| 18:29:45 | bauzas | the fix is simple, but I wonder whether we already have a test suite somewhere for it | |
| 18:30:51 | dansmith | well, generally there should be a test to pass the old version and new version of the call, | |
| 18:31:01 | dansmith | but I think it requires adding two test cases, which probably didn't happen | |
| 19:09:46 | bauzas | can anyone know how to get the full stacktrace returned by the API when we have a HTTP500 ? | |
| 19:10:02 | bauzas | I tried with admin rights and I only get the main exception | |
| 19:10:17 | bauzas | ie. OpenStackApiException('Unexpected status code: {"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n |
|
| 19:10:37 | bauzas | while I would like to see the whole stacktrace | |
| 19:11:33 | bauzas | dansmith: any idea for ^ ? | |
| 19:11:47 | dansmith | that's all we see in the log? | |
| 19:11:51 | bauzas | yup | |
| 19:11:59 | bauzas | I wrote a regression test that works | |
| 19:12:10 | dansmith | I would have thought we log.exception a 500, but I'd guess maybe we've had so many in the past that we had to squelch them | |
| 19:12:21 | bauzas | meh | |
| 19:12:23 | bauzas | anyway | |
| 19:12:37 | bauzas | I'll upload and we could discuss this in the change | |
| 19:13:01 | dansmith | ack | |
| 19:19:19 | dansmith | bauzas: I assume you're just asserting that we get a 500 before the fix and a 2xx after, right? that seems clear enough to me | |
| 19:20:25 | bauzas | dansmith: something like that, yes | |
| 19:20:53 | dansmith | seems fine | |
| 19:21:28 | artom | bauzas, I believe there's precedent for capturing the log and asserting stuff about the content | |
| 19:21:43 | artom | If you want to be really sure you're getting the exact error you expect | |
| 19:23:23 | bauzas | uploading it | |
| 19:23:36 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add a regression test for 5.12 compute API issue https://review.opendev.org/761457 | |
| 19:23:37 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Fix the compute RPC 5.12 issue https://review.opendev.org/761458 | |
| 19:23:48 | bauzas | dansmith: artom: gibi: stephenfin: there it is ^ | |
| 19:24:07 | dansmith | artom: on a functional? | |
| 19:24:20 | artom | dansmith, I'm pretty sure I've seen it done, yeah | |
| 19:24:36 | dansmith | that'd be pretty deep mocking of something that doesn't need to be mocked for behavior | |
| 19:24:53 | dansmith | but even still a 500 that goes to not 500 after the fix seems like plenty to me | |
| 19:25:06 | dansmith | to be clear, I think we need more than just mocking of a log call, | |
| 19:25:18 | dansmith | we need mocking of whatever wraps that exception | |
| 19:25:28 | bauzas | anyway, /me goes off | |
| 19:25:34 | bauzas | 8.25pm here :) | |
| 20:22:02 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Add os-volume_attachments reference docs https://review.opendev.org/760971 | |
| 20:39:47 | openstackgerrit | Billy Olsen proposed openstack/nova stable/ussuri: Set migrate_data.vifs only when using multiple port bindings https://review.opendev.org/759151 | |
| 22:09:35 | openstack | bug 1888395 in OpenStack Compute (nova) ussuri "live migration of a vm using the single port binding work flow is broken in train as a result of the introduction of sriov live migration" [High,In progress] https://launchpad.net/bugs/1888395 - Assigned to Billy Olsen (billy-olsen) | |
| 22:09:35 | openstackgerrit | Billy Olsen proposed openstack/nova stable/train: add functional regression test for bug #1888395 https://review.opendev.org/759533 | |
| #openstack-nova - 2020-11-05 | |||
| 01:09:01 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add os-volume_attachments reference docs https://review.opendev.org/760971 | |
| 01:24:12 | openstackgerrit | MaAoyu proposed openstack/os-traits master: bump py37 to py38 in tox.ini https://review.opendev.org/757432 | |
| 05:43:10 | openstackgerrit | Wenping Song proposed openstack/nova-specs master: Support vGPU management by Cyborg https://review.opendev.org/750116 | |
| 07:47:47 | xinranwang | gibi: Hi gibi, I have replied to your comment about smartnic support, and there are some open question need your suggestion, please check it when you got time. Thanks in advance. https://review.opendev.org/#/c/742785/6/specs/wallaby/approved/support-sriov-smartnic.rst | |
| 08:04:11 | gibi | xinranwang: ack, I will look at it today | |
| 08:04:41 | xinranwang | gibi: great, thanks | |
| 08:13:55 | bauzas | good morning Nova | |
| 08:17:07 | gibi | bauzas: o/ | |
| 08:22:26 | openstackgerrit | Merged openstack/nova master: Use subqueryload() instead of joinedload() for (system_)metadata https://review.opendev.org/758928 | |
| 08:36:31 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Fix the compute RPC 5.12 issue https://review.opendev.org/761458 | |
| 08:36:31 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add a regression test for 5.12 compute API issue https://review.opendev.org/761457 | |
| 08:40:45 | bauzas | gibi: you probably missed my pings yesterday night, but I spotted a critical upgrade issue in victoria | |
| 08:41:27 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 08:41:27 | bauzas | https://bugs.launchpad.net/nova/+bug/1902925 | |
| 08:47:04 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) "Upgrades to compute RPC API 5.12 are broken" [Critical,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 08:47:04 | bauzas | made a better explanation of the impact https://bugs.launchpad.net/nova/+bug/1902925/comments/3 | |
| 08:57:41 | gibi | bauzas: thanks, now I read back. good catch | |
| 08:58:05 | bauzas | well, just found it when writing the RPC major bump | |
| 08:58:34 | bauzas | when you know the RPC usage, it's simple | |
| 08:59:11 | bauzas | oh shit, I forgot to add the conditional I promised to dansmith ^_^ | |
| 09:06:36 | bauzas | actually, we don't need it \o/ | |
| 09:26:31 | gibi | bauzas: I'm confused about the naming here https://review.opendev.org/#/c/761457/2/nova/tests/functional/regressions/test_bug_1902925.py@31 | |
| 09:27:03 | bauzas | that's what happens when you copy/paste some methods... | |
| 09:35:37 | gibi | bauzas: when you respin it, could you update the doc here too https://review.opendev.org/#/c/761458/2/nova/compute/manager.py@3355 | |
| 09:37:14 | gibi | besides these, the fix looks good to me | |
| 09:44:17 | gibi | lyarwood, elod: when the bugfix ^^ is merged to victoria we need to push a point release | |
| 09:44:31 | gibi | as this is a critical upgrade issue to V | |
| 09:46:11 | gibi | bauzas: btw, one more request, could you add an upgrade reno to the fix? It would help making visible that upgrading to V needs this fix | |
| 09:47:28 | bauzas | gibi: sure for both | |
| 09:47:34 | gibi | thanks | |
| 09:47:40 | bauzas | I was just about to upload but I killed it | |
| 09:51:49 | lyarwood | gibi: ack | |
| 10:02:48 | lyarwood | so are we not testing rebuild in grenade? | |
| 10:02:53 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Fix the compute RPC 5.12 issue https://review.opendev.org/761458 | |
| 10:02:53 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add a regression test for 5.12 compute API issue https://review.opendev.org/761457 | |
| 10:03:09 | bauzas | gibi: done ^ | |
| 10:03:36 | elod | gibi: thx, I've planned to propose release patches for stein + train + ussuri + victoria today, but then I'll wait with the victoria release patch :] | |
| 10:03:51 | bauzas | elod: hopefully, we'll merge it today | |
| 10:03:51 | elod | lyarwood: fyi ^^^ | |
| 10:04:03 | bauzas | the fix is simple | |
| 10:04:08 | elod | bauzas: \o/ | |
| 10:04:31 | lyarwood | elod: ack thanks | |
| 10:04:31 | elod | then we just have to wait the gate :] | |
| 10:04:47 | lyarwood | I guess we don't test rebuilds in a mixed upgrade state | |
| 10:05:13 | lyarwood | and that's why grenade multinode didn't hit this | |
| 10:05:33 | gibi | bauzas: looking | |
| 10:06:20 | bauzas | lyarwood: stephenfin: gibi: I'm not telling you were bad about reviewing (i also sometimes misses some issues), but maybe it would be nice for you to look at both the fix https://review.opendev.org/#/c/761458/ but also to review https://review.opendev.org/#/c/761452/ to understand how RPC API works | |
| 10:06:46 | bauzas | again, no worries at all | |
| 10:07:24 | bauzas | it's more for providing a knowledge help for you folks about how RPC versions work | |
| 10:07:34 | stephenfin | ah, I knew that and forgot about it :( | |
| 10:07:45 | bauzas | if you knew it, all good then | |
| 10:08:16 | gibi | bauzas: yeah, thanks for the pointers. I'm wondering if we can make some test enhancements to catch these in the future | |
| 10:08:17 | lyarwood | I didn't even review the broken patch here so I'm not sure what you're trying to say | |
| 10:08:21 | lyarwood | .... | |
| 10:08:27 | bauzas | but hopefully the proxy change I'm providing is nice for knowing how to have a major version | |
| 10:09:14 | bauzas | lyarwood: not about any previous reviews, just for helping you to know what to review when you have a change with a RPC modification | |
| 10:09:56 | lyarwood | sure | |