| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-25 | |||
| 12:49:16 | noonedeadpunk | found the reason here, thanks for the clue! | |
| 12:50:49 | lyarwood | noonedeadpunk: yeah that message above isn't part of the actual request it's from a periodic job on the compute that notices that an instance was scheduled but hasn't started yet | |
| 12:50:56 | lyarwood | noonedeadpunk: np | |
| 13:31:14 | sean-k-mooney | noonedeadpunk: im guessing the compute node was not reciving rpc becuase the topic was lost on the queue? | |
| 13:33:20 | noonedeadpunk | sean-k-mooney: no, because we were using rabbitmq with self-signed SSL but without CA | |
| 13:33:29 | noonedeadpunk | which is not allowed in X | |
| 13:33:35 | noonedeadpunk | but was in V | |
| 13:34:13 | noonedeadpunk | and in that patch we upgraded not only nova, but also u-c, so yeah :( | |
| 13:35:58 | noonedeadpunk | (it's eventually because of the some amqp dependancy) | |
| 13:39:02 | sean-k-mooney | ah ok | |
| 13:44:26 | openstackgerrit | Merged openstack/nova master: Wallaby 23.0.0 prelude section https://review.opendev.org/c/openstack/nova/+/782172 | |
| 13:45:49 | openstackgerrit | Merged openstack/nova master: docs: Add a real-time guide https://review.opendev.org/c/openstack/nova/+/781434 | |
| 13:46:58 | openstackgerrit | Merged openstack/nova master: docs: Remove duplicate TPM extra spec info https://review.opendev.org/c/openstack/nova/+/781435 | |
| 13:48:18 | openstackgerrit | Merged openstack/nova master: docs: Remove duplicated PCI passthrough extra spec info https://review.opendev.org/c/openstack/nova/+/781436 | |
| 13:49:15 | openstackgerrit | Merged openstack/nova master: docs: Add SEV guide https://review.opendev.org/c/openstack/nova/+/781437 | |
| 13:51:25 | gibi | \o/ | |
| 14:12:34 | openstackgerrit | Radomir Dopieralski proposed openstack/python-novaclient master: When creating a client, pass the default logger https://review.opendev.org/c/openstack/python-novaclient/+/783010 | |
| 14:13:18 | Luzi | lyarwood, could you take a look at https://review.opendev.org/c/openstack/nova/+/782430 ? i would like to propose the backport for stein too and start writing the ossn soon | |
| 14:15:37 | openstackgerrit | Merged openstack/nova master: Bump the Compute RPC API to version 6.0 https://review.opendev.org/c/openstack/nova/+/761452 | |
| 14:16:04 | lyarwood | Luzi: done | |
| 14:16:59 | Luzi | lyarwood, thank you :) | |
| 14:21:42 | sean-k-mooney | gibi: actully if there is a followup patch to sev there are a few thigns we shoudl add | |
| 14:23:09 | sean-k-mooney | gibi: multi numa guest may fail to boot due to a qemu issue https://bugzilla.redhat.com/show_bug.cgi?id=1814502 and we shoudl be recommending that hw:mem_page_size be set since sev is locking memory | |
| 14:23:10 | openstack | sean-k-mooney: Error: Error getting bugzilla.redhat.com bug #1814502: NotPermitted | |
| 14:23:36 | sean-k-mooney | oh thats se to private i guess | |
| 14:24:15 | openstackgerrit | Mark Goddard proposed openstack/nova master: Check volume encryption keys prior to resize and migration https://review.opendev.org/c/openstack/nova/+/781079 | |
| 14:28:22 | gibi | sean-k-mooney: you can still propose a reno extension for the RC1. | |
| 14:28:57 | gibi | sean-k-mooney: after today you need a bug that fix on master then backport to stable/wallaby and propose an RC2 | |
| 14:34:41 | sean-k-mooney | well its a docs fix not reno the new admin/sev.rst doc | |
| 14:34:58 | sean-k-mooney | so ya we can fix it later | |
| 14:35:17 | sean-k-mooney | its not urgent | |
| 14:37:32 | gibi | sean-k-mooney: OK | |
| 14:38:26 | bauzas | just looking up fast, have we a regression candidate ? | |
| 14:39:04 | bauzas | and wow for the rcp bump being merged \o/ | |
| 14:44:02 | stephenfin | Hmm, my ZNC bouncer went down :-( | |
| 14:44:16 | stephenfin | Guess that's what I get for 20€ a year or whatever the VPS cost | |
| 14:46:15 | gibi | bauzas: what regression candidate? :) | |
| 14:47:41 | dansmith | bauzas: did you comb through the logs on the RPC bump? https://zuul.opendev.org/t/openstack/build/952de21f47e04866a8a4dcc34f557135/log/logs/screen-n-cpu.txt?severity=3#21829 | |
| 14:48:00 | dansmith | isn't that likely related? I have to go back and look but I thought you touched that method | |
| 14:51:33 | dansmith | looks like it's getting instance and data confused, maybe because you changed the ordering of those params and didn't fix the caller? | |
| 14:52:28 | bauzas | dansmith: eeek, maybe | |
| 14:52:37 | bauzas | lemme look | |
| 14:53:03 | bauzas | and no, while I looked up the logs, I haven't seen those | |
| 14:53:11 | bauzas | but maybe it was for a previous revision | |
| 14:53:57 | dansmith | sorry, I've been trying to schedule a deep dive of the logs, but have been getting distracted.. was surprised to see it merged this morning, so figured I better do that | |
| 14:55:29 | dansmith | bauzas: want me to put up a fix? | |
| 14:55:33 | bauzas | honestly, my fault | |
| 14:55:48 | bauzas | I rushed on fixing a few comments | |
| 14:55:58 | bauzas | but I forgot to lookup the last job run | |
| 14:56:31 | dansmith | well, reviewers should be checking this stuff too | |
| 14:56:56 | bauzas | either way, ship has sailed but we can hold RC1 until we merge the fix | |
| 14:57:00 | bauzas | gibi: ^ | |
| 14:57:17 | bauzas | dansmith: you're already on it ? or you want me to fix the caller ? | |
| 14:57:25 | gibi | we will hodl the RC1 until the fix lands | |
| 14:57:45 | dansmith | bauzas: I'm looking yeah, hang on | |
| 14:57:58 | bauzas | gibi: thanks | |
| 14:58:06 | bauzas | dansmith: AFAIR, we don't have a long list of callers | |
| 14:58:34 | stephenfin | Mea culpa. I didn't think to at the existing CI run past seeing everything was green. I figured breakages would be obvious. I can work on a fix if no one else is? | |
| 14:59:13 | gibi | I don't see the obvious failure in the patch :/ | |
| 14:59:35 | bauzas | dansmith: interesting, looks to me it's dead code | |
| 14:59:49 | bauzas | or I'm blind | |
| 15:00:26 | dansmith | eh? | |
| 15:00:28 | bauzas | oh, wait no | |
| 15:00:30 | bauzas | I'm tired | |
| 15:00:43 | bauzas | _is_instance_storage_shared() | |
| 15:01:06 | bauzas | that's a compute to compute call | |
| 15:01:12 | dansmith | yeah, I'm running tests now to figure out what needs to change in there | |
| 15:02:12 | bauzas | all looks we only mock this caller | |
| 15:02:18 | bauzas | and we don't verify it | |
| 15:03:14 | bauzas | but maybe we verify it's called | |
| 15:10:52 | bauzas | dansmith: gibi: stephenfin: I have to get my kids off school but I'm back in 20-ish minutes | |
| 15:15:03 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix check_instance_shared_storage() call https://review.opendev.org/c/openstack/nova/+/783026 | |
| 15:15:22 | dansmith | bauzas: gibi ^ | |
| 15:15:26 | gibi | ack looking | |
| 15:15:39 | bauzas | dansmith: already on it | |
| 15:15:42 | bauzas | all looks good | |
| 15:15:54 | dansmith | assuming that clears the one fail, we should still examine the logs for more stuff | |
| 15:15:57 | bauzas | I know I reordered a few params | |
| 15:16:11 | bauzas | agreed | |
| 15:16:38 | bauzas | just giving a +2 as it doesn't harm but I agree on the fact we should wait for the next grenade run | |
| 15:17:26 | gibi | ack, I hold my +W until grenade | |
| 15:18:09 | bauzas | a very brief look on https://review.opendev.org/c/openstack/nova/+/761452/14/nova/compute/rpcapi.py makes me think it was the only method I switched the args | |
| 15:20:53 | gibi | bauzas: yes, I agree I don't see any other reordering | |
| 15:22:01 | dansmith | fwiw I was looking for more subtle fails when I found this | |
| 15:22:28 | dansmith | reorderings I would have expected to be caught by unit tests | |
| 15:27:53 | gibi | now I looked through the log you linked and the rest of the warning seems acceptable to me | |
| 15:50:50 | gibi | nova meeting starts in 10 minutes in #openstack-meeting-3 | |
| 15:54:19 | openstackgerrit | Merged openstack/nova stable/train: Add config parameter 'live_migration_scheme' to live migration with tls guide https://review.opendev.org/c/openstack/nova/+/782430 | |
| 16:22:51 | stephenfin | gibi, melwitt: In case you haven't seen it yet, the placement fix is here https://review.opendev.org/c/openstack/placement/+/782988 | |
| 16:23:01 | stephenfin | Might be worth putting into RC1 | |
| 16:23:18 | stephenfin | If not, it should be an immediate backport given how easy this is to hit | |
| 16:23:33 | sean-k-mooney | why was that not caut by the existing testing for duplicates? | |
| 16:23:49 | sean-k-mooney | oh | |
| 16:23:52 | stephenfin | Cos they use SQLite, I'm going to guess | |
| 16:23:54 | sean-k-mooney | that is useing sqlite | |
| 16:24:00 | stephenfin | yarp | |
| 16:24:08 | sean-k-mooney | ya ok makes sense | |
| 16:24:49 | sean-k-mooney | we could look at adding the same opertunistic testign we don in nova but that might eb overkill for the func tests | |
| 16:25:43 | gibi | stephenfin: I don't have cycles today any more | |
| 16:25:51 | sean-k-mooney | it makes sense why we are only seeing this now | |