| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-22 | |||
| 16:39:01 | gibi | I cannot correlate the logs :/ | |
| 16:39:01 | kashyap | gibi: Good news! | |
| 16:39:04 | gibi | oh | |
| 16:39:18 | kashyap | gibi: I just talked to Peter Krempa on #virt, he already identified it as a libvirt bug ... | |
| 16:39:22 | kashyap | gibi: The problem is this, in short: | |
| 16:39:55 | kashyap | [quote] | |
| 16:39:56 | kashyap | libvirt has a piece of code which ensures that thh backing image of the reused destination image is added only when finishing the job. On cancellation of the [copy] job, we want to unplug the image, but the backing image was not yet plugged in. | |
| 16:40:00 | kashyap | [/quote] | |
| 16:40:08 | kashyap | It assumes a bit of libvirt storage knowledge, but Peter is already working on a fix | |
| 16:41:13 | kashyap | gibi: I'm filing RHEL bugs to get it backported to CentOS 8 and 9 | |
| 16:41:43 | gibi | awesome | |
| 16:41:48 | gibi | thanks for the info | |
| 16:44:53 | kashyap | gibi: But Peter is again asking what is the test "exactly doing" | |
| 16:44:57 | kashyap | Is this exactly it? | |
| 16:45:02 | kashyap | "boot servers, then snapshot the server and upload the image to Glance" | |
| 16:45:27 | kashyap | Did you mean boot a "server", as opposed to plural? | |
| 16:47:36 | gibi | kashyap: create a server, and then create a snapshot | |
| 16:47:45 | gibi | kashyap: the test case itself create multiple servers | |
| 16:48:04 | kashyap | I see | |
| 16:57:14 | kashyap | gibi: https://bugzilla.redhat.com/show_bug.cgi?id=2057067 | |
| 16:57:28 | kashyap | gibi: But Peter also says there might be another issue too, because: | |
| 16:57:40 | kashyap | ... "this is on a code path that can't propagate errors to the caller" | |
| 16:57:48 | kashyap | gibi: Anyway, he posted a patch here: https://listman.redhat.com/archives/libvir-list/2022-February/msg00790.html | |
| 17:19:45 | gibi | kashyap: ack. I would need a tripleoo job result where I can correlate logs better to be able to look more into it | |
| 17:20:03 | gibi | kashyap: I serched for the QEMU error in the upstream nova job results and there is a lot of hit | |
| 17:20:16 | gibi | but I guess it is a generic error message | |
| 17:20:51 | gibi | as the QEMU error does not correlate with the tempest test failure | |
| 17:21:11 | gibi | I stop for today but happy to help tomorrow if there are things to look at | |
| 18:53:42 | opendevreview | Rajat Dhasmana proposed openstack/nova master: Add support for volume backed server rebuild https://review.opendev.org/c/openstack/nova/+/820368 | |
| 18:53:58 | opendevreview | Merged openstack/nova master: Support use_multipath for NVME driver https://review.opendev.org/c/openstack/nova/+/823941 | |
| 19:18:20 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Fix migration with remote-managed ports & add FT https://review.opendev.org/c/openstack/nova/+/829974 | |
| 19:33:25 | opendevreview | Rajat Dhasmana proposed openstack/python-novaclient master: Add parameter to rebuild boot volume https://review.opendev.org/c/openstack/python-novaclient/+/827163 | |
| 19:37:04 | whoami-rajat | dansmith, Hey, if you get some time, can you take a look at my feature patches? | |
| 19:37:06 | whoami-rajat | nova: https://review.opendev.org/c/openstack/nova/+/820368 | |
| 19:37:11 | whoami-rajat | novaclient: https://review.opendev.org/c/openstack/python-novaclient/+/827163 | |
| 19:37:53 | dansmith | whoami-rajat: ah I didn't see that you had updated them.. is it working now? | |
| 19:39:33 | dansmith | whoami-rajat: ah I didn't see that you had updated them.. is it working now? | |
| 19:39:45 | dansmith | whoami-rajat: I assume you've got a tempest patch somewhere (or planned) to poke it | |
| 19:39:46 | dansmith | ? | |
| 19:40:28 | opendevreview | Merged openstack/placement master: Add any-traits support for allocation candidates https://review.opendev.org/c/openstack/placement/+/826492 | |
| 19:40:40 | whoami-rajat | dansmith, just updated everything and yes it is working now. thanks! | |
| 19:41:01 | opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 19:41:01 | opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | |
| 19:41:04 | whoami-rajat | dansmith, I've mostly tested this manually but i can work on a tempest test, the problem is deadline is close | |
| 19:41:30 | whoami-rajat | dansmith, If i can get an FFE or propose the test after feature freeze then it will be great | |
| 19:42:18 | dansmith | I mean, I think a tempest test should be part of the requirements here.. was that not in the spec for this? | |
| 19:42:41 | dansmith | I would think it'd be easy right? take an existing rebuild test, make it use BFV, and pass the right version on rebuild? | |
| 19:43:25 | dansmith | if the tempest test shows it works and just needs some work after the deadline to get it into a merge-able shape, I think that's fine | |
| 19:44:15 | whoami-rajat | sure, I haven't looked at existing tests but if there's already one for rebuild, it should be pretty straightforward | |
| 19:44:40 | dansmith | yeah, tempest test is in the testing section of the spec... | |
| 19:48:15 | whoami-rajat | dansmith, I will try and get something working by tomorrow | |
| 19:48:21 | dansmith | ack | |
| 19:48:35 | whoami-rajat | thanks! | |
| 20:20:05 | opendevreview | Merged openstack/placement master: Remove unused compatibility code https://review.opendev.org/c/openstack/placement/+/826493 | |
| 20:23:53 | opendevreview | Merged openstack/placement master: Add microversion 1.39 to support any-trait queries https://review.opendev.org/c/openstack/placement/+/826719 | |
| 20:26:12 | gmann | dansmith: rbac series is ready, added release notes also in last patch please check if that is ok https://review.opendev.org/q/topic:bp/policy-defaults-refresh-2+status:open | |
| #openstack-nova - 2022-02-23 | |||
| 02:14:20 | opendevreview | Ghanshyam proposed openstack/nova master: API change to allow project admin to boot server on specific host https://review.opendev.org/c/openstack/nova/+/830543 | |
| 02:50:28 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Register defaults for undefined hw image properties https://review.opendev.org/c/openstack/nova/+/800708 | |
| 05:09:41 | opendevreview | melanie witt proposed openstack/nova master: WIP manage: Add image_property commands https://review.opendev.org/c/openstack/nova/+/824392 | |
| 09:00:59 | kashyap | gibi: Thank you for looking; just read your messages in the scroll! Yeah, the QEMU error might not correlate with the Tempest failure, but it seems to have uncovered a latent libvirt bug | |
| 09:01:03 | kashyap | Morning, BTW | |
| 09:47:36 | gibi | kashyap: o/ I tried to find the same error in any tempest-integrated-compute-centos-8-stream job but in the last 10 days of run I did not find the same error as reported in https://bugs.launchpad.net/tripleo/+bug/1959014/comments/1 | |
| 10:02:00 | gibi | bauzas: I was about to update the tracking etherpad with the placement feature but I see you updating that in real time :) | |
| 10:02:57 | bauzas | heh | |
| 10:03:02 | bauzas | was about to review the series | |
| 10:03:10 | bauzas | then I saw sean having done it | |
| 10:03:27 | bauzas | gibi: WDYT of https://review.opendev.org/c/openstack/nova/+/808791 ? | |
| 10:04:11 | gibi | bauzas: I'm not worrying about it | |
| 10:04:19 | bauzas | me too | |
| 10:04:34 | gibi | hence my +2 :) | |
| 10:04:37 | bauzas | I'll review it again | |
| 10:05:08 | gibi | ack | |
| 10:05:44 | gibi | I will go back to https://review.opendev.org/q/topic:bp%252Fpick-guest-arch-based-on-host-arch-in-libvirt-driver | |
| 10:05:59 | gibi | I saw chateaulav updated it after my comments | |
| 10:19:03 | bauzas | gibi: well, I'm still torn by https://review.opendev.org/c/openstack/nova/+/808791/7/nova/virt/vmwareapi/constants.py#22 | |
| 10:19:32 | bauzas | I'm a vmware specialist tho but I assume we say the whole driver works with VC 5.1.0 | |
| 10:19:48 | bauzas | I'm not* a vmware specialist, obviously | |
| 10:20:00 | bauzas | can someone know Alban's IRC nick ? | |
| 10:29:11 | opendevreview | melanie witt proposed openstack/nova master: manage: Add image_property commands https://review.opendev.org/c/openstack/nova/+/824392 | |
| 10:35:44 | kashyap | gibi: (Sorry, was out for a run; weather is nice for a change) Hm, I'm feeling like we're 2 steps fwd, 1 step backward | |
| 10:44:16 | gibi | bauzas: Alban responded in review to update the versions so I'm still OK | |
| 10:44:29 | gibi | kashyap: I still in the | |
| 10:44:40 | gibi | kashyap: I'm still in the winter mode an only use the pool | |
| 10:44:48 | gibi | *and | |
| 10:45:55 | kashyap | I can't swim yet, I can do "controlled not drowning" in a calm pool if you know what I mean :D | |
| 10:46:07 | gibi | :) | |
| 10:46:24 | gibi | kashyap: about the bug, I'm not sure how to move forward | |
| 10:46:28 | kashyap | (No worries. I only began after 2 months...but keep walking 8-ish KM each day. But it's no substitue for running) | |
| 10:46:43 | kashyap | gibi: Back to the bug, it looks like it is not reproducible, right? | |
| 10:47:49 | kashyap | gibi: Did you look for this? | |
| 10:47:51 | kashyap | "libvirt.libvirtError: Unable to read from monitor: Connection reset by peer" | |
| 10:48:27 | gibi | I looked for the more specific raise libvirtError..virDomainGetBlockJobInfo.. failed.. regex | |
| 10:48:53 | gibi | but I can check the more generic one | |
| 10:50:31 | kashyap | gibi: Your search makes sense. | |
| 10:50:50 | kashyap | Yeah, what I'm wondering is, we don't see the _crash_ anymore: | |
| 10:50:51 | kashyap | "2022-01-25 13:13:35.058+0000: shutting down, reason=crashed" | |
| 10:51:14 | kashyap | Is that what you found based on your searches? That you can't find the crash anymore? | |
| 10:52:43 | gibi | I did not find the same nova symptom, the libvirtError, in any upstream logs, so I did not look deeper | |
| 10:53:27 | gibi | I can also try to find the same tempest test failure | |
| 10:53:32 | gibi | in upstream jobs | |