| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-10 | |||
| 14:34:33 | kashyap | Okay, so expected error there. | |
| 14:34:56 | gibi | yepp that is OK and the test case is pass, but after the test case tempest cleans up | |
| 14:35:06 | gibi | basically did the actions in revers to move back to the starting state | |
| 14:35:19 | gibi | so as the server in RESCUE state it unrescues it | |
| 14:35:39 | gibi | and as a volume was attached to the server before rescue, it tries to detach the volume after the unrescue | |
| 14:35:52 | gibi | and that detach should remove the volume from the domian but that fails | |
| 14:37:00 | kashyap | I see. And looks like it couldn't find that volume? | |
| 14:37:37 | kashyap | Maybe this goes back to Peter's comment earlier about how libvirt tries "to detach a blockdev node that wasn't ever attached" | |
| 14:38:58 | kashyap | These weird tests are spinning my head. | |
| 14:39:41 | gibi | during this detach: nova first detaches the volume from the persistent domain that succeeds | |
| 14:39:57 | gibi | then nova issue the detach command from the live domian and waits for the event | |
| 14:40:09 | gibi | that event is not received in 20 sec so it issue the command again | |
| 14:40:23 | kashyap | Ah-ha, that makes sense | |
| 14:40:26 | gibi | that commend returns | |
| 14:40:27 | gibi | error message: internal error: unable to execute QEMU command 'device_del': Device virtio-disk1 is already in the process of unplug | |
| 14:40:57 | gibi | then nova retries 6 more times | |
| 14:41:03 | kashyap | Right, and then times out | |
| 14:41:05 | gibi | always getting the same message | |
| 14:41:09 | gibi | and then gives up | |
| 14:41:18 | gibi | this is the log from the detach attempst https://paste.opendev.org/show/be647YeC57HREuAfwwru/ | |
| 14:41:49 | kashyap | gibi: Your last 10-ish messages are a great summary of the prob at hand. Can I take and rephrase them into a paragraph on that mail thread? | |
| 14:41:59 | gibi | sure | |
| 14:48:16 | kashyap | gibi: I've also posted the pimped up version here: https://bugs.launchpad.net/nova/+bug/1960346/comments/8 | |
| 14:48:46 | gibi | thanks | |
| 14:52:05 | gmann | gibi: ack. thanks | |
| 14:52:09 | opendevreview | Andre Aranha proposed openstack/nova stable/xena: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827895 | |
| 14:53:01 | gmann | frickler: I opened it for devstack due to centos9 libvirt version bump but it is ok to add nova too | |
| 14:55:23 | kashyap | frickler: Forgot to respond in the scrollback; yeah, that explains it - why my `wget` didn't work :) Thank you. | |
| 14:55:59 | gibi | kashyap: these are the libvirtd logs from the first and second detach https://paste.opendev.org/show/bKANW2WAzfAzEIGcgJX8/ probably nothing new here I just working through the logs... | |
| 14:59:02 | kashyap | gibi: Nice work narrowing down | |
| 14:59:15 | kashyap | A side-tip is: | |
| 14:59:34 | kashyap | $> grep -Ei '(MONITOR_SEND_MSG|QEMU_MONITOR_RECV_)' libvirtd_log.txt | |
| 14:59:46 | kashyap | (That gets you all the commands and the responses libvirt is sending to QEMU.) | |
| 15:43:52 | spatel | folks i need urgent help to understand what is wrong with nova and rabbitMQ :( | |
| 15:44:13 | spatel | I have rebuild rabbitMQ but now not able to spin up VM | |
| 15:44:32 | spatel | vm getting stuck in BUILD | |
| 15:45:02 | spatel | nova-conductor throwing these errors - https://paste.opendev.org/show/bbSVnr5zGdCCOPdL5tQF/ | |
| 15:48:37 | kashyap | No answer, but please don't count on community to provide "urgent help". That's what vendors are for | |
| 15:49:02 | spatel | kashyap i understand just looking for clue to see what is going on | |
| 16:04:36 | melwitt | gibi: ack, will look | |
| 16:05:26 | gibi | melwitt: thanks | |
| 16:14:36 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| 16:24:14 | opendevreview | Andre Aranha proposed openstack/nova stable/xena: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827895 | |
| 16:29:43 | opendevreview | Andre Aranha proposed openstack/nova stable/wallaby: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827896 | |
| 17:02:31 | opendevreview | Lior Friedman proposed openstack/nova master: Support use_multipath for NVME driver https://review.opendev.org/c/openstack/nova/+/823941 | |
| 17:05:23 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Document remote-managed port usage considerations https://review.opendev.org/c/openstack/nova/+/827513 | |
| 17:12:00 | opendevreview | Lior Friedman proposed openstack/nova master: Support use_multipath for NVME driver https://review.opendev.org/c/openstack/nova/+/823941 | |
| 17:40:04 | gibi | kashyap: fyi there is a smaller reproduction in https://bugs.launchpad.net/nova/+bug/1960346/comments/10 | |
| 17:40:14 | gibi | but I have to drop off now | |
| 18:01:05 | opendevreview | melanie witt proposed openstack/placement master: Make perfload jobs fail if write allocation fails https://review.opendev.org/c/openstack/placement/+/828438 | |
| 18:32:18 | opendevreview | Ghanshyam proposed openstack/nova master: Make more project level APIs scoped to project only https://review.opendev.org/c/openstack/nova/+/828670 | |
| 18:54:17 | chateaulav | gibi: can i get a little more on the backporting of the 1.3 to 1.2. I have been playing around with it but am not quite sure. is this more related to the actual version itself or pulling the new values available in 1.3 to 1.2. this is for https://review.opendev.org/c/openstack/nova/+/828369 and i know that my question seems repeatative | |
| 19:43:55 | opendevreview | Merged openstack/nova master: Join quota exception family trees https://review.opendev.org/c/openstack/nova/+/828185 | |
| 20:27:08 | spatel | kashyap by the way i found issue, it was related to neutron-metadata service which was causing issue and holding VM build.. | |
| 21:35:48 | opendevreview | melanie witt proposed openstack/nova stable/wallaby: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825178 | |
| 23:03:26 | chateaulav | gibi: found the info I needed. Will add the back ports tomorrow. | |
| 23:21:29 | opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | |
| 23:21:51 | opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | |
| #openstack-nova - 2022-02-11 | |||
| 02:05:58 | opendevreview | Merged openstack/nova stable/victoria: Reproduce bug 1953359 https://review.opendev.org/c/openstack/nova/+/820558 | |
| 02:06:05 | opendevreview | Merged openstack/nova stable/victoria: Extend the reproducer for 1953359 and 1952915 https://review.opendev.org/c/openstack/nova/+/820856 | |
| 03:11:03 | opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | |
| 03:11:03 | opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | |
| 03:11:04 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 03:11:04 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 03:11:05 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 03:11:05 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 03:11:06 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 03:11:06 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 03:11:07 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 03:11:08 | opendevreview | melanie witt proposed openstack/nova master: Enable unified limits in the nova-next job https://review.opendev.org/c/openstack/nova/+/789963 | |
| 03:54:03 | opendevreview | Merged openstack/nova stable/xena: Reproduce bug 1952941 https://review.opendev.org/c/openstack/nova/+/827868 | |
| 03:56:59 | opendevreview | Merged openstack/nova stable/xena: Migrate RequestSpec.numa_topology to use pcpuset https://review.opendev.org/c/openstack/nova/+/827869 | |
| 04:03:15 | opendevreview | Merged openstack/nova stable/wallaby: Add functional test for bug 1937375 https://review.opendev.org/c/openstack/nova/+/803717 | |
| 06:17:42 | opendevreview | Minghong Hou proposed openstack/nova master: fix VirtualInterface table can't be update https://review.opendev.org/c/openstack/nova/+/828819 | |
| 08:41:13 | nikparasyr | hello, I have a question regarding shelving/unshelving. We have a flavor with `hw:cpu_policy='dedicated', hw:cpu_thread_policy='isolate'` and pci_passthrough 2 gpus. When we try to unshelve we get this error " Insufficient compute resources: Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology.". My question is to what extent does Nova require to | |
| 08:41:13 | nikparasyr | find the exact same cpu set available on the target host? We have enabled the PCIPassThrough filter for the scheduler but not the NUMATopologyFilter. If I understand well the NUMATopologyFilter will make sure that the scheduler picks a node that has the required topology available. Even so, if Nova requires the exact same cpu set to the target host we will still have an issue even with the numa filter... | |
| 08:41:14 | nikparasyr | So, any idea to what extend does nova require the exact same cpu set when cpu pinning is enabled? | |
| 10:05:18 | kashyap | gibi: Thanks for the link to the smaller repro; also check out Peter's response on that thread | |
| 10:05:32 | kashyap | He points out two possibilities: | |
| 10:05:32 | kashyap | 1) the guest OS didn't confirm the detach | |
| 10:05:33 | kashyap | 2) there was a recent bug in qemu triggered by using JSON syntax for -device | |
| 10:10:16 | kashyap | gibi: That's it: this looks like it -- | |
| 10:10:17 | kashyap | "DEVICE_DELETED event is not delivered for device frontend if -device is configured via JSON" | |
| 10:10:21 | kashyap | https://bugzilla.redhat.com/show_bug.cgi?id=2036669 | |
| 10:19:45 | kashyap | But based on the versions in the CI job, they should already have the fix: | |
| 10:19:48 | kashyap | - libvirt version: 8.0.0, package: 2.el9 | |
| 10:19:51 | kashyap | - qemu-kvm-6.2.0-5.el9 | |
| 10:27:42 | kashyap | gibi: When you're around, to rule out the above bug, I wonder if we could try this workaround: | |
| 10:28:18 | kashyap | On compute nodes, in /etc/libvit/qemu.conf: | |
| 10:28:27 | kashyap | capability_filters = [ "device.json" ] | |
| 10:34:47 | gibi | kashyap: hi! | |
| 10:35:22 | gibi | kashyap: sure, I will try to make that config change via devstack | |
| 10:35:31 | gibi | kashyap: does it require a libvirtd restart? | |
| 10:35:34 | kashyap | gibi: See my latest comment: https://bugs.launchpad.net/nova/+bug/1960346/ | |
| 10:35:47 | kashyap | gibi: Yeah, it is required, afraid | |
| 10:36:09 | gibi | OK | |
| 10:36:10 | gibi | thanks | |
| 11:01:08 | gibi | kashyap: pushed new PS to https://review.opendev.org/c/openstack/devstack/+/828705 with the WA, lets see if it helps or not | |
| 11:01:29 | kashyap | gibi: Thank you! It will at least rule out the 2nd possibility above for sure. | |
| 11:01:46 | gibi | I can try to look at the first | |