| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-10 | |||
| 14:55:23 | kashyap | frickler: Forgot to respond in the scrollback; yeah, that explains it - why my `wget` didn't work :) Thank you. | |
| 14:55:59 | gibi | kashyap: these are the libvirtd logs from the first and second detach https://paste.opendev.org/show/bKANW2WAzfAzEIGcgJX8/ probably nothing new here I just working through the logs... | |
| 14:59:02 | kashyap | gibi: Nice work narrowing down | |
| 14:59:15 | kashyap | A side-tip is: | |
| 14:59:34 | kashyap | $> grep -Ei '(MONITOR_SEND_MSG|QEMU_MONITOR_RECV_)' libvirtd_log.txt | |
| 14:59:46 | kashyap | (That gets you all the commands and the responses libvirt is sending to QEMU.) | |
| 15:43:52 | spatel | folks i need urgent help to understand what is wrong with nova and rabbitMQ :( | |
| 15:44:13 | spatel | I have rebuild rabbitMQ but now not able to spin up VM | |
| 15:44:32 | spatel | vm getting stuck in BUILD | |
| 15:45:02 | spatel | nova-conductor throwing these errors - https://paste.opendev.org/show/bbSVnr5zGdCCOPdL5tQF/ | |
| 15:48:37 | kashyap | No answer, but please don't count on community to provide "urgent help". That's what vendors are for | |
| 15:49:02 | spatel | kashyap i understand just looking for clue to see what is going on | |
| 16:04:36 | melwitt | gibi: ack, will look | |
| 16:05:26 | gibi | melwitt: thanks | |
| 16:14:36 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| 16:24:14 | opendevreview | Andre Aranha proposed openstack/nova stable/xena: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827895 | |
| 16:29:43 | opendevreview | Andre Aranha proposed openstack/nova stable/wallaby: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827896 | |
| 17:02:31 | opendevreview | Lior Friedman proposed openstack/nova master: Support use_multipath for NVME driver https://review.opendev.org/c/openstack/nova/+/823941 | |
| 17:05:23 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Document remote-managed port usage considerations https://review.opendev.org/c/openstack/nova/+/827513 | |
| 17:12:00 | opendevreview | Lior Friedman proposed openstack/nova master: Support use_multipath for NVME driver https://review.opendev.org/c/openstack/nova/+/823941 | |
| 17:40:04 | gibi | kashyap: fyi there is a smaller reproduction in https://bugs.launchpad.net/nova/+bug/1960346/comments/10 | |
| 17:40:14 | gibi | but I have to drop off now | |
| 18:01:05 | opendevreview | melanie witt proposed openstack/placement master: Make perfload jobs fail if write allocation fails https://review.opendev.org/c/openstack/placement/+/828438 | |
| 18:32:18 | opendevreview | Ghanshyam proposed openstack/nova master: Make more project level APIs scoped to project only https://review.opendev.org/c/openstack/nova/+/828670 | |
| 18:54:17 | chateaulav | gibi: can i get a little more on the backporting of the 1.3 to 1.2. I have been playing around with it but am not quite sure. is this more related to the actual version itself or pulling the new values available in 1.3 to 1.2. this is for https://review.opendev.org/c/openstack/nova/+/828369 and i know that my question seems repeatative | |
| 19:43:55 | opendevreview | Merged openstack/nova master: Join quota exception family trees https://review.opendev.org/c/openstack/nova/+/828185 | |
| 20:27:08 | spatel | kashyap by the way i found issue, it was related to neutron-metadata service which was causing issue and holding VM build.. | |
| 21:35:48 | opendevreview | melanie witt proposed openstack/nova stable/wallaby: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825178 | |
| 23:03:26 | chateaulav | gibi: found the info I needed. Will add the back ports tomorrow. | |
| 23:21:29 | opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | |
| 23:21:51 | opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | |
| #openstack-nova - 2022-02-11 | |||
| 02:05:58 | opendevreview | Merged openstack/nova stable/victoria: Reproduce bug 1953359 https://review.opendev.org/c/openstack/nova/+/820558 | |
| 02:06:05 | opendevreview | Merged openstack/nova stable/victoria: Extend the reproducer for 1953359 and 1952915 https://review.opendev.org/c/openstack/nova/+/820856 | |
| 03:11:03 | opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | |
| 03:11:03 | opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | |
| 03:11:04 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 03:11:04 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 03:11:05 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 03:11:05 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 03:11:06 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 03:11:06 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 03:11:07 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 03:11:08 | opendevreview | melanie witt proposed openstack/nova master: Enable unified limits in the nova-next job https://review.opendev.org/c/openstack/nova/+/789963 | |
| 03:54:03 | opendevreview | Merged openstack/nova stable/xena: Reproduce bug 1952941 https://review.opendev.org/c/openstack/nova/+/827868 | |
| 03:56:59 | opendevreview | Merged openstack/nova stable/xena: Migrate RequestSpec.numa_topology to use pcpuset https://review.opendev.org/c/openstack/nova/+/827869 | |
| 04:03:15 | opendevreview | Merged openstack/nova stable/wallaby: Add functional test for bug 1937375 https://review.opendev.org/c/openstack/nova/+/803717 | |
| 06:17:42 | opendevreview | Minghong Hou proposed openstack/nova master: fix VirtualInterface table can't be update https://review.opendev.org/c/openstack/nova/+/828819 | |
| 08:41:13 | nikparasyr | find the exact same cpu set available on the target host? We have enabled the PCIPassThrough filter for the scheduler but not the NUMATopologyFilter. If I understand well the NUMATopologyFilter will make sure that the scheduler picks a node that has the required topology available. Even so, if Nova requires the exact same cpu set to the target host we will still have an issue even with the numa filter... | |
| 08:41:13 | nikparasyr | hello, I have a question regarding shelving/unshelving. We have a flavor with `hw:cpu_policy='dedicated', hw:cpu_thread_policy='isolate'` and pci_passthrough 2 gpus. When we try to unshelve we get this error " Insufficient compute resources: Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology.". My question is to what extent does Nova require to | |
| 08:41:14 | nikparasyr | So, any idea to what extend does nova require the exact same cpu set when cpu pinning is enabled? | |
| 10:05:18 | kashyap | gibi: Thanks for the link to the smaller repro; also check out Peter's response on that thread | |
| 10:05:32 | kashyap | 1) the guest OS didn't confirm the detach | |
| 10:05:32 | kashyap | He points out two possibilities: | |
| 10:05:33 | kashyap | 2) there was a recent bug in qemu triggered by using JSON syntax for -device | |
| 10:10:16 | kashyap | gibi: That's it: this looks like it -- | |
| 10:10:17 | kashyap | "DEVICE_DELETED event is not delivered for device frontend if -device is configured via JSON" | |
| 10:10:21 | kashyap | https://bugzilla.redhat.com/show_bug.cgi?id=2036669 | |
| 10:19:45 | kashyap | But based on the versions in the CI job, they should already have the fix: | |
| 10:19:48 | kashyap | - libvirt version: 8.0.0, package: 2.el9 | |
| 10:19:51 | kashyap | - qemu-kvm-6.2.0-5.el9 | |
| 10:27:42 | kashyap | gibi: When you're around, to rule out the above bug, I wonder if we could try this workaround: | |
| 10:28:18 | kashyap | On compute nodes, in /etc/libvit/qemu.conf: | |
| 10:28:27 | kashyap | capability_filters = [ "device.json" ] | |
| 10:34:47 | gibi | kashyap: hi! | |
| 10:35:22 | gibi | kashyap: sure, I will try to make that config change via devstack | |
| 10:35:31 | gibi | kashyap: does it require a libvirtd restart? | |
| 10:35:34 | kashyap | gibi: See my latest comment: https://bugs.launchpad.net/nova/+bug/1960346/ | |
| 10:35:47 | kashyap | gibi: Yeah, it is required, afraid | |
| 10:36:09 | gibi | OK | |
| 10:36:10 | gibi | thanks | |
| 11:01:08 | gibi | kashyap: pushed new PS to https://review.opendev.org/c/openstack/devstack/+/828705 with the WA, lets see if it helps or not | |
| 11:01:29 | kashyap | gibi: Thank you! It will at least rule out the 2nd possibility above for sure. | |
| 11:01:46 | gibi | I can try to look at the first | |
| 11:01:55 | gibi | we can grab the console log | |
| 11:02:01 | gibi | after the failed detach | |
| 11:03:26 | gibi | hm, we already grabbing it in tempest | |
| 11:03:29 | gibi | let me find it | |
| 11:06:31 | kashyap | I see, I need to be AFK for an hour-ish; will come back and check | |
| 11:15:36 | gibi | added the consol log to the bug https://paste.opendev.org/show/bXXn63wbTPOwiCGC5xDI/ | |
| 11:15:56 | gibi | nothing obviously wrong there | |
| 11:16:10 | gibi | but the guest is still in a state to getting IP from DHCP | |
| 11:16:26 | gibi | so maybe it is not fully boot when the detach was requested | |
| 11:25:27 | gibi | chateaulav: left some suggestions inline about the ovo backports | |
| 11:35:46 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add properties to set advanced QXL video RAM settings https://review.opendev.org/c/openstack/nova/+/828674 | |
| 12:03:44 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add configuration options to set SPICE compression settings https://review.opendev.org/c/openstack/nova/+/828675 | |
| 12:24:58 | chateaulav | gibi: thanks for the follow up, that makes sense i was doing research and reading last night and had found references to `obj_relationships`. but your comments align to what i was trying yesterday, i just had brought in the exception aspect. appreciated | |
| 12:25:33 | gibi | chateaulav: cool | |
| 12:37:44 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add property to set number of screens per video adapter https://review.opendev.org/c/openstack/nova/+/828676 | |
| 12:40:45 | erlon | sean-k-mooney: hey Sean, I believe I have finished implementing all suggested changes in the live migration rollback fix: https://review.opendev.org/q/topic:bug%252F1944619 | |
| 12:41:15 | erlon | when you have a chance to give a look ill appreciate | |
| 13:22:56 | rosmaita | bauzas: fyi, i will be raising the minima in requirements for os-brick release: http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027192.html | |
| 14:16:21 | gibi | bauzas, rosmaita: I quickly checked the os_brick requirements patch I see no major bump in any deps so I think it is not a risky change. | |
| 14:17:08 | rosmaita | gibi: ty | |
| 14:17:37 | gibi | and tempest is green so nova is co-installable with the new os_brick deps | |
| 14:36:43 | gibi | gmann, frickler, bauzas: about the centos-9-steam job failure https://bugs.launchpad.net/nova/+bug/1960346/ I conculded that the cirros guest is not fully booted when the volume detach happens and the guest OS does not release the device. We need https://review.opendev.org/q/topic:wait_until_sshable_pingable to solve this in general | |
| 14:49:08 | kashyap | gibi: So cracked the prob! It's the guest OS indeed - adding a delay helps here? | |
| 14:49:15 | kashyap | s/So/So you/ | |
| 14:53:17 | kashyap | I think for now, going with the extra delay before the detach happens is fine. That saves more time here, before the big Tempest series gets merged | |
| 14:55:49 | gibi | kashyap: I don't have brains any more today but next week I can put up a tempest patch with some selective delays. I'm not sure how well QA will appreciate it | |
| 14:56:39 | gibi | also I can take a look at lyarwood's series and try to move that forward | |