| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-11 | |||
| 03:11:05 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 03:11:06 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 03:11:06 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 03:11:07 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 03:11:08 | opendevreview | melanie witt proposed openstack/nova master: Enable unified limits in the nova-next job https://review.opendev.org/c/openstack/nova/+/789963 | |
| 03:54:03 | opendevreview | Merged openstack/nova stable/xena: Reproduce bug 1952941 https://review.opendev.org/c/openstack/nova/+/827868 | |
| 03:56:59 | opendevreview | Merged openstack/nova stable/xena: Migrate RequestSpec.numa_topology to use pcpuset https://review.opendev.org/c/openstack/nova/+/827869 | |
| 04:03:15 | opendevreview | Merged openstack/nova stable/wallaby: Add functional test for bug 1937375 https://review.opendev.org/c/openstack/nova/+/803717 | |
| 06:17:42 | opendevreview | Minghong Hou proposed openstack/nova master: fix VirtualInterface table can't be update https://review.opendev.org/c/openstack/nova/+/828819 | |
| 08:41:13 | nikparasyr | find the exact same cpu set available on the target host? We have enabled the PCIPassThrough filter for the scheduler but not the NUMATopologyFilter. If I understand well the NUMATopologyFilter will make sure that the scheduler picks a node that has the required topology available. Even so, if Nova requires the exact same cpu set to the target host we will still have an issue even with the numa filter... | |
| 08:41:13 | nikparasyr | hello, I have a question regarding shelving/unshelving. We have a flavor with `hw:cpu_policy='dedicated', hw:cpu_thread_policy='isolate'` and pci_passthrough 2 gpus. When we try to unshelve we get this error " Insufficient compute resources: Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology.". My question is to what extent does Nova require to | |
| 08:41:14 | nikparasyr | So, any idea to what extend does nova require the exact same cpu set when cpu pinning is enabled? | |
| 10:05:18 | kashyap | gibi: Thanks for the link to the smaller repro; also check out Peter's response on that thread | |
| 10:05:32 | kashyap | 1) the guest OS didn't confirm the detach | |
| 10:05:32 | kashyap | He points out two possibilities: | |
| 10:05:33 | kashyap | 2) there was a recent bug in qemu triggered by using JSON syntax for -device | |
| 10:10:16 | kashyap | gibi: That's it: this looks like it -- | |
| 10:10:17 | kashyap | "DEVICE_DELETED event is not delivered for device frontend if -device is configured via JSON" | |
| 10:10:21 | kashyap | https://bugzilla.redhat.com/show_bug.cgi?id=2036669 | |
| 10:19:45 | kashyap | But based on the versions in the CI job, they should already have the fix: | |
| 10:19:48 | kashyap | - libvirt version: 8.0.0, package: 2.el9 | |
| 10:19:51 | kashyap | - qemu-kvm-6.2.0-5.el9 | |
| 10:27:42 | kashyap | gibi: When you're around, to rule out the above bug, I wonder if we could try this workaround: | |
| 10:28:18 | kashyap | On compute nodes, in /etc/libvit/qemu.conf: | |
| 10:28:27 | kashyap | capability_filters = [ "device.json" ] | |
| 10:34:47 | gibi | kashyap: hi! | |
| 10:35:22 | gibi | kashyap: sure, I will try to make that config change via devstack | |
| 10:35:31 | gibi | kashyap: does it require a libvirtd restart? | |
| 10:35:34 | kashyap | gibi: See my latest comment: https://bugs.launchpad.net/nova/+bug/1960346/ | |
| 10:35:47 | kashyap | gibi: Yeah, it is required, afraid | |
| 10:36:09 | gibi | OK | |
| 10:36:10 | gibi | thanks | |
| 11:01:08 | gibi | kashyap: pushed new PS to https://review.opendev.org/c/openstack/devstack/+/828705 with the WA, lets see if it helps or not | |
| 11:01:29 | kashyap | gibi: Thank you! It will at least rule out the 2nd possibility above for sure. | |
| 11:01:46 | gibi | I can try to look at the first | |
| 11:01:55 | gibi | we can grab the console log | |
| 11:02:01 | gibi | after the failed detach | |
| 11:03:26 | gibi | hm, we already grabbing it in tempest | |
| 11:03:29 | gibi | let me find it | |
| 11:06:31 | kashyap | I see, I need to be AFK for an hour-ish; will come back and check | |
| 11:15:36 | gibi | added the consol log to the bug https://paste.opendev.org/show/bXXn63wbTPOwiCGC5xDI/ | |
| 11:15:56 | gibi | nothing obviously wrong there | |
| 11:16:10 | gibi | but the guest is still in a state to getting IP from DHCP | |
| 11:16:26 | gibi | so maybe it is not fully boot when the detach was requested | |
| 11:25:27 | gibi | chateaulav: left some suggestions inline about the ovo backports | |
| 11:35:46 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add properties to set advanced QXL video RAM settings https://review.opendev.org/c/openstack/nova/+/828674 | |
| 12:03:44 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add configuration options to set SPICE compression settings https://review.opendev.org/c/openstack/nova/+/828675 | |
| 12:24:58 | chateaulav | gibi: thanks for the follow up, that makes sense i was doing research and reading last night and had found references to `obj_relationships`. but your comments align to what i was trying yesterday, i just had brought in the exception aspect. appreciated | |
| 12:25:33 | gibi | chateaulav: cool | |
| 12:37:44 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add property to set number of screens per video adapter https://review.opendev.org/c/openstack/nova/+/828676 | |
| 12:40:45 | erlon | sean-k-mooney: hey Sean, I believe I have finished implementing all suggested changes in the live migration rollback fix: https://review.opendev.org/q/topic:bug%252F1944619 | |
| 12:41:15 | erlon | when you have a chance to give a look ill appreciate | |
| 13:22:56 | rosmaita | bauzas: fyi, i will be raising the minima in requirements for os-brick release: http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027192.html | |
| 14:16:21 | gibi | bauzas, rosmaita: I quickly checked the os_brick requirements patch I see no major bump in any deps so I think it is not a risky change. | |
| 14:17:08 | rosmaita | gibi: ty | |
| 14:17:37 | gibi | and tempest is green so nova is co-installable with the new os_brick deps | |
| 14:36:43 | gibi | gmann, frickler, bauzas: about the centos-9-steam job failure https://bugs.launchpad.net/nova/+bug/1960346/ I conculded that the cirros guest is not fully booted when the volume detach happens and the guest OS does not release the device. We need https://review.opendev.org/q/topic:wait_until_sshable_pingable to solve this in general | |
| 14:49:08 | kashyap | gibi: So cracked the prob! It's the guest OS indeed - adding a delay helps here? | |
| 14:49:15 | kashyap | s/So/So you/ | |
| 14:53:17 | kashyap | I think for now, going with the extra delay before the detach happens is fine. That saves more time here, before the big Tempest series gets merged | |
| 14:55:49 | gibi | kashyap: I don't have brains any more today but next week I can put up a tempest patch with some selective delays. I'm not sure how well QA will appreciate it | |
| 14:56:39 | gibi | also I can take a look at lyarwood's series and try to move that forward | |
| 14:56:46 | gibi | kashyap: thanks you for your help! | |
| 15:04:40 | frickler | gibi: thx for the update, do you know why this only occurs on c9s? is booting slower or did previous libvirts not care whether that release actually happens? | |
| 15:19:27 | gibi | frickler: I think older libvirt let nova to restart the detach process but newer libvirt simply rejectes the retry as the original detach is still ongoing | |
| 15:22:29 | opendevreview | Merged openstack/nova stable/victoria: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/818559 | |
| 15:39:10 | sean-k-mooney | gibi: older qemu did not support restart the detach but did not raise an error then qemu started enforcing it | |
| 15:40:15 | gibi | sean-k-mooney: yeah, sorry, s/libvirt/qemu. | |
| 15:45:10 | elodilles | melwitt: whenever you have time, could you review this patch? https://review.opendev.org/c/openstack/nova/+/805628 | |
| 15:45:33 | frickler | sean-k-mooney: gibi: but then it sounds to me that the real fix would still be to make nova not retry the detach, just wait longer? | |
| 15:45:47 | elodilles | melwitt: i think it would reduce the number of rechecks in wallaby and victoria if it merges (and its devstack part) | |
| 15:46:56 | gibi | frickler: if the detach happens while the cirros is booting then the guest OS never releases the device | |
| 15:47:06 | gibi | so right now waiting more is not an option | |
| 15:47:15 | gibi | but in general I agree to remove the retry loop from nova | |
| 15:47:28 | gibi | as it is pointless after qemu starts rejecting the retry | |
| 15:47:49 | sean-k-mooney | really the jobs should wait for the instance to be pingable/sshable | |
| 15:47:53 | sean-k-mooney | and only detach then | |
| 15:48:11 | sean-k-mooney | and nova shoudl not retry if the detach fails and just have the client retry | |
| 15:48:35 | sean-k-mooney | clieht beign tempest or enduser if a retry is needed | |
| 15:50:04 | gibi | sean-k-mooney: yepp | |
| 15:50:40 | gibi | on the other hand if ever a detach is issue by the client right after a boot then that detach will time out in nova, but I'm not sure it ever time outs in qemu | |
| 15:51:15 | gibi | so in that case a client retry will not help either | |
| 15:51:22 | sean-k-mooney | we might be abel to expicitly cancel the job in qemu | |
| 15:51:24 | gibi | as qemu will say that the detach is in progress | |
| 15:51:26 | sean-k-mooney | when we time out | |
| 15:51:57 | opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 15:51:57 | opendevreview | Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369 | |
| 15:51:58 | opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | |
| 15:54:11 | gibi | sean-k-mooney: at least the doc did not mention a way to cancel via libvirt https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainDetachDeviceFlags | |
| 15:55:47 | sean-k-mooney | we could always issue a detach again :) since that will do an abort in qemu | |
| 15:56:22 | sean-k-mooney | but ya we shoudl aske the libirt folks although im on PTO today so im going to drop off irc again soon | |
| 15:56:47 | sean-k-mooney | so kashyap maybe you could folow up and see if there is a way to abort the detach or pass a timeout to qemu via libfirt | |
| 15:57:58 | gibi | sean-k-mooney: if we attach the detach again qemu reject it | |
| 15:58:04 | gibi | sean-k-mooney: if we issue the detach again qemu reject it | |
| 15:58:29 | gibi | as the pervious one is still ongoing | |
| 15:59:08 | sean-k-mooney | yep | |
| 15:59:38 | sean-k-mooney | but we could catch the error | |
| 15:59:58 | gibi | yepp, but that does not make the device actually detached :D | |
| 16:00:01 | sean-k-mooney | if it say devcice not found well presumabel it finsihed before we sent the detach after the time out | |
| 16:00:27 | gibi | sean-k-mooney: it say detach is ongoing | |