| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-10 | |||
| 16:29:43 | opendevreview | Andre Aranha proposed openstack/nova stable/wallaby: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827896 | |
| 17:02:31 | opendevreview | Lior Friedman proposed openstack/nova master: Support use_multipath for NVME driver https://review.opendev.org/c/openstack/nova/+/823941 | |
| 17:05:23 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Document remote-managed port usage considerations https://review.opendev.org/c/openstack/nova/+/827513 | |
| 17:12:00 | opendevreview | Lior Friedman proposed openstack/nova master: Support use_multipath for NVME driver https://review.opendev.org/c/openstack/nova/+/823941 | |
| 17:40:04 | gibi | kashyap: fyi there is a smaller reproduction in https://bugs.launchpad.net/nova/+bug/1960346/comments/10 | |
| 17:40:14 | gibi | but I have to drop off now | |
| 18:01:05 | opendevreview | melanie witt proposed openstack/placement master: Make perfload jobs fail if write allocation fails https://review.opendev.org/c/openstack/placement/+/828438 | |
| 18:32:18 | opendevreview | Ghanshyam proposed openstack/nova master: Make more project level APIs scoped to project only https://review.opendev.org/c/openstack/nova/+/828670 | |
| 18:54:17 | chateaulav | gibi: can i get a little more on the backporting of the 1.3 to 1.2. I have been playing around with it but am not quite sure. is this more related to the actual version itself or pulling the new values available in 1.3 to 1.2. this is for https://review.opendev.org/c/openstack/nova/+/828369 and i know that my question seems repeatative | |
| 19:43:55 | opendevreview | Merged openstack/nova master: Join quota exception family trees https://review.opendev.org/c/openstack/nova/+/828185 | |
| 20:27:08 | spatel | kashyap by the way i found issue, it was related to neutron-metadata service which was causing issue and holding VM build.. | |
| 21:35:48 | opendevreview | melanie witt proposed openstack/nova stable/wallaby: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825178 | |
| 23:03:26 | chateaulav | gibi: found the info I needed. Will add the back ports tomorrow. | |
| 23:21:29 | opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | |
| 23:21:51 | opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | |
| #openstack-nova - 2022-02-11 | |||
| 02:05:58 | opendevreview | Merged openstack/nova stable/victoria: Reproduce bug 1953359 https://review.opendev.org/c/openstack/nova/+/820558 | |
| 02:06:05 | opendevreview | Merged openstack/nova stable/victoria: Extend the reproducer for 1953359 and 1952915 https://review.opendev.org/c/openstack/nova/+/820856 | |
| 03:11:03 | opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | |
| 03:11:03 | opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | |
| 03:11:04 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 03:11:04 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 03:11:05 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 03:11:05 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 03:11:06 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 03:11:06 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 03:11:07 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 03:11:08 | opendevreview | melanie witt proposed openstack/nova master: Enable unified limits in the nova-next job https://review.opendev.org/c/openstack/nova/+/789963 | |
| 03:54:03 | opendevreview | Merged openstack/nova stable/xena: Reproduce bug 1952941 https://review.opendev.org/c/openstack/nova/+/827868 | |
| 03:56:59 | opendevreview | Merged openstack/nova stable/xena: Migrate RequestSpec.numa_topology to use pcpuset https://review.opendev.org/c/openstack/nova/+/827869 | |
| 04:03:15 | opendevreview | Merged openstack/nova stable/wallaby: Add functional test for bug 1937375 https://review.opendev.org/c/openstack/nova/+/803717 | |
| 06:17:42 | opendevreview | Minghong Hou proposed openstack/nova master: fix VirtualInterface table can't be update https://review.opendev.org/c/openstack/nova/+/828819 | |
| 08:41:13 | nikparasyr | find the exact same cpu set available on the target host? We have enabled the PCIPassThrough filter for the scheduler but not the NUMATopologyFilter. If I understand well the NUMATopologyFilter will make sure that the scheduler picks a node that has the required topology available. Even so, if Nova requires the exact same cpu set to the target host we will still have an issue even with the numa filter... | |
| 08:41:13 | nikparasyr | hello, I have a question regarding shelving/unshelving. We have a flavor with `hw:cpu_policy='dedicated', hw:cpu_thread_policy='isolate'` and pci_passthrough 2 gpus. When we try to unshelve we get this error " Insufficient compute resources: Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology.". My question is to what extent does Nova require to | |
| 08:41:14 | nikparasyr | So, any idea to what extend does nova require the exact same cpu set when cpu pinning is enabled? | |
| 10:05:18 | kashyap | gibi: Thanks for the link to the smaller repro; also check out Peter's response on that thread | |
| 10:05:32 | kashyap | 1) the guest OS didn't confirm the detach | |
| 10:05:32 | kashyap | He points out two possibilities: | |
| 10:05:33 | kashyap | 2) there was a recent bug in qemu triggered by using JSON syntax for -device | |
| 10:10:16 | kashyap | gibi: That's it: this looks like it -- | |
| 10:10:17 | kashyap | "DEVICE_DELETED event is not delivered for device frontend if -device is configured via JSON" | |
| 10:10:21 | kashyap | https://bugzilla.redhat.com/show_bug.cgi?id=2036669 | |
| 10:19:45 | kashyap | But based on the versions in the CI job, they should already have the fix: | |
| 10:19:48 | kashyap | - libvirt version: 8.0.0, package: 2.el9 | |
| 10:19:51 | kashyap | - qemu-kvm-6.2.0-5.el9 | |
| 10:27:42 | kashyap | gibi: When you're around, to rule out the above bug, I wonder if we could try this workaround: | |
| 10:28:18 | kashyap | On compute nodes, in /etc/libvit/qemu.conf: | |
| 10:28:27 | kashyap | capability_filters = [ "device.json" ] | |
| 10:34:47 | gibi | kashyap: hi! | |
| 10:35:22 | gibi | kashyap: sure, I will try to make that config change via devstack | |
| 10:35:31 | gibi | kashyap: does it require a libvirtd restart? | |
| 10:35:34 | kashyap | gibi: See my latest comment: https://bugs.launchpad.net/nova/+bug/1960346/ | |
| 10:35:47 | kashyap | gibi: Yeah, it is required, afraid | |
| 10:36:09 | gibi | OK | |
| 10:36:10 | gibi | thanks | |
| 11:01:08 | gibi | kashyap: pushed new PS to https://review.opendev.org/c/openstack/devstack/+/828705 with the WA, lets see if it helps or not | |
| 11:01:29 | kashyap | gibi: Thank you! It will at least rule out the 2nd possibility above for sure. | |
| 11:01:46 | gibi | I can try to look at the first | |
| 11:01:55 | gibi | we can grab the console log | |
| 11:02:01 | gibi | after the failed detach | |
| 11:03:26 | gibi | hm, we already grabbing it in tempest | |
| 11:03:29 | gibi | let me find it | |
| 11:06:31 | kashyap | I see, I need to be AFK for an hour-ish; will come back and check | |
| 11:15:36 | gibi | added the consol log to the bug https://paste.opendev.org/show/bXXn63wbTPOwiCGC5xDI/ | |
| 11:15:56 | gibi | nothing obviously wrong there | |
| 11:16:10 | gibi | but the guest is still in a state to getting IP from DHCP | |
| 11:16:26 | gibi | so maybe it is not fully boot when the detach was requested | |
| 11:25:27 | gibi | chateaulav: left some suggestions inline about the ovo backports | |
| 11:35:46 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add properties to set advanced QXL video RAM settings https://review.opendev.org/c/openstack/nova/+/828674 | |
| 12:03:44 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add configuration options to set SPICE compression settings https://review.opendev.org/c/openstack/nova/+/828675 | |
| 12:24:58 | chateaulav | gibi: thanks for the follow up, that makes sense i was doing research and reading last night and had found references to `obj_relationships`. but your comments align to what i was trying yesterday, i just had brought in the exception aspect. appreciated | |
| 12:25:33 | gibi | chateaulav: cool | |
| 12:37:44 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add property to set number of screens per video adapter https://review.opendev.org/c/openstack/nova/+/828676 | |
| 12:40:45 | erlon | sean-k-mooney: hey Sean, I believe I have finished implementing all suggested changes in the live migration rollback fix: https://review.opendev.org/q/topic:bug%252F1944619 | |
| 12:41:15 | erlon | when you have a chance to give a look ill appreciate | |
| 13:22:56 | rosmaita | bauzas: fyi, i will be raising the minima in requirements for os-brick release: http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027192.html | |
| 14:16:21 | gibi | bauzas, rosmaita: I quickly checked the os_brick requirements patch I see no major bump in any deps so I think it is not a risky change. | |
| 14:17:08 | rosmaita | gibi: ty | |
| 14:17:37 | gibi | and tempest is green so nova is co-installable with the new os_brick deps | |
| 14:36:43 | gibi | gmann, frickler, bauzas: about the centos-9-steam job failure https://bugs.launchpad.net/nova/+bug/1960346/ I conculded that the cirros guest is not fully booted when the volume detach happens and the guest OS does not release the device. We need https://review.opendev.org/q/topic:wait_until_sshable_pingable to solve this in general | |
| 14:49:08 | kashyap | gibi: So cracked the prob! It's the guest OS indeed - adding a delay helps here? | |
| 14:49:15 | kashyap | s/So/So you/ | |
| 14:53:17 | kashyap | I think for now, going with the extra delay before the detach happens is fine. That saves more time here, before the big Tempest series gets merged | |
| 14:55:49 | gibi | kashyap: I don't have brains any more today but next week I can put up a tempest patch with some selective delays. I'm not sure how well QA will appreciate it | |
| 14:56:39 | gibi | also I can take a look at lyarwood's series and try to move that forward | |
| 14:56:46 | gibi | kashyap: thanks you for your help! | |
| 15:04:40 | frickler | gibi: thx for the update, do you know why this only occurs on c9s? is booting slower or did previous libvirts not care whether that release actually happens? | |
| 15:19:27 | gibi | frickler: I think older libvirt let nova to restart the detach process but newer libvirt simply rejectes the retry as the original detach is still ongoing | |
| 15:22:29 | opendevreview | Merged openstack/nova stable/victoria: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/818559 | |
| 15:39:10 | sean-k-mooney | gibi: older qemu did not support restart the detach but did not raise an error then qemu started enforcing it | |
| 15:40:15 | gibi | sean-k-mooney: yeah, sorry, s/libvirt/qemu. | |
| 15:45:10 | elodilles | melwitt: whenever you have time, could you review this patch? https://review.opendev.org/c/openstack/nova/+/805628 | |
| 15:45:33 | frickler | sean-k-mooney: gibi: but then it sounds to me that the real fix would still be to make nova not retry the detach, just wait longer? | |
| 15:45:47 | elodilles | melwitt: i think it would reduce the number of rechecks in wallaby and victoria if it merges (and its devstack part) | |
| 15:46:56 | gibi | frickler: if the detach happens while the cirros is booting then the guest OS never releases the device | |
| 15:47:06 | gibi | so right now waiting more is not an option | |
| 15:47:15 | gibi | but in general I agree to remove the retry loop from nova | |
| 15:47:28 | gibi | as it is pointless after qemu starts rejecting the retry | |
| 15:47:49 | sean-k-mooney | really the jobs should wait for the instance to be pingable/sshable | |
| 15:47:53 | sean-k-mooney | and only detach then | |
| 15:48:11 | sean-k-mooney | and nova shoudl not retry if the detach fails and just have the client retry | |