| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-10 | |||
| 03:01:55 | opendevreview | sean mooney proposed openstack/nova master: [WIP] add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844 | |
| 03:06:45 | spatel | sean-k-mooney are you around? | |
| 05:37:45 | opendevreview | Ghanshyam proposed openstack/nova master: Make more project level APIs scoped to project only https://review.opendev.org/c/openstack/nova/+/828670 | |
| 07:41:58 | frickler | gibi: gmann: wouldn't https://bugs.launchpad.net/devstack/+bug/1960346 rather be a nova issue than devstack? | |
| 07:48:55 | __ministry | hi all, I have a question that is: "Can we plug multiple vGPU at the same time into a compute node?" | |
| 07:49:11 | __ministry | Please, help me. | |
| 07:53:39 | gibi | frickler: it more feels like a libvirt regression between 6.0.0 and 8.0.0 but sure it is probably not a devstack issue | |
| 07:55:53 | gibi | __ministry: you probably want to plug vGPUs to the guest VMs as compute nodes are physical things so they have physical gpus plugged | |
| 07:56:23 | gibi | __ministry: I think nova support multiple pGPUs per compute as well as multiple vGPUs per guest | |
| 07:57:34 | __ministry | yep. I want know to estimate amount of devices to make order. | |
| 07:58:23 | __ministry | thank you. | |
| 08:16:03 | frickler | gibi: how confident are you that it is actually a regression and not an intentional change that nova would have to adapt to? anyway I'll assign it to nova then to find out more details | |
| 08:16:39 | gibi | frickler: yeah, you are right, as there is major version bump, it can be an intended change too | |
| 08:19:35 | __ministry | gibi: multiple physical GPUs which vGPU supported in a compute node, we can? | |
| 08:20:05 | gibi | __ministry: I think so. bauzas do we have limitations around that ^^ ? | |
| 08:21:12 | bauzas | __ministry: gibi: which exact question do you want to know about vGPUs ? | |
| 08:21:47 | gibi | bauzas: as far as I understand __ministry asking in nova supports multiple pGPUs providing vGPUs in a single compute | |
| 08:21:49 | bauzas | most of the limitations are written here https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#caveats | |
| 08:22:01 | bauzas | gibi: oh this totally works | |
| 08:22:37 | bauzas | sometimes with some GPU board (like a Tesla T-something) you actually get more than 1 pGPU per board :) | |
| 08:23:02 | bauzas | since stein, we just create a resource provider per physical PCI device, that's it | |
| 08:23:07 | __ministry | I want plug multiple vGPU to a nova instance. | |
| 08:23:16 | bauzas | oh then the other way around | |
| 08:23:46 | bauzas | __ministry: so you're asking about resources:VGPU=X where X>1 | |
| 08:24:16 | bauzas | I'm unfortunate to say there is some limitation from libvirt due to some nasty bug | |
| 08:24:48 | __ministry | yep. it like we can attach multiple volume to nova instance. | |
| 08:24:50 | __ministry | ok | |
| 08:27:48 | bauzas | __ministry: https://bugs.launchpad.net/nova/+bug/1758086 | |
| 08:33:23 | __ministry | bauzas: thank you. ^.^ | |
| 08:35:16 | bauzas | __ministry: that being said, this is a nvidia driver limitation, you're welcome to test again another newer GRID release version | |
| 08:52:07 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add properties to set advanced QXL video RAM settings https://review.opendev.org/c/openstack/nova/+/828674 | |
| 08:59:17 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add configuration options to set SPICE compression settings https://review.opendev.org/c/openstack/nova/+/828675 | |
| 09:09:56 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add property to set number of screens per video adapter https://review.opendev.org/c/openstack/nova/+/828676 | |
| 10:09:30 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add any-traits support for listing resource providers https://review.opendev.org/c/openstack/placement/+/826491 | |
| 10:09:57 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add any-traits support for allocation candidates https://review.opendev.org/c/openstack/placement/+/826492 | |
| 10:09:58 | opendevreview | Balazs Gibizer proposed openstack/placement master: Remove unused compatibility code https://review.opendev.org/c/openstack/placement/+/826493 | |
| 10:10:08 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add microversion 1.39 to support any-trait queries https://review.opendev.org/c/openstack/placement/+/826719 | |
| 10:11:38 | gibi | gmann: thanks for the comment in the any-traits series. I restored the legacy behavior in https://review.opendev.org/c/openstack/placement/+/826491/8 | |
| 10:11:47 | gibi | melwitt: I had to respin the any-traits series due to ^^ | |
| 10:27:45 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add configuration options to set SPICE compression settings https://review.opendev.org/c/openstack/nova/+/828675 | |
| 10:58:02 | sean-k-mooney | chateaulav: you have 1 local branch for all commits in a feature | |
| 10:58:28 | sean-k-mooney | chateaulav: you should be able to check out that branch locally and run all the code form the top of the branch | |
| 10:58:52 | sean-k-mooney | e.g. run the unit/func tests and or deploy nova and execute the code | |
| 11:02:03 | sean-k-mooney | frickler: im pretty sure any change in the detach behavior is a libvirt/qemu regression likely related to the rework they are doing in this arrea for pci passthtough | |
| 11:02:24 | sean-k-mooney | regarding https://bugs.launchpad.net/nova/+bug/1960346 | |
| 11:05:00 | bauzas | sean-k-mooney: yup we'd appreciate some qemu expert on this one | |
| 11:05:07 | bauzas | kashyap: maybe you can help ? | |
| 11:05:22 | kashyap | bauzas: Hey, reading back | |
| 11:05:45 | bauzas | kashyap: sean-k-mooney: context is some TripleO CI blocked https://bugs.launchpad.net/tripleo/+bug/1960310 | |
| 11:05:55 | kashyap | sean-k-mooney: I wouldn't be so confident about regressions in libvirt/QEMU without evidence. | |
| 11:06:05 | bauzas | which looks to be due b/c of https://bugs.launchpad.net/nova/+bug/1960346 | |
| 11:06:07 | kashyap | When it comes to bugs, I follow "seeing is believing" | |
| 11:06:25 | bauzas | kashyap: if you see the last bug, you'll see gibi saying we use a new libvirt/qemu version | |
| 11:06:41 | bauzas | but it seems to be a regression | |
| 11:06:57 | kashyap | bauzas: Regression where? Me looks | |
| 11:07:24 | frickler | oh, it's not only devstack, but also tripleo, then devstack is even more out of the boat ;) | |
| 11:08:55 | bauzas | kashyap: see https://zuul.openstack.org/build/3e24d977991d4536b6279afd7f3b5d56/log/controller/logs/screen-n-cpu.txt?severity=4#49433 | |
| 11:09:12 | kashyap | bauzas: Yep, already noticed it | |
| 11:09:27 | kashyap | bauzas: Looking for libvirtd logs w/ QEMU filters | |
| 11:09:46 | kashyap | bauzas: Have you got the affected instance ID from the logs | |
| 11:09:58 | bauzas | lemme try to find one | |
| 11:11:47 | kashyap | 1074c6fa-12fe-40a8-b1d5-a47a49018d9f | |
| 11:11:48 | kashyap | ? | |
| 11:12:02 | bauzas | for this job, yes | |
| 11:15:18 | chateaulav | sean-k-mooney: alright then im on the right track, got that all setup and inline. | |
| 11:29:08 | chateaulav | sean-k-mooney: and then with any further changes, i would use interactive rebase to edit and add any files to each specific commit and the submit the topic for review. | |
| 11:29:17 | kashyap | bauzas: Do you know why I'm unable to fetch all the instance log files with a simple `wget`? | |
| 11:29:23 | kashyap | I'm tryin this: | |
| 11:29:40 | kashyap | $> wget -r -nH -nd -np -R "index.html*" https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e2/828280/1/check/devstack-platform-centos-9-stream/3e24d97/controller/logs/libvirt/libvirt/qemu/ | |
| 11:42:59 | kashyap | Anyway, it's not required; ignore the above | |
| 11:56:33 | kashyap | Duh, I wish Launchpad didn't break the complete formatting in text messages by wrapping text | |
| 12:04:17 | sean-k-mooney | chateaulav: yes. and you can always author new commits at the end of the chain and move them with an interactive rebase if need | |
| 12:05:14 | chateaulav | thanks for that last confirmation! your mentorship is much appreciated. | |
| 12:05:30 | sean-k-mooney | but basically gerrit is intended to work with freature branches and it tack each chage to a review with the change-id in the commit message | |
| 12:05:41 | sean-k-mooney | chateaulav: no worries glad to help | |
| 12:06:46 | sean-k-mooney | so rebases or change to a commit will not create a new review if the change id does not change and it will just update teh exsit review with a new revsion | |
| 12:15:14 | gibi | kashyap: so what do you think about "Device virtio-disk1 is already in the process of unplug" error? Should we increase the amount of time we wait before we retry the detach? | |
| 12:17:07 | gibi | it is configurable with CONF.libvirt.device_detach_timeout | |
| 12:17:40 | gibi | it is 20 sec by default | |
| 12:22:48 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Adds regression test for bug LP#1944619 https://review.opendev.org/c/openstack/nova/+/821840 | |
| 12:22:48 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| 12:23:36 | bauzas | kashyap: sorry, was at lunch (isolated but in the kitchen tho) | |
| 12:24:35 | gibi | kashyap: I've pushed https://review.opendev.org/c/openstack/devstack/+/828705 to see if longer timeout helps or not | |
| 12:29:18 | sean-k-mooney | gibi: i tought you implmented an event based retry | |
| 12:29:33 | gibi | sean-k-mooney: it is event based but with a timeout | |
| 12:29:46 | gibi | so if the event came then we stop waiting | |
| 12:29:48 | sean-k-mooney | and when it times out we give up | |
| 12:29:52 | gibi | but if the event never cames we give up | |
| 12:30:00 | sean-k-mooney | ya ok | |
| 12:30:02 | gibi | more preciesly we retry | |
| 12:30:07 | sean-k-mooney | well no | |
| 12:30:13 | sean-k-mooney | retrying would be wrong | |
| 12:30:29 | sean-k-mooney | since we know that is an error in qemu and will abort the detach | |
| 12:30:48 | sean-k-mooney | at least in current verions in old version it was undefiend behavior | |
| 12:30:55 | gibi | I remember that even with libvirt 6.0.0 retry was needed in some cases | |
| 12:31:09 | kashyap | bauzas: Don't worry | |
| 12:31:12 | gibi | but maybe that was just the case of not waiting enough | |
| 12:31:14 | sean-k-mooney | its qemu rather then libvirt that i think is important here | |
| 12:31:19 | kashyap | gibi: Reading back; went for some air | |
| 12:31:51 | gibi | sean-k-mooney: ack, then qemu 4.2.0 vs 6.2.0 | |