| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-12 | |||
| 17:30:42 | colby_ | mdevctl list output: 67e63f1e-07f2-474f-874c-826a024c10ec 0000:21:01.7 nvidia-563 manual | |
| 17:30:42 | colby_ | 3a974d13-5dea-4bfc-b034-533f6e754349 0000:21:03.4 nvidia-563 manual | |
| 17:30:42 | colby_ | d3586a0a-2e56-421e-923f-20797fe74ab5 0000:21:03.7 nvidia-563 manual | |
| 17:30:42 | colby_ | 150c155c-da0b-45a6-8bc1-a8016231b100 0000:21:04.1 nvidia-563 manual | |
| 17:31:36 | colby_ | but spinning up a new instance tried to use the resource provider _pci_0000_21_02_4 (and pci 21 is already full) | |
| 17:32:51 | colby_ | how does nova detect the already created devices and use those? Should it be using the resource provider of those already created mdev (eg _pci_0000_21_01_7) | |
| 17:34:17 | colby_ | We are on Victoria release, Centos 8 Stream, Nvidia A40 GPU | |
| 17:54:13 | opendevreview | Merged openstack/nova master: zuul: Put Centos9 Stream job periodic-weekly and experimental https://review.opendev.org/c/openstack/nova/+/849463 | |
| 18:36:34 | sean-k-mooney | colby_: there is definetly a bug with this we are hitting it downstream too and still investigating | |
| 18:36:50 | sean-k-mooney | colby_: i can see if i can get you the link to where we try to reuse the mdev | |
| 18:37:58 | sean-k-mooney | colby_: https://github.com/openstack/nova/blob/de65131f92ba5ba812e33e6ff63be0991687413a/nova/virt/libvirt/driver.py#L8261-L8278= | |
| 18:40:17 | colby_ | sean-k-mooney: oh good glad to know this is not just us. Is there a bug filed yet that I could follow? | |
| 18:40:55 | sean-k-mooney | downstream definitly ill grab it and see if we have an upstram one. we were still trying to root cause it | |
| 18:41:27 | sean-k-mooney | colby_: basically we were QEing cold migration and noticed that the devices were not being reused | |
| 18:41:42 | sean-k-mooney | so depening on the order the test ran it either worked or failed | |
| 18:42:17 | sean-k-mooney | so we are looking at it as part of https://bugzilla.redhat.com/show_bug.cgi?id=1701281 | |
| 18:42:36 | sean-k-mooney | but i think we are going to break this out as a seperate upstream and downstream bug | |
| 18:43:03 | colby_ | would it be the same root cause as we are seeing just deleting and trying to create new instances? | |
| 18:43:41 | sean-k-mooney | a host reboot or deleteing the unused mdevs is the work around we are usign right now | |
| 18:44:22 | sean-k-mooney | so if you loop over the domain xmls and delete any mdev not used by an xml that "fixes it" temporally | |
| 18:44:24 | colby_ | yea thats what I ended up having to do manually is remove the mdevs that got created then new instances could be spun up | |
| 18:44:28 | sean-k-mooney | but thats not the correct fix | |
| 18:45:46 | colby_ | I suppose I could create a cron job to do that so we can remove the manual part | |
| 18:45:55 | sean-k-mooney | colby_: bauzas is on on pto tomorrow and friday but i think they are here tursday | |
| 18:46:11 | sean-k-mooney | they tought it might be related to who we do the mdev lookup | |
| 18:46:30 | sean-k-mooney | but since you have the issue | |
| 18:46:52 | sean-k-mooney | could you compare the list of mdevs returned by mdevctl and libvirt via virsh | |
| 18:47:11 | sean-k-mooney | libvirt does some caching so one of the guesses we had is it might be getting out of sync | |
| 18:47:48 | sean-k-mooney | actully hum | |
| 18:49:12 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/832489/1/nova/virt/libvirt/utils.py | |
| 18:49:17 | sean-k-mooney | i wonder if its this ^ | |
| 18:49:27 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1951656 | |
| 18:50:00 | sean-k-mooney | colby_: do you know what version fo libvirt you are using | |
| 18:50:20 | colby_ | 7.9.0-1 | |
| 18:50:48 | colby_ | you just want the output form `virsh nodedev-list` ? | |
| 18:50:53 | sean-k-mooney | that was in 7.7 https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d | |
| 18:51:10 | sean-k-mooney | colby_: yes please nodedev-list shoudl have the names | |
| 18:51:15 | sean-k-mooney | and we can check the format | |
| 18:51:21 | sean-k-mooney | to see if it has the parent info or not | |
| 18:54:17 | colby_ | https://pastebin.com/94YfBRH9 | |
| 18:55:00 | sean-k-mooney | there is also https://review.opendev.org/c/openstack/nova/+/838976 as another possible fix | |
| 18:55:39 | sean-k-mooney | odd i dont see any mdevs there | |
| 18:56:03 | colby_ | ha woops sorry | |
| 18:56:05 | colby_ | wrong machine | |
| 18:56:36 | colby_ | https://pastebin.com/NAfjxUt7 | |
| 18:57:18 | colby_ | mdevctl list output: https://pastebin.com/rXmUftzj | |
| 18:57:52 | sean-k-mooney | yep so virsh has the extended names | |
| 18:58:27 | sean-k-mooney | so its not a caching issue but it proably is a parsing issue | |
| 18:58:34 | sean-k-mooney | since both consitent | |
| 18:59:05 | sean-k-mooney | """Note that the lookup of the mdev device by UUID are needed in order | |
| 18:59:08 | sean-k-mooney | to keep the ability to recreate assigned mediated devices on a reboot of | |
| 18:59:10 | sean-k-mooney | the compute node | |
| 18:59:12 | sean-k-mooney | """ | |
| 18:59:27 | sean-k-mooney | but i bet its also needed to be able to reuse the mdevs at all | |
| 19:01:42 | sean-k-mooney | colby_: im not 100% sure this will fix it but i have set https://review.opendev.org/c/openstack/nova/+/838976 as a review priority and ill follow up with sylvain when they are back | |
| 19:03:31 | colby_ | ok sounds good. Im happy to test out the patches on our system if you want | |
| 19:03:42 | colby_ | no one is using this hypervisor right now but the admins | |
| 19:04:04 | sean-k-mooney | if you wanted to test https://review.opendev.org/c/openstack/nova/+/838976 and provide feedback on the review that is the more compelte fix | |
| 19:04:49 | colby_ | sure. Ill get those in place today and let you know if it helps our case at all | |
| 19:04:58 | sean-k-mooney | most of the opencomment are about updating the doc strings but the patch should work as is | |
| 19:05:15 | sean-k-mooney | we might also add a functional repoducer if we can recaret the mdev resue issue | |
| 19:05:29 | sean-k-mooney | colby_: thanks | |
| #openstack-nova - 2022-07-13 | |||
| 04:19:31 | opendevreview | Kashyap Chamarthy proposed openstack/nova stable/wallaby: [nova/libvirt] Support for checking and enabling SMM when needed https://review.opendev.org/c/openstack/nova/+/849610 | |
| 07:43:06 | gibi | good morning | |
| 09:26:53 | opendevreview | Maksim Malchuk proposed openstack/nova stable/xena: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/829804 | |
| 13:28:03 | opendevreview | Kashyap Chamarthy proposed openstack/nova stable/xena: [nova/libvirt] Support for checking and enabling SMM when needed https://review.opendev.org/c/openstack/nova/+/849676 | |
| 13:35:30 | kashyap | sean-k-mooney: gibi: Do we always have to backport in order? Is there anything "off" if we backport from master to wallaby, instead of xena to wallaby, when it's a clean pick? | |
| 13:36:01 | stephenfin | kashyap: Always branch by branch | |
| 13:36:35 | stephenfin | It just ensures that people don't skip things. You will invariably have conflicts for anything but the smallest of backports too | |
| 13:37:42 | kashyap | stephenfin: Heya; nod. I just vaguely recall (although my mind is batterred the last few weeks, so I don't trust it) that we've done backport from main to other branches | |
| 13:38:24 | stephenfin | Downstream, perhaps. I don't recall it happening upstream though (intentionally at least) | |
| 13:38:51 | kashyap | Yeah, downstream definitely | |
| 14:06:12 | opendevreview | Andre Aranha proposed openstack/nova master: Test setting the nova job to centos-9-stream https://review.opendev.org/c/openstack/nova/+/831844 | |
| 14:06:18 | opendevreview | Kashyap Chamarthy proposed openstack/nova stable/wallaby: [nova/libvirt] Support for checking and enabling SMM when needed https://review.opendev.org/c/openstack/nova/+/849610 | |
| 14:06:41 | kashyap | sean-k-mooney: --^ There we go; fixed. Thx for the review! | |
| 14:14:29 | gibi | kashyap: yeah, we need to do it branch by branch. I think tripleo made a decision to EOL some stable branches inbetween active branches | |
| 14:14:35 | gibi | but they are the exception I think | |
| 14:15:09 | kashyap | Nod; thx. I mixed up the downstream way w/ upstream. | |
| 14:16:29 | sean-k-mooney | yes they are the exception but they never followed stable policy and never had the tag in the governance repo | |
| 14:16:32 | sean-k-mooney | we do | |
| 14:17:42 | gibi | ack | |
| 16:59:58 | colby_ | sean-k-mooney: So the patch seemed good yesterday but today I seem to be having trouble reclaiming mdevs again. We were successful yesterday. We do get a different error at least. Previously we just got a message no hosts available. Not we get timeout when building. | |
| 17:01:48 | colby_ | with this on the hypervisor: Insufficient compute resources: vGPU resource is not available | |
| 17:02:00 | sean-k-mooney | ok so perhaps a partial fix. | |
| 17:02:18 | sean-k-mooney | you are seeing that in the nova-compute agent log | |
| 17:02:34 | colby_ | we did notice yesterday that if we tried to spin up right away after delete it would fail and then would work a bit later | |
| 17:02:35 | sean-k-mooney | when its trying to boot? | |
| 17:02:40 | colby_ | yes the nova-compute agent log | |
| 17:02:44 | sean-k-mooney | ack | |
| 17:02:59 | sean-k-mooney | ok that sound like the resuse is still not working | |
| 17:03:28 | sean-k-mooney | it might have fixed the reporting of the resouce to placement | |
| 17:03:45 | sean-k-mooney | but not the consumtion of the exsitng mdev in the driver | |
| 17:03:50 | colby_ | yea and we were able to successfully reclaim some mdevs yesterday | |
| 17:04:06 | colby_ | now its failing and Ive waited a while in case its a caching issue | |
| 17:04:15 | sean-k-mooney | does restarting the comptue agent allow it to work? | |
| 17:04:25 | colby_ | I tried that too. No it got the same error | |
| 17:04:29 | sean-k-mooney | ok | |
| 17:04:41 | sean-k-mooney | so i was wondering if we got out of sync or something | |
| 17:05:01 | sean-k-mooney | like it works after the inital start but then breaks when a perodic runs or something like that | |
| 17:05:14 | sean-k-mooney | colby_: it turns out bauzas is on PTO until monday | |