| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-12 | |||
| 18:44:22 | sean-k-mooney | so if you loop over the domain xmls and delete any mdev not used by an xml that "fixes it" temporally | |
| 18:44:24 | colby_ | yea thats what I ended up having to do manually is remove the mdevs that got created then new instances could be spun up | |
| 18:44:28 | sean-k-mooney | but thats not the correct fix | |
| 18:45:46 | colby_ | I suppose I could create a cron job to do that so we can remove the manual part | |
| 18:45:55 | sean-k-mooney | colby_: bauzas is on on pto tomorrow and friday but i think they are here tursday | |
| 18:46:11 | sean-k-mooney | they tought it might be related to who we do the mdev lookup | |
| 18:46:30 | sean-k-mooney | but since you have the issue | |
| 18:46:52 | sean-k-mooney | could you compare the list of mdevs returned by mdevctl and libvirt via virsh | |
| 18:47:11 | sean-k-mooney | libvirt does some caching so one of the guesses we had is it might be getting out of sync | |
| 18:47:48 | sean-k-mooney | actully hum | |
| 18:49:12 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/832489/1/nova/virt/libvirt/utils.py | |
| 18:49:17 | sean-k-mooney | i wonder if its this ^ | |
| 18:49:27 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1951656 | |
| 18:50:00 | sean-k-mooney | colby_: do you know what version fo libvirt you are using | |
| 18:50:20 | colby_ | 7.9.0-1 | |
| 18:50:48 | colby_ | you just want the output form `virsh nodedev-list` ? | |
| 18:50:53 | sean-k-mooney | that was in 7.7 https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d | |
| 18:51:10 | sean-k-mooney | colby_: yes please nodedev-list shoudl have the names | |
| 18:51:15 | sean-k-mooney | and we can check the format | |
| 18:51:21 | sean-k-mooney | to see if it has the parent info or not | |
| 18:54:17 | colby_ | https://pastebin.com/94YfBRH9 | |
| 18:55:00 | sean-k-mooney | there is also https://review.opendev.org/c/openstack/nova/+/838976 as another possible fix | |
| 18:55:39 | sean-k-mooney | odd i dont see any mdevs there | |
| 18:56:03 | colby_ | ha woops sorry | |
| 18:56:05 | colby_ | wrong machine | |
| 18:56:36 | colby_ | https://pastebin.com/NAfjxUt7 | |
| 18:57:18 | colby_ | mdevctl list output: https://pastebin.com/rXmUftzj | |
| 18:57:52 | sean-k-mooney | yep so virsh has the extended names | |
| 18:58:27 | sean-k-mooney | so its not a caching issue but it proably is a parsing issue | |
| 18:58:34 | sean-k-mooney | since both consitent | |
| 18:59:05 | sean-k-mooney | """Note that the lookup of the mdev device by UUID are needed in order | |
| 18:59:08 | sean-k-mooney | to keep the ability to recreate assigned mediated devices on a reboot of | |
| 18:59:10 | sean-k-mooney | the compute node | |
| 18:59:12 | sean-k-mooney | """ | |
| 18:59:27 | sean-k-mooney | but i bet its also needed to be able to reuse the mdevs at all | |
| 19:01:42 | sean-k-mooney | colby_: im not 100% sure this will fix it but i have set https://review.opendev.org/c/openstack/nova/+/838976 as a review priority and ill follow up with sylvain when they are back | |
| 19:03:31 | colby_ | ok sounds good. Im happy to test out the patches on our system if you want | |
| 19:03:42 | colby_ | no one is using this hypervisor right now but the admins | |
| 19:04:04 | sean-k-mooney | if you wanted to test https://review.opendev.org/c/openstack/nova/+/838976 and provide feedback on the review that is the more compelte fix | |
| 19:04:49 | colby_ | sure. Ill get those in place today and let you know if it helps our case at all | |
| 19:04:58 | sean-k-mooney | most of the opencomment are about updating the doc strings but the patch should work as is | |
| 19:05:15 | sean-k-mooney | we might also add a functional repoducer if we can recaret the mdev resue issue | |
| 19:05:29 | sean-k-mooney | colby_: thanks | |
| #openstack-nova - 2022-07-13 | |||
| 04:19:31 | opendevreview | Kashyap Chamarthy proposed openstack/nova stable/wallaby: [nova/libvirt] Support for checking and enabling SMM when needed https://review.opendev.org/c/openstack/nova/+/849610 | |
| 07:43:06 | gibi | good morning | |
| 09:26:53 | opendevreview | Maksim Malchuk proposed openstack/nova stable/xena: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/829804 | |
| 13:28:03 | opendevreview | Kashyap Chamarthy proposed openstack/nova stable/xena: [nova/libvirt] Support for checking and enabling SMM when needed https://review.opendev.org/c/openstack/nova/+/849676 | |
| 13:35:30 | kashyap | sean-k-mooney: gibi: Do we always have to backport in order? Is there anything "off" if we backport from master to wallaby, instead of xena to wallaby, when it's a clean pick? | |
| 13:36:01 | stephenfin | kashyap: Always branch by branch | |
| 13:36:35 | stephenfin | It just ensures that people don't skip things. You will invariably have conflicts for anything but the smallest of backports too | |
| 13:37:42 | kashyap | stephenfin: Heya; nod. I just vaguely recall (although my mind is batterred the last few weeks, so I don't trust it) that we've done backport from main to other branches | |
| 13:38:24 | stephenfin | Downstream, perhaps. I don't recall it happening upstream though (intentionally at least) | |
| 13:38:51 | kashyap | Yeah, downstream definitely | |
| 14:06:12 | opendevreview | Andre Aranha proposed openstack/nova master: Test setting the nova job to centos-9-stream https://review.opendev.org/c/openstack/nova/+/831844 | |
| 14:06:18 | opendevreview | Kashyap Chamarthy proposed openstack/nova stable/wallaby: [nova/libvirt] Support for checking and enabling SMM when needed https://review.opendev.org/c/openstack/nova/+/849610 | |
| 14:06:41 | kashyap | sean-k-mooney: --^ There we go; fixed. Thx for the review! | |
| 14:14:29 | gibi | kashyap: yeah, we need to do it branch by branch. I think tripleo made a decision to EOL some stable branches inbetween active branches | |
| 14:14:35 | gibi | but they are the exception I think | |
| 14:15:09 | kashyap | Nod; thx. I mixed up the downstream way w/ upstream. | |
| 14:16:29 | sean-k-mooney | yes they are the exception but they never followed stable policy and never had the tag in the governance repo | |
| 14:16:32 | sean-k-mooney | we do | |
| 14:17:42 | gibi | ack | |
| 16:59:58 | colby_ | sean-k-mooney: So the patch seemed good yesterday but today I seem to be having trouble reclaiming mdevs again. We were successful yesterday. We do get a different error at least. Previously we just got a message no hosts available. Not we get timeout when building. | |
| 17:01:48 | colby_ | with this on the hypervisor: Insufficient compute resources: vGPU resource is not available | |
| 17:02:00 | sean-k-mooney | ok so perhaps a partial fix. | |
| 17:02:18 | sean-k-mooney | you are seeing that in the nova-compute agent log | |
| 17:02:34 | colby_ | we did notice yesterday that if we tried to spin up right away after delete it would fail and then would work a bit later | |
| 17:02:35 | sean-k-mooney | when its trying to boot? | |
| 17:02:40 | colby_ | yes the nova-compute agent log | |
| 17:02:44 | sean-k-mooney | ack | |
| 17:02:59 | sean-k-mooney | ok that sound like the resuse is still not working | |
| 17:03:28 | sean-k-mooney | it might have fixed the reporting of the resouce to placement | |
| 17:03:45 | sean-k-mooney | but not the consumtion of the exsitng mdev in the driver | |
| 17:03:50 | colby_ | yea and we were able to successfully reclaim some mdevs yesterday | |
| 17:04:06 | colby_ | now its failing and Ive waited a while in case its a caching issue | |
| 17:04:15 | sean-k-mooney | does restarting the comptue agent allow it to work? | |
| 17:04:25 | colby_ | I tried that too. No it got the same error | |
| 17:04:29 | sean-k-mooney | ok | |
| 17:04:41 | sean-k-mooney | so i was wondering if we got out of sync or something | |
| 17:05:01 | sean-k-mooney | like it works after the inital start but then breaks when a perodic runs or something like that | |
| 17:05:14 | sean-k-mooney | colby_: it turns out bauzas is on PTO until monday | |
| 17:05:28 | sean-k-mooney | so i wont be abel to get his input on this until then | |
| 17:05:44 | sean-k-mooney | have you filed an upstream bug for this | |
| 17:05:53 | colby_ | ok no problem. Ill follow up again on Monday. | |
| 17:06:08 | sean-k-mooney | you coudl use the exsiting one but it might be helpful if you could attach the traceback fo the error | |
| 17:06:09 | colby_ | No Im happy to file a bug so that I can get any logs/info they need to help identify | |
| 17:06:29 | sean-k-mooney | ya that could help use create a repoducer test | |
| 17:07:10 | colby_ | So should I create a new one? If so where is the best place to do that? | |
| 19:39:41 | colby_ | sean-k-mooney: should I file a new bug? Where do I do that? | |
| 20:25:38 | melwitt | colby_: you can file a bug from this page https://bugs.launchpad.net/nova | |
| 20:26:58 | colby_ | thanks I was able to find where I needed to do it | |
| 20:27:08 | colby_ | https://bugs.launchpad.net/nova/+bug/1981631 | |
| 20:48:51 | melwitt | ok great | |
| #openstack-nova - 2022-07-14 | |||
| 00:26:30 | opendevreview | Artom Lifshitz proposed openstack/nova-specs master: Configurable instance domains https://review.opendev.org/c/openstack/nova-specs/+/849765 | |
| 07:38:17 | auniyal | O/ | |
| 07:38:37 | auniyal | tox -e <test> don't always runs same | |
| 07:38:40 | auniyal | tox -re functional-py38 -- regressions.test_bug_1857306.py | |
| 07:38:41 | auniyal | The specified regex doesn't match with anythingERROR: InvocationError for command /opt/stack/nova/.tox/functional-py38/bin/stestr --test-path=./nova/tests/functional run regressions.test_bug_1857306.py (exited with code 1) | |
| 07:39:47 | auniyal | sometime -re works to recreate testing venv, but not always | |
| 07:41:20 | auniyal | tried giving full path as well - nova.tests.functional.regressions.test_bug_1857306.py | |