| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-12 | |||
| 16:12:59 | gibi | #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures | |
| 16:13:07 | gibi | anything else about the gate before I move on? | |
| 16:13:30 | elodilles | - | |
| 16:13:33 | gibi | #topic Release Planning | |
| 16:13:37 | gibi | #link https://releases.openstack.org/zed/schedule.html | |
| 16:13:42 | gibi | #info Zed-2 is in 2 days | |
| 16:13:45 | gibi | #info we'll stop accepting specs by Monday | |
| 16:13:49 | gibi | #action bauzas to send an email on Monday about specs and abandon the yoga specs | |
| 16:13:57 | gibi | we have a small amount of open specs | |
| 16:14:12 | gibi | if you have one then this is the last chance for Zed | |
| 16:14:36 | gibi | feel free to ping me if you need help to land them | |
| 16:15:02 | gibi | is there any other Release info to share? | |
| 16:16:06 | gibi | #topic Review priorities | |
| 16:16:12 | gibi | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+label:Review-Priority%252B1 | |
| 16:16:17 | gibi | #link https://review.opendev.org/c/openstack/project-config/+/837595 Gerrit policy for Review-prio contributors flag. We need project-config cores to merge it. | |
| 16:16:20 | gibi | #link https://docs.openstack.org/nova/latest/contributor/process.html#what-the-review-priority-label-in-gerrit-are-use-for Documentation we already have | |
| 16:16:47 | gibi | at some point we should start to review the list of prio marked review here but not today as so few of us here | |
| 16:16:56 | gibi | #topic Stable Branches | |
| 16:16:58 | gibi | elodilles: ? | |
| 16:17:15 | gibi | or more like <mic> -> elodilles | |
| 16:17:17 | elodilles | unfortunately not so much news, but let me copy them here | |
| 16:17:32 | elodilles | #info stable/train is blocked, fix exists but hasn't merged yet due to intermittent failures | |
| 16:17:40 | elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 16:17:49 | gibi | thanks | |
| 16:18:03 | elodilles | so in short, the intermittent failures are still causing pain for us :( | |
| 16:18:05 | elodilles | np | |
| 16:18:41 | gibi | elodilles: so https://review.opendev.org/c/openstack/nova/+/844530 the one we need for train? | |
| 16:18:56 | elodilles | yes | |
| 16:19:04 | gibi | ack | |
| 16:19:05 | gibi | thanks | |
| 16:19:19 | gibi | #topic Open discussion | |
| 16:19:24 | gibi | (bauzas) Opportunities for low-hanging-fruits, anyone ? (to be punted to next week) | |
| 16:19:31 | gibi | I guess we punt this to next week again | |
| 16:19:42 | gibi | but if you see low hangig fruits then note them for bauzas | |
| 16:19:51 | gibi | any other topic to discuss today? | |
| 16:20:02 | elodilles | nothing from me | |
| 16:20:38 | gibi | it seems we are in summer mode | |
| 16:20:57 | gibi | but at least I can leave in time for a game night ;) | |
| 16:21:18 | elodilles | hahh, have fun then! ; | |
| 16:21:20 | elodilles | :) | |
| 16:21:21 | gibi | thanks | |
| 16:21:32 | gibi | so lets close this | |
| 16:21:35 | gibi | #endmeeting | |
| 16:21:35 | opendevmeet | Meeting ended Tue Jul 12 16:21:35 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:21:35 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-07-12-16.00.html | |
| 16:21:35 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-07-12-16.00.txt | |
| 16:21:35 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-07-12-16.00.log.html | |
| 16:21:47 | Uggla | gibi, thx for running the meeting | |
| 16:21:49 | gibi | Uggla, elodilles: thanks for joining :) | |
| 16:22:05 | elodilles | o/ | |
| 16:22:06 | elodilles | :) | |
| 16:22:13 | Uggla | gibi, have fun | |
| 17:29:54 | colby_ | Hey All. Ive messaged before about our vGPU issues. Ive been working on trying to get this to work for a while now and we would really like to be able to offer this product to our users. Someone here mentioned that Nova should reuse the mdev devices that get created. The problem is that it is not. After spinning up vgpu instances then removing the mdev devices stay, but when we try to spin up a new instance its trying | |
| 17:29:54 | colby_ | to create another medev device (I think it is since its trying to use a resource provide of a different pci address on the card that already has all the mdevs created). | |
| 17:30:42 | colby_ | mdevctl list output: 67e63f1e-07f2-474f-874c-826a024c10ec 0000:21:01.7 nvidia-563 manual | |
| 17:30:42 | colby_ | 3a974d13-5dea-4bfc-b034-533f6e754349 0000:21:03.4 nvidia-563 manual | |
| 17:30:42 | colby_ | d3586a0a-2e56-421e-923f-20797fe74ab5 0000:21:03.7 nvidia-563 manual | |
| 17:30:42 | colby_ | 150c155c-da0b-45a6-8bc1-a8016231b100 0000:21:04.1 nvidia-563 manual | |
| 17:31:36 | colby_ | but spinning up a new instance tried to use the resource provider _pci_0000_21_02_4 (and pci 21 is already full) | |
| 17:32:51 | colby_ | how does nova detect the already created devices and use those? Should it be using the resource provider of those already created mdev (eg _pci_0000_21_01_7) | |
| 17:34:17 | colby_ | We are on Victoria release, Centos 8 Stream, Nvidia A40 GPU | |
| 17:54:13 | opendevreview | Merged openstack/nova master: zuul: Put Centos9 Stream job periodic-weekly and experimental https://review.opendev.org/c/openstack/nova/+/849463 | |
| 18:36:34 | sean-k-mooney | colby_: there is definetly a bug with this we are hitting it downstream too and still investigating | |
| 18:36:50 | sean-k-mooney | colby_: i can see if i can get you the link to where we try to reuse the mdev | |
| 18:37:58 | sean-k-mooney | colby_: https://github.com/openstack/nova/blob/de65131f92ba5ba812e33e6ff63be0991687413a/nova/virt/libvirt/driver.py#L8261-L8278= | |
| 18:40:17 | colby_ | sean-k-mooney: oh good glad to know this is not just us. Is there a bug filed yet that I could follow? | |
| 18:40:55 | sean-k-mooney | downstream definitly ill grab it and see if we have an upstram one. we were still trying to root cause it | |
| 18:41:27 | sean-k-mooney | colby_: basically we were QEing cold migration and noticed that the devices were not being reused | |
| 18:41:42 | sean-k-mooney | so depening on the order the test ran it either worked or failed | |
| 18:42:17 | sean-k-mooney | so we are looking at it as part of https://bugzilla.redhat.com/show_bug.cgi?id=1701281 | |
| 18:42:36 | sean-k-mooney | but i think we are going to break this out as a seperate upstream and downstream bug | |
| 18:43:03 | colby_ | would it be the same root cause as we are seeing just deleting and trying to create new instances? | |
| 18:43:41 | sean-k-mooney | a host reboot or deleteing the unused mdevs is the work around we are usign right now | |
| 18:44:22 | sean-k-mooney | so if you loop over the domain xmls and delete any mdev not used by an xml that "fixes it" temporally | |
| 18:44:24 | colby_ | yea thats what I ended up having to do manually is remove the mdevs that got created then new instances could be spun up | |
| 18:44:28 | sean-k-mooney | but thats not the correct fix | |
| 18:45:46 | colby_ | I suppose I could create a cron job to do that so we can remove the manual part | |
| 18:45:55 | sean-k-mooney | colby_: bauzas is on on pto tomorrow and friday but i think they are here tursday | |
| 18:46:11 | sean-k-mooney | they tought it might be related to who we do the mdev lookup | |
| 18:46:30 | sean-k-mooney | but since you have the issue | |
| 18:46:52 | sean-k-mooney | could you compare the list of mdevs returned by mdevctl and libvirt via virsh | |
| 18:47:11 | sean-k-mooney | libvirt does some caching so one of the guesses we had is it might be getting out of sync | |
| 18:47:48 | sean-k-mooney | actully hum | |
| 18:49:12 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/832489/1/nova/virt/libvirt/utils.py | |
| 18:49:17 | sean-k-mooney | i wonder if its this ^ | |
| 18:49:27 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1951656 | |
| 18:50:00 | sean-k-mooney | colby_: do you know what version fo libvirt you are using | |
| 18:50:20 | colby_ | 7.9.0-1 | |
| 18:50:48 | colby_ | you just want the output form `virsh nodedev-list` ? | |
| 18:50:53 | sean-k-mooney | that was in 7.7 https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d | |
| 18:51:10 | sean-k-mooney | colby_: yes please nodedev-list shoudl have the names | |
| 18:51:15 | sean-k-mooney | and we can check the format | |
| 18:51:21 | sean-k-mooney | to see if it has the parent info or not | |
| 18:54:17 | colby_ | https://pastebin.com/94YfBRH9 | |
| 18:55:00 | sean-k-mooney | there is also https://review.opendev.org/c/openstack/nova/+/838976 as another possible fix | |
| 18:55:39 | sean-k-mooney | odd i dont see any mdevs there | |
| 18:56:03 | colby_ | ha woops sorry | |
| 18:56:05 | colby_ | wrong machine | |
| 18:56:36 | colby_ | https://pastebin.com/NAfjxUt7 | |
| 18:57:18 | colby_ | mdevctl list output: https://pastebin.com/rXmUftzj | |