| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-15 | |||
| 14:36:50 | mnaser | yeah but unfortunately it fully blocks nova from going back up, so that was a little part of how we ended up with no libvirt domain xml | |
| 14:37:07 | sean-k-mooney | yep understandable | |
| 14:37:07 | sean-k-mooney | yep understandable | |
| 14:37:18 | sean-k-mooney | mnaser: do dyou have multiple mdev types per host | |
| 14:37:18 | sean-k-mooney | mnaser: do dyou have multiple mdev types per host | |
| 14:37:21 | sean-k-mooney | or just one | |
| 14:37:21 | sean-k-mooney | or just one | |
| 14:37:24 | mnaser | in my case no, just one | |
| 14:37:24 | mnaser | in my case no, just one | |
| 14:37:29 | mnaser | by 'handle host reboots' == 'handle mdev devices disappearing' | |
| 14:37:29 | mnaser | by 'handle host reboots' == 'handle mdev devices disappearing' | |
| 14:37:38 | sean-k-mooney | ok then the workaround for you is simple | |
| 14:37:38 | sean-k-mooney | ok then the workaround for you is simple | |
| 14:37:44 | bauzas | mnaser: the paste you showed is just the bug you hit | |
| 14:37:44 | bauzas | mnaser: the paste you showed is just the bug you hit | |
| 14:38:12 | mnaser | create mdevs manually with the uuids from libvirt, i guess | |
| 14:38:12 | mnaser | create mdevs manually with the uuids from libvirt, i guess | |
| 14:38:15 | sean-k-mooney | mnaser: do you have multiple pGPUs per host | |
| 14:38:15 | sean-k-mooney | mnaser: do you have multiple pGPUs per host | |
| 14:38:25 | mnaser | nope, single pGPU, single vGPU type | |
| 14:38:25 | mnaser | nope, single pGPU, single vGPU type | |
| 14:38:26 | sean-k-mooney | mnaser: yep that is the workaround | |
| 14:38:26 | sean-k-mooney | mnaser: yep that is the workaround | |
| 14:38:31 | bauzas | I stupidely wrote something like "if the mdev doesn't exist; get its name from the xml and then lookup the non-existing mdev" | |
| 14:38:31 | bauzas | I stupidely wrote something like "if the mdev doesn't exist; get its name from the xml and then lookup the non-existing mdev" | |
| 14:38:33 | sean-k-mooney | perfect | |
| 14:38:33 | sean-k-mooney | perfect | |
| 14:38:50 | sean-k-mooney | so for your case then loop over the domains and grep all the mdevs | |
| 14:38:50 | sean-k-mooney | so for your case then loop over the domains and grep all the mdevs | |
| 14:38:57 | bauzas | right | |
| 14:38:57 | bauzas | right | |
| 14:39:00 | sean-k-mooney | then just create them with the hardcoded mdev type and parent | |
| 14:39:00 | sean-k-mooney | then just create them with the hardcoded mdev type and parent | |
| 14:39:05 | bauzas | if you only use one type, that's trivial | |
| 14:39:05 | bauzas | if you only use one type, that's trivial | |
| 14:39:10 | sean-k-mooney | by ecoing into /sys | |
| 14:39:10 | sean-k-mooney | by ecoing into /sys | |
| 14:39:22 | bauzas | just make sure to recreate the mdev with the right uuid | |
| 14:39:22 | bauzas | just make sure to recreate the mdev with the right uuid | |
| 14:39:45 | bauzas | echo "myuuid" > /sys/bus/mdev_bus/<mdev_type>/create | |
| 14:39:45 | bauzas | echo "myuuid" > /sys/bus/mdev_bus/<mdev_type>/create | |
| 14:39:53 | sean-k-mooney | yep that | |
| 14:39:53 | sean-k-mooney | yep that | |
| 14:39:57 | bauzas | (can't remember the exact sysfs path) | |
| 14:39:57 | bauzas | (can't remember the exact sysfs path) | |
| 14:39:58 | mnaser | got it, so that's our workaround until something placemet-y lands | |
| 14:39:58 | mnaser | got it, so that's our workaround until something placemet-y lands | |
| 14:40:08 | sean-k-mooney | yep | |
| 14:40:11 | bauzas | yup | |
| 14:40:18 | bauzas | YUUUUUUP even | |
| 14:40:18 | bauzas | YUUUUUUP even | |
| 14:40:23 | sean-k-mooney | :) | |
| 14:40:40 | sean-k-mooney | * sean-k-mooney thinks of the land before time | |
| 14:41:27 | bauzas | mnaser: sorry for the nasty bug, I should have wrote some docs describing the workaround | |
| 14:41:27 | bauzas | mnaser: sorry for the nasty bug, I should have wrote some docs describing the workaround | |
| 14:41:28 | sean-k-mooney | https://www.youtube.com/watch?v=cAEVzJnHv2c | |
| 14:41:28 | sean-k-mooney | https://www.youtube.com/watch?v=cAEVzJnHv2c | |
| 14:41:43 | bauzas | I'll at least amend the bug report | |
| 14:41:43 | bauzas | I'll at least amend the bug report | |
| 14:42:37 | mnaser | bauzas: hey no worries at all :D | |
| 14:42:37 | mnaser | bauzas: hey no worries at all :D | |
| 14:42:57 | mnaser | bauzas: i amended an eavesdrop link pointing to our conversation | |
| 14:42:57 | mnaser | bauzas: i amended an eavesdrop link pointing to our conversation | |
| 14:43:28 | mnaser | bauzas: i can throw something in an etherpad about a potential workaround with using single pgpu with single vgpu type | |
| 14:43:28 | mnaser | bauzas: i can throw something in an etherpad about a potential workaround with using single pgpu with single vgpu type | |
| 14:43:36 | mnaser | i need to write up something for my team anyways :) | |
| 14:43:36 | mnaser | i need to write up something for my team anyways :) | |
| 14:47:52 | mnaser | working here https://etherpad.opendev.org/p/nova-vgpu-sys-reboot and ill post it as a comment after i run it by y'all :) | |
| 14:47:52 | mnaser | working here https://etherpad.opendev.org/p/nova-vgpu-sys-reboot and ill post it as a comment after i run it by y'all :) | |
| 14:48:27 | bauzas | mnaser: https://bugs.launchpad.net/nova/+bug/1900800/comments/4 | |
| 14:48:27 | bauzas | mnaser: https://bugs.launchpad.net/nova/+bug/1900800/comments/4 | |
| 14:48:30 | openstack | Launchpad bug 1900800 in OpenStack Compute (nova) "VGPUs is not recreated on host reboot" [Low,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 14:48:46 | mnaser | oh heck, even better, ill refernce that :) | |
| 14:48:46 | mnaser | oh heck, even better, ill refernce that :) | |
| 14:49:17 | bauzas | you can even ask libvirt to give you all the domains that have mdevs | |
| 14:49:17 | bauzas | you can even ask libvirt to give you all the domains that have mdevs | |
| 14:49:52 | bauzas | no need to lookup all your instances, just the ones that have mdevs | |
| 14:49:52 | bauzas | no need to lookup all your instances, just the ones that have mdevs | |
| 14:52:22 | sean-k-mooney | bauzas: can you? i didnt know that | |
| 14:52:22 | sean-k-mooney | bauzas: can you? i didnt know that | |
| 14:52:33 | bauzas | oh, nevermind, call me stupid | |
| 14:52:33 | bauzas | oh, nevermind, call me stupid | |
| 14:52:37 | sean-k-mooney | bauzas: can you do that with virsh? | |
| 14:52:37 | sean-k-mooney | bauzas: can you do that with virsh? | |
| 14:52:40 | bauzas | that's the PCI devices you can get | |
| 14:52:40 | bauzas | that's the PCI devices you can get | |
| 14:52:48 | bauzas | the ones supporting mdev caps | |
| 14:52:48 | bauzas | the ones supporting mdev caps | |
| 14:53:04 | bauzas | in nova, I'm just blindly iterating over all instances | |
| 14:53:04 | bauzas | in nova, I'm just blindly iterating over all instances | |
| 14:53:07 | bauzas | ... :( | |
| 14:53:07 | bauzas | ... :( | |
| 14:53:27 | sean-k-mooney | ya i think that is what you have to do | |
| 14:53:27 | sean-k-mooney | ya i think that is what you have to do | |
| 14:53:40 | sean-k-mooney | im not aware of an api that allows you to fileter domains by the content of there xml | |
| 14:53:40 | sean-k-mooney | im not aware of an api that allows you to fileter domains by the content of there xml | |
| 14:59:56 | bauzas | sean-k-mooney: I guess looking up the instances by their flavors would be then better for mnaser ;) | |
| 14:59:56 | bauzas | sean-k-mooney: I guess looking up the instances by their flavors would be then better for mnaser ;) | |
| 15:00:06 | bauzas | using the nova api | |
| 15:00:06 | bauzas | using the nova api | |