| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-15 | |||
| 14:38:33 | sean-k-mooney | perfect | |
| 14:38:33 | sean-k-mooney | perfect | |
| 14:38:50 | sean-k-mooney | so for your case then loop over the domains and grep all the mdevs | |
| 14:38:50 | sean-k-mooney | so for your case then loop over the domains and grep all the mdevs | |
| 14:38:57 | bauzas | right | |
| 14:38:57 | bauzas | right | |
| 14:39:00 | sean-k-mooney | then just create them with the hardcoded mdev type and parent | |
| 14:39:00 | sean-k-mooney | then just create them with the hardcoded mdev type and parent | |
| 14:39:05 | bauzas | if you only use one type, that's trivial | |
| 14:39:05 | bauzas | if you only use one type, that's trivial | |
| 14:39:10 | sean-k-mooney | by ecoing into /sys | |
| 14:39:10 | sean-k-mooney | by ecoing into /sys | |
| 14:39:22 | bauzas | just make sure to recreate the mdev with the right uuid | |
| 14:39:22 | bauzas | just make sure to recreate the mdev with the right uuid | |
| 14:39:45 | bauzas | echo "myuuid" > /sys/bus/mdev_bus/<mdev_type>/create | |
| 14:39:45 | bauzas | echo "myuuid" > /sys/bus/mdev_bus/<mdev_type>/create | |
| 14:39:53 | sean-k-mooney | yep that | |
| 14:39:53 | sean-k-mooney | yep that | |
| 14:39:57 | bauzas | (can't remember the exact sysfs path) | |
| 14:39:57 | bauzas | (can't remember the exact sysfs path) | |
| 14:39:58 | mnaser | got it, so that's our workaround until something placemet-y lands | |
| 14:39:58 | mnaser | got it, so that's our workaround until something placemet-y lands | |
| 14:40:08 | sean-k-mooney | yep | |
| 14:40:11 | bauzas | yup | |
| 14:40:18 | bauzas | YUUUUUUP even | |
| 14:40:18 | bauzas | YUUUUUUP even | |
| 14:40:23 | sean-k-mooney | :) | |
| 14:40:40 | sean-k-mooney | * sean-k-mooney thinks of the land before time | |
| 14:41:27 | bauzas | mnaser: sorry for the nasty bug, I should have wrote some docs describing the workaround | |
| 14:41:27 | bauzas | mnaser: sorry for the nasty bug, I should have wrote some docs describing the workaround | |
| 14:41:28 | sean-k-mooney | https://www.youtube.com/watch?v=cAEVzJnHv2c | |
| 14:41:28 | sean-k-mooney | https://www.youtube.com/watch?v=cAEVzJnHv2c | |
| 14:41:43 | bauzas | I'll at least amend the bug report | |
| 14:41:43 | bauzas | I'll at least amend the bug report | |
| 14:42:37 | mnaser | bauzas: hey no worries at all :D | |
| 14:42:37 | mnaser | bauzas: hey no worries at all :D | |
| 14:42:57 | mnaser | bauzas: i amended an eavesdrop link pointing to our conversation | |
| 14:42:57 | mnaser | bauzas: i amended an eavesdrop link pointing to our conversation | |
| 14:43:28 | mnaser | bauzas: i can throw something in an etherpad about a potential workaround with using single pgpu with single vgpu type | |
| 14:43:28 | mnaser | bauzas: i can throw something in an etherpad about a potential workaround with using single pgpu with single vgpu type | |
| 14:43:36 | mnaser | i need to write up something for my team anyways :) | |
| 14:43:36 | mnaser | i need to write up something for my team anyways :) | |
| 14:47:52 | mnaser | working here https://etherpad.opendev.org/p/nova-vgpu-sys-reboot and ill post it as a comment after i run it by y'all :) | |
| 14:47:52 | mnaser | working here https://etherpad.opendev.org/p/nova-vgpu-sys-reboot and ill post it as a comment after i run it by y'all :) | |
| 14:48:27 | bauzas | mnaser: https://bugs.launchpad.net/nova/+bug/1900800/comments/4 | |
| 14:48:27 | bauzas | mnaser: https://bugs.launchpad.net/nova/+bug/1900800/comments/4 | |
| 14:48:30 | openstack | Launchpad bug 1900800 in OpenStack Compute (nova) "VGPUs is not recreated on host reboot" [Low,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 14:48:46 | mnaser | oh heck, even better, ill refernce that :) | |
| 14:48:46 | mnaser | oh heck, even better, ill refernce that :) | |
| 14:49:17 | bauzas | you can even ask libvirt to give you all the domains that have mdevs | |
| 14:49:17 | bauzas | you can even ask libvirt to give you all the domains that have mdevs | |
| 14:49:52 | bauzas | no need to lookup all your instances, just the ones that have mdevs | |
| 14:49:52 | bauzas | no need to lookup all your instances, just the ones that have mdevs | |
| 14:52:22 | sean-k-mooney | bauzas: can you? i didnt know that | |
| 14:52:22 | sean-k-mooney | bauzas: can you? i didnt know that | |
| 14:52:33 | bauzas | oh, nevermind, call me stupid | |
| 14:52:33 | bauzas | oh, nevermind, call me stupid | |
| 14:52:37 | sean-k-mooney | bauzas: can you do that with virsh? | |
| 14:52:37 | sean-k-mooney | bauzas: can you do that with virsh? | |
| 14:52:40 | bauzas | that's the PCI devices you can get | |
| 14:52:40 | bauzas | that's the PCI devices you can get | |
| 14:52:48 | bauzas | the ones supporting mdev caps | |
| 14:52:48 | bauzas | the ones supporting mdev caps | |
| 14:53:04 | bauzas | in nova, I'm just blindly iterating over all instances | |
| 14:53:04 | bauzas | in nova, I'm just blindly iterating over all instances | |
| 14:53:07 | bauzas | ... :( | |
| 14:53:07 | bauzas | ... :( | |
| 14:53:27 | sean-k-mooney | ya i think that is what you have to do | |
| 14:53:27 | sean-k-mooney | ya i think that is what you have to do | |
| 14:53:40 | sean-k-mooney | im not aware of an api that allows you to fileter domains by the content of there xml | |
| 14:53:40 | sean-k-mooney | im not aware of an api that allows you to fileter domains by the content of there xml | |
| 14:59:56 | bauzas | sean-k-mooney: I guess looking up the instances by their flavors would be then better for mnaser ;) | |
| 14:59:56 | bauzas | sean-k-mooney: I guess looking up the instances by their flavors would be then better for mnaser ;) | |
| 15:00:06 | bauzas | using the nova api | |
| 15:00:06 | bauzas | using the nova api | |
| 15:00:16 | bauzas | and then getting the instance name | |
| 15:00:16 | bauzas | and then getting the instance name | |
| 15:01:49 | sean-k-mooney | not really | |
| 15:01:49 | sean-k-mooney | not really | |
| 15:01:58 | sean-k-mooney | that would be much more expensive | |
| 15:01:58 | sean-k-mooney | that would be much more expensive | |
| 15:02:32 | sean-k-mooney | mnaser should now which host has gpus so its really not that hard to loop over the domains on those hosts in a script | |
| 15:02:32 | sean-k-mooney | mnaser should now which host has gpus so its really not that hard to loop over the domains on those hosts in a script | |
| 15:03:24 | sean-k-mooney | doing that will be much faster then hitting keytone to get a tokken then listing the instance on host and filtering by flaovr then looking it up in libvirt to get the mdev | |
| 15:03:24 | sean-k-mooney | doing that will be much faster then hitting keytone to get a tokken then listing the instance on host and filtering by flaovr then looking it up in libvirt to get the mdev | |
| 15:15:36 | mnaser | yeah i think we'll have a bash script because its single pgpu per system so if the libvirt domains werent destroyed yet | |
| 15:15:36 | mnaser | yeah i think we'll have a bash script because its single pgpu per system so if the libvirt domains werent destroyed yet | |
| 15:15:40 | mnaser | it should be easy to rebuild | |
| 15:15:40 | mnaser | it should be easy to rebuild | |
| 15:16:19 | sean-k-mooney | mnaser: a hard reboot might fix your instances that dont have gpus | |
| 15:16:19 | sean-k-mooney | mnaser: a hard reboot might fix your instances that dont have gpus | |
| 15:16:28 | sean-k-mooney | or a cold migrate it that is not enough | |
| 15:16:28 | sean-k-mooney | or a cold migrate it that is not enough | |
| 15:16:32 | mnaser | sean-k-mooney: yeah those are fine but init_host() failing means we have to fix them all first | |
| 15:16:32 | mnaser | sean-k-mooney: yeah those are fine but init_host() failing means we have to fix them all first | |
| 15:16:36 | mnaser | cause nova wont go up | |
| 15:16:36 | mnaser | cause nova wont go up | |
| 15:16:48 | sean-k-mooney | yep i ment for the ones you undifed | |
| 15:16:48 | sean-k-mooney | yep i ment for the ones you undifed | |
| 15:21:29 | openstackgerrit | Rodrigo Barbieri proposed openstack/nova master: Error anti-affinity violation on migrations https://review.opendev.org/c/openstack/nova/+/784166 | |