| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-15 | |||
| 14:33:04 | bauzas | mnaser: that call to placement is made at service restart | |
| 14:33:04 | bauzas | mnaser: that call to placement is made at service restart | |
| 14:33:17 | bauzas | not during hard reboots | |
| 14:33:17 | bauzas | not during hard reboots | |
| 14:33:41 | bauzas | for reboots, we just recreate the XML as we *already* have the allocations | |
| 14:33:42 | bauzas | for reboots, we just recreate the XML as we *already* have the allocations | |
| 14:34:11 | mnaser | bauzas: i do have another fun thing to add to it though | |
| 14:34:11 | mnaser | bauzas: i do have another fun thing to add to it though | |
| 14:34:21 | bauzas | shoot (/me hides) | |
| 14:34:21 | bauzas | shoot (/me hides) | |
| 14:34:31 | mnaser | http://paste.openstack.org/show/804516/ | |
| 14:34:31 | mnaser | http://paste.openstack.org/show/804516/ | |
| 14:34:59 | mnaser | that was actually why we had to undefine the domain to let nova start | |
| 14:34:59 | mnaser | that was actually why we had to undefine the domain to let nova start | |
| 14:35:07 | sean-k-mooney | well | |
| 14:35:17 | mnaser | so mdev is gone but on init_host we try to look it up | |
| 14:35:17 | mnaser | so mdev is gone but on init_host we try to look it up | |
| 14:35:24 | sean-k-mooney | that is just becasue the way we try to recrate the mdev today | |
| 14:35:24 | sean-k-mooney | that is just becasue the way we try to recrate the mdev today | |
| 14:35:41 | sean-k-mooney | mdev is still listed in the domain xml | |
| 14:35:41 | sean-k-mooney | mdev is still listed in the domain xml | |
| 14:35:50 | sean-k-mooney | but nothing has created in in sysfs yet | |
| 14:35:50 | sean-k-mooney | but nothing has created in in sysfs yet | |
| 14:36:06 | sean-k-mooney | which is why nodeDeviceLookupByName fails | |
| 14:36:06 | sean-k-mooney | which is why nodeDeviceLookupByName fails | |
| 14:36:27 | sean-k-mooney | mnaser: the current code only account for nova-compute restarts | |
| 14:36:27 | sean-k-mooney | mnaser: the current code only account for nova-compute restarts | |
| 14:36:36 | sean-k-mooney | it does not properly handel host reboots | |
| 14:36:36 | sean-k-mooney | it does not properly handel host reboots | |
| 14:36:50 | mnaser | yeah but unfortunately it fully blocks nova from going back up, so that was a little part of how we ended up with no libvirt domain xml | |
| 14:36:50 | mnaser | yeah but unfortunately it fully blocks nova from going back up, so that was a little part of how we ended up with no libvirt domain xml | |
| 14:37:07 | sean-k-mooney | yep understandable | |
| 14:37:07 | sean-k-mooney | yep understandable | |
| 14:37:18 | sean-k-mooney | mnaser: do dyou have multiple mdev types per host | |
| 14:37:18 | sean-k-mooney | mnaser: do dyou have multiple mdev types per host | |
| 14:37:21 | sean-k-mooney | or just one | |
| 14:37:21 | sean-k-mooney | or just one | |
| 14:37:24 | mnaser | in my case no, just one | |
| 14:37:24 | mnaser | in my case no, just one | |
| 14:37:29 | mnaser | by 'handle host reboots' == 'handle mdev devices disappearing' | |
| 14:37:29 | mnaser | by 'handle host reboots' == 'handle mdev devices disappearing' | |
| 14:37:38 | sean-k-mooney | ok then the workaround for you is simple | |
| 14:37:38 | sean-k-mooney | ok then the workaround for you is simple | |
| 14:37:44 | bauzas | mnaser: the paste you showed is just the bug you hit | |
| 14:37:44 | bauzas | mnaser: the paste you showed is just the bug you hit | |
| 14:38:12 | mnaser | create mdevs manually with the uuids from libvirt, i guess | |
| 14:38:12 | mnaser | create mdevs manually with the uuids from libvirt, i guess | |
| 14:38:15 | sean-k-mooney | mnaser: do you have multiple pGPUs per host | |
| 14:38:15 | sean-k-mooney | mnaser: do you have multiple pGPUs per host | |
| 14:38:25 | mnaser | nope, single pGPU, single vGPU type | |
| 14:38:25 | mnaser | nope, single pGPU, single vGPU type | |
| 14:38:26 | sean-k-mooney | mnaser: yep that is the workaround | |
| 14:38:26 | sean-k-mooney | mnaser: yep that is the workaround | |
| 14:38:31 | bauzas | I stupidely wrote something like "if the mdev doesn't exist; get its name from the xml and then lookup the non-existing mdev" | |
| 14:38:31 | bauzas | I stupidely wrote something like "if the mdev doesn't exist; get its name from the xml and then lookup the non-existing mdev" | |
| 14:38:33 | sean-k-mooney | perfect | |
| 14:38:33 | sean-k-mooney | perfect | |
| 14:38:50 | sean-k-mooney | so for your case then loop over the domains and grep all the mdevs | |
| 14:38:50 | sean-k-mooney | so for your case then loop over the domains and grep all the mdevs | |
| 14:38:57 | bauzas | right | |
| 14:38:57 | bauzas | right | |
| 14:39:00 | sean-k-mooney | then just create them with the hardcoded mdev type and parent | |
| 14:39:00 | sean-k-mooney | then just create them with the hardcoded mdev type and parent | |
| 14:39:05 | bauzas | if you only use one type, that's trivial | |
| 14:39:05 | bauzas | if you only use one type, that's trivial | |
| 14:39:10 | sean-k-mooney | by ecoing into /sys | |
| 14:39:10 | sean-k-mooney | by ecoing into /sys | |
| 14:39:22 | bauzas | just make sure to recreate the mdev with the right uuid | |
| 14:39:22 | bauzas | just make sure to recreate the mdev with the right uuid | |
| 14:39:45 | bauzas | echo "myuuid" > /sys/bus/mdev_bus/<mdev_type>/create | |
| 14:39:45 | bauzas | echo "myuuid" > /sys/bus/mdev_bus/<mdev_type>/create | |
| 14:39:53 | sean-k-mooney | yep that | |
| 14:39:53 | sean-k-mooney | yep that | |
| 14:39:57 | bauzas | (can't remember the exact sysfs path) | |
| 14:39:57 | bauzas | (can't remember the exact sysfs path) | |
| 14:39:58 | mnaser | got it, so that's our workaround until something placemet-y lands | |
| 14:39:58 | mnaser | got it, so that's our workaround until something placemet-y lands | |
| 14:40:08 | sean-k-mooney | yep | |
| 14:40:11 | bauzas | yup | |
| 14:40:18 | bauzas | YUUUUUUP even | |
| 14:40:18 | bauzas | YUUUUUUP even | |
| 14:40:23 | sean-k-mooney | :) | |
| 14:40:40 | sean-k-mooney | * sean-k-mooney thinks of the land before time | |
| 14:41:27 | bauzas | mnaser: sorry for the nasty bug, I should have wrote some docs describing the workaround | |
| 14:41:27 | bauzas | mnaser: sorry for the nasty bug, I should have wrote some docs describing the workaround | |
| 14:41:28 | sean-k-mooney | https://www.youtube.com/watch?v=cAEVzJnHv2c | |
| 14:41:28 | sean-k-mooney | https://www.youtube.com/watch?v=cAEVzJnHv2c | |
| 14:41:43 | bauzas | I'll at least amend the bug report | |
| 14:41:43 | bauzas | I'll at least amend the bug report | |
| 14:42:37 | mnaser | bauzas: hey no worries at all :D | |
| 14:42:37 | mnaser | bauzas: hey no worries at all :D | |
| 14:42:57 | mnaser | bauzas: i amended an eavesdrop link pointing to our conversation | |
| 14:42:57 | mnaser | bauzas: i amended an eavesdrop link pointing to our conversation | |
| 14:43:28 | mnaser | bauzas: i can throw something in an etherpad about a potential workaround with using single pgpu with single vgpu type | |
| 14:43:28 | mnaser | bauzas: i can throw something in an etherpad about a potential workaround with using single pgpu with single vgpu type | |
| 14:43:36 | mnaser | i need to write up something for my team anyways :) | |
| 14:43:36 | mnaser | i need to write up something for my team anyways :) | |
| 14:47:52 | mnaser | working here https://etherpad.opendev.org/p/nova-vgpu-sys-reboot and ill post it as a comment after i run it by y'all :) | |
| 14:47:52 | mnaser | working here https://etherpad.opendev.org/p/nova-vgpu-sys-reboot and ill post it as a comment after i run it by y'all :) | |
| 14:48:27 | bauzas | mnaser: https://bugs.launchpad.net/nova/+bug/1900800/comments/4 | |