| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-15 | |||
| 14:27:10 | bauzas | sean-k-mooney: you know what ? I'll start filling a spec for drafting mdev management in nova | |
| 14:27:21 | bauzas | and exposing them as raw resources | |
| 14:27:21 | bauzas | and exposing them as raw resources | |
| 14:27:23 | mnaser | so pretty much regenerate state from placement | |
| 14:27:23 | mnaser | so pretty much regenerate state from placement | |
| 14:27:40 | sean-k-mooney | bauzas: i think if we want to do the stateless mdev work it would make sense to do that anyway | |
| 14:27:40 | sean-k-mooney | bauzas: i think if we want to do the stateless mdev work it would make sense to do that anyway | |
| 14:27:46 | bauzas | sean-k-mooney: we could discuss the oppportunity of persisting them in the spec | |
| 14:27:46 | bauzas | sean-k-mooney: we could discuss the oppportunity of persisting them in the spec | |
| 14:27:51 | sean-k-mooney | we could keep them seperate or combine them | |
| 14:27:51 | sean-k-mooney | we could keep them seperate or combine them | |
| 14:27:56 | mnaser | i feel like that would be inline with VFs since you have to create them beforehand | |
| 14:27:56 | mnaser | i feel like that would be inline with VFs since you have to create them beforehand | |
| 14:28:26 | sean-k-mooney | ya i think there are two thing we should do | |
| 14:28:26 | sean-k-mooney | ya i think there are two thing we should do | |
| 14:28:29 | bauzas | mnaser: yeah and honestly we regressed on this, so I feel responsible for closing the bug | |
| 14:28:29 | bauzas | mnaser: yeah and honestly we regressed on this, so I feel responsible for closing the bug | |
| 14:28:47 | sean-k-mooney | 1 try an come up with a backporatable thing to adress the bug | |
| 14:28:47 | sean-k-mooney | 1 try an come up with a backporatable thing to adress the bug | |
| 14:29:00 | sean-k-mooney | and 2 figure out how to do it better longterm in the spec | |
| 14:29:00 | sean-k-mooney | and 2 figure out how to do it better longterm in the spec | |
| 14:29:21 | sean-k-mooney | while also discussing generic stateless mdevs for non gpu usecases | |
| 14:29:21 | sean-k-mooney | while also discussing generic stateless mdevs for non gpu usecases | |
| 14:29:52 | sean-k-mooney | the placement way can work for the backportable solution | |
| 14:29:52 | sean-k-mooney | the placement way can work for the backportable solution | |
| 14:30:14 | sean-k-mooney | im not sure we want to do that long term since i dont know what the performance of that will be like | |
| 14:30:14 | sean-k-mooney | im not sure we want to do that long term since i dont know what the performance of that will be like | |
| 14:30:25 | sean-k-mooney | i assume worse then a straight db lookup | |
| 14:30:25 | sean-k-mooney | i assume worse then a straight db lookup | |
| 14:30:48 | mnaser | yeah but only hitting on a hard_reboot() that involves regenerating xml | |
| 14:30:48 | mnaser | yeah but only hitting on a hard_reboot() that involves regenerating xml | |
| 14:31:55 | sean-k-mooney | mnaser: actully only on init host | |
| 14:31:55 | sean-k-mooney | mnaser: actully only on init host | |
| 14:32:07 | sean-k-mooney | we dont need to hit placment on hard reboot nessialy | |
| 14:32:07 | sean-k-mooney | we dont need to hit placment on hard reboot nessialy | |
| 14:32:25 | mnaser | ah right yes, unless someone is undefinning domains while nova is running | |
| 14:32:25 | mnaser | ah right yes, unless someone is undefinning domains while nova is running | |
| 14:32:31 | mnaser | in that case, that's on them =P | |
| 14:32:31 | mnaser | in that case, that's on them =P | |
| 14:33:04 | bauzas | mnaser: that call to placement is made at service restart | |
| 14:33:04 | bauzas | mnaser: that call to placement is made at service restart | |
| 14:33:17 | bauzas | not during hard reboots | |
| 14:33:17 | bauzas | not during hard reboots | |
| 14:33:41 | bauzas | for reboots, we just recreate the XML as we *already* have the allocations | |
| 14:33:42 | bauzas | for reboots, we just recreate the XML as we *already* have the allocations | |
| 14:34:11 | mnaser | bauzas: i do have another fun thing to add to it though | |
| 14:34:11 | mnaser | bauzas: i do have another fun thing to add to it though | |
| 14:34:21 | bauzas | shoot (/me hides) | |
| 14:34:21 | bauzas | shoot (/me hides) | |
| 14:34:31 | mnaser | http://paste.openstack.org/show/804516/ | |
| 14:34:31 | mnaser | http://paste.openstack.org/show/804516/ | |
| 14:34:59 | mnaser | that was actually why we had to undefine the domain to let nova start | |
| 14:34:59 | mnaser | that was actually why we had to undefine the domain to let nova start | |
| 14:35:07 | sean-k-mooney | well | |
| 14:35:17 | mnaser | so mdev is gone but on init_host we try to look it up | |
| 14:35:17 | mnaser | so mdev is gone but on init_host we try to look it up | |
| 14:35:24 | sean-k-mooney | that is just becasue the way we try to recrate the mdev today | |
| 14:35:24 | sean-k-mooney | that is just becasue the way we try to recrate the mdev today | |
| 14:35:41 | sean-k-mooney | mdev is still listed in the domain xml | |
| 14:35:41 | sean-k-mooney | mdev is still listed in the domain xml | |
| 14:35:50 | sean-k-mooney | but nothing has created in in sysfs yet | |
| 14:35:50 | sean-k-mooney | but nothing has created in in sysfs yet | |
| 14:36:06 | sean-k-mooney | which is why nodeDeviceLookupByName fails | |
| 14:36:06 | sean-k-mooney | which is why nodeDeviceLookupByName fails | |
| 14:36:27 | sean-k-mooney | mnaser: the current code only account for nova-compute restarts | |
| 14:36:27 | sean-k-mooney | mnaser: the current code only account for nova-compute restarts | |
| 14:36:36 | sean-k-mooney | it does not properly handel host reboots | |
| 14:36:36 | sean-k-mooney | it does not properly handel host reboots | |
| 14:36:50 | mnaser | yeah but unfortunately it fully blocks nova from going back up, so that was a little part of how we ended up with no libvirt domain xml | |
| 14:36:50 | mnaser | yeah but unfortunately it fully blocks nova from going back up, so that was a little part of how we ended up with no libvirt domain xml | |
| 14:37:07 | sean-k-mooney | yep understandable | |
| 14:37:07 | sean-k-mooney | yep understandable | |
| 14:37:18 | sean-k-mooney | mnaser: do dyou have multiple mdev types per host | |
| 14:37:18 | sean-k-mooney | mnaser: do dyou have multiple mdev types per host | |
| 14:37:21 | sean-k-mooney | or just one | |
| 14:37:21 | sean-k-mooney | or just one | |
| 14:37:24 | mnaser | in my case no, just one | |
| 14:37:24 | mnaser | in my case no, just one | |
| 14:37:29 | mnaser | by 'handle host reboots' == 'handle mdev devices disappearing' | |
| 14:37:29 | mnaser | by 'handle host reboots' == 'handle mdev devices disappearing' | |
| 14:37:38 | sean-k-mooney | ok then the workaround for you is simple | |
| 14:37:38 | sean-k-mooney | ok then the workaround for you is simple | |
| 14:37:44 | bauzas | mnaser: the paste you showed is just the bug you hit | |
| 14:37:44 | bauzas | mnaser: the paste you showed is just the bug you hit | |
| 14:38:12 | mnaser | create mdevs manually with the uuids from libvirt, i guess | |
| 14:38:12 | mnaser | create mdevs manually with the uuids from libvirt, i guess | |
| 14:38:15 | sean-k-mooney | mnaser: do you have multiple pGPUs per host | |
| 14:38:15 | sean-k-mooney | mnaser: do you have multiple pGPUs per host | |
| 14:38:25 | mnaser | nope, single pGPU, single vGPU type | |
| 14:38:25 | mnaser | nope, single pGPU, single vGPU type | |
| 14:38:26 | sean-k-mooney | mnaser: yep that is the workaround | |
| 14:38:26 | sean-k-mooney | mnaser: yep that is the workaround | |
| 14:38:31 | bauzas | I stupidely wrote something like "if the mdev doesn't exist; get its name from the xml and then lookup the non-existing mdev" | |
| 14:38:31 | bauzas | I stupidely wrote something like "if the mdev doesn't exist; get its name from the xml and then lookup the non-existing mdev" | |
| 14:38:33 | sean-k-mooney | perfect | |
| 14:38:33 | sean-k-mooney | perfect | |
| 14:38:50 | sean-k-mooney | so for your case then loop over the domains and grep all the mdevs | |
| 14:38:50 | sean-k-mooney | so for your case then loop over the domains and grep all the mdevs | |
| 14:38:57 | bauzas | right | |
| 14:38:57 | bauzas | right | |