| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-15 | |||
| 14:23:31 | bauzas | sean-k-mooney: that's actually a great point | |
| 14:23:32 | bauzas | sean-k-mooney: that's actually a great point | |
| 14:23:33 | mnaser | is there anthing else we store in the xml as a state? | |
| 14:23:33 | mnaser | is there anthing else we store in the xml as a state? | |
| 14:23:50 | sean-k-mooney | mnaser: no | |
| 14:23:50 | sean-k-mooney | mnaser: no | |
| 14:24:02 | sean-k-mooney | long term we want to get rid of persitent domains | |
| 14:24:02 | sean-k-mooney | long term we want to get rid of persitent domains | |
| 14:24:07 | mnaser | only thing with this is if something goes wrong with libvirt or anything, you would lose all your gpus | |
| 14:24:07 | mnaser | only thing with this is if something goes wrong with libvirt or anything, you would lose all your gpus | |
| 14:24:11 | sean-k-mooney | e.g. the domain xml on disk | |
| 14:24:11 | sean-k-mooney | e.g. the domain xml on disk | |
| 14:24:15 | bauzas | sean-k-mooney: the only problem with metadata is that we won't recreate it on move operations | |
| 14:24:15 | bauzas | sean-k-mooney: the only problem with metadata is that we won't recreate it on move operations | |
| 14:24:39 | mnaser | thats how this bit us, nova wouldn't start, so we tried to undefine the domain to let nova recreate it, and here we are with no vgpus | |
| 14:24:39 | mnaser | thats how this bit us, nova wouldn't start, so we tried to undefine the domain to let nova recreate it, and here we are with no vgpus | |
| 14:24:59 | bauzas | mnaser: nova just binds mdevs | |
| 14:24:59 | bauzas | mnaser: nova just binds mdevs | |
| 14:25:24 | mnaser | but if libvirt domain is gone, it doesnt know which mdevs were assigned to that vm, even on a hard reboot | |
| 14:25:24 | mnaser | but if libvirt domain is gone, it doesnt know which mdevs were assigned to that vm, even on a hard reboot | |
| 14:25:30 | sean-k-mooney | so fundimentally i think we need to revisit using the xml for state storage | |
| 14:25:30 | sean-k-mooney | so fundimentally i think we need to revisit using the xml for state storage | |
| 14:25:49 | bauzas | mnaser: sure, but why would you undefine the domain ? | |
| 14:25:49 | bauzas | mnaser: sure, but why would you undefine the domain ? | |
| 14:25:51 | sean-k-mooney | and just store the inf in the nova db eventualy | |
| 14:25:51 | sean-k-mooney | and just store the inf in the nova db eventualy | |
| 14:25:57 | bauzas | sean-k-mooney: eeeek | |
| 14:25:57 | bauzas | sean-k-mooney: eeeek | |
| 14:26:07 | mnaser | bauzas: we had other issues why the domain would not start, because a call to libvirt was failing because the mdev was missing | |
| 14:26:07 | mnaser | bauzas: we had other issues why the domain would not start, because a call to libvirt was failing because the mdev was missing | |
| 14:26:14 | sean-k-mooney | bauzas: i didnt like using the xml for this in the first place | |
| 14:26:14 | sean-k-mooney | bauzas: i didnt like using the xml for this in the first place | |
| 14:26:26 | mnaser | so mdev was missing so nova couldnt start | |
| 14:26:26 | mnaser | so mdev was missing so nova couldnt start | |
| 14:26:29 | sean-k-mooney | bauzas: this probalem is just another rasons to not do it this way | |
| 14:26:29 | sean-k-mooney | bauzas: this probalem is just another rasons to not do it this way | |
| 14:26:50 | bauzas | mnaser: again, I can try to fix the logic by looking up placement | |
| 14:26:50 | bauzas | mnaser: again, I can try to fix the logic by looking up placement | |
| 14:27:10 | bauzas | sean-k-mooney: you know what ? I'll start filling a spec for drafting mdev management in nova | |
| 14:27:10 | bauzas | sean-k-mooney: you know what ? I'll start filling a spec for drafting mdev management in nova | |
| 14:27:21 | bauzas | and exposing them as raw resources | |
| 14:27:21 | bauzas | and exposing them as raw resources | |
| 14:27:23 | mnaser | so pretty much regenerate state from placement | |
| 14:27:23 | mnaser | so pretty much regenerate state from placement | |
| 14:27:40 | sean-k-mooney | bauzas: i think if we want to do the stateless mdev work it would make sense to do that anyway | |
| 14:27:40 | sean-k-mooney | bauzas: i think if we want to do the stateless mdev work it would make sense to do that anyway | |
| 14:27:46 | bauzas | sean-k-mooney: we could discuss the oppportunity of persisting them in the spec | |
| 14:27:46 | bauzas | sean-k-mooney: we could discuss the oppportunity of persisting them in the spec | |
| 14:27:51 | sean-k-mooney | we could keep them seperate or combine them | |
| 14:27:51 | sean-k-mooney | we could keep them seperate or combine them | |
| 14:27:56 | mnaser | i feel like that would be inline with VFs since you have to create them beforehand | |
| 14:27:56 | mnaser | i feel like that would be inline with VFs since you have to create them beforehand | |
| 14:28:26 | sean-k-mooney | ya i think there are two thing we should do | |
| 14:28:26 | sean-k-mooney | ya i think there are two thing we should do | |
| 14:28:29 | bauzas | mnaser: yeah and honestly we regressed on this, so I feel responsible for closing the bug | |
| 14:28:29 | bauzas | mnaser: yeah and honestly we regressed on this, so I feel responsible for closing the bug | |
| 14:28:47 | sean-k-mooney | 1 try an come up with a backporatable thing to adress the bug | |
| 14:28:47 | sean-k-mooney | 1 try an come up with a backporatable thing to adress the bug | |
| 14:29:00 | sean-k-mooney | and 2 figure out how to do it better longterm in the spec | |
| 14:29:00 | sean-k-mooney | and 2 figure out how to do it better longterm in the spec | |
| 14:29:21 | sean-k-mooney | while also discussing generic stateless mdevs for non gpu usecases | |
| 14:29:21 | sean-k-mooney | while also discussing generic stateless mdevs for non gpu usecases | |
| 14:29:52 | sean-k-mooney | the placement way can work for the backportable solution | |
| 14:29:52 | sean-k-mooney | the placement way can work for the backportable solution | |
| 14:30:14 | sean-k-mooney | im not sure we want to do that long term since i dont know what the performance of that will be like | |
| 14:30:14 | sean-k-mooney | im not sure we want to do that long term since i dont know what the performance of that will be like | |
| 14:30:25 | sean-k-mooney | i assume worse then a straight db lookup | |
| 14:30:25 | sean-k-mooney | i assume worse then a straight db lookup | |
| 14:30:48 | mnaser | yeah but only hitting on a hard_reboot() that involves regenerating xml | |
| 14:30:48 | mnaser | yeah but only hitting on a hard_reboot() that involves regenerating xml | |
| 14:31:55 | sean-k-mooney | mnaser: actully only on init host | |
| 14:31:55 | sean-k-mooney | mnaser: actully only on init host | |
| 14:32:07 | sean-k-mooney | we dont need to hit placment on hard reboot nessialy | |
| 14:32:07 | sean-k-mooney | we dont need to hit placment on hard reboot nessialy | |
| 14:32:25 | mnaser | ah right yes, unless someone is undefinning domains while nova is running | |
| 14:32:25 | mnaser | ah right yes, unless someone is undefinning domains while nova is running | |
| 14:32:31 | mnaser | in that case, that's on them =P | |
| 14:32:31 | mnaser | in that case, that's on them =P | |
| 14:33:04 | bauzas | mnaser: that call to placement is made at service restart | |
| 14:33:04 | bauzas | mnaser: that call to placement is made at service restart | |
| 14:33:17 | bauzas | not during hard reboots | |
| 14:33:17 | bauzas | not during hard reboots | |
| 14:33:41 | bauzas | for reboots, we just recreate the XML as we *already* have the allocations | |
| 14:33:42 | bauzas | for reboots, we just recreate the XML as we *already* have the allocations | |
| 14:34:11 | mnaser | bauzas: i do have another fun thing to add to it though | |
| 14:34:11 | mnaser | bauzas: i do have another fun thing to add to it though | |
| 14:34:21 | bauzas | shoot (/me hides) | |
| 14:34:21 | bauzas | shoot (/me hides) | |
| 14:34:31 | mnaser | http://paste.openstack.org/show/804516/ | |
| 14:34:31 | mnaser | http://paste.openstack.org/show/804516/ | |
| 14:34:59 | mnaser | that was actually why we had to undefine the domain to let nova start | |
| 14:34:59 | mnaser | that was actually why we had to undefine the domain to let nova start | |
| 14:35:07 | sean-k-mooney | well | |
| 14:35:17 | mnaser | so mdev is gone but on init_host we try to look it up | |
| 14:35:17 | mnaser | so mdev is gone but on init_host we try to look it up | |
| 14:35:24 | sean-k-mooney | that is just becasue the way we try to recrate the mdev today | |
| 14:35:24 | sean-k-mooney | that is just becasue the way we try to recrate the mdev today | |
| 14:35:41 | sean-k-mooney | mdev is still listed in the domain xml | |
| 14:35:41 | sean-k-mooney | mdev is still listed in the domain xml | |
| 14:35:50 | sean-k-mooney | but nothing has created in in sysfs yet | |