| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-15 | |||
| 14:18:55 | sean-k-mooney | bauzas: and track the mdevs in the pci_devices table or similar | |
| 14:18:55 | sean-k-mooney | bauzas: and track the mdevs in the pci_devices table or similar | |
| 14:19:10 | bauzas | sean-k-mooney: honestly, mdevs are like VFs | |
| 14:19:10 | bauzas | sean-k-mooney: honestly, mdevs are like VFs | |
| 14:19:14 | sean-k-mooney | yep | |
| 14:19:17 | bauzas | and we don't persist the latter | |
| 14:19:17 | bauzas | and we don't persist the latter | |
| 14:19:22 | sean-k-mooney | we do | |
| 14:19:22 | sean-k-mooney | we do | |
| 14:19:26 | bauzas | in nova ? | |
| 14:19:26 | bauzas | in nova ? | |
| 14:19:29 | sean-k-mooney | in the pci_devices table in nova | |
| 14:19:29 | sean-k-mooney | in the pci_devices table in nova | |
| 14:19:40 | bauzas | but you need to precreate them, right? | |
| 14:19:40 | bauzas | but you need to precreate them, right? | |
| 14:19:43 | sean-k-mooney | thats what the pci_tracker does | |
| 14:19:43 | sean-k-mooney | thats what the pci_tracker does | |
| 14:19:49 | sean-k-mooney | bauzas: oh yes you do | |
| 14:19:49 | sean-k-mooney | bauzas: oh yes you do | |
| 14:19:55 | bauzas | the pci trackers tracks the VFs | |
| 14:19:55 | bauzas | the pci trackers tracks the VFs | |
| 14:20:05 | bauzas | but it doesn't create them, right? | |
| 14:20:05 | bauzas | but it doesn't create them, right? | |
| 14:20:07 | sean-k-mooney | yep so for vf the operator has to precreate them | |
| 14:20:07 | sean-k-mooney | yep so for vf the operator has to precreate them | |
| 14:20:13 | bauzas | that's my point | |
| 14:20:13 | mnaser | would it make sense to have something like if len(mdev) == 0: <check with placement if vm has vgpu>; if <system-has-vgpu>: find_an_unused_mdev_or_create_a_new_one(); | |
| 14:20:13 | bauzas | that's my point | |
| 14:20:13 | mnaser | would it make sense to have something like if len(mdev) == 0: <check with placement if vm has vgpu>; if <system-has-vgpu>: find_an_unused_mdev_or_create_a_new_one(); | |
| 14:20:23 | sean-k-mooney | we chose not to do that for mdevs for some reason | |
| 14:20:23 | sean-k-mooney | we chose not to do that for mdevs for some reason | |
| 14:20:37 | bauzas | mnaser: what I *could* do is to work on what sean-k-mooney and I said | |
| 14:20:37 | bauzas | mnaser: what I *could* do is to work on what sean-k-mooney and I said | |
| 14:20:39 | sean-k-mooney | but since we chose to create them it meas we should alwasy do it | |
| 14:20:39 | sean-k-mooney | but since we chose to create them it meas we should alwasy do it | |
| 14:20:53 | bauzas | mnaser: ie. looking up the placement DB | |
| 14:20:59 | bauzas | mnaser: ie. looking up the placement DB | |
| 14:21:05 | bauzas | and magically recreating them | |
| 14:21:05 | bauzas | and magically recreating them | |
| 14:21:16 | bauzas | mnaser: that's why I left the bug open | |
| 14:21:16 | bauzas | mnaser: that's why I left the bug open | |
| 14:21:26 | mnaser | and that would pretty much get rid of the statefulness of libvirt domain xml again | |
| 14:21:26 | mnaser | and that would pretty much get rid of the statefulness of libvirt domain xml again | |
| 14:21:35 | bauzas | mnaser: but the fact is, maybe eventually we would remove this whole recreate method | |
| 14:21:35 | bauzas | mnaser: but the fact is, maybe eventually we would remove this whole recreate method | |
| 14:22:09 | sean-k-mooney | bauzas: i would be ok using mdevctl if we moved mdev to the pci track or resouces table | |
| 14:22:09 | sean-k-mooney | bauzas: i would be ok using mdevctl if we moved mdev to the pci track or resouces table | |
| 14:22:15 | bauzas | mnaser: if the libvirt domain information would persist the mdev type, that'd be awesome | |
| 14:22:15 | bauzas | mnaser: if the libvirt domain information would persist the mdev type, that'd be awesome | |
| 14:22:23 | sean-k-mooney | and then just getting rid of this code and not needing the domain | |
| 14:22:23 | sean-k-mooney | and then just getting rid of this code and not needing the domain | |
| 14:22:27 | bauzas | mnaser: but it doesn't | |
| 14:22:27 | bauzas | mnaser: but it doesn't | |
| 14:23:03 | bauzas | sean-k-mooney: IIRC, aw (the mdevctl developer) was against using it for upper tooling | |
| 14:23:03 | bauzas | sean-k-mooney: IIRC, aw (the mdevctl developer) was against using it for upper tooling | |
| 14:23:08 | sean-k-mooney | bauzas: if we really needed too we could store it in the metadta section of the xml | |
| 14:23:08 | sean-k-mooney | bauzas: if we really needed too we could store it in the metadta section of the xml | |
| 14:23:31 | bauzas | sean-k-mooney: that's actually a great point | |
| 14:23:32 | bauzas | sean-k-mooney: that's actually a great point | |
| 14:23:33 | mnaser | is there anthing else we store in the xml as a state? | |
| 14:23:33 | mnaser | is there anthing else we store in the xml as a state? | |
| 14:23:50 | sean-k-mooney | mnaser: no | |
| 14:23:50 | sean-k-mooney | mnaser: no | |
| 14:24:02 | sean-k-mooney | long term we want to get rid of persitent domains | |
| 14:24:02 | sean-k-mooney | long term we want to get rid of persitent domains | |
| 14:24:07 | mnaser | only thing with this is if something goes wrong with libvirt or anything, you would lose all your gpus | |
| 14:24:07 | mnaser | only thing with this is if something goes wrong with libvirt or anything, you would lose all your gpus | |
| 14:24:11 | sean-k-mooney | e.g. the domain xml on disk | |
| 14:24:11 | sean-k-mooney | e.g. the domain xml on disk | |
| 14:24:15 | bauzas | sean-k-mooney: the only problem with metadata is that we won't recreate it on move operations | |
| 14:24:15 | bauzas | sean-k-mooney: the only problem with metadata is that we won't recreate it on move operations | |
| 14:24:39 | mnaser | thats how this bit us, nova wouldn't start, so we tried to undefine the domain to let nova recreate it, and here we are with no vgpus | |
| 14:24:39 | mnaser | thats how this bit us, nova wouldn't start, so we tried to undefine the domain to let nova recreate it, and here we are with no vgpus | |
| 14:24:59 | bauzas | mnaser: nova just binds mdevs | |
| 14:24:59 | bauzas | mnaser: nova just binds mdevs | |
| 14:25:24 | mnaser | but if libvirt domain is gone, it doesnt know which mdevs were assigned to that vm, even on a hard reboot | |
| 14:25:24 | mnaser | but if libvirt domain is gone, it doesnt know which mdevs were assigned to that vm, even on a hard reboot | |
| 14:25:30 | sean-k-mooney | so fundimentally i think we need to revisit using the xml for state storage | |
| 14:25:30 | sean-k-mooney | so fundimentally i think we need to revisit using the xml for state storage | |
| 14:25:49 | bauzas | mnaser: sure, but why would you undefine the domain ? | |
| 14:25:49 | bauzas | mnaser: sure, but why would you undefine the domain ? | |
| 14:25:51 | sean-k-mooney | and just store the inf in the nova db eventualy | |
| 14:25:51 | sean-k-mooney | and just store the inf in the nova db eventualy | |
| 14:25:57 | bauzas | sean-k-mooney: eeeek | |
| 14:25:57 | bauzas | sean-k-mooney: eeeek | |
| 14:26:07 | mnaser | bauzas: we had other issues why the domain would not start, because a call to libvirt was failing because the mdev was missing | |
| 14:26:07 | mnaser | bauzas: we had other issues why the domain would not start, because a call to libvirt was failing because the mdev was missing | |
| 14:26:14 | sean-k-mooney | bauzas: i didnt like using the xml for this in the first place | |
| 14:26:14 | sean-k-mooney | bauzas: i didnt like using the xml for this in the first place | |
| 14:26:26 | mnaser | so mdev was missing so nova couldnt start | |
| 14:26:26 | mnaser | so mdev was missing so nova couldnt start | |
| 14:26:29 | sean-k-mooney | bauzas: this probalem is just another rasons to not do it this way | |
| 14:26:29 | sean-k-mooney | bauzas: this probalem is just another rasons to not do it this way | |
| 14:26:50 | bauzas | mnaser: again, I can try to fix the logic by looking up placement | |
| 14:26:50 | bauzas | mnaser: again, I can try to fix the logic by looking up placement | |
| 14:27:10 | bauzas | sean-k-mooney: you know what ? I'll start filling a spec for drafting mdev management in nova | |
| 14:27:10 | bauzas | sean-k-mooney: you know what ? I'll start filling a spec for drafting mdev management in nova | |
| 14:27:21 | bauzas | and exposing them as raw resources | |
| 14:27:21 | bauzas | and exposing them as raw resources | |
| 14:27:23 | mnaser | so pretty much regenerate state from placement | |