| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-15 | |||
| 14:16:48 | mnaser | sean-k-mooney / bauzas: i guess if i'm understanding correctly, the instance <=> mdev mapping is not stored inside nova anywhere so we rely on the state in the libvirt domain | |
| 14:16:48 | mnaser | sean-k-mooney / bauzas: i guess if i'm understanding correctly, the instance <=> mdev mapping is not stored inside nova anywhere so we rely on the state in the libvirt domain | |
| 14:17:05 | bauzas | that's right | |
| 14:17:05 | bauzas | that's right | |
| 14:17:17 | bauzas | there is an existing tool tho | |
| 14:17:17 | bauzas | there is an existing tool tho | |
| 14:17:31 | bauzas | mnaser: https://github.com/mdevctl/mdevctl | |
| 14:17:31 | bauzas | mnaser: https://github.com/mdevctl/mdevctl | |
| 14:17:47 | bauzas | mnaser: you could use it for precreating the mdevs and persist them (using systemctl) | |
| 14:17:47 | bauzas | mnaser: you could use it for precreating the mdevs and persist them (using systemctl) | |
| 14:18:05 | mnaser | "When a known parent device add udev event occurs (or, for more recent kernels, change events with MDEV_STATE values), mdevctl is called by a udev rule to create defined devices with "start": "auto" configured." interesting | |
| 14:18:05 | mnaser | "When a known parent device add udev event occurs (or, for more recent kernels, change events with MDEV_STATE values), mdevctl is called by a udev rule to create defined devices with "start": "auto" configured." interesting | |
| 14:18:06 | bauzas | (whoops, systemd) | |
| 14:18:06 | bauzas | (whoops, systemd) | |
| 14:18:10 | sean-k-mooney | you could but long term we dont want peopel to do that | |
| 14:18:10 | sean-k-mooney | you could but long term we dont want peopel to do that | |
| 14:18:38 | sean-k-mooney | bauzas: if we were to go down that route we shoudl remvoe the code for nova to do it | |
| 14:18:38 | sean-k-mooney | bauzas: if we were to go down that route we shoudl remvoe the code for nova to do it | |
| 14:18:54 | bauzas | sean-k-mooney: we said this before | |
| 14:18:54 | bauzas | sean-k-mooney: we said this before | |
| 14:18:55 | sean-k-mooney | bauzas: and track the mdevs in the pci_devices table or similar | |
| 14:18:55 | sean-k-mooney | bauzas: and track the mdevs in the pci_devices table or similar | |
| 14:19:10 | bauzas | sean-k-mooney: honestly, mdevs are like VFs | |
| 14:19:10 | bauzas | sean-k-mooney: honestly, mdevs are like VFs | |
| 14:19:14 | sean-k-mooney | yep | |
| 14:19:17 | bauzas | and we don't persist the latter | |
| 14:19:17 | bauzas | and we don't persist the latter | |
| 14:19:22 | sean-k-mooney | we do | |
| 14:19:22 | sean-k-mooney | we do | |
| 14:19:26 | bauzas | in nova ? | |
| 14:19:26 | bauzas | in nova ? | |
| 14:19:29 | sean-k-mooney | in the pci_devices table in nova | |
| 14:19:29 | sean-k-mooney | in the pci_devices table in nova | |
| 14:19:40 | bauzas | but you need to precreate them, right? | |
| 14:19:40 | bauzas | but you need to precreate them, right? | |
| 14:19:43 | sean-k-mooney | thats what the pci_tracker does | |
| 14:19:43 | sean-k-mooney | thats what the pci_tracker does | |
| 14:19:49 | sean-k-mooney | bauzas: oh yes you do | |
| 14:19:49 | sean-k-mooney | bauzas: oh yes you do | |
| 14:19:55 | bauzas | the pci trackers tracks the VFs | |
| 14:19:55 | bauzas | the pci trackers tracks the VFs | |
| 14:20:05 | bauzas | but it doesn't create them, right? | |
| 14:20:05 | bauzas | but it doesn't create them, right? | |
| 14:20:07 | sean-k-mooney | yep so for vf the operator has to precreate them | |
| 14:20:07 | sean-k-mooney | yep so for vf the operator has to precreate them | |
| 14:20:13 | bauzas | that's my point | |
| 14:20:13 | mnaser | would it make sense to have something like if len(mdev) == 0: <check with placement if vm has vgpu>; if <system-has-vgpu>: find_an_unused_mdev_or_create_a_new_one(); | |
| 14:20:13 | bauzas | that's my point | |
| 14:20:13 | mnaser | would it make sense to have something like if len(mdev) == 0: <check with placement if vm has vgpu>; if <system-has-vgpu>: find_an_unused_mdev_or_create_a_new_one(); | |
| 14:20:23 | sean-k-mooney | we chose not to do that for mdevs for some reason | |
| 14:20:23 | sean-k-mooney | we chose not to do that for mdevs for some reason | |
| 14:20:37 | bauzas | mnaser: what I *could* do is to work on what sean-k-mooney and I said | |
| 14:20:37 | bauzas | mnaser: what I *could* do is to work on what sean-k-mooney and I said | |
| 14:20:39 | sean-k-mooney | but since we chose to create them it meas we should alwasy do it | |
| 14:20:39 | sean-k-mooney | but since we chose to create them it meas we should alwasy do it | |
| 14:20:53 | bauzas | mnaser: ie. looking up the placement DB | |
| 14:20:59 | bauzas | mnaser: ie. looking up the placement DB | |
| 14:21:05 | bauzas | and magically recreating them | |
| 14:21:05 | bauzas | and magically recreating them | |
| 14:21:16 | bauzas | mnaser: that's why I left the bug open | |
| 14:21:16 | bauzas | mnaser: that's why I left the bug open | |
| 14:21:26 | mnaser | and that would pretty much get rid of the statefulness of libvirt domain xml again | |
| 14:21:26 | mnaser | and that would pretty much get rid of the statefulness of libvirt domain xml again | |
| 14:21:35 | bauzas | mnaser: but the fact is, maybe eventually we would remove this whole recreate method | |
| 14:21:35 | bauzas | mnaser: but the fact is, maybe eventually we would remove this whole recreate method | |
| 14:22:09 | sean-k-mooney | bauzas: i would be ok using mdevctl if we moved mdev to the pci track or resouces table | |
| 14:22:09 | sean-k-mooney | bauzas: i would be ok using mdevctl if we moved mdev to the pci track or resouces table | |
| 14:22:15 | bauzas | mnaser: if the libvirt domain information would persist the mdev type, that'd be awesome | |
| 14:22:15 | bauzas | mnaser: if the libvirt domain information would persist the mdev type, that'd be awesome | |
| 14:22:23 | sean-k-mooney | and then just getting rid of this code and not needing the domain | |
| 14:22:23 | sean-k-mooney | and then just getting rid of this code and not needing the domain | |
| 14:22:27 | bauzas | mnaser: but it doesn't | |
| 14:22:27 | bauzas | mnaser: but it doesn't | |
| 14:23:03 | bauzas | sean-k-mooney: IIRC, aw (the mdevctl developer) was against using it for upper tooling | |
| 14:23:03 | bauzas | sean-k-mooney: IIRC, aw (the mdevctl developer) was against using it for upper tooling | |
| 14:23:08 | sean-k-mooney | bauzas: if we really needed too we could store it in the metadta section of the xml | |
| 14:23:08 | sean-k-mooney | bauzas: if we really needed too we could store it in the metadta section of the xml | |
| 14:23:31 | bauzas | sean-k-mooney: that's actually a great point | |
| 14:23:32 | bauzas | sean-k-mooney: that's actually a great point | |
| 14:23:33 | mnaser | is there anthing else we store in the xml as a state? | |
| 14:23:33 | mnaser | is there anthing else we store in the xml as a state? | |
| 14:23:50 | sean-k-mooney | mnaser: no | |
| 14:23:50 | sean-k-mooney | mnaser: no | |
| 14:24:02 | sean-k-mooney | long term we want to get rid of persitent domains | |
| 14:24:02 | sean-k-mooney | long term we want to get rid of persitent domains | |
| 14:24:07 | mnaser | only thing with this is if something goes wrong with libvirt or anything, you would lose all your gpus | |
| 14:24:07 | mnaser | only thing with this is if something goes wrong with libvirt or anything, you would lose all your gpus | |
| 14:24:11 | sean-k-mooney | e.g. the domain xml on disk | |
| 14:24:11 | sean-k-mooney | e.g. the domain xml on disk | |
| 14:24:15 | bauzas | sean-k-mooney: the only problem with metadata is that we won't recreate it on move operations | |
| 14:24:15 | bauzas | sean-k-mooney: the only problem with metadata is that we won't recreate it on move operations | |
| 14:24:39 | mnaser | thats how this bit us, nova wouldn't start, so we tried to undefine the domain to let nova recreate it, and here we are with no vgpus | |
| 14:24:39 | mnaser | thats how this bit us, nova wouldn't start, so we tried to undefine the domain to let nova recreate it, and here we are with no vgpus | |
| 14:24:59 | bauzas | mnaser: nova just binds mdevs | |
| 14:24:59 | bauzas | mnaser: nova just binds mdevs | |
| 14:25:24 | mnaser | but if libvirt domain is gone, it doesnt know which mdevs were assigned to that vm, even on a hard reboot | |
| 14:25:24 | mnaser | but if libvirt domain is gone, it doesnt know which mdevs were assigned to that vm, even on a hard reboot | |
| 14:25:30 | sean-k-mooney | so fundimentally i think we need to revisit using the xml for state storage | |
| 14:25:30 | sean-k-mooney | so fundimentally i think we need to revisit using the xml for state storage | |
| 14:25:49 | bauzas | mnaser: sure, but why would you undefine the domain ? | |