| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-15 | |||
| 13:58:25 | bauzas | for a RP | |
| 13:58:25 | bauzas | for a RP | |
| 13:58:32 | bauzas | which is a pGPU | |
| 13:58:32 | bauzas | which is a pGPU | |
| 13:58:52 | bauzas | so then we would need to look at the conf option to know which type it uses | |
| 13:58:52 | bauzas | so then we would need to look at the conf option to know which type it uses | |
| 13:58:59 | sean-k-mooney | yep | |
| 13:59:03 | bauzas | that *could* work | |
| 13:59:03 | bauzas | that *could* work | |
| 13:59:23 | bauzas | but that's a long change I think | |
| 13:59:23 | bauzas | but that's a long change I think | |
| 13:59:55 | sean-k-mooney | you mean complex to write/test | |
| 13:59:55 | sean-k-mooney | you mean complex to write/test | |
| 14:00:19 | sean-k-mooney | i think its what is required though unless we start storing the infor in the nova db in the resouces table for example | |
| 14:00:19 | sean-k-mooney | i think its what is required though unless we start storing the infor in the nova db in the resouces table for example | |
| 14:00:52 | sean-k-mooney | those are our two options caulate it form plamcnet or record mdevs in the db like pmem or pcidevices | |
| 14:00:52 | sean-k-mooney | those are our two options caulate it form plamcnet or record mdevs in the db like pmem or pcidevices | |
| 14:01:17 | sean-k-mooney | so that we can just look it up | |
| 14:01:17 | sean-k-mooney | so that we can just look it up | |
| 14:03:12 | bauzas | I can try to help | |
| 14:03:12 | bauzas | I can try to help | |
| 14:05:01 | sean-k-mooney | mnaser: it would be a bit of a hack but you could proably fix this with a bash script executed by a systemd service file tempoarlly | |
| 14:05:01 | sean-k-mooney | mnaser: it would be a bit of a hack but you could proably fix this with a bash script executed by a systemd service file tempoarlly | |
| 14:05:29 | sean-k-mooney | basically implemented the same logic | |
| 14:05:29 | sean-k-mooney | basically implemented the same logic | |
| 14:06:37 | sean-k-mooney | loop over the domains and for each with an mdev look up the placment allocation and get the rp with the vgpu resouces | |
| 14:06:37 | sean-k-mooney | loop over the domains and for each with an mdev look up the placment allocation and get the rp with the vgpu resouces | |
| 14:07:11 | sean-k-mooney | then get the mdev type and create it with the same mdev uuid as the xml currenly has | |
| 14:07:11 | sean-k-mooney | then get the mdev type and create it with the same mdev uuid as the xml currenly has | |
| 14:08:04 | sean-k-mooney | you could use systemd's "before" and "after" requirements to ensure it runs before nova-compute and after libvirt start | |
| 14:08:04 | sean-k-mooney | you could use systemd's "before" and "after" requirements to ensure it runs before nova-compute and after libvirt start | |
| 14:08:58 | sean-k-mooney | really nova should do that but that the bug your hitting i guess. | |
| 14:08:58 | sean-k-mooney | really nova should do that but that the bug your hitting i guess. | |
| 14:16:48 | mnaser | sean-k-mooney / bauzas: i guess if i'm understanding correctly, the instance <=> mdev mapping is not stored inside nova anywhere so we rely on the state in the libvirt domain | |
| 14:16:48 | mnaser | sean-k-mooney / bauzas: i guess if i'm understanding correctly, the instance <=> mdev mapping is not stored inside nova anywhere so we rely on the state in the libvirt domain | |
| 14:17:05 | bauzas | that's right | |
| 14:17:05 | bauzas | that's right | |
| 14:17:17 | bauzas | there is an existing tool tho | |
| 14:17:17 | bauzas | there is an existing tool tho | |
| 14:17:31 | bauzas | mnaser: https://github.com/mdevctl/mdevctl | |
| 14:17:31 | bauzas | mnaser: https://github.com/mdevctl/mdevctl | |
| 14:17:47 | bauzas | mnaser: you could use it for precreating the mdevs and persist them (using systemctl) | |
| 14:17:47 | bauzas | mnaser: you could use it for precreating the mdevs and persist them (using systemctl) | |
| 14:18:05 | mnaser | "When a known parent device add udev event occurs (or, for more recent kernels, change events with MDEV_STATE values), mdevctl is called by a udev rule to create defined devices with "start": "auto" configured." interesting | |
| 14:18:05 | mnaser | "When a known parent device add udev event occurs (or, for more recent kernels, change events with MDEV_STATE values), mdevctl is called by a udev rule to create defined devices with "start": "auto" configured." interesting | |
| 14:18:06 | bauzas | (whoops, systemd) | |
| 14:18:06 | bauzas | (whoops, systemd) | |
| 14:18:10 | sean-k-mooney | you could but long term we dont want peopel to do that | |
| 14:18:10 | sean-k-mooney | you could but long term we dont want peopel to do that | |
| 14:18:38 | sean-k-mooney | bauzas: if we were to go down that route we shoudl remvoe the code for nova to do it | |
| 14:18:38 | sean-k-mooney | bauzas: if we were to go down that route we shoudl remvoe the code for nova to do it | |
| 14:18:54 | bauzas | sean-k-mooney: we said this before | |
| 14:18:54 | bauzas | sean-k-mooney: we said this before | |
| 14:18:55 | sean-k-mooney | bauzas: and track the mdevs in the pci_devices table or similar | |
| 14:18:55 | sean-k-mooney | bauzas: and track the mdevs in the pci_devices table or similar | |
| 14:19:10 | bauzas | sean-k-mooney: honestly, mdevs are like VFs | |
| 14:19:10 | bauzas | sean-k-mooney: honestly, mdevs are like VFs | |
| 14:19:14 | sean-k-mooney | yep | |
| 14:19:17 | bauzas | and we don't persist the latter | |
| 14:19:17 | bauzas | and we don't persist the latter | |
| 14:19:22 | sean-k-mooney | we do | |
| 14:19:22 | sean-k-mooney | we do | |
| 14:19:26 | bauzas | in nova ? | |
| 14:19:26 | bauzas | in nova ? | |
| 14:19:29 | sean-k-mooney | in the pci_devices table in nova | |
| 14:19:29 | sean-k-mooney | in the pci_devices table in nova | |
| 14:19:40 | bauzas | but you need to precreate them, right? | |
| 14:19:40 | bauzas | but you need to precreate them, right? | |
| 14:19:43 | sean-k-mooney | thats what the pci_tracker does | |
| 14:19:43 | sean-k-mooney | thats what the pci_tracker does | |
| 14:19:49 | sean-k-mooney | bauzas: oh yes you do | |
| 14:19:49 | sean-k-mooney | bauzas: oh yes you do | |
| 14:19:55 | bauzas | the pci trackers tracks the VFs | |
| 14:19:55 | bauzas | the pci trackers tracks the VFs | |
| 14:20:05 | bauzas | but it doesn't create them, right? | |
| 14:20:05 | bauzas | but it doesn't create them, right? | |
| 14:20:07 | sean-k-mooney | yep so for vf the operator has to precreate them | |
| 14:20:07 | sean-k-mooney | yep so for vf the operator has to precreate them | |
| 14:20:13 | bauzas | that's my point | |
| 14:20:13 | mnaser | would it make sense to have something like if len(mdev) == 0: <check with placement if vm has vgpu>; if <system-has-vgpu>: find_an_unused_mdev_or_create_a_new_one(); | |
| 14:20:13 | bauzas | that's my point | |
| 14:20:13 | mnaser | would it make sense to have something like if len(mdev) == 0: <check with placement if vm has vgpu>; if <system-has-vgpu>: find_an_unused_mdev_or_create_a_new_one(); | |
| 14:20:23 | sean-k-mooney | we chose not to do that for mdevs for some reason | |
| 14:20:23 | sean-k-mooney | we chose not to do that for mdevs for some reason | |
| 14:20:37 | bauzas | mnaser: what I *could* do is to work on what sean-k-mooney and I said | |
| 14:20:37 | bauzas | mnaser: what I *could* do is to work on what sean-k-mooney and I said | |
| 14:20:39 | sean-k-mooney | but since we chose to create them it meas we should alwasy do it | |
| 14:20:39 | sean-k-mooney | but since we chose to create them it meas we should alwasy do it | |
| 14:20:53 | bauzas | mnaser: ie. looking up the placement DB | |
| 14:20:59 | bauzas | mnaser: ie. looking up the placement DB | |
| 14:21:05 | bauzas | and magically recreating them | |
| 14:21:05 | bauzas | and magically recreating them | |
| 14:21:16 | bauzas | mnaser: that's why I left the bug open | |
| 14:21:16 | bauzas | mnaser: that's why I left the bug open | |
| 14:21:26 | mnaser | and that would pretty much get rid of the statefulness of libvirt domain xml again | |
| 14:21:26 | mnaser | and that would pretty much get rid of the statefulness of libvirt domain xml again | |
| 14:21:35 | bauzas | mnaser: but the fact is, maybe eventually we would remove this whole recreate method | |
| 14:21:35 | bauzas | mnaser: but the fact is, maybe eventually we would remove this whole recreate method | |
| 14:22:09 | sean-k-mooney | bauzas: i would be ok using mdevctl if we moved mdev to the pci track or resouces table | |
| 14:22:09 | sean-k-mooney | bauzas: i would be ok using mdevctl if we moved mdev to the pci track or resouces table | |