| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-17 | |||
| 12:44:50 | sean-k-mooney | /stell/still/ | |
| 12:46:40 | bauzas | Yumeng: fwiw, I already provided my thoughts | |
| 12:46:52 | bauzas | and i prefer a solution with cyborg precreating the mdevs | |
| 12:47:13 | bauzas | (and just ask nova to use an existing mdev) | |
| 12:48:46 | gibi | bauzas, Yumeng: I've also just pushed my comments on the vGPU spec | |
| 12:49:43 | Yumeng | bauzas: I am reviewing your comments now. yes, solution 2 in the spec is a cyborg precreating the mdevs. seems that's what you guys like. | |
| 12:50:39 | Yumeng | bauzas: ok.thank you, I will take a look first | |
| 12:50:50 | Yumeng | gibi: ok.thank you, I will take a look first | |
| 13:01:07 | gibi | bauzas: if you want an easy spec: https://review.opendev.org/#/c/755477/ :) | |
| 13:11:10 | bauzas | gibi: nit : s/Sylvian/Sylvain (Sylvian(e) is a woman name ;) ) | |
| 13:11:24 | gibi | bauzas: sorry | |
| 13:11:54 | bauzas | gibi: heh no worries ;) | |
| 13:12:22 | sean-k-mooney | gibi: bauzas i dont think we shoudl be treating mdevs and vf similarly unless you are suggesting we start tracking mdevs in teh pci tracker | |
| 13:12:47 | sean-k-mooney | so that we persist them there and have nova precreate them | |
| 13:13:02 | sean-k-mooney | to provide stable names | |
| 13:13:09 | sean-k-mooney | and other things like numa affinity | |
| 13:13:29 | sean-k-mooney | e.g. just add a type_mdev to the list | |
| 13:13:37 | bauzas | why ? | |
| 13:13:46 | bauzas | anyway, I need to look at other specs | |
| 13:14:08 | gibi | sean-k-mooney: I don't suggest to plug mdevs into the pci tracker, especially that cyborgs would like to track them | |
| 13:14:21 | sean-k-mooney | because mdev are not ment to work like vf in that they framework was desigined to have them dynmicaly created at runtime by higher level orcestors | |
| 13:14:59 | sean-k-mooney | gibi: well my point is that without cycborg i dont think its valid to say they should be precreated | |
| 13:15:00 | bauzas | gibi: me too | |
| 13:15:13 | sean-k-mooney | that is just not how they were ment to be used | |
| 13:15:20 | bauzas | gibi: sean-k-mooney: honestly, I would like to stop supporting to create mdevs in nova | |
| 13:15:38 | bauzas | but first, we would need to support it by TripleO | |
| 13:15:40 | gibi | sean-k-mooney: does libvirt handles mdevs differently than pci? | |
| 13:15:49 | sean-k-mooney | right i dont think we shoudl do that not without providing a real alternitive | |
| 13:15:56 | sean-k-mooney | gibi: yes it does | |
| 13:16:03 | bauzas | gibi: mdevs are a bit different | |
| 13:16:06 | bauzas | but, | |
| 13:16:19 | sean-k-mooney | gibi: libvirt alows them to be allcoated dymically ti does not do the same for vfs | |
| 13:16:20 | bauzas | I don't see why we couldn't just tell 'sorry, but please precreate them' | |
| 13:16:41 | sean-k-mooney | bauzas: that would be a major regression in functionality for one | |
| 13:16:48 | sean-k-mooney | and be an upgrade issue | |
| 13:16:51 | bauzas | given we only support one type for GPU, when you tell which type you want to get for a pGPU, then we could just create them | |
| 13:17:01 | sean-k-mooney | but the main reason is most device dont support one type | |
| 13:17:04 | bauzas | anyway, I need to be off for 10 mins | |
| 13:17:13 | bauzas | (someone is around my home) | |
| 13:18:24 | sean-k-mooney | the whole we only support one type per PGPU is a limiatieon in nvidias solution that is not representivie of the mdev framework as a whole | |
| 13:18:58 | sean-k-mooney | if we support generic mdevs in teh future and in the cyborg case in particalar it dose not hold true that all device will only support 1 mdev type | |
| 13:22:42 | tacco | sean-k-mooney: fyi, still no luck. but all in all some more informations. I can see 248available cpus in the vm but everything above id63 are marked as offline. | |
| 13:22:57 | gibi | hm, I more and more like the idea to hide this complexity from nova in cyborg. Then cyborg is free to pre-create or create on the fly the mdev at arq mind. Nova don't have to worry about it (except the recreation case at reboot, see my comment in the spec) | |
| 13:23:09 | gibi | s/mind/bind/ | |
| 13:23:25 | sean-k-mooney | tacco: ok so that is pointing to the guest kernel | |
| 13:23:43 | sean-k-mooney | tacco: well most likely the kernel | |
| 13:23:53 | sean-k-mooney | have you tried onlining them via /sys | |
| 13:24:18 | sean-k-mooney | gibi: yep i was noting that in the spec | |
| 13:24:37 | sean-k-mooney | solution 2 id fine if cyborg does the precreate when we bind at teh conductor | |
| 13:24:45 | sean-k-mooney | its not ok if we need to do an api call on the compute | |
| 13:25:13 | sean-k-mooney | they can also do option 3 and precreate them at agent start | |
| 13:25:16 | gibi | sean-k-mooney: yeah, I assumed that cyborg provides the mdev to as in the same way as provides the pci for an fpga | |
| 13:25:30 | gibi | s/to as/ to us/ | |
| 13:25:37 | sean-k-mooney | yep in the arq when it completes the binding | |
| 13:25:38 | tacco | sean-k-mooney: yes, but then i got a mem error. | |
| 13:25:44 | sean-k-mooney | whcih we poll for on the compute | |
| 13:25:45 | gibi | sean-k-mooney: exactly | |
| 13:25:58 | sean-k-mooney | so its really just the uuid which we then stick in the xml | |
| 13:27:20 | gibi | sean-k-mooney: and this uuid needs to be stable from nova perspective at reboot, or else nova needs to update the xml | |
| 13:27:40 | sean-k-mooney | well we regenerate teh xml form scratch on reboot so | |
| 13:27:52 | sean-k-mooney | it could change | |
| 13:27:56 | sean-k-mooney | we wont notice | |
| 13:28:13 | openstackgerrit | Merged openstack/nova-specs master: Support interface attach with qos ports https://review.opendev.org/755477 | |
| 13:28:19 | gibi | I mean hypervisor reboot not VM reboot | |
| 13:28:40 | sean-k-mooney | oh same power-on for reasons calls hard reboot | |
| 13:29:09 | gibi | hm, that could work then | |
| 13:29:11 | sean-k-mooney | reasons being we required spwan and hard-reboot before we required stop start in the virt driver | |
| 13:29:26 | sean-k-mooney | and we never changed libvirt because it just worked | |
| 13:30:17 | gibi | do we rebind or requery the arq from cyborg at VM hard reboot? | |
| 13:30:32 | gibi | we need that to pick up a the new uuid | |
| 13:31:05 | sean-k-mooney | that was a question i was going to ask in the spec | |
| 13:31:22 | sean-k-mooney | i cant rember if we get the arq or not | |
| 13:31:23 | bauzas | (still on and off, but I see your convo, will be back in 20 mins-ish) | |
| 13:31:44 | sean-k-mooney | we do not rebind ports on hard reboot | |
| 13:32:27 | sean-k-mooney | for cinder block devices lyarwood would have to correct me but i think we cached the attachmemnt but maybe we recreate them | |
| 13:32:43 | sean-k-mooney | gibi: point being that we had to solve this for fpgas already | |
| 13:32:55 | sean-k-mooney | so i think what ever we did there should be suffienct | |
| 13:33:04 | gibi | sean-k-mooney: yeah, I agree | |
| 13:33:22 | gibi | I have to jump on a call back later | |
| 13:33:34 | sean-k-mooney | cool | |
| 13:34:03 | sean-k-mooney | im leaving comments on the spec but i think we agree cyborg should handel the mdev creation and just tell us | |
| 13:34:44 | sean-k-mooney | where it dose that (agent start or bind) is up to them so long as it work for all lifecycle events | |
| 13:35:27 | lyarwood | sean-k-mooney: we don't recreate volume attachments on hard reboot | |
| 13:35:39 | gibi | sean-k-mooney: agre | |
| 13:35:40 | sean-k-mooney | we cache the conection info in the db | |
| 13:35:47 | sean-k-mooney | ? | |
| 13:35:48 | lyarwood | sean-k-mooney: the libvirt driver disconnects volumes from the host and reconnects them but that's it | |
| 13:35:58 | lyarwood | sean-k-mooney: yeah we cache the attachment id and associated connection_info | |
| 13:36:04 | sean-k-mooney | ya | |
| 13:36:16 | sean-k-mooney | so i tought we could hard reboot without external api calls | |
| 13:36:25 | lyarwood | sean-k-mooney: the only way to refresh that at the moment is via a move operation, I've argued that we should do it during a hard reboot in the past | |
| 13:36:28 | sean-k-mooney | but im not sure what the case is for cyborg | |
| 13:37:03 | sean-k-mooney | lyarwood: yep that is why i was asking i knew you wanted to change it but didnt knwo if you wanted to add or remove the caching | |
| 13:37:52 | lyarwood | sean-k-mooney: remove the connection_info caching | |
| 13:38:11 | lyarwood | sean-k-mooney: but if we ever get there is another thing | |
| 13:38:13 | sean-k-mooney | so we would then have a call to cinder on every hard reboot | |
| 13:38:22 | lyarwood | sean-k-mooney: yeah | |
| 13:38:41 | sean-k-mooney | i guess if the cinder service is down the storeage backend may also be down | |
| 13:39:01 | sean-k-mooney | so even with the cache we are not guareteed to be able to connect | |
| 13:39:23 | lyarwood | sean-k-mooney: really depends on the backend, most could be up still | |