| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-17 | |||
| 13:15:38 | bauzas | but first, we would need to support it by TripleO | |
| 13:15:40 | gibi | sean-k-mooney: does libvirt handles mdevs differently than pci? | |
| 13:15:49 | sean-k-mooney | right i dont think we shoudl do that not without providing a real alternitive | |
| 13:15:56 | sean-k-mooney | gibi: yes it does | |
| 13:16:03 | bauzas | gibi: mdevs are a bit different | |
| 13:16:06 | bauzas | but, | |
| 13:16:19 | sean-k-mooney | gibi: libvirt alows them to be allcoated dymically ti does not do the same for vfs | |
| 13:16:20 | bauzas | I don't see why we couldn't just tell 'sorry, but please precreate them' | |
| 13:16:41 | sean-k-mooney | bauzas: that would be a major regression in functionality for one | |
| 13:16:48 | sean-k-mooney | and be an upgrade issue | |
| 13:16:51 | bauzas | given we only support one type for GPU, when you tell which type you want to get for a pGPU, then we could just create them | |
| 13:17:01 | sean-k-mooney | but the main reason is most device dont support one type | |
| 13:17:04 | bauzas | anyway, I need to be off for 10 mins | |
| 13:17:13 | bauzas | (someone is around my home) | |
| 13:18:24 | sean-k-mooney | the whole we only support one type per PGPU is a limiatieon in nvidias solution that is not representivie of the mdev framework as a whole | |
| 13:18:58 | sean-k-mooney | if we support generic mdevs in teh future and in the cyborg case in particalar it dose not hold true that all device will only support 1 mdev type | |
| 13:22:42 | tacco | sean-k-mooney: fyi, still no luck. but all in all some more informations. I can see 248available cpus in the vm but everything above id63 are marked as offline. | |
| 13:22:57 | gibi | hm, I more and more like the idea to hide this complexity from nova in cyborg. Then cyborg is free to pre-create or create on the fly the mdev at arq mind. Nova don't have to worry about it (except the recreation case at reboot, see my comment in the spec) | |
| 13:23:09 | gibi | s/mind/bind/ | |
| 13:23:25 | sean-k-mooney | tacco: ok so that is pointing to the guest kernel | |
| 13:23:43 | sean-k-mooney | tacco: well most likely the kernel | |
| 13:23:53 | sean-k-mooney | have you tried onlining them via /sys | |
| 13:24:18 | sean-k-mooney | gibi: yep i was noting that in the spec | |
| 13:24:37 | sean-k-mooney | solution 2 id fine if cyborg does the precreate when we bind at teh conductor | |
| 13:24:45 | sean-k-mooney | its not ok if we need to do an api call on the compute | |
| 13:25:13 | sean-k-mooney | they can also do option 3 and precreate them at agent start | |
| 13:25:16 | gibi | sean-k-mooney: yeah, I assumed that cyborg provides the mdev to as in the same way as provides the pci for an fpga | |
| 13:25:30 | gibi | s/to as/ to us/ | |
| 13:25:37 | sean-k-mooney | yep in the arq when it completes the binding | |
| 13:25:38 | tacco | sean-k-mooney: yes, but then i got a mem error. | |
| 13:25:44 | sean-k-mooney | whcih we poll for on the compute | |
| 13:25:45 | gibi | sean-k-mooney: exactly | |
| 13:25:58 | sean-k-mooney | so its really just the uuid which we then stick in the xml | |
| 13:27:20 | gibi | sean-k-mooney: and this uuid needs to be stable from nova perspective at reboot, or else nova needs to update the xml | |
| 13:27:40 | sean-k-mooney | well we regenerate teh xml form scratch on reboot so | |
| 13:27:52 | sean-k-mooney | it could change | |
| 13:27:56 | sean-k-mooney | we wont notice | |
| 13:28:13 | openstackgerrit | Merged openstack/nova-specs master: Support interface attach with qos ports https://review.opendev.org/755477 | |
| 13:28:19 | gibi | I mean hypervisor reboot not VM reboot | |
| 13:28:40 | sean-k-mooney | oh same power-on for reasons calls hard reboot | |
| 13:29:09 | gibi | hm, that could work then | |
| 13:29:11 | sean-k-mooney | reasons being we required spwan and hard-reboot before we required stop start in the virt driver | |
| 13:29:26 | sean-k-mooney | and we never changed libvirt because it just worked | |
| 13:30:17 | gibi | do we rebind or requery the arq from cyborg at VM hard reboot? | |
| 13:30:32 | gibi | we need that to pick up a the new uuid | |
| 13:31:05 | sean-k-mooney | that was a question i was going to ask in the spec | |
| 13:31:22 | sean-k-mooney | i cant rember if we get the arq or not | |
| 13:31:23 | bauzas | (still on and off, but I see your convo, will be back in 20 mins-ish) | |
| 13:31:44 | sean-k-mooney | we do not rebind ports on hard reboot | |
| 13:32:27 | sean-k-mooney | for cinder block devices lyarwood would have to correct me but i think we cached the attachmemnt but maybe we recreate them | |
| 13:32:43 | sean-k-mooney | gibi: point being that we had to solve this for fpgas already | |
| 13:32:55 | sean-k-mooney | so i think what ever we did there should be suffienct | |
| 13:33:04 | gibi | sean-k-mooney: yeah, I agree | |
| 13:33:22 | gibi | I have to jump on a call back later | |
| 13:33:34 | sean-k-mooney | cool | |
| 13:34:03 | sean-k-mooney | im leaving comments on the spec but i think we agree cyborg should handel the mdev creation and just tell us | |
| 13:34:44 | sean-k-mooney | where it dose that (agent start or bind) is up to them so long as it work for all lifecycle events | |
| 13:35:27 | lyarwood | sean-k-mooney: we don't recreate volume attachments on hard reboot | |
| 13:35:39 | gibi | sean-k-mooney: agre | |
| 13:35:40 | sean-k-mooney | we cache the conection info in the db | |
| 13:35:47 | sean-k-mooney | ? | |
| 13:35:48 | lyarwood | sean-k-mooney: the libvirt driver disconnects volumes from the host and reconnects them but that's it | |
| 13:35:58 | lyarwood | sean-k-mooney: yeah we cache the attachment id and associated connection_info | |
| 13:36:04 | sean-k-mooney | ya | |
| 13:36:16 | sean-k-mooney | so i tought we could hard reboot without external api calls | |
| 13:36:25 | lyarwood | sean-k-mooney: the only way to refresh that at the moment is via a move operation, I've argued that we should do it during a hard reboot in the past | |
| 13:36:28 | sean-k-mooney | but im not sure what the case is for cyborg | |
| 13:37:03 | sean-k-mooney | lyarwood: yep that is why i was asking i knew you wanted to change it but didnt knwo if you wanted to add or remove the caching | |
| 13:37:52 | lyarwood | sean-k-mooney: remove the connection_info caching | |
| 13:38:11 | lyarwood | sean-k-mooney: but if we ever get there is another thing | |
| 13:38:13 | sean-k-mooney | so we would then have a call to cinder on every hard reboot | |
| 13:38:22 | lyarwood | sean-k-mooney: yeah | |
| 13:38:41 | sean-k-mooney | i guess if the cinder service is down the storeage backend may also be down | |
| 13:39:01 | sean-k-mooney | so even with the cache we are not guareteed to be able to connect | |
| 13:39:23 | lyarwood | sean-k-mooney: really depends on the backend, most could be up still | |
| 13:39:40 | lyarwood | sean-k-mooney: we could at least attempt to refresh tbh | |
| 13:41:17 | sean-k-mooney | ya ceph and other likely are not running on the same hosts as the cinder services | |
| 14:01:29 | tacco | hm.. struggling aroung with aggregates, strange behavior. i tought if i create a aggreagate with a key and attatch that key to a flavor only hosts with the specific flavors are on the aggregate hosts. | |
| 14:02:06 | sean-k-mooney | tacco: only if you use the placement version | |
| 14:02:09 | tacco | or is there anything else to do? For now only a couple of HVs already have the new filter. But i guess only the hosts in the aggregate group needs this filter, or to be aware of. | |
| 14:02:18 | sean-k-mooney | if you use the fitler you need to add key=false to every other flavor | |
| 14:02:37 | tacco | ah.. ok, thanks. that could help for the moment :) | |
| 14:02:45 | sean-k-mooney | https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html | |
| 14:03:03 | tacco | i tought absence of this flavor also implies that it can not be on the hosts. | |
| 14:03:12 | tacco | thanks for pointing me there. | |
| 14:03:33 | sean-k-mooney | tacco: nope that what many assume but its not how it works | |
| 14:03:48 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: WIP - Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284 | |
| 14:04:19 | sean-k-mooney | tacco: this is basically unmaintained and im not sure if it still works(it should) but https://opendev.org/x/nfv-filters provdes a filter that does what you want | |
| 14:04:42 | sean-k-mooney | https://opendev.org/x/nfv-filters/src/branch/master/nfv_filters/nova/scheduler/filters/aggregate_instance_type_filter.py | |
| 14:04:58 | sean-k-mooney | docs are here https://opendev.org/x/nfv-filters/src/branch/master/doc/source/scheduler_filters/aggregate-instance-type-filter.rst | |
| 14:05:13 | sean-k-mooney | if you dont have placment avaiable with the prefilter then that shoudl bridge the gap | |
| 14:05:27 | tacco | ok, will read first and decide then. :) but.. yeah.. found a bug that was already reported. yay | |
| 14:06:08 | sean-k-mooney | its actully more powerful the what you can do with placemnt or the code in tree but its out of tree and not maintained by me really anymore | |
| 14:08:09 | tacco | yes.. can understand that. I would like to stick as close as i can to upstream and maintained versions. | |
| 14:09:21 | tacco | only if there is no other way out i would think about that.. but our first solution we had was way to full with cherry-picked stuff.. and deployment was only done by manual actions.. i hate it. Now we have a nice fully automated and functional setup i don't want to miss that anymore. :D | |
| 14:15:08 | lyarwood | exit | |
| 14:15:10 | lyarwood | argh | |
| 14:16:47 | kashyap | "You can leave anytime, but you can never exit." | |
| 14:50:22 | tacco | sean-k-mooney: where do i have to set the scheduler.enable_isolated_aggregate_filtering nova.conf on the HVs? | |
| 14:50:53 | sean-k-mooney | no nova.conf on the schduler | |