| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-21 | |||
| 10:30:17 | sean-k-mooney | and bug/<bug number> for bugs | |
| 10:30:39 | sean-k-mooney | otherwise its freeform | |
| 10:31:08 | sean-k-mooney | for feature that require changes in multiple project we try to use the same topic across all of them to be able to see all reated patches quickly | |
| 10:31:42 | sean-k-mooney | so now that you updated it https://review.opendev.org/q/topic:bp%252Fvirtio-packedring-configuration-support | |
| 10:32:00 | sean-k-mooney | we can see the spec, nova and os-traits patches all in one view | |
| 10:33:08 | sean-k-mooney | that tells me you are missing a patch to glance to update the metadefs https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json | |
| 10:33:48 | sean-k-mooney | dvo-plv_: ^ the metadefs are what horizon and heat use to generate the drop down for adding traits automatically | |
| 10:34:09 | sean-k-mooney | sorry not triats extra_specs and image properties | |
| 10:35:49 | dvo-plv_ | okay, I will investigate and fix | |
| 10:36:23 | sean-k-mooney | you basically jsut need to copy this https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L30-L35 | |
| 10:36:45 | sean-k-mooney | updatign mem_encyption to packed_ring | |
| 10:37:07 | sean-k-mooney | the namespce/prefix is automaticaly handeled by https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L7-L16 | |
| 10:37:49 | dvo-plv_ | okay, thanks. I would liek to check it on the horizon ui too | |
| 10:37:52 | sean-k-mooney | its also good to update the userful image properies doc https://github.com/openstack/glance/commit/3a281b9bc62a1b8b0f1468bc641105a5662f8ecd is an example | |
| 10:38:11 | dvo-plv_ | sure | |
| 10:38:18 | dvo-plv_ | Could you also clarify me with your scheduler according to the upstream work? We would like to start work over multiqueue support for hardware offloading | |
| 10:39:55 | sean-k-mooney | other can review your code not jsut me. but currently i have around 20% of my time for upstream work less fo the last 3-4 weeks because i was busy with downstream planing activies due to the ptg and other issues | |
| 10:40:54 | sean-k-mooney | i have been trying to capture all the outcomes of the ptg in our downstream jira and plannign the work for our team there | |
| 10:41:15 | sean-k-mooney | normally i have more time for upstream review so that shoudl go back to normal soon | |
| 10:41:48 | sean-k-mooney | regarding multi queue | |
| 10:42:14 | sean-k-mooney | do you mean https://review.opendev.org/c/openstack/nova-specs/+/855514 | |
| 10:42:26 | dvo-plv_ | Whom I should ping regarding review process. I saw gibi on some review process | |
| 10:43:13 | dvo-plv_ | Yes, we would like to analyze it better and provide some vision, how we can improve this in the OpenStack for hardware offloading, expecially for our nic | |
| 10:43:58 | sean-k-mooney | so multi queue is only partly implemented for hardware offload | |
| 10:44:15 | sean-k-mooney | i dont think it actully works we cannot enable it for nics that use sriov | |
| 10:44:26 | sean-k-mooney | i.e. nics that present the vf directly to the guest | |
| 10:44:37 | sean-k-mooney | but it might eb possibel for macvtap or vdpa | |
| 10:45:03 | sean-k-mooney | dvo-plv_: in terms of pings the active member of the nova core team | |
| 10:46:13 | sean-k-mooney | dvo-plv_: so gibi, bauzas, melwitt, gmann and dansmith are you best bet in addtion to me. stephenfin is also around somethimes but they mainly work on non nova related thigns day to day | |
| 10:47:51 | dvo-plv_ | okay, thanks | |
| 10:49:32 | sean-k-mooney | so looping back to multi queue | |
| 10:50:06 | sean-k-mooney | implementing this in the way you wanted is not going to be easy or really desireable form a nova point of view | |
| 10:51:19 | sean-k-mooney | there are a few parts to this problem | |
| 10:51:56 | sean-k-mooney | first we need to detach and recored the number of queues avaialbel in the vf | |
| 10:52:28 | sean-k-mooney | second we need to be able to schudle based on that (either updatign the pci filter or recording this in placement) | |
| 10:53:16 | sean-k-mooney | third we need a way to request a device with a min number of queuse multiup queuse out side the falvoar/image (likely on the nutron port) | |
| 10:54:09 | sean-k-mooney | finally we need to take the resouce request form teh port and include that in our scueduling reeust and wonce we find a host/device that meets that need we need to ensure that the qemu device is cofnigured correctly | |
| 10:54:37 | dvo-plv_ | regarding first question, we investigated it and the best option on our opinio is to parse other config. we configure queues like that -a 0000:65:00.0,representor=[4-6],portqueues=[4:2,5:2,6:2]. Yes it will work only for our nic | |
| 10:54:57 | sean-k-mooney | which config | |
| 10:55:01 | sean-k-mooney | the pci config space | |
| 10:55:15 | sean-k-mooney | you can useuslly get this form sysfs i tought | |
| 10:55:15 | dvo-plv_ | ovs. other_config | |
| 10:55:31 | sean-k-mooney | we cant do that | |
| 10:55:42 | sean-k-mooney | the ovs port wont exist at that point | |
| 10:55:45 | dvo-plv_ | no we can not, this is untrivial task for our dpdk driver | |
| 10:56:05 | sean-k-mooney | oh right so honestly you cant start this work | |
| 10:56:24 | sean-k-mooney | until the basic work of supporting the dpdk represtors is done | |
| 10:56:28 | dvo-plv_ | ovs port not, but vf yes. We would like to parse this config and fill it to the device_spec | |
| 10:56:54 | sean-k-mooney | the ovs port will be created and added by os-vif | |
| 10:57:11 | sean-k-mooney | only after we ahve selected a vf | |
| 10:57:46 | sean-k-mooney | so i think we need https://review.opendev.org/c/openstack/nova-specs/+/859290 to be done before we can talk about multiqueue | |
| 10:59:17 | dvo-plv_ | I see, we thought we can start to find solution for all comments to the blueprint in the parallel at the moment | |
| 10:59:47 | sean-k-mooney | well we could but its going to be diffuctly to compelte jsut one of the 3-4 specs you have propsoed this cycle | |
| 10:59:51 | sean-k-mooney | maybe 2 | |
| 11:00:00 | sean-k-mooney | its very unlikely that all of them will land | |
| 11:00:41 | dvo-plv_ | You mentined that multiqueue functional will be possible for vdpa. So maybe it will be better to move from virtio-forwarder to the vdpa nvic type for future purposes | |
| 11:01:17 | sean-k-mooney | well there is work in dpdk to supprot vdpa | |
| 11:01:31 | sean-k-mooney | i was expecting to have a vdpa-user type at some point for that | |
| 11:02:47 | sean-k-mooney | https://doc.dpdk.org/guides/vdpadevs/features_overview.html | |
| 11:02:52 | sean-k-mooney | i have not looked into it much | |
| 11:03:25 | sean-k-mooney | i dont think that is supported by ovs-dpdk currenlty but i have not really been following it closely | |
| 11:04:08 | sean-k-mooney | dvo-plv_: so for the basic enablement we are going to be trackign napatec VF which we will add to ovs as dpdk prots corret | |
| 11:04:34 | sean-k-mooney | and then those will be exposed to the guest as vhost-user ports | |
| 11:05:01 | sean-k-mooney | so for multi queue we would need to read the number of queues on the vf ideally | |
| 11:05:07 | dvo-plv_ | This multiqueue functional with queue mq and vector is in the our ovs fork at the moment | |
| 11:05:08 | sean-k-mooney | because we need that info for schduling | |
| 11:05:27 | sean-k-mooney | ok so thats kind of a problem | |
| 11:05:52 | sean-k-mooney | we do not really allow enablment of forked functionality in nova | |
| 11:06:11 | dvo-plv_ | Yes, I remember that it requires for placement to handle scheduler with queues number | |
| 11:06:57 | sean-k-mooney | if we can do it generically we enabel it so i was hoppng we coudl do somehting liek read /sys/bus/pci/device/<address>/num_queus or somehting like that | |
| 11:07:09 | sean-k-mooney | ideally vai libvirt nodedev api | |
| 11:07:15 | sean-k-mooney | not reading sys directly | |
| 11:09:18 | sean-k-mooney | so you can get the queue like this https://paste.opendev.org/show/bVSM5IDtJTRwhcIcuTFs/ | |
| 11:09:27 | sean-k-mooney | that is a pf | |
| 11:09:33 | sean-k-mooney | but i belive the same is true for VFs | |
| 11:11:44 | sean-k-mooney | i done see the queues in libvirt https://paste.opendev.org/show/bJHNfZR8JNpxJVvLggCI/ | |
| 11:12:18 | sean-k-mooney | so the first step woudl really be to add the ablityu to get the queus form libvirt to libvirt | |
| 11:14:39 | sean-k-mooney | dvo-plv_: do you need the VFs to be bound to vfio-pci | |
| 11:15:05 | sean-k-mooney | i assume use so i geuss this infor will not be aviable via the vf since it wont have a netdev | |
| 11:16:14 | dvo-plv_ | we probe vfio-pci driver modprobe vfio-pci enable_sriov=1 | |
| 11:16:25 | dvo-plv_ | ane then allocate vf echo "$NUMVFS" > /sys/bus/pci/devices/0000:$BUS:00.0/sriov_numvfs | |
| 11:17:14 | sean-k-mooney | yep thats pretty standard for dpdk although the enable_sriov bit is relitvly recent | |
| 11:17:18 | dvo-plv_ | we don not have netdev devices for that. so this is hard to get queue at linux layer | |
| 11:17:26 | sean-k-mooney | ya | |
| 11:17:46 | sean-k-mooney | so the probelm is the device spec is not intended for configuration | |
| 11:17:52 | sean-k-mooney | it was orgianly just for filtering | |
| 11:18:02 | sean-k-mooney | we have since added some metadta to it | |
| 11:18:14 | sean-k-mooney | im not sure hwo peopel would fell about adding the number of queues | |
| 11:18:35 | dvo-plv_ | yes, so this is why we firstly decided that user can fill device_spec with additional rapameter for filtering | |
| 11:18:58 | dvo-plv_ | queue_number | |
| 11:19:10 | sean-k-mooney | dvo-plv_: right so that approch has been rejected in the past | |
| 11:19:17 | dvo-plv_ | yes | |
| 11:19:26 | sean-k-mooney | i mean before you propsoed it | |
| 11:19:55 | sean-k-mooney | there have been attempts to do this in teh past and it was rejected | |
| 11:20:07 | sean-k-mooney | that said we now have enough things like this that we might be ok with it | |
| 11:20:26 | sean-k-mooney | we now have things liek remote_managed and resouce class | |
| 11:20:51 | dvo-plv_ | I see. now you would like to see that queue parameter gets automatically like metadata and placemnet filter nodes according to the required queue. But you would not like to increase resource provider | |
| 11:20:58 | sean-k-mooney | so addign queue_pairs=<count> might be ok | |
| 11:21:58 | sean-k-mooney | am no we can model this in placment but it would need use to have one RP per vf | |
| 11:22:05 | sean-k-mooney | which is not soemthgn we wanted to do if we coudl avoid it | |