| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-21 | |||
| 09:53:59 | sean-k-mooney | dvo-plv_: im happy with the spec as is. others might ask for more info but i think you have the main points. | |
| 09:54:15 | sean-k-mooney | i changed the topic in gerrit to bp/virtio-packedring-configuration-support | |
| 09:54:23 | sean-k-mooney | can you use the same topic for the code | |
| 10:19:35 | dvo-plv_ | okay, I will update topic for code. Currently it has https://review.opendev.org/q/topic:VirtIO_PackedRing | |
| 10:19:42 | dvo-plv_ | What does this topic means ? | |
| 10:29:09 | sean-k-mooney | we can use that on the spec | |
| 10:29:18 | sean-k-mooney | the topic is just wa way to group related patches together | |
| 10:29:27 | sean-k-mooney | so the code and spec should use the same one | |
| 10:30:07 | sean-k-mooney | by convention we use bp/<bluprint name> for features that are tracked by a bluepirnt or spec | |
| 10:30:17 | sean-k-mooney | and bug/<bug number> for bugs | |
| 10:30:39 | sean-k-mooney | otherwise its freeform | |
| 10:31:08 | sean-k-mooney | for feature that require changes in multiple project we try to use the same topic across all of them to be able to see all reated patches quickly | |
| 10:31:42 | sean-k-mooney | so now that you updated it https://review.opendev.org/q/topic:bp%252Fvirtio-packedring-configuration-support | |
| 10:32:00 | sean-k-mooney | we can see the spec, nova and os-traits patches all in one view | |
| 10:33:08 | sean-k-mooney | that tells me you are missing a patch to glance to update the metadefs https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json | |
| 10:33:48 | sean-k-mooney | dvo-plv_: ^ the metadefs are what horizon and heat use to generate the drop down for adding traits automatically | |
| 10:34:09 | sean-k-mooney | sorry not triats extra_specs and image properties | |
| 10:35:49 | dvo-plv_ | okay, I will investigate and fix | |
| 10:36:23 | sean-k-mooney | you basically jsut need to copy this https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L30-L35 | |
| 10:36:45 | sean-k-mooney | updatign mem_encyption to packed_ring | |
| 10:37:07 | sean-k-mooney | the namespce/prefix is automaticaly handeled by https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L7-L16 | |
| 10:37:49 | dvo-plv_ | okay, thanks. I would liek to check it on the horizon ui too | |
| 10:37:52 | sean-k-mooney | its also good to update the userful image properies doc https://github.com/openstack/glance/commit/3a281b9bc62a1b8b0f1468bc641105a5662f8ecd is an example | |
| 10:38:11 | dvo-plv_ | sure | |
| 10:38:18 | dvo-plv_ | Could you also clarify me with your scheduler according to the upstream work? We would like to start work over multiqueue support for hardware offloading | |
| 10:39:55 | sean-k-mooney | other can review your code not jsut me. but currently i have around 20% of my time for upstream work less fo the last 3-4 weeks because i was busy with downstream planing activies due to the ptg and other issues | |
| 10:40:54 | sean-k-mooney | i have been trying to capture all the outcomes of the ptg in our downstream jira and plannign the work for our team there | |
| 10:41:15 | sean-k-mooney | normally i have more time for upstream review so that shoudl go back to normal soon | |
| 10:41:48 | sean-k-mooney | regarding multi queue | |
| 10:42:14 | sean-k-mooney | do you mean https://review.opendev.org/c/openstack/nova-specs/+/855514 | |
| 10:42:26 | dvo-plv_ | Whom I should ping regarding review process. I saw gibi on some review process | |
| 10:43:13 | dvo-plv_ | Yes, we would like to analyze it better and provide some vision, how we can improve this in the OpenStack for hardware offloading, expecially for our nic | |
| 10:43:58 | sean-k-mooney | so multi queue is only partly implemented for hardware offload | |
| 10:44:15 | sean-k-mooney | i dont think it actully works we cannot enable it for nics that use sriov | |
| 10:44:26 | sean-k-mooney | i.e. nics that present the vf directly to the guest | |
| 10:44:37 | sean-k-mooney | but it might eb possibel for macvtap or vdpa | |
| 10:45:03 | sean-k-mooney | dvo-plv_: in terms of pings the active member of the nova core team | |
| 10:46:13 | sean-k-mooney | dvo-plv_: so gibi, bauzas, melwitt, gmann and dansmith are you best bet in addtion to me. stephenfin is also around somethimes but they mainly work on non nova related thigns day to day | |
| 10:47:51 | dvo-plv_ | okay, thanks | |
| 10:49:32 | sean-k-mooney | so looping back to multi queue | |
| 10:50:06 | sean-k-mooney | implementing this in the way you wanted is not going to be easy or really desireable form a nova point of view | |
| 10:51:19 | sean-k-mooney | there are a few parts to this problem | |
| 10:51:56 | sean-k-mooney | first we need to detach and recored the number of queues avaialbel in the vf | |
| 10:52:28 | sean-k-mooney | second we need to be able to schudle based on that (either updatign the pci filter or recording this in placement) | |
| 10:53:16 | sean-k-mooney | third we need a way to request a device with a min number of queuse multiup queuse out side the falvoar/image (likely on the nutron port) | |
| 10:54:09 | sean-k-mooney | finally we need to take the resouce request form teh port and include that in our scueduling reeust and wonce we find a host/device that meets that need we need to ensure that the qemu device is cofnigured correctly | |
| 10:54:37 | dvo-plv_ | regarding first question, we investigated it and the best option on our opinio is to parse other config. we configure queues like that -a 0000:65:00.0,representor=[4-6],portqueues=[4:2,5:2,6:2]. Yes it will work only for our nic | |
| 10:54:57 | sean-k-mooney | which config | |
| 10:55:01 | sean-k-mooney | the pci config space | |
| 10:55:15 | dvo-plv_ | ovs. other_config | |
| 10:55:15 | sean-k-mooney | you can useuslly get this form sysfs i tought | |
| 10:55:31 | sean-k-mooney | we cant do that | |
| 10:55:42 | sean-k-mooney | the ovs port wont exist at that point | |
| 10:55:45 | dvo-plv_ | no we can not, this is untrivial task for our dpdk driver | |
| 10:56:05 | sean-k-mooney | oh right so honestly you cant start this work | |
| 10:56:24 | sean-k-mooney | until the basic work of supporting the dpdk represtors is done | |
| 10:56:28 | dvo-plv_ | ovs port not, but vf yes. We would like to parse this config and fill it to the device_spec | |
| 10:56:54 | sean-k-mooney | the ovs port will be created and added by os-vif | |
| 10:57:11 | sean-k-mooney | only after we ahve selected a vf | |
| 10:57:46 | sean-k-mooney | so i think we need https://review.opendev.org/c/openstack/nova-specs/+/859290 to be done before we can talk about multiqueue | |
| 10:59:17 | dvo-plv_ | I see, we thought we can start to find solution for all comments to the blueprint in the parallel at the moment | |
| 10:59:47 | sean-k-mooney | well we could but its going to be diffuctly to compelte jsut one of the 3-4 specs you have propsoed this cycle | |
| 10:59:51 | sean-k-mooney | maybe 2 | |
| 11:00:00 | sean-k-mooney | its very unlikely that all of them will land | |
| 11:00:41 | dvo-plv_ | You mentined that multiqueue functional will be possible for vdpa. So maybe it will be better to move from virtio-forwarder to the vdpa nvic type for future purposes | |
| 11:01:17 | sean-k-mooney | well there is work in dpdk to supprot vdpa | |
| 11:01:31 | sean-k-mooney | i was expecting to have a vdpa-user type at some point for that | |
| 11:02:47 | sean-k-mooney | https://doc.dpdk.org/guides/vdpadevs/features_overview.html | |
| 11:02:52 | sean-k-mooney | i have not looked into it much | |
| 11:03:25 | sean-k-mooney | i dont think that is supported by ovs-dpdk currenlty but i have not really been following it closely | |
| 11:04:08 | sean-k-mooney | dvo-plv_: so for the basic enablement we are going to be trackign napatec VF which we will add to ovs as dpdk prots corret | |
| 11:04:34 | sean-k-mooney | and then those will be exposed to the guest as vhost-user ports | |
| 11:05:01 | sean-k-mooney | so for multi queue we would need to read the number of queues on the vf ideally | |
| 11:05:07 | dvo-plv_ | This multiqueue functional with queue mq and vector is in the our ovs fork at the moment | |
| 11:05:08 | sean-k-mooney | because we need that info for schduling | |
| 11:05:27 | sean-k-mooney | ok so thats kind of a problem | |
| 11:05:52 | sean-k-mooney | we do not really allow enablment of forked functionality in nova | |
| 11:06:11 | dvo-plv_ | Yes, I remember that it requires for placement to handle scheduler with queues number | |
| 11:06:57 | sean-k-mooney | if we can do it generically we enabel it so i was hoppng we coudl do somehting liek read /sys/bus/pci/device/<address>/num_queus or somehting like that | |
| 11:07:09 | sean-k-mooney | ideally vai libvirt nodedev api | |
| 11:07:15 | sean-k-mooney | not reading sys directly | |
| 11:09:18 | sean-k-mooney | so you can get the queue like this https://paste.opendev.org/show/bVSM5IDtJTRwhcIcuTFs/ | |
| 11:09:27 | sean-k-mooney | that is a pf | |
| 11:09:33 | sean-k-mooney | but i belive the same is true for VFs | |
| 11:11:44 | sean-k-mooney | i done see the queues in libvirt https://paste.opendev.org/show/bJHNfZR8JNpxJVvLggCI/ | |
| 11:12:18 | sean-k-mooney | so the first step woudl really be to add the ablityu to get the queus form libvirt to libvirt | |
| 11:14:39 | sean-k-mooney | dvo-plv_: do you need the VFs to be bound to vfio-pci | |
| 11:15:05 | sean-k-mooney | i assume use so i geuss this infor will not be aviable via the vf since it wont have a netdev | |
| 11:16:14 | dvo-plv_ | we probe vfio-pci driver modprobe vfio-pci enable_sriov=1 | |
| 11:16:25 | dvo-plv_ | ane then allocate vf echo "$NUMVFS" > /sys/bus/pci/devices/0000:$BUS:00.0/sriov_numvfs | |
| 11:17:14 | sean-k-mooney | yep thats pretty standard for dpdk although the enable_sriov bit is relitvly recent | |
| 11:17:18 | dvo-plv_ | we don not have netdev devices for that. so this is hard to get queue at linux layer | |
| 11:17:26 | sean-k-mooney | ya | |
| 11:17:46 | sean-k-mooney | so the probelm is the device spec is not intended for configuration | |
| 11:17:52 | sean-k-mooney | it was orgianly just for filtering | |
| 11:18:02 | sean-k-mooney | we have since added some metadta to it | |
| 11:18:14 | sean-k-mooney | im not sure hwo peopel would fell about adding the number of queues | |
| 11:18:35 | dvo-plv_ | yes, so this is why we firstly decided that user can fill device_spec with additional rapameter for filtering | |
| 11:18:58 | dvo-plv_ | queue_number | |
| 11:19:10 | sean-k-mooney | dvo-plv_: right so that approch has been rejected in the past | |