Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-04-21
09:49:08 sean-k-mooney i think i was more or less ok with it the last time i looked so ill take a look again
09:53:59 sean-k-mooney dvo-plv_: im happy with the spec as is. others might ask for more info but i think you have the main points.
09:54:15 sean-k-mooney i changed the topic in gerrit to bp/virtio-packedring-configuration-support
09:54:23 sean-k-mooney can you use the same topic for the code
10:19:35 dvo-plv_ okay, I will update topic for code. Currently it has https://review.opendev.org/q/topic:VirtIO_PackedRing
10:19:42 dvo-plv_ What does this topic means ?
10:29:09 sean-k-mooney we can use that on the spec
10:29:18 sean-k-mooney the topic is just wa way to group related patches together
10:29:27 sean-k-mooney so the code and spec should use the same one
10:30:07 sean-k-mooney by convention we use bp/<bluprint name> for features that are tracked by a bluepirnt or spec
10:30:17 sean-k-mooney and bug/<bug number> for bugs
10:30:39 sean-k-mooney otherwise its freeform
10:31:08 sean-k-mooney for feature that require changes in multiple project we try to use the same topic across all of them to be able to see all reated patches quickly
10:31:42 sean-k-mooney so now that you updated it https://review.opendev.org/q/topic:bp%252Fvirtio-packedring-configuration-support
10:32:00 sean-k-mooney we can see the spec, nova and os-traits patches all in one view
10:33:08 sean-k-mooney that tells me you are missing a patch to glance to update the metadefs https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json
10:33:48 sean-k-mooney dvo-plv_: ^ the metadefs are what horizon and heat use to generate the drop down for adding traits automatically
10:34:09 sean-k-mooney sorry not triats extra_specs and image properties
10:35:49 dvo-plv_ okay, I will investigate and fix
10:36:23 sean-k-mooney you basically jsut need to copy this https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L30-L35
10:36:45 sean-k-mooney updatign mem_encyption to packed_ring
10:37:07 sean-k-mooney the namespce/prefix is automaticaly handeled by https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L7-L16
10:37:49 dvo-plv_ okay, thanks. I would liek to check it on the horizon ui too
10:37:52 sean-k-mooney its also good to update the userful image properies doc https://github.com/openstack/glance/commit/3a281b9bc62a1b8b0f1468bc641105a5662f8ecd is an example
10:38:11 dvo-plv_ sure
10:38:18 dvo-plv_ Could you also clarify me with your scheduler according to the upstream work? We would like to start work over multiqueue support for hardware offloading
10:39:55 sean-k-mooney other can review your code not jsut me. but currently i have around 20% of my time for upstream work less fo the last 3-4 weeks because i was busy with downstream planing activies due to the ptg and other issues
10:40:54 sean-k-mooney i have been trying to capture all the outcomes of the ptg in our downstream jira and plannign the work for our team there
10:41:15 sean-k-mooney normally i have more time for upstream review so that shoudl go back to normal soon
10:41:48 sean-k-mooney regarding multi queue
10:42:14 sean-k-mooney do you mean https://review.opendev.org/c/openstack/nova-specs/+/855514
10:42:26 dvo-plv_ Whom I should ping regarding review process. I saw gibi on some review process
10:43:13 dvo-plv_ Yes, we would like to analyze it better and provide some vision, how we can improve this in the OpenStack for hardware offloading, expecially for our nic
10:43:58 sean-k-mooney so multi queue is only partly implemented for hardware offload
10:44:15 sean-k-mooney i dont think it actully works we cannot enable it for nics that use sriov
10:44:26 sean-k-mooney i.e. nics that present the vf directly to the guest
10:44:37 sean-k-mooney but it might eb possibel for macvtap or vdpa
10:45:03 sean-k-mooney dvo-plv_: in terms of pings the active member of the nova core team
10:46:13 sean-k-mooney dvo-plv_: so gibi, bauzas, melwitt, gmann and dansmith are you best bet in addtion to me. stephenfin is also around somethimes but they mainly work on non nova related thigns day to day
10:47:51 dvo-plv_ okay, thanks
10:49:32 sean-k-mooney so looping back to multi queue
10:50:06 sean-k-mooney implementing this in the way you wanted is not going to be easy or really desireable form a nova point of view
10:51:19 sean-k-mooney there are a few parts to this problem
10:51:56 sean-k-mooney first we need to detach and recored the number of queues avaialbel in the vf
10:52:28 sean-k-mooney second we need to be able to schudle based on that (either updatign the pci filter or recording this in placement)
10:53:16 sean-k-mooney third we need a way to request a device with a min number of queuse multiup queuse out side the falvoar/image (likely on the nutron port)
10:54:09 sean-k-mooney finally we need to take the resouce request form teh port and include that in our scueduling reeust and wonce we find a host/device that meets that need we need to ensure that the qemu device is cofnigured correctly
10:54:37 dvo-plv_ regarding first question, we investigated it and the best option on our opinio is to parse other config. we configure queues like that -a 0000:65:00.0,representor=[4-6],portqueues=[4:2,5:2,6:2]. Yes it will work only for our nic
10:54:57 sean-k-mooney which config
10:55:01 sean-k-mooney the pci config space
10:55:15 dvo-plv_ ovs. other_config
10:55:15 sean-k-mooney you can useuslly get this form sysfs i tought
10:55:31 sean-k-mooney we cant do that
10:55:42 sean-k-mooney the ovs port wont exist at that point
10:55:45 dvo-plv_ no we can not, this is untrivial task for our dpdk driver
10:56:05 sean-k-mooney oh right so honestly you cant start this work
10:56:24 sean-k-mooney until the basic work of supporting the dpdk represtors is done
10:56:28 dvo-plv_ ovs port not, but vf yes. We would like to parse this config and fill it to the device_spec
10:56:54 sean-k-mooney the ovs port will be created and added by os-vif
10:57:11 sean-k-mooney only after we ahve selected a vf
10:57:46 sean-k-mooney so i think we need https://review.opendev.org/c/openstack/nova-specs/+/859290 to be done before we can talk about multiqueue
10:59:17 dvo-plv_ I see, we thought we can start to find solution for all comments to the blueprint in the parallel at the moment
10:59:47 sean-k-mooney well we could but its going to be diffuctly to compelte jsut one of the 3-4 specs you have propsoed this cycle
10:59:51 sean-k-mooney maybe 2
11:00:00 sean-k-mooney its very unlikely that all of them will land
11:00:41 dvo-plv_ You mentined that multiqueue functional will be possible for vdpa. So maybe it will be better to move from virtio-forwarder to the vdpa nvic type for future purposes
11:01:17 sean-k-mooney well there is work in dpdk to supprot vdpa
11:01:31 sean-k-mooney i was expecting to have a vdpa-user type at some point for that
11:02:47 sean-k-mooney https://doc.dpdk.org/guides/vdpadevs/features_overview.html
11:02:52 sean-k-mooney i have not looked into it much
11:03:25 sean-k-mooney i dont think that is supported by ovs-dpdk currenlty but i have not really been following it closely
11:04:08 sean-k-mooney dvo-plv_: so for the basic enablement we are going to be trackign napatec VF which we will add to ovs as dpdk prots corret
11:04:34 sean-k-mooney and then those will be exposed to the guest as vhost-user ports
11:05:01 sean-k-mooney so for multi queue we would need to read the number of queues on the vf ideally
11:05:07 dvo-plv_ This multiqueue functional with queue mq and vector is in the our ovs fork at the moment
11:05:08 sean-k-mooney because we need that info for schduling
11:05:27 sean-k-mooney ok so thats kind of a problem
11:05:52 sean-k-mooney we do not really allow enablment of forked functionality in nova
11:06:11 dvo-plv_ Yes, I remember that it requires for placement to handle scheduler with queues number
11:06:57 sean-k-mooney if we can do it generically we enabel it so i was hoppng we coudl do somehting liek read /sys/bus/pci/device/<address>/num_queus or somehting like that
11:07:09 sean-k-mooney ideally vai libvirt nodedev api
11:07:15 sean-k-mooney not reading sys directly
11:09:18 sean-k-mooney so you can get the queue like this https://paste.opendev.org/show/bVSM5IDtJTRwhcIcuTFs/
11:09:27 sean-k-mooney that is a pf
11:09:33 sean-k-mooney but i belive the same is true for VFs
11:11:44 sean-k-mooney i done see the queues in libvirt https://paste.opendev.org/show/bJHNfZR8JNpxJVvLggCI/
11:12:18 sean-k-mooney so the first step woudl really be to add the ablityu to get the queus form libvirt to libvirt
11:14:39 sean-k-mooney dvo-plv_: do you need the VFs to be bound to vfio-pci
11:15:05 sean-k-mooney i assume use so i geuss this infor will not be aviable via the vf since it wont have a netdev
11:16:14 dvo-plv_ we probe vfio-pci driver modprobe vfio-pci enable_sriov=1
11:16:25 dvo-plv_ ane then allocate vf echo "$NUMVFS" > /sys/bus/pci/devices/0000:$BUS:00.0/sriov_numvfs
11:17:14 sean-k-mooney yep thats pretty standard for dpdk although the enable_sriov bit is relitvly recent
11:17:18 dvo-plv_ we don not have netdev devices for that. so this is hard to get queue at linux layer
11:17:26 sean-k-mooney ya
11:17:46 sean-k-mooney so the probelm is the device spec is not intended for configuration
11:17:52 sean-k-mooney it was orgianly just for filtering
11:18:02 sean-k-mooney we have since added some metadta to it
11:18:14 sean-k-mooney im not sure hwo peopel would fell about adding the number of queues
11:18:35 dvo-plv_ yes, so this is why we firstly decided that user can fill device_spec with additional rapameter for filtering
11:18:58 dvo-plv_ queue_number

Earlier   Later