Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-25
17:25:53 sean-k-mooney jhinman: the tag i think is the interface tagging feature where you can associate a generic name to an interface and look it up in the metadata service
17:27:15 jhinman but how does the tag get assigned? is it something passed from neutron. or maybe the tag is not the problem? is there supposed to be a vlan associated
17:27:45 sean-k-mooney its passed in by the user on nova boot. it is not related to the neutron segmentation id
17:27:55 sean-k-mooney its a value like "wan" or "lan"
17:28:30 efried stephenfin: Are we waiting to merge https://review.openstack.org/#/c/645991/ ?
17:28:33 sean-k-mooney so this is the code that is failing https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L9284-L9286
17:29:44 sean-k-mooney looks like we just merged somthing
17:29:46 sean-k-mooney https://github.com/openstack/nova/blob/0dfbcd74642b16432b800b51f304a89d762e3ff1/nova/virt/libvirt/driver.py#L9290-L9292
17:30:54 sean-k-mooney jhinman: there should be nothing ovs-dpdk specific in that code. it shoudl be totally generic
17:34:59 sean-k-mooney jhinman: actully if that is the code that is runnign that should not run for vhost-user interfaces
17:35:00 jhinman ultimately, it ends up with line 9204: LOG.debug('No VIF found with MAC %s, not building metadata', dev.mac_addr)
17:35:09 sean-k-mooney the section i linked if for sriov
17:37:03 sean-k-mooney jhinman: can you provide a paste of the trackback on paste.openstack.org
17:37:34 sean-k-mooney vifs_to_expose = {vif.address: vif for vif in vifs
17:37:36 sean-k-mooney if ('tag' in vif and vif.tag) or
17:37:38 sean-k-mooney vlans_by_mac.get(vif.address)}
17:37:52 sean-k-mooney is in the sriov code path and should not be called when handeling vhost-user
17:39:08 sean-k-mooney well unless we use that fucntion to do generic dispatch to the others. lets see where is it called
17:39:39 sean-k-mooney actully so yes that is how it is working
17:39:50 sean-k-mooney we get the interface metadata here
17:39:52 sean-k-mooney https://github.com/openstack/nova/blob/0dfbcd74642b16432b800b51f304a89d762e3ff1/nova/virt/libvirt/driver.py#L9310
17:41:23 sean-k-mooney jhinman: we emiit that debug message when the user does not specify any device role tags
17:41:53 sean-k-mooney so its not a error to see that in the logs
17:42:11 openstackgerrit Elod Illes proposed openstack/nova stable/pike: Fix incompatible version handling in BuildRequest https://review.openstack.org/647557
17:43:57 sean-k-mooney jhinman: this code is for this spec https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-device-role-tagging.html are you having a specic issue or did you jsut see that debug message and were trying to figure out how to scilance it?
17:45:03 jhinman allright, I was looking at that because I was going back from another error: I can't find any reason for qemu not to have permission to write the socket: Failed to connect socket /var/run/openvswitch/vhu7a90d3cd-d7: Permission denied
17:46:26 sean-k-mooney jhinman: the socket error can happen for two reasons. first apparmor can block it, that should not be an issue with rocky but it was in the past
17:46:57 sean-k-mooney second /var/run/openvswitch need to be writabel by the user that qemu is running with
17:46:58 jhinman I disable apparmor
17:47:34 jhinman but nova is the user of qemu, so should have permission for anything on the instances, right?
17:47:54 sean-k-mooney qemu and nova need to be runign in the same group
17:47:59 sean-k-mooney *qemu and ovs
17:49:13 sean-k-mooney jhinman: on rocky qemu should be acting as teh vhost-user server and will create the vhost-user socket
17:49:41 sean-k-mooney as such qemu need to have write permission to the /var/run/openvswith directory that is owned by ovs
17:51:08 jhinman so I need to add qemu to the ovs group?
17:51:53 sean-k-mooney jhinman: this should all be done by kolla ansible
17:53:12 openstackgerrit Merged openstack/nova stable/queens: Replace openstack.org git:// URLs with https:// https://review.openstack.org/646685
17:53:21 openstackgerrit Merged openstack/nova stable/ocata: Replace openstack.org git:// URLs with https:// https://review.openstack.org/646683
17:54:30 sean-k-mooney jhinman: its been a few years since i implented the support in kolla-ansibel so i cant remember of the top of my head how i fixed this issue
17:54:37 jhinman I think I remember a patch for qemu.conf, to set user to nova. but I think its in the nova group. is there a common group?
17:56:45 sean-k-mooney jhinman: this is where the contain is created https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/ovs-dpdk/handlers/main.yml#L56-L76
17:57:36 sean-k-mooney we are not setting a user in the yaml or in teh command using su no
17:57:43 sean-k-mooney https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/ovs-dpdk/templates/ovsdpdk-vswitchd.json.j2
17:58:18 sean-k-mooney so i think that means ovs woudl be running as root maybe
17:59:22 jhinman not obvious to me what you see, but I'll study it
17:59:53 sean-k-mooney qemu is running as nova
17:59:54 sean-k-mooney https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/nova/templates/qemu.conf.j2
18:00:31 jhinman yes, so how to give it permission to access ovs's file?
18:00:43 sean-k-mooney so nova would have to have write permission to /var/run/openvswitch
18:00:49 sean-k-mooney ya
18:04:29 jhinman sean: in order to get hostconfig stuff working, I added a new start-ovs script, overriding the embedded script. Is that the right way, or can the commands be added to the script in the tools directory (ovs-dpdkctl.sh)?
18:06:00 sean-k-mooney before or after this issue
18:06:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Delete allocations even if _confirm_resize raises https://review.openstack.org/647566
18:06:37 jhinman if you have a quick answer. I could submit a patch for you to review
18:07:18 sean-k-mooney sure its simpler to anwer questiosn on a review but i would proably put it here https://github.com/openstack/kolla/blob/master/docker/ovsdpdk/ovsdpdk/extend_start.sh
18:07:33 dansmith mriedem: you should be okay to backport a default= change on an object field It hink
18:08:31 mriedem that affects the version of the object doesn't it?
18:08:50 dansmith mriedem: it shouldn't.. it has nothing to do with the wire format
18:09:25 sean-k-mooney mriedem: technicall i think it wont casue our ovo tests to fail as the filde dont change and the default value is not considered part of teh field type
18:09:46 dansmith mriedem: some remote host could not specify a value and then break if a different default is chosen, but that'd be pretty silly
18:10:36 sean-k-mooney jhinman: oh wait its already here https://github.com/openstack/kolla/blob/master/docker/ovsdpdk/ovsdpdk-vswitchd/extend_start.sh
18:11:11 dansmith mriedem: I'm not saying you need to change your patch, but if you want to and/or it's better, then I think you're in the clear
18:11:21 mriedem dansmith: ok i guess i was thinking of takashi's change here https://review.openstack.org/#/c/641200/3/nova/objects/request_spec.py but he's also setting nullable=True on network_metadata
18:11:24 mriedem so maybe i got confused
18:11:34 dansmith mriedem: yeah, nullable is fo' sho' a problem
18:11:44 mriedem fo real fo sho?
18:11:51 sean-k-mooney mriedem: nullable will change the db schma for the column
18:12:07 dansmith sean-k-mooney: no it won't
18:12:13 mriedem was just about to say ^ :)
18:12:14 mriedem exact words
18:12:26 sean-k-mooney well it requires you too chagne it right
18:12:28 dansmith no
18:12:31 sean-k-mooney no
18:12:46 dansmith we have many fields which intentionally have different nullability for the db and object
18:12:52 dansmith including one we just did for the quota stuff
18:13:00 sean-k-mooney allowing an object to have a nullable field that was previousl not nullable does not require the db to allow it to be null?
18:13:08 sean-k-mooney ohok
18:13:12 dansmith not necessarily
18:13:18 dansmith totally depends on the thing
18:15:54 jhinman sean: that doesn't affect the instance's socket perms, does it?
18:16:14 sean-k-mooney jhinman: its not he socket perms that is the issue
18:16:41 sean-k-mooney qemu is createing the socket so it will be the owner and will have read and write permissions
18:16:47 sean-k-mooney ovs is runnign as root
18:16:56 sean-k-mooney so the issue is the permission on the directory
18:17:09 sean-k-mooney nova need read write permission to create the unix socket
18:17:54 sean-k-mooney 777 should give all users and groups read wite permission to create files there
18:20:39 jhinman but that script has been there all along
18:20:58 sean-k-mooney jhinman: we bind mount the host /run into both the nova-libvirt and ovs vswtichd container
18:21:30 sean-k-mooney can you check the permission on the host
18:21:32 sean-k-mooney ls -al /run/openvswitch/
18:22:04 jhinman ok, will do. co-worker just told me she has it working on master branch, so something got fixed
18:22:39 sean-k-mooney jhinman: perhaps but none of this code has change in well over 8 month and most is 2+ years old
18:23:29 sean-k-mooney jhinman: that said i wrote most of the ovs-dpdk code in kolla so im not surpised if there are bugs :)
18:24:06 jhinman host has 777 on /run/openvswitch
18:26:02 jhinman I thought the log is saying that it's creating a socket in the VM
18:26:24 sean-k-mooney no its creating a socket for the vm. not in the vm.
18:26:59 jhinman a socket in nova-compute?
18:27:00 sean-k-mooney its created on the host by qemu i recent release. it was previously created by ovs and connected to by qemu in old releases
18:27:32 sean-k-mooney the socket wil be create in the nova-libvirt container which also has the same directory bind monted shared form the host
18:27:47 sean-k-mooney jhinman: https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/nova/defaults/main.yml#L16

Earlier   Later