Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-25
17:43:57 sean-k-mooney jhinman: this code is for this spec https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-device-role-tagging.html are you having a specic issue or did you jsut see that debug message and were trying to figure out how to scilance it?
17:45:03 jhinman allright, I was looking at that because I was going back from another error: I can't find any reason for qemu not to have permission to write the socket: Failed to connect socket /var/run/openvswitch/vhu7a90d3cd-d7: Permission denied
17:46:26 sean-k-mooney jhinman: the socket error can happen for two reasons. first apparmor can block it, that should not be an issue with rocky but it was in the past
17:46:57 sean-k-mooney second /var/run/openvswitch need to be writabel by the user that qemu is running with
17:46:58 jhinman I disable apparmor
17:47:34 jhinman but nova is the user of qemu, so should have permission for anything on the instances, right?
17:47:54 sean-k-mooney qemu and nova need to be runign in the same group
17:47:59 sean-k-mooney *qemu and ovs
17:49:13 sean-k-mooney jhinman: on rocky qemu should be acting as teh vhost-user server and will create the vhost-user socket
17:49:41 sean-k-mooney as such qemu need to have write permission to the /var/run/openvswith directory that is owned by ovs
17:51:08 jhinman so I need to add qemu to the ovs group?
17:51:53 sean-k-mooney jhinman: this should all be done by kolla ansible
17:53:12 openstackgerrit Merged openstack/nova stable/queens: Replace openstack.org git:// URLs with https:// https://review.openstack.org/646685
17:53:21 openstackgerrit Merged openstack/nova stable/ocata: Replace openstack.org git:// URLs with https:// https://review.openstack.org/646683
17:54:30 sean-k-mooney jhinman: its been a few years since i implented the support in kolla-ansibel so i cant remember of the top of my head how i fixed this issue
17:54:37 jhinman I think I remember a patch for qemu.conf, to set user to nova. but I think its in the nova group. is there a common group?
17:56:45 sean-k-mooney jhinman: this is where the contain is created https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/ovs-dpdk/handlers/main.yml#L56-L76
17:57:36 sean-k-mooney we are not setting a user in the yaml or in teh command using su no
17:57:43 sean-k-mooney https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/ovs-dpdk/templates/ovsdpdk-vswitchd.json.j2
17:58:18 sean-k-mooney so i think that means ovs woudl be running as root maybe
17:59:22 jhinman not obvious to me what you see, but I'll study it
17:59:53 sean-k-mooney qemu is running as nova
17:59:54 sean-k-mooney https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/nova/templates/qemu.conf.j2
18:00:31 jhinman yes, so how to give it permission to access ovs's file?
18:00:43 sean-k-mooney so nova would have to have write permission to /var/run/openvswitch
18:00:49 sean-k-mooney ya
18:04:29 jhinman sean: in order to get hostconfig stuff working, I added a new start-ovs script, overriding the embedded script. Is that the right way, or can the commands be added to the script in the tools directory (ovs-dpdkctl.sh)?
18:06:00 sean-k-mooney before or after this issue
18:06:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Delete allocations even if _confirm_resize raises https://review.openstack.org/647566
18:06:37 jhinman if you have a quick answer. I could submit a patch for you to review
18:07:18 sean-k-mooney sure its simpler to anwer questiosn on a review but i would proably put it here https://github.com/openstack/kolla/blob/master/docker/ovsdpdk/ovsdpdk/extend_start.sh
18:07:33 dansmith mriedem: you should be okay to backport a default= change on an object field It hink
18:08:31 mriedem that affects the version of the object doesn't it?
18:08:50 dansmith mriedem: it shouldn't.. it has nothing to do with the wire format
18:09:25 sean-k-mooney mriedem: technicall i think it wont casue our ovo tests to fail as the filde dont change and the default value is not considered part of teh field type
18:09:46 dansmith mriedem: some remote host could not specify a value and then break if a different default is chosen, but that'd be pretty silly
18:10:36 sean-k-mooney jhinman: oh wait its already here https://github.com/openstack/kolla/blob/master/docker/ovsdpdk/ovsdpdk-vswitchd/extend_start.sh
18:11:11 dansmith mriedem: I'm not saying you need to change your patch, but if you want to and/or it's better, then I think you're in the clear
18:11:21 mriedem dansmith: ok i guess i was thinking of takashi's change here https://review.openstack.org/#/c/641200/3/nova/objects/request_spec.py but he's also setting nullable=True on network_metadata
18:11:24 mriedem so maybe i got confused
18:11:34 dansmith mriedem: yeah, nullable is fo' sho' a problem
18:11:44 mriedem fo real fo sho?
18:11:51 sean-k-mooney mriedem: nullable will change the db schma for the column
18:12:07 dansmith sean-k-mooney: no it won't
18:12:13 mriedem was just about to say ^ :)
18:12:14 mriedem exact words
18:12:26 sean-k-mooney well it requires you too chagne it right
18:12:28 dansmith no
18:12:31 sean-k-mooney no
18:12:46 dansmith we have many fields which intentionally have different nullability for the db and object
18:12:52 dansmith including one we just did for the quota stuff
18:13:00 sean-k-mooney allowing an object to have a nullable field that was previousl not nullable does not require the db to allow it to be null?
18:13:08 sean-k-mooney ohok
18:13:12 dansmith not necessarily
18:13:18 dansmith totally depends on the thing
18:15:54 jhinman sean: that doesn't affect the instance's socket perms, does it?
18:16:14 sean-k-mooney jhinman: its not he socket perms that is the issue
18:16:41 sean-k-mooney qemu is createing the socket so it will be the owner and will have read and write permissions
18:16:47 sean-k-mooney ovs is runnign as root
18:16:56 sean-k-mooney so the issue is the permission on the directory
18:17:09 sean-k-mooney nova need read write permission to create the unix socket
18:17:54 sean-k-mooney 777 should give all users and groups read wite permission to create files there
18:20:39 jhinman but that script has been there all along
18:20:58 sean-k-mooney jhinman: we bind mount the host /run into both the nova-libvirt and ovs vswtichd container
18:21:30 sean-k-mooney can you check the permission on the host
18:21:32 sean-k-mooney ls -al /run/openvswitch/
18:22:04 jhinman ok, will do. co-worker just told me she has it working on master branch, so something got fixed
18:22:39 sean-k-mooney jhinman: perhaps but none of this code has change in well over 8 month and most is 2+ years old
18:23:29 sean-k-mooney jhinman: that said i wrote most of the ovs-dpdk code in kolla so im not surpised if there are bugs :)
18:24:06 jhinman host has 777 on /run/openvswitch
18:26:02 jhinman I thought the log is saying that it's creating a socket in the VM
18:26:24 sean-k-mooney no its creating a socket for the vm. not in the vm.
18:26:59 jhinman a socket in nova-compute?
18:27:00 sean-k-mooney its created on the host by qemu i recent release. it was previously created by ovs and connected to by qemu in old releases
18:27:32 sean-k-mooney the socket wil be create in the nova-libvirt container which also has the same directory bind monted shared form the host
18:27:47 sean-k-mooney jhinman: https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/nova/defaults/main.yml#L16
18:28:14 jhinman ok. I'll figure it out. thanks for your help!
18:29:24 sean-k-mooney sorry i couldnt help more. i shoudl really test that code again but i have not been working on kolla for about 18 months so i havent really been aroudn to take care of it
18:30:47 sean-k-mooney jhinman: if you have issue im also on the #openstack-kolla channel so feel free to ping me there not to bother the nova folks. if its not specific feel free to ping me here
18:53:40 openstackgerrit sean mooney proposed openstack/nova-specs master: add spec for image metadata prefiltering https://review.openstack.org/647578
18:56:49 mnaser would it be ok to add https://review.openstack.org/#/c/645352/ to the rc2 list? and https://review.openstack.org/#/c/641907/ ?
18:59:30 sean-k-mooney mnaser: oh that the fix for the null issue you were looking at last week?
19:01:22 mnaser sean-k-mooney: correct, the first one is that, the second one is Oslo.service doing a full restart of the service instead of reload, which breaks some assumptions nova does
19:01:29 mnaser which means reloading to refresh info leaves you in a broken state
19:02:35 sean-k-mooney mnaser: right i brifly skimed that thread on the mailing list
19:02:54 sean-k-mooney oh so that was how this was getting to None in the first place
19:03:39 mnaser sean-k-mooney: yeah I could see it in the DB going from "NULL" (sql null) to "null" (son null)
19:03:40 sean-k-mooney or rather how self._bdm_obj.connection_info was
19:03:42 mnaser json*
19:08:19 sean-k-mooney mnaser: by the way on the olso service thing my intiall reaction was make oslos reload not call stop and start but i have no idea what other effect that would have
19:08:42 sean-k-mooney mnaser: but that is more have sane names for things then because it definetly the right thing to do
19:10:34 openstackgerrit sean mooney proposed openstack/nova-specs master: re add for train https://review.openstack.org/647581
19:15:08 openstackgerrit sean mooney proposed openstack/nova-specs master: forward ported form stien https://review.openstack.org/647582
19:17:42 sean-k-mooney mriedem: do ^ need to go back through full review or do can they be fast approved. i litrally just copied them updated the history footer? i know we are still focusing on RC2 but im hoping we can land that code before the ptg or m1
19:18:20 sean-k-mooney mriedem: i know we want to talk about openlab for the numa testing at the ptg but just said i would ask
19:19:47 efried sean-k-mooney: If they were approved in stein, and they're verbatim (except the footer and s/stein/train/i) then they can be fast approved.
19:20:24 sean-k-mooney efried: ya they are
19:20:29 efried sean-k-mooney: However, I would request that you make some reference to the actual blueprint being proposed in the commit title and message body.
19:20:42 melwitt mnaser: are they regressions? I think we're only doing regression fixes for RC2
19:20:46 efried sean-k-mooney: those titles are going to kill me

Earlier   Later