Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-06
12:36:45 sahid did you noticed that for newton only hybrid vif was supported ? and for hybrid vif we have a special condition to check whether the hybrid vif already exists and so in that case we do not execute that command again
12:38:00 sean-k-mooney sahid: in the non hybrid case the vif is plugged by libvirt not os-vif
12:39:10 sahid can you point me the code where that issue could happen?
12:39:58 sean-k-mooney sahid: its a race condtion between ovs and vm tap devices being created its not an issue in the code itself
12:40:38 sahid sean-k-mooney: yes but we do not create any tap device when restarting nova-compue service
12:40:41 sahid that is what i mean
12:42:20 sean-k-mooney correct the "bridge|WARN|could not open network device tap2cf7dbad-9d (No such device)" only happens if the ovs-db has an entry for an interface that does not currently exist in the root namespace when ovs-vswitchd is starting
12:42:48 sean-k-mooney the fact the interface is prefixed with tap mean that hybrid plug is not in use meaning libvirt is pluging the interface not os-vif
12:43:33 sean-k-mooney so this will happen if ovs start before libvirt starts the vm after a reboot
12:43:40 sean-k-mooney but only on ovs <2.6
12:44:26 sahid sean-k-mooney: after a reboot of what?
12:44:40 sean-k-mooney sahid: the host
12:46:11 sahid how possible that happen?
12:46:42 sahid the tap device would have been loaded by the kernel before ovs and libvirt
12:46:53 sahid then i don't understnad why libvirt is in relation here?
12:47:04 sean-k-mooney basicaly when ovs trys to add all ports in the ovsdb to the ovs kernel datapath it skips any it does not find, e.g. not running vms. the ovs revalidtor tread does not redo this check untill another port is added to ovs before 2.6
12:47:17 sean-k-mooney sahid: tap devices are not persited across reboots
12:47:36 sean-k-mooney they are created by libvirt/qemu when the vm starts
12:47:51 sahid oh yes you are right
12:48:14 sean-k-mooney but the ovsdb info is preserved across reboot so ovs is expecting it to exist but libvirt/qemu have not created it yet
12:49:15 sahid yes that is a possible case
12:49:30 sean-k-mooney sahid: anyway im going to approve the patch as it is for master and request you add a check for the interface type on backport as we technically support ovs older then 2.6 in ocata. is that ok with you?
12:51:21 sahid sean-k-mooney: ok so please take the time to ack it soon as possible
12:52:29 sean-k-mooney sahid: i have it open on another tab. i was currently reading neutron release notes to see if they required ovs 2.6+ but i could not find any assertion to that effect for ocata
12:53:00 sean-k-mooney i need to run to a meeting at 1 but ill try to ack it before then
12:54:17 sahid sean-k-mooney: yes actually using 2.5 looks really old
12:54:21 sean-k-mooney looking at osp 11 redhat shiped ovs 2.6 for the ocata release if i can validate that suse,mirantis and canonical also did the same then perhaps we can skip the check but that is up to the stable mainance team to decide
12:54:37 sahid sean-k-mooney: that would be the ideal
12:56:30 sahid stephenfin, sean-k-mooney can we work together to ensure that for newton all the major openstack dist are using OVS2.6
12:57:24 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/550057
12:58:14 sahid s/netwon/ocata
13:02:56 sahid sean-k-mooney: I'm thinking about something, the scenario you have indicated is not in relation with the patch
13:15:35 sean-k-mooney sahid: correct however that was the reason that https://github.com/openstack/nova/commit/33cc64fb817 was merged to resolve https://bugs.launchpad.net/nova/+bug/1270973
13:15:36 openstack Launchpad bug 1270973 in OpenStack Compute (nova) "Remove and recreate interface if already exists" [Medium,Fix released] - Assigned to Aaron Rosen (arosen)
13:18:00 sean-k-mooney sahid: so your fix would be reverting that previous change. but that bug can nolonger happen after ovs 2.6 so its safe to revert the old change on any moderen openstack deployment
13:20:06 sahid sean-k-mooney: the fix will be backported for stable queens only - i think we are ok
13:21:41 sean-k-mooney oh yes well the addtionally check would only be required for stable/ocata if it was backported to ocata. for queens and pike the patch can be applied as is
13:30:48 sahid sean-k-mooney: i'm just going to backport it for stable/queens and stable/pike, I don't think we can backport the patch in 3 release whether it's not a security issue
13:31:02 sean-k-mooney sahid: i have +w https://review.openstack.org/#/c/546588/1. i confirmed that both osp and the ubuntu cloud archive ship ovs 2.6 for ocata i would find it strange if other did not also as it was required for the contrack firewall driver, vhost-user reconnect and the netive python bindings for ovs which where three of the big feature in ocata for ovs
13:32:01 sean-k-mooney sahid: technicall if its not a security bug it has to be easily provale to not have any sidefects and is at the sole descretion of the stable maintainer. generally that means docs changes
13:32:51 stephenfin sean-k-mooney: Excellent. Good to get that closed off 🎉
13:33:09 sahid sean-k-mooney: i don't want add functional change to a backport
13:33:26 sahid so if we want ot backport it for ocata we want it as it is
13:33:55 sahid if not so distribution will do some downstream only backport
13:34:40 stephenfin sahid: Yeah, my suggestion was to add the check to the current patch and then immediately remove it on master. This would let us backport the first patch all the way back
13:34:42 sean-k-mooney sahid: i think an downstream only backport would be best for ocata if required but i dont think may distros shipped 2.5 with ocata if any
13:35:01 stephenfin I think that might have got lost in translation :)
13:35:01 sahid sean-k-mooney: seems reasonable
13:35:15 stephenfin However, downstream only seems the way to go with Ocata and previous
13:35:17 sahid let have it backported for queens and pike so
13:36:16 stephenfin sahid: +1
13:37:20 openstackgerrit Merged openstack/os-vif master: ovs: do not delete port if already exists https://review.openstack.org/546588
13:37:48 openstackgerrit sahid proposed openstack/os-vif stable/queens: ovs: do not delete port if already exists https://review.openstack.org/550079
13:38:17 openstackgerrit sahid proposed openstack/os-vif stable/pike: ovs: do not delete port if already exists https://review.openstack.org/550080
13:39:43 sean-k-mooney sahid sound good to me. stephen and i are not in the nova-stable-maint group so i suggest you ping sylvain and perhaps matt to review for the backports
13:42:11 lvdombrkr89 hello folks
13:42:12 lvdombrkr89 when try to create instance with multiatach volume get error: Multiattach volumes are only supported starting with compute API version 2.60. (HTTP 400) (Request-ID: req-8babdaa8-29d9-4fd2-b3c4-9129ae3ad0db)
13:42:18 lvdombrkr89 any ideas why?
13:42:54 sean-k-mooney lvdombrkr89: are you using an old nova client that dose not default to the 2.60+ microversion
13:43:11 stephenfin sean-k-mooney, sahid: lyarwood too. I'm sure I could trade a review of the stable patch for this https://review.openstack.org/#/c/550046/ :)
13:43:31 lvdombrkr89 sean-k-mooney : i use dashboard to launch instance
13:43:33 lvdombrkr89 not client
13:44:49 sean-k-mooney lvdombrkr89: i assume you are using queens release of horizon? multi attace lanned very late in queens if i remember correctly so perhaps horizon has not been updated with that micro version
13:45:49 lvdombrkr89 sean-k-mooney: yes i use queens. ok thanks so i would try to use client
13:46:23 stephenfin sean-k-mooney: Does Horizon support microversions? I recall someone saying it did not
13:46:53 bauzas no, I don't think so
13:47:04 bauzas like OSC
13:47:14 sean-k-mooney stephenfin: not sure but if it did not it would default to whatever was the nova clint default correct?
13:47:21 sean-k-mooney or just latest?
13:47:55 stephenfin sean-k-mooney: Not a clue :) I don't tend to use Horizon at all
13:52:13 bauzas stephenfin: sean-k-mooney: sec, see https://github.com/openstack/horizon/blob/master/openstack_dashboard/api/nova.py#L62
13:54:20 sean-k-mooney bauzas: from that it should cache the micro virsion range supported.
13:54:44 sean-k-mooney lvdombrkr89: did you upgrade your nova depoloyment without restarting horizon by any chance?
13:55:11 sean-k-mooney lvdombrkr89: if you did then perhaps horizon is still cacheing the old microversion
13:56:59 lvdombrkr89 sean-k-mooney: i was deploying queens from the scratch not upgrading
13:59:07 sean-k-mooney lvdombrkr89: that is strang. i only took a passing glance over the multi attach patch sets. my advice would be try to replicate using osc. if you cannot this is a horizon bug if you can reporduce then you should check what micro version is being retrun by the nova api.
14:01:30 lvdombrkr89 sean-k-mooney: okey i will try, will give feedback soon
14:27:30 openstackgerrit Surya Seetharaman proposed openstack/nova master: Add disabled column/field to cell_mappings table/object https://review.openstack.org/550090
14:29:02 openstackgerrit Surya Seetharaman proposed openstack/nova master: [WIP] Add disabled column/field to cell_mappings table/object https://review.openstack.org/550090
14:38:12 stephenfin I'm seeing logs from the nova-novncproxy service but I can't figure out what module/package is throwing them. Anyone know if it's possible to modify the output of the 'logging' module on a global basis?
14:41:58 gibi stephenfin: global on what level? you can change the logging config via the nova.conf
14:42:27 stephenfin gibi: Global in the context of a service (nova-novncproxy) and its dependent modules
14:42:57 stephenfin I'm thinking it's the websockify package throwing the INFO log but grep isn't helping me much, heh
14:44:04 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: Detach volumes when VM creation fails https://review.openstack.org/544144
14:44:40 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: Ensure the JSON-Schema covers the legacy v2 API https://review.openstack.org/543490
14:45:01 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: Allow force-delete even if task_state is not None https://review.openstack.org/543343
14:45:36 gibi stephenfin: have you tried changing default_log_levels https://docs.openstack.org/oslo.log/latest/configuration/index.html#DEFAULT.default_log_levels
14:46:14 stephenfin gibi: That's _exactly_ what I was looking for :)
14:46:55 gibi stephenfin: cool :)
14:48:54 gibi sahid: thanks for updating the rxt/tx spec. I left some questions inline https://review.openstack.org/#/c/539605/
14:49:21 stephenfin gibi: Got the bugger (it was websockify). Ta :)
14:49:47 gibi stephenfin: :)
14:50:19 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Add extra-specs to the flavor show and detail API calls https://review.openstack.org/544158
14:54:11 mriedem sahid: i wonder if you'd have any ideas about this bug https://bugs.launchpad.net/cinder/+bug/1753394
14:54:12 openstack Launchpad bug 1753394 in OpenStack Compute (nova) "block_device_mapping_v2 cannot work with Pike" [Undecided,New]
14:54:26 mriedem lyarwood: or you
14:56:25 odyssey4me mriedem Not sure if you've seen this, but I expect that anyone using upgrade_levels/compute=auto will find that the upgrades fail. We have a patch up - I wonder if we could discuss whether that's appropriate or whether there's an alternative we should explore? https://review.openstack.org/#/c/549737/1
14:57:39 mriedem odyssey4me: dansmith should take a look
14:58:07 mriedem sahid: my guess in that bug is something in https://github.com/openstack/nova/commit/c25629f85feb53b5be0347f68c43b3b55fb9f137 caused a regression
15:02:01 sahid mriedem: yes that seems to be related, i can't really figure that out right now, but i will try to look at it soon

Earlier   Later