| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-06 | |||
| 11:41:08 | sean-k-mooney | well both bugs are resolved with a vm hard reboot so i dont think this new bug is more serious | |
| 11:41:26 | sean-k-mooney | they both resolt in the same user observable behavior | |
| 11:41:32 | stephenfin | sean-k-mooney: So Newton was released 6 October 2016 (per Wikipedia). I wonder if it's fair to say anyone using...Ocata and upwards would be using OVS 2.6+ ? | |
| 11:41:51 | stephenfin | We could include a release note to that effect (do we do release notes for os-vif?) | |
| 11:42:06 | sean-k-mooney | i belive neutron has a minium version for ovs as we have for libvirt | |
| 11:42:37 | sean-k-mooney | well what im considering is should we confige this change to only when plugging a dpdkvhostuser port | |
| 11:43:19 | sean-k-mooney | s/confige/confine/ | |
| 11:43:23 | stephenfin | Hmm, that could be possible | |
| 11:43:47 | stephenfin | But if we do, I'd like to remove that check immediately afterwards and not backport that | |
| 11:44:13 | stephenfin | That could be discussed separately though | |
| 11:44:17 | sean-k-mooney | stephenfin: well no i would like it in the backport as the old bug was specifically for kernel ovs | |
| 11:45:02 | stephenfin | You mean you would like the check to be included in the backport? | |
| 11:45:03 | sean-k-mooney | when https://github.com/openstack/nova/commit/33cc64fb817 was done this fuction did not support vhost user | |
| 11:45:09 | stephenfin | If so, that's what I mean :) | |
| 11:45:24 | stephenfin | Add the check, backport that patch, then remove that check in a follow-up patch | |
| 11:45:50 | sean-k-mooney | ya so in the backport i think we should have the check so that we only change the behavior for vhost user ports | |
| 11:45:53 | stephenfin | ...given that we don't need it anymore (since OVS 2.6.0), and using '--if-exists' seems far saner to me | |
| 11:46:05 | stephenfin | OK, if that's possible then I've no objections | |
| 11:46:09 | stephenfin | sahid: Thoughts? ^ | |
| 11:46:56 | sean-k-mooney | stephenfin: did you mean '--may-exists' seams saner? | |
| 11:47:03 | sean-k-mooney | for master | |
| 11:47:54 | stephenfin | I did 🙈 | |
| 11:48:22 | stephenfin | i.e. don't special case dpdkvhostuser on master | |
| 11:48:23 | openstackgerrit | Pranab proposed openstack/os-vif master: Add native implementation OVSDB API https://review.openstack.org/482226 | |
| 11:50:47 | sean-k-mooney | stephenfin: yes agreed. also i would like to tackover rodolfo patches to add support for using the python binding so that we can stop using the commandline clients | |
| 11:51:58 | stephenfin | sean-k-mooney: +1 from me. I saw an email on openstack-dev about that earlier this week | |
| 11:52:14 | stephenfin | I'll review...probably not this week, but definitely next week | |
| 11:54:13 | sean-k-mooney | ill leave a comment regarding the backport and +w the patch for master. | |
| 11:55:22 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.openstack.org/507486 | |
| 12:25:18 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Don't wait for direct vnic plugged events during a reboot https://review.openstack.org/550046 | |
| 12:25:38 | lyarwood | stephenfin: ^ re the downstream ping | |
| 12:34:49 | sahid | sean-k-mooney, stephenfin - i don't think it make sense to have a special condition. I think that old fix was merged without real issue but in prevention | |
| 12:35:35 | sean-k-mooney | the old bug happens if a vm tap device does not exist during ovs startup | |
| 12:36:04 | sean-k-mooney | this happens if libvirt has not finished spawning the vms before ovs starts | |
| 12:36:42 | sean-k-mooney | so there is a race between libvirt/qemu starting the instance and ovs starting for ovs <2.6 | |
| 12:36:45 | sahid | did you noticed that for newton only hybrid vif was supported ? and for hybrid vif we have a special condition to check whether the hybrid vif already exists and so in that case we do not execute that command again | |
| 12:38:00 | sean-k-mooney | sahid: in the non hybrid case the vif is plugged by libvirt not os-vif | |
| 12:39:10 | sahid | can you point me the code where that issue could happen? | |
| 12:39:58 | sean-k-mooney | sahid: its a race condtion between ovs and vm tap devices being created its not an issue in the code itself | |
| 12:40:38 | sahid | sean-k-mooney: yes but we do not create any tap device when restarting nova-compue service | |
| 12:40:41 | sahid | that is what i mean | |
| 12:42:20 | sean-k-mooney | correct the "bridge|WARN|could not open network device tap2cf7dbad-9d (No such device)" only happens if the ovs-db has an entry for an interface that does not currently exist in the root namespace when ovs-vswitchd is starting | |
| 12:42:48 | sean-k-mooney | the fact the interface is prefixed with tap mean that hybrid plug is not in use meaning libvirt is pluging the interface not os-vif | |
| 12:43:33 | sean-k-mooney | so this will happen if ovs start before libvirt starts the vm after a reboot | |
| 12:43:40 | sean-k-mooney | but only on ovs <2.6 | |
| 12:44:26 | sahid | sean-k-mooney: after a reboot of what? | |
| 12:44:40 | sean-k-mooney | sahid: the host | |
| 12:46:11 | sahid | how possible that happen? | |
| 12:46:42 | sahid | the tap device would have been loaded by the kernel before ovs and libvirt | |
| 12:46:53 | sahid | then i don't understnad why libvirt is in relation here? | |
| 12:47:04 | sean-k-mooney | basicaly when ovs trys to add all ports in the ovsdb to the ovs kernel datapath it skips any it does not find, e.g. not running vms. the ovs revalidtor tread does not redo this check untill another port is added to ovs before 2.6 | |
| 12:47:17 | sean-k-mooney | sahid: tap devices are not persited across reboots | |
| 12:47:36 | sean-k-mooney | they are created by libvirt/qemu when the vm starts | |
| 12:47:51 | sahid | oh yes you are right | |
| 12:48:14 | sean-k-mooney | but the ovsdb info is preserved across reboot so ovs is expecting it to exist but libvirt/qemu have not created it yet | |
| 12:49:15 | sahid | yes that is a possible case | |
| 12:49:30 | sean-k-mooney | sahid: anyway im going to approve the patch as it is for master and request you add a check for the interface type on backport as we technically support ovs older then 2.6 in ocata. is that ok with you? | |
| 12:51:21 | sahid | sean-k-mooney: ok so please take the time to ack it soon as possible | |
| 12:52:29 | sean-k-mooney | sahid: i have it open on another tab. i was currently reading neutron release notes to see if they required ovs 2.6+ but i could not find any assertion to that effect for ocata | |
| 12:53:00 | sean-k-mooney | i need to run to a meeting at 1 but ill try to ack it before then | |
| 12:54:17 | sahid | sean-k-mooney: yes actually using 2.5 looks really old | |
| 12:54:21 | sean-k-mooney | looking at osp 11 redhat shiped ovs 2.6 for the ocata release if i can validate that suse,mirantis and canonical also did the same then perhaps we can skip the check but that is up to the stable mainance team to decide | |
| 12:54:37 | sahid | sean-k-mooney: that would be the ideal | |
| 12:56:30 | sahid | stephenfin, sean-k-mooney can we work together to ensure that for newton all the major openstack dist are using OVS2.6 | |
| 12:57:24 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/550057 | |
| 12:58:14 | sahid | s/netwon/ocata | |
| 13:02:56 | sahid | sean-k-mooney: I'm thinking about something, the scenario you have indicated is not in relation with the patch | |
| 13:15:35 | sean-k-mooney | sahid: correct however that was the reason that https://github.com/openstack/nova/commit/33cc64fb817 was merged to resolve https://bugs.launchpad.net/nova/+bug/1270973 | |
| 13:15:36 | openstack | Launchpad bug 1270973 in OpenStack Compute (nova) "Remove and recreate interface if already exists" [Medium,Fix released] - Assigned to Aaron Rosen (arosen) | |
| 13:18:00 | sean-k-mooney | sahid: so your fix would be reverting that previous change. but that bug can nolonger happen after ovs 2.6 so its safe to revert the old change on any moderen openstack deployment | |
| 13:20:06 | sahid | sean-k-mooney: the fix will be backported for stable queens only - i think we are ok | |
| 13:21:41 | sean-k-mooney | oh yes well the addtionally check would only be required for stable/ocata if it was backported to ocata. for queens and pike the patch can be applied as is | |
| 13:30:48 | sahid | sean-k-mooney: i'm just going to backport it for stable/queens and stable/pike, I don't think we can backport the patch in 3 release whether it's not a security issue | |
| 13:31:02 | sean-k-mooney | sahid: i have +w https://review.openstack.org/#/c/546588/1. i confirmed that both osp and the ubuntu cloud archive ship ovs 2.6 for ocata i would find it strange if other did not also as it was required for the contrack firewall driver, vhost-user reconnect and the netive python bindings for ovs which where three of the big feature in ocata for ovs | |
| 13:32:01 | sean-k-mooney | sahid: technicall if its not a security bug it has to be easily provale to not have any sidefects and is at the sole descretion of the stable maintainer. generally that means docs changes | |
| 13:32:51 | stephenfin | sean-k-mooney: Excellent. Good to get that closed off 🎉 | |
| 13:33:09 | sahid | sean-k-mooney: i don't want add functional change to a backport | |
| 13:33:26 | sahid | so if we want ot backport it for ocata we want it as it is | |
| 13:33:55 | sahid | if not so distribution will do some downstream only backport | |
| 13:34:40 | stephenfin | sahid: Yeah, my suggestion was to add the check to the current patch and then immediately remove it on master. This would let us backport the first patch all the way back | |
| 13:34:42 | sean-k-mooney | sahid: i think an downstream only backport would be best for ocata if required but i dont think may distros shipped 2.5 with ocata if any | |
| 13:35:01 | sahid | sean-k-mooney: seems reasonable | |
| 13:35:01 | stephenfin | I think that might have got lost in translation :) | |
| 13:35:15 | stephenfin | However, downstream only seems the way to go with Ocata and previous | |
| 13:35:17 | sahid | let have it backported for queens and pike so | |
| 13:36:16 | stephenfin | sahid: +1 | |
| 13:37:20 | openstackgerrit | Merged openstack/os-vif master: ovs: do not delete port if already exists https://review.openstack.org/546588 | |
| 13:37:48 | openstackgerrit | sahid proposed openstack/os-vif stable/queens: ovs: do not delete port if already exists https://review.openstack.org/550079 | |
| 13:38:17 | openstackgerrit | sahid proposed openstack/os-vif stable/pike: ovs: do not delete port if already exists https://review.openstack.org/550080 | |
| 13:39:43 | sean-k-mooney | sahid sound good to me. stephen and i are not in the nova-stable-maint group so i suggest you ping sylvain and perhaps matt to review for the backports | |
| 13:42:11 | lvdombrkr89 | hello folks | |
| 13:42:12 | lvdombrkr89 | when try to create instance with multiatach volume get error: Multiattach volumes are only supported starting with compute API version 2.60. (HTTP 400) (Request-ID: req-8babdaa8-29d9-4fd2-b3c4-9129ae3ad0db) | |
| 13:42:18 | lvdombrkr89 | any ideas why? | |
| 13:42:54 | sean-k-mooney | lvdombrkr89: are you using an old nova client that dose not default to the 2.60+ microversion | |
| 13:43:11 | stephenfin | sean-k-mooney, sahid: lyarwood too. I'm sure I could trade a review of the stable patch for this https://review.openstack.org/#/c/550046/ :) | |
| 13:43:31 | lvdombrkr89 | sean-k-mooney : i use dashboard to launch instance | |
| 13:43:33 | lvdombrkr89 | not client | |
| 13:44:49 | sean-k-mooney | lvdombrkr89: i assume you are using queens release of horizon? multi attace lanned very late in queens if i remember correctly so perhaps horizon has not been updated with that micro version | |
| 13:45:49 | lvdombrkr89 | sean-k-mooney: yes i use queens. ok thanks so i would try to use client | |
| 13:46:23 | stephenfin | sean-k-mooney: Does Horizon support microversions? I recall someone saying it did not | |