| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-18 | |||
| 18:48:28 | melwitt | sean-k-mooney: whatever is easier | |
| 18:49:14 | sean-k-mooney | experimental is eaiser but it was a bunch of resouce so ill do a dnm patch with both the os-vif and neutron iptables tempest job | |
| 18:49:29 | sean-k-mooney | they are the same but the os-vif on uses master and the neutorn one uses the released version | |
| 18:49:59 | melwitt | ah ok, sounds cool | |
| 18:50:55 | sean-k-mooney | im kind of surprised we dont have any iptables testing in nova actully but i guess it has not been the default for a few releases at this point | |
| 18:51:41 | sean-k-mooney | ya even the greade job uses the ovs firewall at this point https://zuul.opendev.org/t/openstack/build/ec9d4bf702c84bcf9ae355ae0a735cd1/log/controller/logs/etc/neutron/plugins/ml2/ml2_conf.ini#299 | |
| 18:52:12 | sean-k-mooney | although to be faire we also dont have any ovn testing so its not surprising | |
| 18:53:52 | melwitt | interesting. so the original bug only affects deployments using iptables. I hadn't realized that | |
| 18:55:02 | sean-k-mooney | oh wait sorry | |
| 18:55:08 | sean-k-mooney | i had that backwards | |
| 18:55:18 | sean-k-mooney | it only affect deployment not using iptables | |
| 18:55:23 | sean-k-mooney | so ya all the jobs test this | |
| 18:55:34 | sean-k-mooney | for iptables ovs already plugs the interface | |
| 18:55:39 | sean-k-mooney | when not using iptables libvirt does | |
| 18:56:09 | sean-k-mooney | now os-vif will always do it regardesll fo if you use iptables or not | |
| 18:56:52 | sean-k-mooney | melwitt: we fixed it for iptables alreeady via os-vif but we could not fix it via os-vif in this case since os-vif did not do the plugging libvirt did | |
| 18:57:12 | melwitt | I see, thanks | |
| 19:00:12 | sean-k-mooney | for context the vlan specification defines vlan 4095 as reserved for implemantion use. in ovs its used for the "dead vlan" per the specification swithc whould not forward packets with vlan 4095 so ovs adopted vlan 4095 as a reserved vlan on which all traffic will be droped | |
| 19:00:41 | sean-k-mooney | so the mitigation is to place allports on vlan 4095 until they are moved to the tenant vlan by neutron | |
| 19:01:48 | sean-k-mooney | you enable that behavior by setting isolate_vif but we cant do it via libvirt since it will conflict with neutron if you set vlan 4095 in the xml | |
| 19:02:26 | sean-k-mooney | * isolate_vif is set in the nova.conf in the os vif plugin section | |
| 19:04:34 | sean-k-mooney | so [os_vif_ovs]/isolate_vif=true which i should proably have destack set by default at somepoint | |
| 19:04:45 | melwitt | ah cool, thanks for those details, nice context for the fix approach | |
| 19:08:57 | sean-k-mooney | anyway to your orginial question all the tempest jobs are exefcuting the code change so i wont submit a DNM patch | |
| 19:12:16 | melwitt | sean-k-mooney: is there any change we would see in the logs as a result of the os-vif plug vs the old libvirt plug? | |
| 19:26:39 | melwitt | looks like in the guest xml the device changes to <interface type="ethernet"> from <interface type="bridge"> | |
| 19:29:01 | sean-k-mooney | yes the xml would be different | |
| 19:29:12 | sean-k-mooney | and that is in the nova compute log | |
| 19:29:20 | melwitt | cool, yeah I am looking at it | |
| 19:30:21 | sean-k-mooney | sorry was away just putting on dinner | |
| 19:30:38 | melwitt | np, I found it :) | |
| 19:34:09 | sean-k-mooney | huh i wonder if infra changed something in zuul | |
| 19:34:28 | sean-k-mooney | its not rendering the xmls correctly anymore | |
| 19:34:46 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/00ff397ca5374a0b9441036d43ee3416/log/controller/logs/screen-n-cpu.txt#7371 should be the full xml but you can only see it in the raw view | |
| 19:34:58 | melwitt | I'm looking at the raw files so I didn't see the zuul thing. but I did notice the UI changed recently in other ways | |
| 19:35:23 | sean-k-mooney | ya i might need to repot that as a bug since you cant link to lines in the raw view | |
| 19:35:38 | sean-k-mooney | i guess they have an issue with multi line stringes or soemthing | |
| 19:36:47 | melwitt | hm yeah, need to ask in -infra | |
| 20:41:25 | melwitt | sean-k-mooney: I'm trying to confirm that the nova-grenade-multinode fails on PS25 is unrelated to the change. I can't tell why the live migrations failed. do you expect there would be any issues with a live migration going from N to N+1 or vice versa during an upgrade wrt to the plugging mechanism? | |
| 21:13:36 | sean-k-mooney | the port would be created on ovs twice but that is ok | |
| 21:14:08 | sean-k-mooney | os-vif would create it via pre live migate on the new host then libvirt would create it again by deleteing it and readding it | |
| 21:14:14 | sean-k-mooney | that should not break anything | |
| 21:14:48 | sean-k-mooney | libvirt basiclaly does ovs-vsctl del-port --if-exist ... ovs-vsctl add-port | |
| #openstack-nova - 2020-11-19 | |||
| 00:42:24 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Re-proposes 'Proposal for a safer remote console with password authentication https://review.opendev.org/759828 | |
| 01:07:16 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Remove tenant_id https://review.opendev.org/737241 | |
| 01:10:42 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: [Trivial] update the upgrade release goal https://review.opendev.org/763294 | |
| 06:20:31 | ninad | Hi | |
| 06:20:32 | ninad | I am using linuxbridge for the neutron services and follow the installation guide as documented but during manila instance creation I am getting [Errno 113] EHOSTUNREACH _test_server_connection | |
| 06:21:03 | ninad | can someone please help me? | |
| 07:58:40 | openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | |
| 08:06:20 | openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | |
| 09:24:40 | lyarwood | aarents: apologies, had a fun day downstream yesterday, your test changes LGTM, I missed that you had already pulled in the devstack change within the tempest change so assuming it's still passing this should be good to go now. | |
| 09:31:16 | aarents | lyarwood: many thanks, I just repush with nit fix sugested by gmann | |
| 10:21:35 | stephenfin | sean-k-mooney: I think there's something broken in https://review.opendev.org/#/c/602432/. It appears grenade hasn't passed on that for the last couple of revisions. Haven't gone diving through logs yet | |
| 11:13:45 | openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | |
| 11:14:24 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Add devstack-plugin-ceph-compute-local-ephemeral to experimental https://review.opendev.org/743220 | |
| 11:45:38 | sean-k-mooney | stephenfin: if we were to go from new to old i guess it would break | |
| 11:46:09 | sean-k-mooney | the xml would be generated on the souce with type ethernet | |
| 11:46:28 | sean-k-mooney | and the os-vif on the dest would be a noop | |
| 11:46:38 | sean-k-mooney | so nothing would plug the interface | |
| 11:48:02 | sean-k-mooney | the singel node grenade have been passing but not multinode | |
| 11:50:22 | sean-k-mooney | the only way i think of to fix that is to first backport the neutron change then flip the default for the option in os-vif/depercate it for removal and backport that too before we merge on master | |
| 11:50:55 | sean-k-mooney | and i would then have to keep backporting the neutron and os-vif patches first for every release. | |
| 11:51:56 | sean-k-mooney | stephenfin: lyarwood ^ is there a better way around that | |
| 12:06:50 | lyarwood | sean-k-mooney: we shouldn't be going from new to old in the tests right? | |
| 12:07:02 | lyarwood | sean-k-mooney: only old to new, or is that not defined in grenade | |
| 12:07:07 | sean-k-mooney | we do live migration back and fort | |
| 12:07:14 | sean-k-mooney | in the multi node job | |
| 12:08:06 | stephenfin | sean-k-mooney: You could make this dependent on a service version check | |
| 12:08:23 | sean-k-mooney | not really i would have to keep changin it | |
| 12:08:50 | stephenfin | why? Once everything's on Wallaby with a suitably new version of os-vif, we're good, no? | |
| 12:08:52 | sean-k-mooney | i thory this should go to osp 10 so all the way back to queens | |
| 12:08:55 | stephenfin | ah | |
| 12:09:22 | sean-k-mooney | im thinking of stoping at train to be honest | |
| 12:09:34 | sean-k-mooney | but it defiently needs to be backported a few releases | |
| 12:09:41 | stephenfin | (osp 10 is newton, fwiw) | |
| 12:09:55 | sean-k-mooney | ... of course it is | |
| 12:10:32 | sean-k-mooney | since its related to a security issue we are still ment to backport it althou technically i guess 13 would be enough at this point | |
| 12:10:33 | stephenfin | what you've proposed would require os-vif be upgraded before nova, right? I don't think we can count on that | |
| 12:10:45 | stephenfin | *os-vif and neutron | |
| 12:11:34 | sean-k-mooney | without the neturon patch you just get erros in the neutron logs but it fixes its self | |
| 12:11:44 | sean-k-mooney | the os-vif change however would be needed first ya | |
| 12:13:05 | sean-k-mooney | lyarwood: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ff2/602432/26/check/nova-grenade-multinode/ff2bd9a/logs/new/tempest_conf.txt | |
| 12:13:12 | sean-k-mooney | live_migrate_back_and_forth = True | |
| 12:13:37 | sean-k-mooney | that is why its failing | |
| 12:14:06 | gibi | stephenfin: do you have a lauchpad bp filled for this? https://review.opendev.org/#/c/755109 | |
| 12:14:24 | sean-k-mooney | its off in live migration job and that all the new version so it passes the https://zuul.opendev.org/t/openstack/build/e2fa8ae7802f41309877b537f51ba660/log/controller/logs/tempest_conf.txt#76 | |
| 12:14:26 | stephenfin | gibi: I though I did but I'm not sure | |
| 12:14:39 | gibi | the bp link in the specs does not seem to work for me | |
| 12:16:28 | sean-k-mooney | ya cant find it either | |
| 12:16:45 | sean-k-mooney | but it should be a simple copy past of the into section | |
| 12:17:01 | stephenfin | gibi: Didn't look like it. Apologies. Have created one now https://blueprints.launchpad.net/nova/+spec/modernize-os-hypervisors-api | |
| 12:17:07 | gibi | thanks | |
| 12:19:13 | sean-k-mooney | weird i cant add the spec via the spec link so i put it in the whiteborad | |
| 12:19:34 | sean-k-mooney | i tought i could do that once i was on the bug team but i guess not | |
| 12:28:30 | gibi | sean-k-mooney: I guess we edited the whiteboard in parallel and my change overwrote yours, sorry. | |
| 12:30:52 | sean-k-mooney | no launchpad gave me a permission error | |
| 12:31:38 | gibi | interesting, I got the following mail from launchpad | |
| 12:31:38 | gibi | Blueprint changed by sean mooney: | |
| 12:31:39 | gibi | Whiteboard set to: | |