| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-04 | |||
| 11:32:18 | sean-k-mooney | normally it compeltes first so perhaps there is a bug where it wont send the event if the port is already active or something like that | |
| 11:39:13 | gibi | sean-k-mooney: thanks for looking into this, could you please add these details to the bug | |
| 11:39:43 | sean-k-mooney | sure just responeing to a review comment and ill add them | |
| 11:40:03 | gibi | thank you | |
| 11:55:07 | sean-k-mooney | gibi: actully the dhcp agent has marked it as complete | |
| 11:55:17 | sean-k-mooney | in the neutron server log | |
| 12:02:46 | sean-k-mooney | gibi: neutron did send the event | |
| 12:02:48 | sean-k-mooney | INFO neutron.notifiers.nova [-] Nova event matching ['req-17e4fee0-ad06-4350-af09-1db0d331d6b5'] response: {'server_uuid': '3a81145d-d263-4e1d-8ec3-faf38fed34f2', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': 'b6dc2b79-ed38-4907-86e2-bdff1c5a9b9f', 'code': 200} | |
| 12:05:18 | gibi | sean-k-mooney: timing doesn't add up | |
| 12:05:37 | gibi | nova plugged the vif at May 03 16:48:41.237538 | |
| 12:05:51 | sean-k-mooney | we recvie it at 16:47:22 | |
| 12:06:11 | sean-k-mooney | ya so this is proably because of the dhcp agent race | |
| 12:06:25 | gibi | neutron should not send the plugged event _before_ nova plugs the vif | |
| 12:06:28 | sean-k-mooney | i bet we dont have the config option set in neutron | |
| 12:06:56 | sean-k-mooney | gibi: there was a race in neutron where it would not wait for both the dhcp and l2 agent to finish | |
| 12:07:35 | sean-k-mooney | it was fixed by https://review.opendev.org/c/openstack/neutron/+/766277 | |
| 12:07:57 | sean-k-mooney | although hum | |
| 12:08:02 | gibi | this is not a live migration | |
| 12:08:02 | sean-k-mooney | that ws for live migration | |
| 12:08:07 | gibi | this is evacuate | |
| 12:08:16 | sean-k-mooney | the same could happen there | |
| 12:08:39 | gibi | and I still not get it. Can neutron send a vif-plugged event _before_ nova even plugs the vif via os-vif? | |
| 12:08:43 | sean-k-mooney | let me check if that is enabled or not | |
| 12:09:00 | sean-k-mooney | gibi: its simple | |
| 12:09:20 | sean-k-mooney | the port was active on the host we are evacuating form | |
| 12:09:36 | sean-k-mooney | so it thinks the l2 agent is finished doign its work | |
| 12:09:47 | sean-k-mooney | so when the dhcp agent responds it sends the event | |
| 12:09:54 | sean-k-mooney | that is proably what is happening here | |
| 12:10:06 | sean-k-mooney | that is what happend for live migration | |
| 12:10:15 | sean-k-mooney | i expect the same behavior for evacuate | |
| 12:11:44 | sean-k-mooney | gibi: https://zuul.opendev.org/t/openstack/build/518f8641b9a7419391b0f99f795f26bd/log/controller/logs/etc/neutron/neutron_conf.txt#1322-1334 | |
| 12:11:47 | sean-k-mooney | its disabled | |
| 12:11:57 | sean-k-mooney | we should try enabling that and see if it fixes the problem | |
| 12:12:59 | sean-k-mooney | gibi: the fix in neutron is based on the presence of migratin_to in the port profile | |
| 12:13:22 | gibi | I thought the expected sequence would be: 1) nova bounds the port to the target host 2) nova plugs the vif on the target host 3) neutron agents plugs the other end of the vif on the target host 4) neutron sends the vif-plugged event to nova | |
| 12:14:32 | gibi | but based on what you said 3) and 4) happens before 2) | |
| 12:14:35 | sean-k-mooney | right but what actuly happens is the l2 agent on the souce host say the port is alreay bound the dhcp agent say the dhcp configurtion is correct and then neuton sends the event | |
| 12:14:45 | gibi | I see, | |
| 12:14:53 | sean-k-mooney | https://review.opendev.org/c/openstack/neutron/+/766277/10/neutron/agent/rpc.py | |
| 12:15:39 | sean-k-mooney | add filterign so that we only consider updates form the host that migrating_to points too | |
| 12:15:43 | gibi | OK | |
| 12:16:08 | gibi | based on the comment in the config, the live_migration_events flag should be removed in neutron in Zed already | |
| 12:16:23 | sean-k-mooney | well it will be removed in zed and always enabled | |
| 12:16:29 | sean-k-mooney | but i dont know if they have done that yet | |
| 12:16:35 | gibi | OK | |
| 12:17:15 | sean-k-mooney | https://github.com/openstack/neutron/blob/master/neutron/conf/common.py#L182-L199= | |
| 12:17:17 | sean-k-mooney | still there | |
| 12:17:31 | gibi | OK, I will push a patch to enable that flag in the hybrid plug job | |
| 12:17:59 | sean-k-mooney | ack ralonsoh do you have patches to remove https://github.com/openstack/neutron/blob/master/neutron/conf/common.py#L182-L199= | |
| 12:18:17 | sean-k-mooney | and alway smake that the correct behvior | |
| 12:23:34 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable live_migration_events in nova-ovs-hybrid-plug https://review.opendev.org/c/openstack/nova/+/840446 | |
| 12:23:39 | gibi | sean-k-mooney: ^^ | |
| 12:24:04 | sean-k-mooney | ack just realised im meent to be on a call | |
| 12:24:08 | sean-k-mooney | ill take a look after | |
| 12:24:12 | gibi | thanks | |
| 12:24:24 | gibi | btw this error was pretty infrequent so we might not know if this fixes it | |
| 12:26:02 | sean-k-mooney | ok ya if its what i think it is its an race that we ocationly loose | |
| 12:26:27 | sean-k-mooney | normaly the event arrives after we start waiting but not always | |
| 12:34:24 | ralonsoh | sean-k-mooney, I'll do it now | |
| 12:35:06 | sean-k-mooney | ralonsoh: thanks no rush but better to land that earlier then late in the cycle | |
| 12:35:12 | ralonsoh | sean-k-mooney, btw, just to confirm: the value will be True now | |
| 12:35:19 | sean-k-mooney | yes | |
| 12:35:21 | ralonsoh | perfect | |
| 12:35:25 | sean-k-mooney | well you are removing the option yes | |
| 12:45:54 | ralonsoh | sean-k-mooney, https://review.opendev.org/c/openstack/neutron/+/840448 | |
| 12:46:03 | ralonsoh | I'll wait until your reviews | |
| 12:47:02 | sean-k-mooney | ralonsoh: it looks good to me but one nit | |
| 12:47:08 | ralonsoh | sure | |
| 12:47:09 | sean-k-mooney | you do not have a release note for this | |
| 12:47:19 | ralonsoh | right, it deserves one | |
| 12:47:21 | ralonsoh | I'll add it | |
| 12:48:31 | sean-k-mooney | +0 while you adress that but otherwise +1 | |
| 12:48:49 | sean-k-mooney | i do want to also see the ci run on this too but it should be fine | |
| 12:49:14 | ralonsoh | perfect | |
| 13:09:18 | gmann | bauzas: ah, good point on yoga spec link. in that case, let me keep it same name then and in detail i can mention what all things this BP is targeting. I will update zed proposed spec file | |
| 13:09:33 | bauzas | gmann: ack ok | |
| 13:47:12 | opendevreview | Rico Lin proposed openstack/nova-specs master: Add vIOMMU device support for libvirt driver https://review.opendev.org/c/openstack/nova-specs/+/840310 | |
| 14:11:00 | sean-k-mooney | gibi: so rodolfo has a patch to make this the default but do we ant to proceed with https://review.opendev.org/c/openstack/nova/+/840446 anyway and perhaps backport that to the relevent branches ? | |
| 14:11:55 | gibi | I haven't see hits of this bug on stable. do we have the hybrid job on stable? | |
| 14:12:29 | sean-k-mooney | i think artom is adding it we had before yoga too since that was the devstack default | |
| 14:14:38 | sean-k-mooney | gibi: https://github.com/openstack/nova/blob/stable/yoga/.zuul.yaml#L650= | |
| 14:15:17 | gibi | then I think it make sense to land this now and backport it | |
| 14:15:28 | gibi | then we can drop the flag from master when ralonsoh's patch lands | |
| 14:16:05 | sean-k-mooney | this https://review.opendev.org/c/openstack/nova/+/828413/2 and https://review.opendev.org/c/openstack/nova/+/828418 will be adding it to xena and wallaby | |
| 14:16:18 | sean-k-mooney | cool | |
| 16:04:28 | opendevreview | Rico Lin proposed openstack/nova-specs master: Add vIOMMU device support for libvirt driver https://review.opendev.org/c/openstack/nova-specs/+/840310 | |
| 16:05:56 | ricolin | sean-k-mooney: Thanks for your very detailed review, just update the spec accordingly :) | |
| 16:34:35 | opendevreview | Merged openstack/nova-specs master: Re-propose remove tenant_id https://review.opendev.org/c/openstack/nova-specs/+/837789 | |
| 16:44:10 | opendevreview | Ghanshyam proposed openstack/nova-specs master: Re-propose allow Project admin to list allowed hypervisors https://review.opendev.org/c/openstack/nova-specs/+/833165 | |
| 16:45:25 | gmann | gibi: dansmith ^^ as you reviewed it in Yoga cycle. re-proposing the spec. | |
| #openstack-nova - 2022-05-05 | |||
| 09:00:45 | kashyap | gibi: Hi, the 'py310' and 'nova-next' failures seem nothing to do with my patch, yeah? - https://review.opendev.org/c/openstack/nova/+/838926 | |
| 09:01:46 | gibi | py310 is definitely not, as it fails at setting up mysql | |
| 09:02:07 | bauzas | wow, python 31.0 release number, time flies | |
| 09:02:16 | bauzas | :) | |
| 09:02:44 | gibi | the nova-next fails with some rsync issue so that is also not your patch | |
| 09:03:00 | gibi | I'm not sure we have a tracking bug for the py310 issue but we should | |
| 09:03:17 | gibi | the nova-next issue might be intermittent, but worth to check if it happens regurarly | |
| 09:03:50 | gibi | bauzas: yeah, I push the problem out to time when py 31.0 will be released until then py310 is py 3.10 for me :D | |
| 09:03:52 | kashyap | I see, yeah - I'll file a tracking bz for py310 'mysql' fail | |
| 09:04:22 | gibi | kashyap: try to check if the same job fails for other repos too maybe there is also a tracking bug somewhere outside of nova | |