| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-19 | |||
| 10:27:24 | gibi | fairly recent master | |
| 10:28:00 | sean-k-mooney | i belive to stop the race you need to set a neutron config option | |
| 10:28:00 | sean-k-mooney | i belive to stop the race you need to set a neutron config option | |
| 10:30:22 | sean-k-mooney | gibi: i think https://review.opendev.org/c/openstack/nova/+/770745 should help with the issue | |
| 10:30:22 | sean-k-mooney | gibi: i think https://review.opendev.org/c/openstack/nova/+/770745 should help with the issue | |
| 10:30:47 | sean-k-mooney | but you would still need to enable https://review.opendev.org/c/openstack/neutron/+/790702/1/neutron/conf/common.py | |
| 10:30:47 | sean-k-mooney | but you would still need to enable https://review.opendev.org/c/openstack/neutron/+/790702/1/neutron/conf/common.py | |
| 10:31:04 | sean-k-mooney | [nova]/live_migration_events=true | |
| 10:31:04 | sean-k-mooney | [nova]/live_migration_events=true | |
| 10:31:25 | sean-k-mooney | gibi: i have not fully looked at the bug yet so ill do that shortly and try an confirm if its the same thing | |
| 10:31:25 | sean-k-mooney | gibi: i have not fully looked at the bug yet so ill do that shortly and try an confirm if its the same thing | |
| 10:31:43 | sean-k-mooney | gibi: but im pretty sure its https://bugzilla.redhat.com/show_bug.cgi?id=1930432 | |
| 10:31:45 | openstack | bugzilla.redhat.com bug 1930432 in openstack-nova "Nova evacuate fails due to timeout waiting for a network-vif-plugged event for instance" [Medium,Verified] - Assigned to smooney | |
| 10:31:45 | openstack | bugzilla.redhat.com bug 1930432 in openstack-nova "Nova evacuate fails due to timeout waiting for a network-vif-plugged event for instance" [Medium,Verified] - Assigned to smooney | |
| 10:31:45 | sean-k-mooney | gibi: but im pretty sure its https://bugzilla.redhat.com/show_bug.cgi?id=1930432 | |
| 10:31:46 | gibi | does live_migration_event affects evacutaion? | |
| 10:31:46 | gibi | does live_migration_event affects evacutaion? | |
| 10:33:51 | sean-k-mooney | kind of the patch has 2 fixes. one it only send events form the l2 agent to nova and second if fixes the filtering in nueton to allow procing the port if it has any port binding for the current host. previously it only did it for active port bindign which was wrong | |
| 10:33:51 | sean-k-mooney | kind of the patch has 2 fixes. one it only send events form the l2 agent to nova and second if fixes the filtering in nueton to allow procing the port if it has any port binding for the current host. previously it only did it for active port bindign which was wrong | |
| 10:34:20 | sean-k-mooney | gibi: are you able to repoduce this reliably | |
| 10:34:20 | sean-k-mooney | gibi: are you able to repoduce this reliably | |
| 10:34:37 | gibi | sean-k-mooney: nope, it is random and seemingly infrequent | |
| 10:34:37 | gibi | sean-k-mooney: nope, it is random and seemingly infrequent | |
| 10:35:21 | sean-k-mooney | ok i was going to suggest chanig the default of that config option since its ment to be removed in Y it shoudl default to true in Xena anyway | |
| 10:35:21 | sean-k-mooney | ok i was going to suggest chanig the default of that config option since its ment to be removed in Y it shoudl default to true in Xena anyway | |
| 10:35:32 | sean-k-mooney | and then using a depends on patch to test | |
| 10:35:32 | sean-k-mooney | and then using a depends on patch to test | |
| 10:35:45 | sean-k-mooney | but if its infrequent we might not see a difference | |
| 10:35:45 | sean-k-mooney | but if its infrequent we might not see a difference | |
| 10:36:58 | gibi | looking at the live migration fix, I think we could have a similar race during evacuation causing the vent to arrive too early | |
| 10:36:58 | gibi | looking at the live migration fix, I think we could have a similar race during evacuation causing the vent to arrive too early | |
| 10:37:45 | sean-k-mooney | gibi: one casue fo this in the past was that during evac in the ci we were previously just stoping the nova compute agent not the neutron l2 agent so wehn we did the port update it would respond | |
| 10:37:45 | sean-k-mooney | gibi: one casue fo this in the past was that during evac in the ci we were previously just stoping the nova compute agent not the neutron l2 agent so wehn we did the port update it would respond | |
| 10:37:58 | sean-k-mooney | e.g. the souce agent woudl say yep its already wired | |
| 10:37:58 | sean-k-mooney | e.g. the souce agent woudl say yep its already wired | |
| 10:38:10 | gibi | sean-k-mooney: I confirmed that the q-agt is dead on the source host during this run | |
| 10:38:10 | gibi | sean-k-mooney: I confirmed that the q-agt is dead on the source host during this run | |
| 10:38:12 | sean-k-mooney | i think we fixed that in our job | |
| 10:38:12 | sean-k-mooney | i think we fixed that in our job | |
| 10:38:21 | sean-k-mooney | ok good | |
| 10:38:21 | gibi | yes, it is fixed in the test | |
| 10:38:21 | sean-k-mooney | ok good | |
| 10:38:21 | gibi | yes, it is fixed in the test | |
| 10:38:35 | sean-k-mooney | am did we also ensure the job is not swapted to ovn | |
| 10:38:35 | sean-k-mooney | am did we also ensure the job is not swapted to ovn | |
| 10:38:38 | sean-k-mooney | i belvie we did | |
| 10:38:38 | sean-k-mooney | i belvie we did | |
| 10:38:50 | gibi | checking... | |
| 10:38:50 | gibi | checking... | |
| 10:41:14 | sean-k-mooney | it looks like ml2/ovs | |
| 10:41:14 | sean-k-mooney | it looks like ml2/ovs | |
| 10:41:16 | gibi | we only held back nova-next on ovs https://review.opendev.org/c/openstack/nova/+/776944 but nova-live-migration moved to OVN as far as I see | |
| 10:41:16 | gibi | we only held back nova-next on ovs https://review.opendev.org/c/openstack/nova/+/776944 but nova-live-migration moved to OVN as far as I see | |
| 10:41:47 | sean-k-mooney | really the build you linked to has teh screen-q-agt.txt | |
| 10:41:47 | sean-k-mooney | really the build you linked to has teh screen-q-agt.txt | |
| 10:43:49 | sean-k-mooney | i guess it has not run since the default change maybe? | |
| 10:43:49 | sean-k-mooney | i guess it has not run since the default change maybe? | |
| 10:43:59 | gibi | hm it run last week | |
| 10:43:59 | gibi | hm it run last week | |
| 10:44:09 | gibi | let me check a more recent run... | |
| 10:44:09 | sean-k-mooney | this was from friday yes | |
| 10:44:09 | gibi | let me check a more recent run... | |
| 10:44:09 | sean-k-mooney | this was from friday yes | |
| 10:44:47 | gibi | hm a todays run also has q-agt.txt | |
| 10:44:47 | gibi | hm a todays run also has q-agt.txt | |
| 10:45:13 | gibi | but I don't see in the zuul config where we set the Q_AGENT option to be openvswitch for this job | |
| 10:45:13 | gibi | but I don't see in the zuul config where we set the Q_AGENT option to be openvswitch for this job | |
| 10:45:14 | sean-k-mooney | yep also looking at it | |
| 10:45:14 | sean-k-mooney | yep also looking at it | |
| 10:45:30 | sean-k-mooney | is this still zullv2? | |
| 10:45:30 | sean-k-mooney | is this still zullv2? | |
| 10:46:18 | sean-k-mooney | no? https://github.com/openstack/nova/blob/master/.zuul.yaml#L53-L87 | |
| 10:46:18 | sean-k-mooney | no? https://github.com/openstack/nova/blob/master/.zuul.yaml#L53-L87 | |
| 10:46:24 | sean-k-mooney | that looks like it v3 | |
| 10:46:24 | sean-k-mooney | that looks like it v3 | |
| 10:46:58 | sean-k-mooney | ok well that is a different mistery to look at after | |
| 10:46:58 | sean-k-mooney | ok well that is a different mistery to look at after | |
| 10:48:03 | sean-k-mooney | maybe it has something to do with neutron-trunk: true but that seams odd if it does | |
| 10:48:03 | sean-k-mooney | maybe it has something to do with neutron-trunk: true but that seams odd if it does | |
| 10:48:24 | gibi | yeah, that is the only odd thing in the job config | |
| 10:48:24 | gibi | yeah, that is the only odd thing in the job config | |
| 10:48:54 | sean-k-mooney | do you think the previos live_migation_evetns patch would fix evacuate? | |
| 10:48:54 | sean-k-mooney | do you think the previos live_migation_evetns patch would fix evacuate? | |
| 10:49:22 | sean-k-mooney | we were planning to backport that to train as the race exits basically since netuuron has been a thing | |
| 10:49:22 | sean-k-mooney | we were planning to backport that to train as the race exits basically since netuuron has been a thing | |
| 10:50:12 | gibi | I don't think it fixes the evacuate as we see the error on master _after_ the live migration event patch merged | |
| 10:50:12 | gibi | I don't think it fixes the evacuate as we see the error on master _after_ the live migration event patch merged | |
| 10:50:23 | gibi | but the logic behind the failure can be the same | |
| 10:50:23 | gibi | but the logic behind the failure can be the same | |
| 10:50:38 | sean-k-mooney | gibi: well its disabled by default | |
| 10:50:38 | sean-k-mooney | gibi: well its disabled by default | |
| 10:51:12 | sean-k-mooney | because it need the nova patch to be present to enable it | |
| 10:51:12 | sean-k-mooney | because it need the nova patch to be present to enable it | |
| 10:51:19 | gibi | also it only changes _get_neutron_events_for_live_migration() which is fairly live migration specific | |
| 10:51:19 | gibi | also it only changes _get_neutron_events_for_live_migration() which is fairly live migration specific | |
| 10:51:55 | gibi | ohh you talk about the neutron fixes | |
| 10:51:55 | gibi | ohh you talk about the neutron fixes | |
| 10:52:04 | sean-k-mooney | yes | |
| 10:52:11 | gibi | we can try that | |
| 10:52:11 | gibi | we can try that | |