| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-19 | |||
| 09:33:54 | ignaziocassano | So, the solution at this time is waiting for someone solves the bug, right ? | |
| 09:33:54 | ignaziocassano | So, the solution at this time is waiting for someone solves the bug, right ? | |
| 09:34:18 | kashyap | ignaziocassano: Not really; as upstream QEMU claims these problems should not occur with newer QEMU releases | |
| 09:34:18 | kashyap | ignaziocassano: Not really; as upstream QEMU claims these problems should not occur with newer QEMU releases | |
| 09:34:47 | kashyap | But if you can generate a reproducer that'll help. But usually reproducers in this case are difficult to come by. | |
| 09:34:47 | kashyap | But if you can generate a reproducer that'll help. But usually reproducers in this case are difficult to come by. | |
| 09:35:22 | ignaziocassano | kashyap: let me know if I am wrong: new qemu version come with centos 8, right ? | |
| 09:35:22 | ignaziocassano | kashyap: let me know if I am wrong: new qemu version come with centos 8, right ? | |
| 09:35:54 | kashyap | ignaziocassano: Yes. Or even a potentially updated QEMU on CentOS7 (if you're running older ones; I haven't checked) | |
| 09:35:55 | kashyap | ignaziocassano: Yes. Or even a potentially updated QEMU on CentOS7 (if you're running older ones; I haven't checked) | |
| 09:37:56 | ignaziocassano | kashyap: I am running the last available for centos 7 repos. I cannot migrate to centos 8 before upgrading to train, but with train there is another bug for live migration because loss packages during it | |
| 09:37:56 | ignaziocassano | kashyap: I am running the last available for centos 7 repos. I cannot migrate to centos 8 before upgrading to train, but with train there is another bug for live migration because loss packages during it | |
| 09:38:57 | ignaziocassano | I stein I solved with Seean Mooney workaround to force port legacy binding. On train I have not found any workaround yet | |
| 09:38:57 | ignaziocassano | I stein I solved with Seean Mooney workaround to force port legacy binding. On train I have not found any workaround yet | |
| 09:39:32 | kashyap | I see; I don't know about that problem, afraid | |
| 09:39:32 | kashyap | I see; I don't know about that problem, afraid | |
| 09:41:11 | gibi | sean-k-mooney: hi! If you have time then please check https://bugs.launchpad.net/nova/+bug/1928922 Is the observation correct? If yes then why neutron sends vif-plugged before nova even requested the plug? | |
| 09:41:16 | openstack | Launchpad bug 1928922 in OpenStack Compute (nova) "evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it." [Medium,New] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:41:16 | openstack | Launchpad bug 1928922 in OpenStack Compute (nova) "evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it." [Medium,New] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:41:16 | gibi | sean-k-mooney: hi! If you have time then please check https://bugs.launchpad.net/nova/+bug/1928922 Is the observation correct? If yes then why neutron sends vif-plugged before nova even requested the plug? | |
| 09:42:17 | ignaziocassano | kashyap: thanks, I will keep informed with the bug you mentioned | |
| 09:42:17 | ignaziocassano | kashyap: thanks, I will keep informed with the bug you mentioned | |
| 09:52:50 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 09:52:51 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 09:52:52 | openstack | bug 1897528 in OpenStack Compute (nova) "32bit pci domain number is not supported" [High,In progress] https://launchpad.net/bugs/1897528 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:52:52 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 09:52:53 | openstack | bug 1897528 in OpenStack Compute (nova) "32bit pci domain number is not supported" [High,In progress] https://launchpad.net/bugs/1897528 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:52:53 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 10:19:51 | sean-k-mooney | gibi: that should be fixed now | |
| 10:19:51 | sean-k-mooney | gibi: that should be fixed now | |
| 10:20:36 | sean-k-mooney | gibi: i have not read it fully but there was a race with the neutron dhcp server | |
| 10:20:37 | sean-k-mooney | gibi: i have not read it fully but there was a race with the neutron dhcp server | |
| 10:20:57 | sean-k-mooney | gibi: what release was this | |
| 10:20:57 | sean-k-mooney | gibi: what release was this | |
| 10:27:10 | gibi | sean-k-mooney: this is from master | |
| 10:27:11 | gibi | sean-k-mooney: this is from master | |
| 10:27:24 | gibi | fairly recent master | |
| 10:27:24 | gibi | fairly recent master | |
| 10:28:00 | sean-k-mooney | i belive to stop the race you need to set a neutron config option | |
| 10:28:00 | sean-k-mooney | i belive to stop the race you need to set a neutron config option | |
| 10:30:22 | sean-k-mooney | gibi: i think https://review.opendev.org/c/openstack/nova/+/770745 should help with the issue | |
| 10:30:22 | sean-k-mooney | gibi: i think https://review.opendev.org/c/openstack/nova/+/770745 should help with the issue | |
| 10:30:47 | sean-k-mooney | but you would still need to enable https://review.opendev.org/c/openstack/neutron/+/790702/1/neutron/conf/common.py | |
| 10:30:47 | sean-k-mooney | but you would still need to enable https://review.opendev.org/c/openstack/neutron/+/790702/1/neutron/conf/common.py | |
| 10:31:04 | sean-k-mooney | [nova]/live_migration_events=true | |
| 10:31:04 | sean-k-mooney | [nova]/live_migration_events=true | |
| 10:31:25 | sean-k-mooney | gibi: i have not fully looked at the bug yet so ill do that shortly and try an confirm if its the same thing | |
| 10:31:25 | sean-k-mooney | gibi: i have not fully looked at the bug yet so ill do that shortly and try an confirm if its the same thing | |
| 10:31:43 | sean-k-mooney | gibi: but im pretty sure its https://bugzilla.redhat.com/show_bug.cgi?id=1930432 | |
| 10:31:45 | openstack | bugzilla.redhat.com bug 1930432 in openstack-nova "Nova evacuate fails due to timeout waiting for a network-vif-plugged event for instance" [Medium,Verified] - Assigned to smooney | |
| 10:31:45 | openstack | bugzilla.redhat.com bug 1930432 in openstack-nova "Nova evacuate fails due to timeout waiting for a network-vif-plugged event for instance" [Medium,Verified] - Assigned to smooney | |
| 10:31:45 | sean-k-mooney | gibi: but im pretty sure its https://bugzilla.redhat.com/show_bug.cgi?id=1930432 | |
| 10:31:46 | gibi | does live_migration_event affects evacutaion? | |
| 10:31:46 | gibi | does live_migration_event affects evacutaion? | |
| 10:33:51 | sean-k-mooney | kind of the patch has 2 fixes. one it only send events form the l2 agent to nova and second if fixes the filtering in nueton to allow procing the port if it has any port binding for the current host. previously it only did it for active port bindign which was wrong | |
| 10:33:51 | sean-k-mooney | kind of the patch has 2 fixes. one it only send events form the l2 agent to nova and second if fixes the filtering in nueton to allow procing the port if it has any port binding for the current host. previously it only did it for active port bindign which was wrong | |
| 10:34:20 | sean-k-mooney | gibi: are you able to repoduce this reliably | |
| 10:34:20 | sean-k-mooney | gibi: are you able to repoduce this reliably | |
| 10:34:37 | gibi | sean-k-mooney: nope, it is random and seemingly infrequent | |
| 10:34:37 | gibi | sean-k-mooney: nope, it is random and seemingly infrequent | |
| 10:35:21 | sean-k-mooney | ok i was going to suggest chanig the default of that config option since its ment to be removed in Y it shoudl default to true in Xena anyway | |
| 10:35:21 | sean-k-mooney | ok i was going to suggest chanig the default of that config option since its ment to be removed in Y it shoudl default to true in Xena anyway | |
| 10:35:32 | sean-k-mooney | and then using a depends on patch to test | |
| 10:35:32 | sean-k-mooney | and then using a depends on patch to test | |
| 10:35:45 | sean-k-mooney | but if its infrequent we might not see a difference | |
| 10:35:45 | sean-k-mooney | but if its infrequent we might not see a difference | |
| 10:36:58 | gibi | looking at the live migration fix, I think we could have a similar race during evacuation causing the vent to arrive too early | |
| 10:36:58 | gibi | looking at the live migration fix, I think we could have a similar race during evacuation causing the vent to arrive too early | |
| 10:37:45 | sean-k-mooney | gibi: one casue fo this in the past was that during evac in the ci we were previously just stoping the nova compute agent not the neutron l2 agent so wehn we did the port update it would respond | |
| 10:37:45 | sean-k-mooney | gibi: one casue fo this in the past was that during evac in the ci we were previously just stoping the nova compute agent not the neutron l2 agent so wehn we did the port update it would respond | |
| 10:37:58 | sean-k-mooney | e.g. the souce agent woudl say yep its already wired | |
| 10:37:58 | sean-k-mooney | e.g. the souce agent woudl say yep its already wired | |
| 10:38:10 | gibi | sean-k-mooney: I confirmed that the q-agt is dead on the source host during this run | |
| 10:38:10 | gibi | sean-k-mooney: I confirmed that the q-agt is dead on the source host during this run | |
| 10:38:12 | sean-k-mooney | i think we fixed that in our job | |
| 10:38:12 | sean-k-mooney | i think we fixed that in our job | |
| 10:38:21 | sean-k-mooney | ok good | |
| 10:38:21 | gibi | yes, it is fixed in the test | |
| 10:38:21 | sean-k-mooney | ok good | |
| 10:38:21 | gibi | yes, it is fixed in the test | |
| 10:38:35 | sean-k-mooney | am did we also ensure the job is not swapted to ovn | |
| 10:38:35 | sean-k-mooney | am did we also ensure the job is not swapted to ovn | |
| 10:38:38 | sean-k-mooney | i belvie we did | |
| 10:38:38 | sean-k-mooney | i belvie we did | |
| 10:38:50 | gibi | checking... | |
| 10:38:50 | gibi | checking... | |
| 10:41:14 | sean-k-mooney | it looks like ml2/ovs | |
| 10:41:14 | sean-k-mooney | it looks like ml2/ovs | |
| 10:41:16 | gibi | we only held back nova-next on ovs https://review.opendev.org/c/openstack/nova/+/776944 but nova-live-migration moved to OVN as far as I see | |
| 10:41:16 | gibi | we only held back nova-next on ovs https://review.opendev.org/c/openstack/nova/+/776944 but nova-live-migration moved to OVN as far as I see | |
| 10:41:47 | sean-k-mooney | really the build you linked to has teh screen-q-agt.txt | |
| 10:41:47 | sean-k-mooney | really the build you linked to has teh screen-q-agt.txt | |
| 10:43:49 | sean-k-mooney | i guess it has not run since the default change maybe? | |
| 10:43:49 | sean-k-mooney | i guess it has not run since the default change maybe? | |
| 10:43:59 | gibi | hm it run last week | |
| 10:43:59 | gibi | hm it run last week | |
| 10:44:09 | gibi | let me check a more recent run... | |
| 10:44:09 | sean-k-mooney | this was from friday yes | |
| 10:44:09 | gibi | let me check a more recent run... | |
| 10:44:09 | sean-k-mooney | this was from friday yes | |