| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-19 | |||
| 09:25:45 | kashyap | ignaziocassano: Ha, see this: | |
| 09:25:51 | kashyap | --- | |
| 09:25:51 | kashyap | 2021-05-19T08:12:30.397606Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:08.0/virtio-blk' | |
| 09:25:52 | kashyap | 2021-05-19T08:12:30.397606Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:08.0/virtio-blk' | |
| 09:25:55 | kashyap | 2021-05-19T08:12:30.399542Z qemu-kvm: load of migration failed: Input/output error | |
| 09:25:55 | kashyap | 2021-05-19T08:12:30.399542Z qemu-kvm: load of migration failed: Input/output error | |
| 09:25:58 | kashyap | 2021-05-19 08:12:31.022+0000: shutting down, reason=crashed | |
| 09:25:58 | kashyap | 2021-05-19 08:12:31.022+0000: shutting down, reason=crashed | |
| 09:26:00 | kashyap | --- | |
| 09:26:00 | kashyap | --- | |
| 09:26:03 | kashyap | I've pretty sure seen that 'virtio-blk' error before | |
| 09:26:04 | kashyap | I've pretty sure seen that 'virtio-blk' error before | |
| 09:27:12 | ignaziocassano | kashyap: so do you think instanance not worked fine also before live migration ? | |
| 09:27:12 | ignaziocassano | kashyap: so do you think instanance not worked fine also before live migration ? | |
| 09:27:43 | kashyap | ignaziocassano: See this bug (comments are complex; browse it from the bottom): https://bugs.launchpad.net/nova/+bug/1761798 | |
| 09:27:45 | kashyap | ignaziocassano: See this bug (comments are complex; browse it from the bottom): https://bugs.launchpad.net/nova/+bug/1761798 | |
| 09:27:45 | openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | |
| 09:27:45 | openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | |
| 09:28:25 | kashyap | ignaziocassano: No, not really; see this bit from the logs: | |
| 09:28:25 | kashyap | ignaziocassano: No, not really; see this bit from the logs: | |
| 09:28:26 | kashyap | [quote] | |
| 09:28:26 | kashyap | [quote] | |
| 09:28:27 | kashyap | we get this "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted. | |
| 09:28:28 | kashyap | we get this "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted. | |
| 09:28:32 | kashyap | [/quote] | |
| 09:28:32 | kashyap | [/quote] | |
| 09:28:59 | kashyap | (From my comment#11) | |
| 09:29:00 | kashyap | (From my comment#11) | |
| 09:30:38 | ignaziocassano | kashyap: my englush is poor but I did not read a conclusion. | |
| 09:30:38 | ignaziocassano | kashyap: my englush is poor but I did not read a conclusion. | |
| 09:31:22 | ignaziocassano | kayshap: do you think post_copy and autoconverge can help ? | |
| 09:31:22 | ignaziocassano | kayshap: do you think post_copy and autoconverge can help ? | |
| 09:31:25 | kashyap | ignaziocassano: Your English is not poor; there's no conclusion yet, as the problem is complex. | |
| 09:31:25 | kashyap | ignaziocassano: Your English is not poor; there's no conclusion yet, as the problem is complex. | |
| 09:31:55 | kashyap | ignaziocassano: They can help if you're guest is doing I/O faster than your migration can keep up | |
| 09:31:55 | kashyap | ignaziocassano: They can help if you're guest is doing I/O faster than your migration can keep up | |
| 09:33:54 | ignaziocassano | So, the solution at this time is waiting for someone solves the bug, right ? | |
| 09:33:54 | ignaziocassano | So, the solution at this time is waiting for someone solves the bug, right ? | |
| 09:34:18 | kashyap | ignaziocassano: Not really; as upstream QEMU claims these problems should not occur with newer QEMU releases | |
| 09:34:18 | kashyap | ignaziocassano: Not really; as upstream QEMU claims these problems should not occur with newer QEMU releases | |
| 09:34:47 | kashyap | But if you can generate a reproducer that'll help. But usually reproducers in this case are difficult to come by. | |
| 09:34:47 | kashyap | But if you can generate a reproducer that'll help. But usually reproducers in this case are difficult to come by. | |
| 09:35:22 | ignaziocassano | kashyap: let me know if I am wrong: new qemu version come with centos 8, right ? | |
| 09:35:22 | ignaziocassano | kashyap: let me know if I am wrong: new qemu version come with centos 8, right ? | |
| 09:35:54 | kashyap | ignaziocassano: Yes. Or even a potentially updated QEMU on CentOS7 (if you're running older ones; I haven't checked) | |
| 09:35:55 | kashyap | ignaziocassano: Yes. Or even a potentially updated QEMU on CentOS7 (if you're running older ones; I haven't checked) | |
| 09:37:56 | ignaziocassano | kashyap: I am running the last available for centos 7 repos. I cannot migrate to centos 8 before upgrading to train, but with train there is another bug for live migration because loss packages during it | |
| 09:37:56 | ignaziocassano | kashyap: I am running the last available for centos 7 repos. I cannot migrate to centos 8 before upgrading to train, but with train there is another bug for live migration because loss packages during it | |
| 09:38:57 | ignaziocassano | I stein I solved with Seean Mooney workaround to force port legacy binding. On train I have not found any workaround yet | |
| 09:38:57 | ignaziocassano | I stein I solved with Seean Mooney workaround to force port legacy binding. On train I have not found any workaround yet | |
| 09:39:32 | kashyap | I see; I don't know about that problem, afraid | |
| 09:39:32 | kashyap | I see; I don't know about that problem, afraid | |
| 09:41:11 | gibi | sean-k-mooney: hi! If you have time then please check https://bugs.launchpad.net/nova/+bug/1928922 Is the observation correct? If yes then why neutron sends vif-plugged before nova even requested the plug? | |
| 09:41:16 | gibi | sean-k-mooney: hi! If you have time then please check https://bugs.launchpad.net/nova/+bug/1928922 Is the observation correct? If yes then why neutron sends vif-plugged before nova even requested the plug? | |
| 09:41:16 | openstack | Launchpad bug 1928922 in OpenStack Compute (nova) "evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it." [Medium,New] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:41:16 | openstack | Launchpad bug 1928922 in OpenStack Compute (nova) "evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it." [Medium,New] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:42:17 | ignaziocassano | kashyap: thanks, I will keep informed with the bug you mentioned | |
| 09:42:17 | ignaziocassano | kashyap: thanks, I will keep informed with the bug you mentioned | |
| 09:52:50 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 09:52:51 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 09:52:52 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 09:52:52 | openstack | bug 1897528 in OpenStack Compute (nova) "32bit pci domain number is not supported" [High,In progress] https://launchpad.net/bugs/1897528 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:52:53 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 09:52:53 | openstack | bug 1897528 in OpenStack Compute (nova) "32bit pci domain number is not supported" [High,In progress] https://launchpad.net/bugs/1897528 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 10:19:51 | sean-k-mooney | gibi: that should be fixed now | |
| 10:19:51 | sean-k-mooney | gibi: that should be fixed now | |
| 10:20:36 | sean-k-mooney | gibi: i have not read it fully but there was a race with the neutron dhcp server | |
| 10:20:37 | sean-k-mooney | gibi: i have not read it fully but there was a race with the neutron dhcp server | |
| 10:20:57 | sean-k-mooney | gibi: what release was this | |
| 10:20:57 | sean-k-mooney | gibi: what release was this | |
| 10:27:10 | gibi | sean-k-mooney: this is from master | |
| 10:27:11 | gibi | sean-k-mooney: this is from master | |
| 10:27:24 | gibi | fairly recent master | |
| 10:27:24 | gibi | fairly recent master | |
| 10:28:00 | sean-k-mooney | i belive to stop the race you need to set a neutron config option | |
| 10:28:00 | sean-k-mooney | i belive to stop the race you need to set a neutron config option | |
| 10:30:22 | sean-k-mooney | gibi: i think https://review.opendev.org/c/openstack/nova/+/770745 should help with the issue | |
| 10:30:22 | sean-k-mooney | gibi: i think https://review.opendev.org/c/openstack/nova/+/770745 should help with the issue | |
| 10:30:47 | sean-k-mooney | but you would still need to enable https://review.opendev.org/c/openstack/neutron/+/790702/1/neutron/conf/common.py | |
| 10:30:47 | sean-k-mooney | but you would still need to enable https://review.opendev.org/c/openstack/neutron/+/790702/1/neutron/conf/common.py | |
| 10:31:04 | sean-k-mooney | [nova]/live_migration_events=true | |
| 10:31:04 | sean-k-mooney | [nova]/live_migration_events=true | |
| 10:31:25 | sean-k-mooney | gibi: i have not fully looked at the bug yet so ill do that shortly and try an confirm if its the same thing | |
| 10:31:25 | sean-k-mooney | gibi: i have not fully looked at the bug yet so ill do that shortly and try an confirm if its the same thing | |
| 10:31:43 | sean-k-mooney | gibi: but im pretty sure its https://bugzilla.redhat.com/show_bug.cgi?id=1930432 | |
| 10:31:45 | sean-k-mooney | gibi: but im pretty sure its https://bugzilla.redhat.com/show_bug.cgi?id=1930432 | |
| 10:31:45 | openstack | bugzilla.redhat.com bug 1930432 in openstack-nova "Nova evacuate fails due to timeout waiting for a network-vif-plugged event for instance" [Medium,Verified] - Assigned to smooney | |
| 10:31:45 | openstack | bugzilla.redhat.com bug 1930432 in openstack-nova "Nova evacuate fails due to timeout waiting for a network-vif-plugged event for instance" [Medium,Verified] - Assigned to smooney | |
| 10:31:46 | gibi | does live_migration_event affects evacutaion? | |
| 10:31:46 | gibi | does live_migration_event affects evacutaion? | |
| 10:33:51 | sean-k-mooney | kind of the patch has 2 fixes. one it only send events form the l2 agent to nova and second if fixes the filtering in nueton to allow procing the port if it has any port binding for the current host. previously it only did it for active port bindign which was wrong | |
| 10:33:51 | sean-k-mooney | kind of the patch has 2 fixes. one it only send events form the l2 agent to nova and second if fixes the filtering in nueton to allow procing the port if it has any port binding for the current host. previously it only did it for active port bindign which was wrong | |
| 10:34:20 | sean-k-mooney | gibi: are you able to repoduce this reliably | |
| 10:34:20 | sean-k-mooney | gibi: are you able to repoduce this reliably | |
| 10:34:37 | gibi | sean-k-mooney: nope, it is random and seemingly infrequent | |
| 10:34:37 | gibi | sean-k-mooney: nope, it is random and seemingly infrequent | |
| 10:35:21 | sean-k-mooney | ok i was going to suggest chanig the default of that config option since its ment to be removed in Y it shoudl default to true in Xena anyway | |
| 10:35:21 | sean-k-mooney | ok i was going to suggest chanig the default of that config option since its ment to be removed in Y it shoudl default to true in Xena anyway | |
| 10:35:32 | sean-k-mooney | and then using a depends on patch to test | |
| 10:35:32 | sean-k-mooney | and then using a depends on patch to test | |