| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-19 | |||
| 09:20:38 | ignaziocassano | kashyap, unfortunately since the affected vm remained in pause on source and destination, I had to destroy them so the log files contains only: | |
| 09:20:38 | ignaziocassano | kashyap, unfortunately since the affected vm remained in pause on source and destination, I had to destroy them so the log files contains only: | |
| 09:20:42 | ignaziocassano | 021-05-19 08:11:19.800+0000: initiating migration | |
| 09:20:42 | ignaziocassano | 021-05-19 08:11:19.800+0000: initiating migration | |
| 09:20:42 | ignaziocassano | 2021-05-19 08:12:30.446+0000: shutting down, reason=destroyed | |
| 09:20:43 | ignaziocassano | 2021-05-19 08:12:30.446+0000: shutting down, reason=destroyed | |
| 09:21:00 | kashyap | Hm; that won't help us much, afraid | |
| 09:21:00 | kashyap | Hm; that won't help us much, afraid | |
| 09:21:12 | ignaziocassano | I could try on destination host if there is something else | |
| 09:21:12 | ignaziocassano | I could try on destination host if there is something else | |
| 09:22:10 | admin0 | hi guys .. i have a strange issue: openstack server show $uuid shows its host on hypervisor => h7 . on the hypervisor, hostname, hostname -f and virsh hostname returns h7 , but during migration of this instance ( ceph backed ) to say h9 or h10, it says h7 host not found .. .. how can i address/solve this ? | |
| 09:22:10 | admin0 | hi guys .. i have a strange issue: openstack server show $uuid shows its host on hypervisor => h7 . on the hypervisor, hostname, hostname -f and virsh hostname returns h7 , but during migration of this instance ( ceph backed ) to say h9 or h10, it says h7 host not found .. .. how can i address/solve this ? | |
| 09:22:19 | ignaziocassano | yes | |
| 09:22:19 | ignaziocassano | yes | |
| 09:22:41 | ignaziocassano | I post it on openstack pastebin | |
| 09:22:41 | ignaziocassano | I post it on openstack pastebin | |
| 09:22:46 | ignaziocassano | :q! | |
| 09:22:46 | ignaziocassano | :q! | |
| 09:22:47 | admin0 | https://gist.github.com/a1git/c22ec0c17aaa9dcf95fd7485eb76af2f looks like my hostnames cycled from h7, h7. h7.openstack.local .. so now i am unable to migrate anything off of the hosts | |
| 09:22:47 | admin0 | https://gist.github.com/a1git/c22ec0c17aaa9dcf95fd7485eb76af2f looks like my hostnames cycled from h7, h7. h7.openstack.local .. so now i am unable to migrate anything off of the hosts | |
| 09:24:06 | ignaziocassano | kashyap: http://paste.openstack.org/show/805476/ | |
| 09:24:06 | ignaziocassano | kashyap: http://paste.openstack.org/show/805476/ | |
| 09:25:44 | kashyap | ignaziocassano: Ha, see this: | |
| 09:25:45 | kashyap | ignaziocassano: Ha, see this: | |
| 09:25:51 | kashyap | --- | |
| 09:25:51 | kashyap | 2021-05-19T08:12:30.397606Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:08.0/virtio-blk' | |
| 09:25:52 | kashyap | 2021-05-19T08:12:30.397606Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:08.0/virtio-blk' | |
| 09:25:55 | kashyap | 2021-05-19T08:12:30.399542Z qemu-kvm: load of migration failed: Input/output error | |
| 09:25:55 | kashyap | 2021-05-19T08:12:30.399542Z qemu-kvm: load of migration failed: Input/output error | |
| 09:25:58 | kashyap | 2021-05-19 08:12:31.022+0000: shutting down, reason=crashed | |
| 09:25:58 | kashyap | 2021-05-19 08:12:31.022+0000: shutting down, reason=crashed | |
| 09:26:00 | kashyap | --- | |
| 09:26:00 | kashyap | --- | |
| 09:26:03 | kashyap | I've pretty sure seen that 'virtio-blk' error before | |
| 09:26:04 | kashyap | I've pretty sure seen that 'virtio-blk' error before | |
| 09:27:12 | ignaziocassano | kashyap: so do you think instanance not worked fine also before live migration ? | |
| 09:27:12 | ignaziocassano | kashyap: so do you think instanance not worked fine also before live migration ? | |
| 09:27:43 | kashyap | ignaziocassano: See this bug (comments are complex; browse it from the bottom): https://bugs.launchpad.net/nova/+bug/1761798 | |
| 09:27:45 | kashyap | ignaziocassano: See this bug (comments are complex; browse it from the bottom): https://bugs.launchpad.net/nova/+bug/1761798 | |
| 09:27:45 | openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | |
| 09:27:45 | openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | |
| 09:28:25 | kashyap | ignaziocassano: No, not really; see this bit from the logs: | |
| 09:28:25 | kashyap | ignaziocassano: No, not really; see this bit from the logs: | |
| 09:28:26 | kashyap | [quote] | |
| 09:28:26 | kashyap | [quote] | |
| 09:28:27 | kashyap | we get this "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted. | |
| 09:28:28 | kashyap | we get this "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted. | |
| 09:28:32 | kashyap | [/quote] | |
| 09:28:32 | kashyap | [/quote] | |
| 09:28:59 | kashyap | (From my comment#11) | |
| 09:29:00 | kashyap | (From my comment#11) | |
| 09:30:38 | ignaziocassano | kashyap: my englush is poor but I did not read a conclusion. | |
| 09:30:38 | ignaziocassano | kashyap: my englush is poor but I did not read a conclusion. | |
| 09:31:22 | ignaziocassano | kayshap: do you think post_copy and autoconverge can help ? | |
| 09:31:22 | ignaziocassano | kayshap: do you think post_copy and autoconverge can help ? | |
| 09:31:25 | kashyap | ignaziocassano: Your English is not poor; there's no conclusion yet, as the problem is complex. | |
| 09:31:25 | kashyap | ignaziocassano: Your English is not poor; there's no conclusion yet, as the problem is complex. | |
| 09:31:55 | kashyap | ignaziocassano: They can help if you're guest is doing I/O faster than your migration can keep up | |
| 09:31:55 | kashyap | ignaziocassano: They can help if you're guest is doing I/O faster than your migration can keep up | |
| 09:33:54 | ignaziocassano | So, the solution at this time is waiting for someone solves the bug, right ? | |
| 09:33:54 | ignaziocassano | So, the solution at this time is waiting for someone solves the bug, right ? | |
| 09:34:18 | kashyap | ignaziocassano: Not really; as upstream QEMU claims these problems should not occur with newer QEMU releases | |
| 09:34:18 | kashyap | ignaziocassano: Not really; as upstream QEMU claims these problems should not occur with newer QEMU releases | |
| 09:34:47 | kashyap | But if you can generate a reproducer that'll help. But usually reproducers in this case are difficult to come by. | |
| 09:34:47 | kashyap | But if you can generate a reproducer that'll help. But usually reproducers in this case are difficult to come by. | |
| 09:35:22 | ignaziocassano | kashyap: let me know if I am wrong: new qemu version come with centos 8, right ? | |
| 09:35:22 | ignaziocassano | kashyap: let me know if I am wrong: new qemu version come with centos 8, right ? | |
| 09:35:54 | kashyap | ignaziocassano: Yes. Or even a potentially updated QEMU on CentOS7 (if you're running older ones; I haven't checked) | |
| 09:35:55 | kashyap | ignaziocassano: Yes. Or even a potentially updated QEMU on CentOS7 (if you're running older ones; I haven't checked) | |
| 09:37:56 | ignaziocassano | kashyap: I am running the last available for centos 7 repos. I cannot migrate to centos 8 before upgrading to train, but with train there is another bug for live migration because loss packages during it | |
| 09:37:56 | ignaziocassano | kashyap: I am running the last available for centos 7 repos. I cannot migrate to centos 8 before upgrading to train, but with train there is another bug for live migration because loss packages during it | |
| 09:38:57 | ignaziocassano | I stein I solved with Seean Mooney workaround to force port legacy binding. On train I have not found any workaround yet | |
| 09:38:57 | ignaziocassano | I stein I solved with Seean Mooney workaround to force port legacy binding. On train I have not found any workaround yet | |
| 09:39:32 | kashyap | I see; I don't know about that problem, afraid | |
| 09:39:32 | kashyap | I see; I don't know about that problem, afraid | |
| 09:41:11 | gibi | sean-k-mooney: hi! If you have time then please check https://bugs.launchpad.net/nova/+bug/1928922 Is the observation correct? If yes then why neutron sends vif-plugged before nova even requested the plug? | |
| 09:41:16 | gibi | sean-k-mooney: hi! If you have time then please check https://bugs.launchpad.net/nova/+bug/1928922 Is the observation correct? If yes then why neutron sends vif-plugged before nova even requested the plug? | |
| 09:41:16 | openstack | Launchpad bug 1928922 in OpenStack Compute (nova) "evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it." [Medium,New] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:41:16 | openstack | Launchpad bug 1928922 in OpenStack Compute (nova) "evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it." [Medium,New] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:42:17 | ignaziocassano | kashyap: thanks, I will keep informed with the bug you mentioned | |
| 09:42:17 | ignaziocassano | kashyap: thanks, I will keep informed with the bug you mentioned | |
| 09:52:50 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 09:52:51 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 09:52:52 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 09:52:52 | openstack | bug 1897528 in OpenStack Compute (nova) "32bit pci domain number is not supported" [High,In progress] https://launchpad.net/bugs/1897528 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 09:52:53 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 09:52:53 | openstack | bug 1897528 in OpenStack Compute (nova) "32bit pci domain number is not supported" [High,In progress] https://launchpad.net/bugs/1897528 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 10:19:51 | sean-k-mooney | gibi: that should be fixed now | |
| 10:19:51 | sean-k-mooney | gibi: that should be fixed now | |
| 10:20:36 | sean-k-mooney | gibi: i have not read it fully but there was a race with the neutron dhcp server | |
| 10:20:37 | sean-k-mooney | gibi: i have not read it fully but there was a race with the neutron dhcp server | |
| 10:20:57 | sean-k-mooney | gibi: what release was this | |
| 10:20:57 | sean-k-mooney | gibi: what release was this | |
| 10:27:10 | gibi | sean-k-mooney: this is from master | |
| 10:27:11 | gibi | sean-k-mooney: this is from master | |
| 10:27:24 | gibi | fairly recent master | |
| 10:27:24 | gibi | fairly recent master | |
| 10:28:00 | sean-k-mooney | i belive to stop the race you need to set a neutron config option | |
| 10:28:00 | sean-k-mooney | i belive to stop the race you need to set a neutron config option | |
| 10:30:22 | sean-k-mooney | gibi: i think https://review.opendev.org/c/openstack/nova/+/770745 should help with the issue | |