| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-19 | |||
| 08:50:09 | ignaziocassano | a openstack-discuss | |
| 08:50:09 | ignaziocassano | Hello Guys, | |
| 08:50:09 | ignaziocassano | a openstack-discuss | |
| 08:50:10 | ignaziocassano | The error reported is: | |
| 08:50:10 | ignaziocassano | on train centos7 I am facing live migration issue only for some instances (not all). | |
| 08:50:10 | ignaziocassano | The error reported is: | |
| 08:50:10 | ignaziocassano | on train centos7 I am facing live migration issue only for some instances (not all). | |
| 08:50:11 | ignaziocassano | 2021-05-19 08:45:57.096 142537 ERROR nova.compute.manager [-] [instance: b18450e8-b3db-4886-a737-c161d99c6a46] Live migration failed.: libvirtError: Unable to read from monitor: Connection reset by peer | |
| 08:51:16 | ignaziocassano | some instances migrate without errors. I tried to stop end restart libvirtd end nova-compute without solving | |
| 08:51:16 | ignaziocassano | some instances migrate without errors. I tried to stop end restart libvirtd end nova-compute without solving | |
| 08:52:48 | ignaziocassano | On instances where migration failed, If I stop them and I start on another node, If migrate them on original node and migrate again, it works | |
| 08:52:48 | ignaziocassano | On instances where migration failed, If I stop them and I start on another node, If migrate them on original node and migrate again, it works | |
| 08:55:03 | ignaziocassano | Sorry, the version is stein | |
| 08:55:03 | ignaziocassano | Sorry, the version is stein | |
| 08:58:19 | stephenfin | ignaziocassano: Have you looked into the libvirt logs directly or investigated syslog? | |
| 08:58:20 | stephenfin | ignaziocassano: Have you looked into the libvirt logs directly or investigated syslog? | |
| 08:58:54 | stephenfin | kashyap can correct me if I'm wrong, but that error usually implies the connection between QEMU and libvirt has died | |
| 08:58:54 | stephenfin | kashyap can correct me if I'm wrong, but that error usually implies the connection between QEMU and libvirt has died | |
| 08:59:07 | kashyap | * kashyap blinks and looks | |
| 08:59:26 | kashyap | stephenfin: Yes, you're right | |
| 08:59:26 | kashyap | stephenfin: Yes, you're right | |
| 09:00:13 | kashyap | But the underlying problem could be anywhere ... and needs more details to debug | |
| 09:00:13 | kashyap | But the underlying problem could be anywhere ... and needs more details to debug | |
| 09:00:25 | kashyap | ignaziocassano: Are you migrating including your storage? (I.e. "block migration"?) | |
| 09:00:25 | kashyap | ignaziocassano: Are you migrating including your storage? (I.e. "block migration"?) | |
| 09:01:05 | ignaziocassano | kashyap the storage is shared on netapp nfs | |
| 09:01:05 | ignaziocassano | kashyap the storage is shared on netapp nfs | |
| 09:01:50 | ignaziocassano | no errors on openvswitch agent | |
| 09:01:51 | ignaziocassano | no errors on openvswitch agent | |
| 09:03:32 | ignaziocassano | kashyap: I presume the connection between QEMU and libvirt ha died for some instances, but I do not know how I can verify it | |
| 09:03:32 | ignaziocassano | kashyap: I presume the connection between QEMU and libvirt ha died for some instances, but I do not know how I can verify it | |
| 09:04:08 | kashyap | ignaziocassano: Right; that's the reason. So one way to debug this is to obtain libvirt debug log filters that track interactions b/n QEMU and libvirt | |
| 09:04:08 | kashyap | ignaziocassano: Right; that's the reason. So one way to debug this is to obtain libvirt debug log filters that track interactions b/n QEMU and libvirt | |
| 09:04:14 | kashyap | ignaziocassano: You can get it this way: | |
| 09:04:14 | kashyap | ignaziocassano: You can get it this way: | |
| 09:04:31 | kashyap | On the relevant compute nodes (both source and dest): | |
| 09:04:31 | kashyap | On the relevant compute nodes (both source and dest): | |
| 09:04:40 | kashyap | (1) Set the log output file: $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 09:04:40 | kashyap | (1) Set the log output file: $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 09:04:49 | kashyap | (2) Configure the dynamic filters: | |
| 09:04:49 | kashyap | (2) Configure the dynamic filters: | |
| 09:04:54 | kashyap | $> virt-admin daemon-log-filters \ "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 09:04:55 | kashyap | $> virt-admin daemon-log-filters \ "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 09:05:05 | kashyap | (Remove that "\") | |
| 09:05:05 | kashyap | (Remove that "\") | |
| 09:05:14 | ignaziocassano | kashyap: on messages it reports: 19:22:29 podvc-kvm04 libvirtd: 2021-05-16 17:22:29.562+0000: 166295: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor | |
| 09:05:14 | ignaziocassano | kashyap: on messages it reports: 19:22:29 podvc-kvm04 libvirtd: 2021-05-16 17:22:29.562+0000: 166295: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor | |
| 09:05:17 | kashyap | And then re-run the migration of the affected instances | |
| 09:05:17 | kashyap | And then re-run the migration of the affected instances | |
| 09:05:41 | kashyap | ignaziocassano: Yeah; that's also "normal" / "expected"; doesn't tell much still | |
| 09:05:41 | kashyap | ignaziocassano: Yeah; that's also "normal" / "expected"; doesn't tell much still | |
| 09:06:06 | kashyap | ignaziocassano: BTW, can you quickly try this: | |
| 09:06:06 | kashyap | ignaziocassano: BTW, can you quickly try this: | |
| 09:06:14 | kashyap | $> journalctl -u libvirtd -l --since=yesterday -p err | |
| 09:06:14 | kashyap | $> journalctl -u libvirtd -l --since=yesterday -p err | |
| 09:08:11 | ignaziocassano | error : qemuDomainObjBeginJobInternal:6825 : Timed out during operation: cannot acquire state change lock (held by monitor=remo | |
| 09:08:11 | ignaziocassano | error : qemuDomainObjBeginJobInternal:6825 : Timed out during operation: cannot acquire state change lock (held by monitor=remo | |
| 09:08:53 | kashyap | Ugh, this one... | |
| 09:08:53 | kashyap | Ugh, this one... | |
| 09:09:31 | ignaziocassano | unfortunately I cannot try to move further vm because they are in production and I must inform my customers | |
| 09:09:31 | ignaziocassano | unfortunately I cannot try to move further vm because they are in production and I must inform my customers | |
| 09:10:11 | ignaziocassano | it us very strange because for some instances it works fine | |
| 09:10:11 | ignaziocassano | it us very strange because for some instances it works fine | |
| 09:10:33 | kashyap | ignaziocassano: So this is one of the most painful errors to debug from libvirt. I recently wrote a response on a Red Hat bugzilla | |
| 09:10:34 | kashyap | ignaziocassano: So this is one of the most painful errors to debug from libvirt. I recently wrote a response on a Red Hat bugzilla | |
| 09:10:54 | kashyap | Let me get that for you (beware: no easy solution here :-() | |
| 09:10:54 | kashyap | Let me get that for you (beware: no easy solution here :-() | |
| 09:11:42 | ignaziocassano | kashyap, I read some red hat suggestions, but they include the instance restart :-( | |
| 09:11:42 | ignaziocassano | kashyap, I read some red hat suggestions, but they include the instance restart :-( | |
| 09:12:11 | kashyap | ignaziocassano: Yes, you're not wrong, afraid. | |
| 09:12:11 | kashyap | ignaziocassano: Yes, you're not wrong, afraid. | |
| 09:12:39 | kashyap | ignaziocassano: These "cannot acquire state change lock" errors are notorious; and it could be due to QEMU getting hung, which in turn could be caused by stuck I/O | |
| 09:12:39 | kashyap | ignaziocassano: These "cannot acquire state change lock" errors are notorious; and it could be due to QEMU getting hung, which in turn could be caused by stuck I/O | |
| 09:14:07 | ignaziocassano | I did not see any nfs stale on my compute node | |
| 09:14:08 | ignaziocassano | I did not see any nfs stale on my compute node | |
| 09:15:52 | kashyap | ignaziocassano: Can you post the affected guest log from /var/log/libvir/qemu/instance-YYYY.log? | |
| 09:15:52 | kashyap | ignaziocassano: Can you post the affected guest log from /var/log/libvir/qemu/instance-YYYY.log? | |
| 09:16:11 | kashyap | It _might_ sometimes havea clue | |
| 09:16:11 | kashyap | It _might_ sometimes havea clue | |
| 09:16:26 | openstackgerrit | liuzhuangzhuang proposed openstack/nova master: Fix RBD timeout https://review.opendev.org/c/openstack/nova/+/786588 | |
| 09:16:35 | kashyap | And also this (if the output is long, use a paste-bin): `journalctl -u libvirtd -r` | |
| 09:16:35 | kashyap | And also this (if the output is long, use a paste-bin): `journalctl -u libvirtd -r` | |
| 09:20:38 | ignaziocassano | kashyap, unfortunately since the affected vm remained in pause on source and destination, I had to destroy them so the log files contains only: | |
| 09:20:38 | ignaziocassano | kashyap, unfortunately since the affected vm remained in pause on source and destination, I had to destroy them so the log files contains only: | |
| 09:20:42 | ignaziocassano | 2021-05-19 08:12:30.446+0000: shutting down, reason=destroyed | |
| 09:20:42 | ignaziocassano | 021-05-19 08:11:19.800+0000: initiating migration | |
| 09:20:42 | ignaziocassano | 021-05-19 08:11:19.800+0000: initiating migration | |
| 09:20:43 | ignaziocassano | 2021-05-19 08:12:30.446+0000: shutting down, reason=destroyed | |
| 09:21:00 | kashyap | Hm; that won't help us much, afraid | |
| 09:21:00 | kashyap | Hm; that won't help us much, afraid | |
| 09:21:12 | ignaziocassano | I could try on destination host if there is something else | |
| 09:21:12 | ignaziocassano | I could try on destination host if there is something else | |
| 09:22:10 | admin0 | hi guys .. i have a strange issue: openstack server show $uuid shows its host on hypervisor => h7 . on the hypervisor, hostname, hostname -f and virsh hostname returns h7 , but during migration of this instance ( ceph backed ) to say h9 or h10, it says h7 host not found .. .. how can i address/solve this ? | |
| 09:22:10 | admin0 | hi guys .. i have a strange issue: openstack server show $uuid shows its host on hypervisor => h7 . on the hypervisor, hostname, hostname -f and virsh hostname returns h7 , but during migration of this instance ( ceph backed ) to say h9 or h10, it says h7 host not found .. .. how can i address/solve this ? | |
| 09:22:19 | ignaziocassano | yes | |
| 09:22:19 | ignaziocassano | yes | |
| 09:22:41 | ignaziocassano | I post it on openstack pastebin | |
| 09:22:41 | ignaziocassano | I post it on openstack pastebin | |
| 09:22:46 | ignaziocassano | :q! | |
| 09:22:46 | ignaziocassano | :q! | |