| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-18 | |||
| 17:04:34 | bauzas | given we only have the RP uuid, and given we don't have yet the provider tree, I'll pass both the allocations *and the RPs to the new method I'm creating | |
| 17:04:34 | bauzas | given we only have the RP uuid, and given we don't have yet the provider tree, I'll pass both the allocations *and the RPs to the new method I'm creating | |
| 17:11:04 | sean-k-mooney | im not sure but i can see potially supporting a list of uuids | |
| 17:11:04 | sean-k-mooney | im not sure but i can see potially supporting a list of uuids | |
| 20:07:35 | admin0 | hi sean-k-mooney .. any ideas how i can fix this mess: https://gist.github.com/a1git/c22ec0c17aaa9dcf95fd7485eb76af2f | |
| 20:07:35 | admin0 | hi sean-k-mooney .. any ideas how i can fix this mess: https://gist.github.com/a1git/c22ec0c17aaa9dcf95fd7485eb76af2f | |
| 20:08:27 | admin0 | is there a way to know under which "hostname" the instance I am trying to migrate is listed under .. so that i can set the hostname to that exact and migrate it off | |
| 20:08:27 | admin0 | is there a way to know under which "hostname" the instance I am trying to migrate is listed under .. so that i can set the hostname to that exact and migrate it off | |
| 20:09:15 | admin0 | openstack server show $uuid shows it as h7 . while the hostname is also h7, but during migration is telling h7 not found .. so even though i have everything set as h7 and restarted nova-compute, i think its thinking it as different | |
| 20:09:16 | admin0 | openstack server show $uuid shows it as h7 . while the hostname is also h7, but during migration is telling h7 not found .. so even though i have everything set as h7 and restarted nova-compute, i think its thinking it as different | |
| #openstack-nova - 2021-05-19 | |||
| 00:48:59 | openstackgerrit | norman shen proposed openstack/nova master: Saving security group to info_cache https://review.opendev.org/c/openstack/nova/+/786348 | |
| 02:16:08 | openstackgerrit | norman shen proposed openstack/nova-specs master: Speed up server details https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| 03:12:57 | openstackgerrit | norman shen proposed openstack/nova-specs master: Speed up server details https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| 03:38:01 | openstackgerrit | norman shen proposed openstack/nova-specs master: Speed up server details https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| 08:23:20 | openstackgerrit | Daniel Bengtsson proposed openstack/nova master: Use the new type HostDomainOpt. https://review.opendev.org/c/openstack/nova/+/788240 | |
| 08:23:20 | openstackgerrit | Daniel Bengtsson proposed openstack/nova master: Use the new type HostDomainOpt. https://review.opendev.org/c/openstack/nova/+/788240 | |
| 08:50:08 | ignaziocassano | 10:48 (1 minuto fa) | |
| 08:50:08 | ignaziocassano | Ignazio Cassano <ignaziocassano@gmail.com> | |
| 08:50:08 | ignaziocassano | 10:48 (1 minuto fa) | |
| 08:50:08 | ignaziocassano | Ignazio Cassano <ignaziocassano@gmail.com> | |
| 08:50:09 | ignaziocassano | Hello Guys, | |
| 08:50:09 | ignaziocassano | a openstack-discuss | |
| 08:50:09 | ignaziocassano | Hello Guys, | |
| 08:50:09 | ignaziocassano | a openstack-discuss | |
| 08:50:10 | ignaziocassano | The error reported is: | |
| 08:50:10 | ignaziocassano | on train centos7 I am facing live migration issue only for some instances (not all). | |
| 08:50:10 | ignaziocassano | The error reported is: | |
| 08:50:10 | ignaziocassano | on train centos7 I am facing live migration issue only for some instances (not all). | |
| 08:50:11 | ignaziocassano | 2021-05-19 08:45:57.096 142537 ERROR nova.compute.manager [-] [instance: b18450e8-b3db-4886-a737-c161d99c6a46] Live migration failed.: libvirtError: Unable to read from monitor: Connection reset by peer | |
| 08:51:16 | ignaziocassano | some instances migrate without errors. I tried to stop end restart libvirtd end nova-compute without solving | |
| 08:51:16 | ignaziocassano | some instances migrate without errors. I tried to stop end restart libvirtd end nova-compute without solving | |
| 08:52:48 | ignaziocassano | On instances where migration failed, If I stop them and I start on another node, If migrate them on original node and migrate again, it works | |
| 08:52:48 | ignaziocassano | On instances where migration failed, If I stop them and I start on another node, If migrate them on original node and migrate again, it works | |
| 08:55:03 | ignaziocassano | Sorry, the version is stein | |
| 08:55:03 | ignaziocassano | Sorry, the version is stein | |
| 08:58:19 | stephenfin | ignaziocassano: Have you looked into the libvirt logs directly or investigated syslog? | |
| 08:58:20 | stephenfin | ignaziocassano: Have you looked into the libvirt logs directly or investigated syslog? | |
| 08:58:54 | stephenfin | kashyap can correct me if I'm wrong, but that error usually implies the connection between QEMU and libvirt has died | |
| 08:58:54 | stephenfin | kashyap can correct me if I'm wrong, but that error usually implies the connection between QEMU and libvirt has died | |
| 08:59:07 | kashyap | * kashyap blinks and looks | |
| 08:59:26 | kashyap | stephenfin: Yes, you're right | |
| 08:59:26 | kashyap | stephenfin: Yes, you're right | |
| 09:00:13 | kashyap | But the underlying problem could be anywhere ... and needs more details to debug | |
| 09:00:13 | kashyap | But the underlying problem could be anywhere ... and needs more details to debug | |
| 09:00:25 | kashyap | ignaziocassano: Are you migrating including your storage? (I.e. "block migration"?) | |
| 09:00:25 | kashyap | ignaziocassano: Are you migrating including your storage? (I.e. "block migration"?) | |
| 09:01:05 | ignaziocassano | kashyap the storage is shared on netapp nfs | |
| 09:01:05 | ignaziocassano | kashyap the storage is shared on netapp nfs | |
| 09:01:50 | ignaziocassano | no errors on openvswitch agent | |
| 09:01:51 | ignaziocassano | no errors on openvswitch agent | |
| 09:03:32 | ignaziocassano | kashyap: I presume the connection between QEMU and libvirt ha died for some instances, but I do not know how I can verify it | |
| 09:03:32 | ignaziocassano | kashyap: I presume the connection between QEMU and libvirt ha died for some instances, but I do not know how I can verify it | |
| 09:04:08 | kashyap | ignaziocassano: Right; that's the reason. So one way to debug this is to obtain libvirt debug log filters that track interactions b/n QEMU and libvirt | |
| 09:04:08 | kashyap | ignaziocassano: Right; that's the reason. So one way to debug this is to obtain libvirt debug log filters that track interactions b/n QEMU and libvirt | |
| 09:04:14 | kashyap | ignaziocassano: You can get it this way: | |
| 09:04:14 | kashyap | ignaziocassano: You can get it this way: | |
| 09:04:31 | kashyap | On the relevant compute nodes (both source and dest): | |
| 09:04:31 | kashyap | On the relevant compute nodes (both source and dest): | |
| 09:04:40 | kashyap | (1) Set the log output file: $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 09:04:40 | kashyap | (1) Set the log output file: $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 09:04:49 | kashyap | (2) Configure the dynamic filters: | |
| 09:04:49 | kashyap | (2) Configure the dynamic filters: | |
| 09:04:54 | kashyap | $> virt-admin daemon-log-filters \ "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 09:04:55 | kashyap | $> virt-admin daemon-log-filters \ "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 09:05:05 | kashyap | (Remove that "\") | |
| 09:05:05 | kashyap | (Remove that "\") | |
| 09:05:14 | ignaziocassano | kashyap: on messages it reports: 19:22:29 podvc-kvm04 libvirtd: 2021-05-16 17:22:29.562+0000: 166295: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor | |
| 09:05:14 | ignaziocassano | kashyap: on messages it reports: 19:22:29 podvc-kvm04 libvirtd: 2021-05-16 17:22:29.562+0000: 166295: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor | |
| 09:05:17 | kashyap | And then re-run the migration of the affected instances | |
| 09:05:17 | kashyap | And then re-run the migration of the affected instances | |
| 09:05:41 | kashyap | ignaziocassano: Yeah; that's also "normal" / "expected"; doesn't tell much still | |
| 09:05:41 | kashyap | ignaziocassano: Yeah; that's also "normal" / "expected"; doesn't tell much still | |
| 09:06:06 | kashyap | ignaziocassano: BTW, can you quickly try this: | |
| 09:06:06 | kashyap | ignaziocassano: BTW, can you quickly try this: | |
| 09:06:14 | kashyap | $> journalctl -u libvirtd -l --since=yesterday -p err | |
| 09:06:14 | kashyap | $> journalctl -u libvirtd -l --since=yesterday -p err | |
| 09:08:11 | ignaziocassano | error : qemuDomainObjBeginJobInternal:6825 : Timed out during operation: cannot acquire state change lock (held by monitor=remo | |
| 09:08:11 | ignaziocassano | error : qemuDomainObjBeginJobInternal:6825 : Timed out during operation: cannot acquire state change lock (held by monitor=remo | |
| 09:08:53 | kashyap | Ugh, this one... | |
| 09:08:53 | kashyap | Ugh, this one... | |
| 09:09:31 | ignaziocassano | unfortunately I cannot try to move further vm because they are in production and I must inform my customers | |
| 09:09:31 | ignaziocassano | unfortunately I cannot try to move further vm because they are in production and I must inform my customers | |
| 09:10:11 | ignaziocassano | it us very strange because for some instances it works fine | |
| 09:10:11 | ignaziocassano | it us very strange because for some instances it works fine | |
| 09:10:33 | kashyap | ignaziocassano: So this is one of the most painful errors to debug from libvirt. I recently wrote a response on a Red Hat bugzilla | |
| 09:10:34 | kashyap | ignaziocassano: So this is one of the most painful errors to debug from libvirt. I recently wrote a response on a Red Hat bugzilla | |
| 09:10:54 | kashyap | Let me get that for you (beware: no easy solution here :-() | |
| 09:10:54 | kashyap | Let me get that for you (beware: no easy solution here :-() | |
| 09:11:42 | ignaziocassano | kashyap, I read some red hat suggestions, but they include the instance restart :-( | |
| 09:11:42 | ignaziocassano | kashyap, I read some red hat suggestions, but they include the instance restart :-( | |
| 09:12:11 | kashyap | ignaziocassano: Yes, you're not wrong, afraid. | |
| 09:12:11 | kashyap | ignaziocassano: Yes, you're not wrong, afraid. | |
| 09:12:39 | kashyap | ignaziocassano: These "cannot acquire state change lock" errors are notorious; and it could be due to QEMU getting hung, which in turn could be caused by stuck I/O | |
| 09:12:39 | kashyap | ignaziocassano: These "cannot acquire state change lock" errors are notorious; and it could be due to QEMU getting hung, which in turn could be caused by stuck I/O | |
| 09:14:07 | ignaziocassano | I did not see any nfs stale on my compute node | |
| 09:14:08 | ignaziocassano | I did not see any nfs stale on my compute node | |
| 09:15:52 | kashyap | ignaziocassano: Can you post the affected guest log from /var/log/libvir/qemu/instance-YYYY.log? | |
| 09:15:52 | kashyap | ignaziocassano: Can you post the affected guest log from /var/log/libvir/qemu/instance-YYYY.log? | |
| 09:16:11 | kashyap | It _might_ sometimes havea clue | |
| 09:16:11 | kashyap | It _might_ sometimes havea clue | |