| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-18 | |||
| 16:09:05 | sean-k-mooney | which is not a change you can do on an existing cloud without breaking a lot of things | |
| 16:10:02 | sean-k-mooney | admin0: i would guess that your automation tooling tried to set canonical hostname to the fqdn but coudl not fiure out the domain but it still added the . | |
| 16:10:02 | sean-k-mooney | admin0: i would guess that your automation tooling tried to set canonical hostname to the fqdn but coudl not fiure out the domain but it still added the . | |
| 16:10:12 | admin0 | yeah | |
| 16:28:11 | admin0 | hostname => h7 ; hostname -f => h7 ; virsh hostname => h7 ; openstack hypervisor list => h7 ; migrate command still says => Compute host h7 could not be found. | |
| 16:28:11 | admin0 | hostname => h7 ; hostname -f => h7 ; virsh hostname => h7 ; openstack hypervisor list => h7 ; migrate command still says => Compute host h7 could not be found. | |
| 16:40:23 | admin0 | sean-k-mooney, is it possible to fix associations directly from the database ? | |
| 16:40:23 | admin0 | sean-k-mooney, is it possible to fix associations directly from the database ? | |
| 16:40:43 | admin0 | for example rename h7 -> say h77 for example, and then in nova instances update the instances as h77 as their new host | |
| 16:40:43 | admin0 | for example rename h7 -> say h77 for example, and then in nova instances update the instances as h77 as their new host | |
| 16:50:25 | sean-k-mooney | admin0: its possible but a really bad idea | |
| 16:50:25 | sean-k-mooney | admin0: its possible but a really bad idea | |
| 16:51:04 | sean-k-mooney | admin0: you need to fix it in multiple dbs nova api db , nova cell db, neutron, cinder and posibley others | |
| 16:51:04 | sean-k-mooney | admin0: you need to fix it in multiple dbs nova api db , nova cell db, neutron, cinder and posibley others | |
| 17:00:43 | bauzas | have someone tested https://docs.openstack.org/api-ref/placement/?expanded=list-resource-providers-detail#list-resource-providers with multiple 'uuid' query strings ? | |
| 17:00:43 | bauzas | have someone tested https://docs.openstack.org/api-ref/placement/?expanded=list-resource-providers-detail#list-resource-providers with multiple 'uuid' query strings ? | |
| 17:01:12 | bauzas | like /resource_providers?uuid=<uuid1>&uuid=<uuid2> ? | |
| 17:01:12 | bauzas | like /resource_providers?uuid=<uuid1>&uuid=<uuid2> ? | |
| 17:01:25 | bauzas | sean-k-mooney: do you know whether it works ? ^ | |
| 17:01:25 | bauzas | sean-k-mooney: do you know whether it works ? ^ | |
| 17:02:00 | bauzas | I'd like to get a list of RPs by their UUIDs | |
| 17:02:00 | bauzas | I'd like to get a list of RPs by their UUIDs | |
| 17:02:37 | bauzas | of course, I can see we don't support uuid=<uuid1>&<uuid2> | |
| 17:02:38 | bauzas | of course, I can see we don't support uuid=<uuid1>&<uuid2> | |
| 17:03:30 | bauzas | (tbc, I need to get the RP name in the libvirt driver when calling to look at the no longer mdevs | |
| 17:03:30 | bauzas | (tbc, I need to get the RP name in the libvirt driver when calling to look at the no longer mdevs | |
| 17:04:34 | bauzas | given we only have the RP uuid, and given we don't have yet the provider tree, I'll pass both the allocations *and the RPs to the new method I'm creating | |
| 17:04:34 | bauzas | given we only have the RP uuid, and given we don't have yet the provider tree, I'll pass both the allocations *and the RPs to the new method I'm creating | |
| 17:11:04 | sean-k-mooney | im not sure but i can see potially supporting a list of uuids | |
| 17:11:04 | sean-k-mooney | im not sure but i can see potially supporting a list of uuids | |
| 20:07:35 | admin0 | hi sean-k-mooney .. any ideas how i can fix this mess: https://gist.github.com/a1git/c22ec0c17aaa9dcf95fd7485eb76af2f | |
| 20:07:35 | admin0 | hi sean-k-mooney .. any ideas how i can fix this mess: https://gist.github.com/a1git/c22ec0c17aaa9dcf95fd7485eb76af2f | |
| 20:08:27 | admin0 | is there a way to know under which "hostname" the instance I am trying to migrate is listed under .. so that i can set the hostname to that exact and migrate it off | |
| 20:08:27 | admin0 | is there a way to know under which "hostname" the instance I am trying to migrate is listed under .. so that i can set the hostname to that exact and migrate it off | |
| 20:09:15 | admin0 | openstack server show $uuid shows it as h7 . while the hostname is also h7, but during migration is telling h7 not found .. so even though i have everything set as h7 and restarted nova-compute, i think its thinking it as different | |
| 20:09:16 | admin0 | openstack server show $uuid shows it as h7 . while the hostname is also h7, but during migration is telling h7 not found .. so even though i have everything set as h7 and restarted nova-compute, i think its thinking it as different | |
| #openstack-nova - 2021-05-19 | |||
| 00:48:59 | openstackgerrit | norman shen proposed openstack/nova master: Saving security group to info_cache https://review.opendev.org/c/openstack/nova/+/786348 | |
| 02:16:08 | openstackgerrit | norman shen proposed openstack/nova-specs master: Speed up server details https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| 03:12:57 | openstackgerrit | norman shen proposed openstack/nova-specs master: Speed up server details https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| 03:38:01 | openstackgerrit | norman shen proposed openstack/nova-specs master: Speed up server details https://review.opendev.org/c/openstack/nova-specs/+/791620 | |
| 08:23:20 | openstackgerrit | Daniel Bengtsson proposed openstack/nova master: Use the new type HostDomainOpt. https://review.opendev.org/c/openstack/nova/+/788240 | |
| 08:23:20 | openstackgerrit | Daniel Bengtsson proposed openstack/nova master: Use the new type HostDomainOpt. https://review.opendev.org/c/openstack/nova/+/788240 | |
| 08:50:08 | ignaziocassano | 10:48 (1 minuto fa) | |
| 08:50:08 | ignaziocassano | Ignazio Cassano <ignaziocassano@gmail.com> | |
| 08:50:08 | ignaziocassano | 10:48 (1 minuto fa) | |
| 08:50:08 | ignaziocassano | Ignazio Cassano <ignaziocassano@gmail.com> | |
| 08:50:09 | ignaziocassano | Hello Guys, | |
| 08:50:09 | ignaziocassano | a openstack-discuss | |
| 08:50:09 | ignaziocassano | Hello Guys, | |
| 08:50:09 | ignaziocassano | a openstack-discuss | |
| 08:50:10 | ignaziocassano | The error reported is: | |
| 08:50:10 | ignaziocassano | on train centos7 I am facing live migration issue only for some instances (not all). | |
| 08:50:10 | ignaziocassano | The error reported is: | |
| 08:50:10 | ignaziocassano | on train centos7 I am facing live migration issue only for some instances (not all). | |
| 08:50:11 | ignaziocassano | 2021-05-19 08:45:57.096 142537 ERROR nova.compute.manager [-] [instance: b18450e8-b3db-4886-a737-c161d99c6a46] Live migration failed.: libvirtError: Unable to read from monitor: Connection reset by peer | |
| 08:51:16 | ignaziocassano | some instances migrate without errors. I tried to stop end restart libvirtd end nova-compute without solving | |
| 08:51:16 | ignaziocassano | some instances migrate without errors. I tried to stop end restart libvirtd end nova-compute without solving | |
| 08:52:48 | ignaziocassano | On instances where migration failed, If I stop them and I start on another node, If migrate them on original node and migrate again, it works | |
| 08:52:48 | ignaziocassano | On instances where migration failed, If I stop them and I start on another node, If migrate them on original node and migrate again, it works | |
| 08:55:03 | ignaziocassano | Sorry, the version is stein | |
| 08:55:03 | ignaziocassano | Sorry, the version is stein | |
| 08:58:19 | stephenfin | ignaziocassano: Have you looked into the libvirt logs directly or investigated syslog? | |
| 08:58:20 | stephenfin | ignaziocassano: Have you looked into the libvirt logs directly or investigated syslog? | |
| 08:58:54 | stephenfin | kashyap can correct me if I'm wrong, but that error usually implies the connection between QEMU and libvirt has died | |
| 08:58:54 | stephenfin | kashyap can correct me if I'm wrong, but that error usually implies the connection between QEMU and libvirt has died | |
| 08:59:07 | kashyap | * kashyap blinks and looks | |
| 08:59:26 | kashyap | stephenfin: Yes, you're right | |
| 08:59:26 | kashyap | stephenfin: Yes, you're right | |
| 09:00:13 | kashyap | But the underlying problem could be anywhere ... and needs more details to debug | |
| 09:00:13 | kashyap | But the underlying problem could be anywhere ... and needs more details to debug | |
| 09:00:25 | kashyap | ignaziocassano: Are you migrating including your storage? (I.e. "block migration"?) | |
| 09:00:25 | kashyap | ignaziocassano: Are you migrating including your storage? (I.e. "block migration"?) | |
| 09:01:05 | ignaziocassano | kashyap the storage is shared on netapp nfs | |
| 09:01:05 | ignaziocassano | kashyap the storage is shared on netapp nfs | |
| 09:01:50 | ignaziocassano | no errors on openvswitch agent | |
| 09:01:51 | ignaziocassano | no errors on openvswitch agent | |
| 09:03:32 | ignaziocassano | kashyap: I presume the connection between QEMU and libvirt ha died for some instances, but I do not know how I can verify it | |
| 09:03:32 | ignaziocassano | kashyap: I presume the connection between QEMU and libvirt ha died for some instances, but I do not know how I can verify it | |
| 09:04:08 | kashyap | ignaziocassano: Right; that's the reason. So one way to debug this is to obtain libvirt debug log filters that track interactions b/n QEMU and libvirt | |
| 09:04:08 | kashyap | ignaziocassano: Right; that's the reason. So one way to debug this is to obtain libvirt debug log filters that track interactions b/n QEMU and libvirt | |
| 09:04:14 | kashyap | ignaziocassano: You can get it this way: | |
| 09:04:14 | kashyap | ignaziocassano: You can get it this way: | |
| 09:04:31 | kashyap | On the relevant compute nodes (both source and dest): | |
| 09:04:31 | kashyap | On the relevant compute nodes (both source and dest): | |
| 09:04:40 | kashyap | (1) Set the log output file: $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 09:04:40 | kashyap | (1) Set the log output file: $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 09:04:49 | kashyap | (2) Configure the dynamic filters: | |
| 09:04:49 | kashyap | (2) Configure the dynamic filters: | |
| 09:04:54 | kashyap | $> virt-admin daemon-log-filters \ "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 09:04:55 | kashyap | $> virt-admin daemon-log-filters \ "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 09:05:05 | kashyap | (Remove that "\") | |
| 09:05:05 | kashyap | (Remove that "\") | |
| 09:05:14 | ignaziocassano | kashyap: on messages it reports: 19:22:29 podvc-kvm04 libvirtd: 2021-05-16 17:22:29.562+0000: 166295: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor | |
| 09:05:14 | ignaziocassano | kashyap: on messages it reports: 19:22:29 podvc-kvm04 libvirtd: 2021-05-16 17:22:29.562+0000: 166295: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor | |
| 09:05:17 | kashyap | And then re-run the migration of the affected instances | |
| 09:05:17 | kashyap | And then re-run the migration of the affected instances | |
| 09:05:41 | kashyap | ignaziocassano: Yeah; that's also "normal" / "expected"; doesn't tell much still | |
| 09:05:41 | kashyap | ignaziocassano: Yeah; that's also "normal" / "expected"; doesn't tell much still | |
| 09:06:06 | kashyap | ignaziocassano: BTW, can you quickly try this: | |
| 09:06:06 | kashyap | ignaziocassano: BTW, can you quickly try this: | |