| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-30 | |||
| 11:09:58 | ierdem | it says "Queue.declare: (404) NOT_FOUND". I can not reach the VMs. I don't know why rabbitmq queues deleted from the list, or why these queues can not recreated by service requests? Can you help me? How can I solve this queue problem and what is the reason of this? btw, this is nova-compute logs: http://paste.openstack.org/show/804880/ , Thanks | |
| 11:09:58 | ierdem | Hi everyone, I have an OSA Victoria environment. After 14 days from installation, some problems occured on rabbitmq and galera clusters. There are connection problems between galera containers but these problems almost solved in 1 minutes, at the same time with these problems, rabbitmq containers throwed connection errors. Now, in nova-compute logs | |
| 11:09:58 | ierdem | Hi everyone, I have an OSA Victoria environment. After 14 days from installation, some problems occured on rabbitmq and galera clusters. There are connection problems between galera containers but these problems almost solved in 1 minutes, at the same time with these problems, rabbitmq containers throwed connection errors. Now, in nova-compute logs | |
| 11:09:59 | ierdem | it says "Queue.declare: (404) NOT_FOUND". I can not reach the VMs. I don't know why rabbitmq queues deleted from the list, or why these queues can not recreated by service requests? Can you help me? How can I solve this queue problem and what is the reason of this? btw, this is nova-compute logs: http://paste.openstack.org/show/804880/ , Thanks | |
| 11:11:04 | ierdem | Rabbitmq and other services are working now, but problem still exist | |
| 11:11:05 | ierdem | Rabbitmq and other services are working now, but problem still exist | |
| 11:21:15 | noonedeadpunk | I'm not really sure that there's no problem with cluster now... | |
| 11:21:15 | noonedeadpunk | I'm not really sure that there's no problem with cluster now... | |
| 11:22:31 | noonedeadpunk | Does other services work properly? Or it's only nova that complain? | |
| 11:22:31 | noonedeadpunk | Does other services work properly? Or it's only nova that complain? | |
| 11:26:59 | ierdem | After restarting all services, nova-compute services seems work fine but neutron-server containers have the same error log now, "queue declare(404) not found". I could not inspect the all services for now, I am checking one by one but you probably right, there will be errors in cluster I think | |
| 11:26:59 | ierdem | After restarting all services, nova-compute services seems work fine but neutron-server containers have the same error log now, "queue declare(404) not found". I could not inspect the all services for now, I am checking one by one but you probably right, there will be errors in cluster I think | |
| 11:29:45 | noonedeadpunk | ierdem: jsut try running rabbitmq-install.yml -e rabbitmq_upgrade=true - it should fix issue with missing queues (with several mins of rabbit cluster downtime) | |
| 11:29:45 | noonedeadpunk | ierdem: jsut try running rabbitmq-install.yml -e rabbitmq_upgrade=true - it should fix issue with missing queues (with several mins of rabbit cluster downtime) | |
| 11:34:13 | ierdem | I am running now, thanks | |
| 11:34:13 | ierdem | I am running now, thanks | |
| 11:34:58 | stephenfin | lyarwood: Just a reminder to take a punt at https://review.opendev.org/c/openstack/nova/+/676209 if you could today | |
| 11:34:59 | stephenfin | lyarwood: Just a reminder to take a punt at https://review.opendev.org/c/openstack/nova/+/676209 if you could today | |
| 11:41:10 | lyarwood | stephenfin: ack still on my list, if not today it might have to wait until after PTO | |
| 11:41:10 | lyarwood | stephenfin: ack still on my list, if not today it might have to wait until after PTO | |
| 11:46:45 | stephenfin | (y) | |
| 11:46:45 | stephenfin | (y) | |
| 11:54:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Delegate OVS plug to os-vif https://review.opendev.org/c/openstack/nova/+/602432 | |
| 11:54:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Delegate OVS plug to os-vif https://review.opendev.org/c/openstack/nova/+/602432 | |
| 12:35:53 | damiandabrowski | But do we have any fix for already running VMs? Because now, they fails to spawn after reboot. | |
| 12:35:53 | damiandabrowski | We have merged a fix for newly created disks: https://review.opendev.org/c/openstack/nova/+/710790/ | |
| 12:35:53 | damiandabrowski | hey! libvirt v6.0.0 fails to launch a domain when using qcow2 disks where the backing file format is not explicitly specified in the qcow2 metadata. | |
| 12:35:53 | damiandabrowski | But do we have any fix for already running VMs? Because now, they fails to spawn after reboot. | |
| 12:35:53 | damiandabrowski | We have merged a fix for newly created disks: https://review.opendev.org/c/openstack/nova/+/710790/ | |
| 12:35:53 | damiandabrowski | hey! libvirt v6.0.0 fails to launch a domain when using qcow2 disks where the backing file format is not explicitly specified in the qcow2 metadata. | |
| 12:45:43 | lyarwood | damiandabrowski: I don't think we had to fix that as the libvirt devs eventually stopped enforcing that | |
| 12:45:44 | lyarwood | damiandabrowski: I don't think we had to fix that as the libvirt devs eventually stopped enforcing that | |
| 12:46:02 | lyarwood | damiandabrowski: which base OS is this? | |
| 12:46:03 | lyarwood | damiandabrowski: which base OS is this? | |
| 12:47:30 | damiandabrowski | <lyarwood "damiandabrowski: which base OS i"> ubuntu 18.04, libvirt 6.0.0-0ubuntu8.8~cloud0 | |
| 12:47:30 | damiandabrowski | <lyarwood "damiandabrowski: which base OS i"> ubuntu 18.04, libvirt 6.0.0-0ubuntu8.8~cloud0 | |
| 12:48:00 | lyarwood | kashyap: ^ the libvirt devs eventually removed that right? | |
| 12:48:00 | lyarwood | kashyap: ^ the libvirt devs eventually removed that right? | |
| 12:48:14 | kashyap | * kashyap reads | |
| 12:48:52 | kashyap | lyarwood: Yes, I recall; but I need to find trhe right commit | |
| 12:48:52 | kashyap | lyarwood: Yes, I recall; but I need to find trhe right commit | |
| 12:49:19 | kashyap | * kashyap digs up | |
| 13:08:52 | kashyap | damiandabrowski: lyarwood: There we go: "Allow format probing under special circumstances": https://github.com/libvirt/libvirt/commit/ae9e6c2a2b | |
| 13:08:52 | kashyap | damiandabrowski: lyarwood: There we go: "Allow format probing under special circumstances": https://github.com/libvirt/libvirt/commit/ae9e6c2a2b | |
| 13:09:02 | kashyap | damiandabrowski: lyarwood: That's the libvirt commit that relaxes the restriction | |
| 13:09:02 | kashyap | damiandabrowski: lyarwood: That's the libvirt commit that relaxes the restriction | |
| 13:09:50 | lyarwood | so you need >=6.1.0 | |
| 13:09:51 | lyarwood | so you need >=6.1.0 | |
| 13:09:53 | kashyap | damiandabrowski: That should be in 6.1.0 and above. (I checked via `git tag --contains`) | |
| 13:09:53 | kashyap | damiandabrowski: That should be in 6.1.0 and above. (I checked via `git tag --contains`) | |
| 13:09:56 | kashyap | lyarwood: Yep | |
| 13:09:56 | kashyap | lyarwood: Yep | |
| 13:10:31 | lyarwood | kashyap: I guess we could write a known issues releasenote about that | |
| 13:10:31 | lyarwood | kashyap: I guess we could write a known issues releasenote about that | |
| 13:11:06 | kashyap | lyarwood: Yeah; that's reasonable. I can volunteer to write one. | |
| 13:11:06 | kashyap | lyarwood: Yeah; that's reasonable. I can volunteer to write one. | |
| 13:13:54 | lyarwood | kashyap: awesome thanks | |
| 13:13:54 | lyarwood | kashyap: awesome thanks | |
| 13:16:31 | damiandabrowski | thanks a lot for the input! | |
| 13:16:31 | damiandabrowski | thanks a lot for the input! | |
| 13:16:40 | kashyap | I'd call it a "resolved known issue" :-) | |
| 13:16:40 | kashyap | I'd call it a "resolved known issue" :-) | |
| 14:09:39 | dansmith | gibi: but we should be checking for the *last* victoria version to determine if something is victoria, right? | |
| 14:09:39 | dansmith | gibi: but we should be checking for the *last* victoria version to determine if something is victoria, right? | |
| 14:12:59 | noonedeadpunk | hey there! | |
| 14:12:59 | noonedeadpunk | hey there! | |
| 14:13:48 | gibi | dansmith: when we says that Wallaby controller supports Victoria compute, doesn't that means it supports even first Victoria service version? | |
| 14:13:48 | gibi | dansmith: when we says that Wallaby controller supports Victoria compute, doesn't that means it supports even first Victoria service version? | |
| 14:14:30 | gibi | ~ the oldest thing we support is the oldest Victoria pieces | |
| 14:14:30 | gibi | ~ the oldest thing we support is the oldest Victoria pieces | |
| 14:15:03 | noonedeadpunk | We've faced nasty thing on bionic, regarding libvirt virStorageFileGetMetadataRecurse. While new images are created properly thanks to https://review.opendev.org/c/openstack/nova/+/710790/ I'm not sure about the way forward for existing ones | |
| 14:15:03 | noonedeadpunk | We've faced nasty thing on bionic, regarding libvirt virStorageFileGetMetadataRecurse. While new images are created properly thanks to https://review.opendev.org/c/openstack/nova/+/710790/ I'm not sure about the way forward for existing ones | |
| 14:15:20 | noonedeadpunk | as once VM stopps, it can't be started anymore | |
| 14:15:20 | noonedeadpunk | as once VM stopps, it can't be started anymore | |
| 14:16:06 | noonedeadpunk | I believe I'm not the first who comes with this, but yeah :( | |
| 14:16:06 | noonedeadpunk | I believe I'm not the first who comes with this, but yeah :( | |
| 14:16:24 | dansmith | gibi: that's not how we've ever done it for RPC, for example, but I guess it's fair.. it's just confusing though, | |
| 14:16:24 | dansmith | gibi: that's not how we've ever done it for RPC, for example, but I guess it's fair.. it's just confusing though, | |
| 14:16:38 | lyarwood | noonedeadpunk: it was discussed earlier in the channel, you need libvirt >= 6.1.0 | |
| 14:16:39 | lyarwood | noonedeadpunk: it was discussed earlier in the channel, you need libvirt >= 6.1.0 | |
| 14:16:45 | dansmith | because from the perspective of someone trying to figure out what it's looking for, the numbers don't match up | |
| 14:16:45 | dansmith | because from the perspective of someone trying to figure out what it's looking for, the numbers don't match up | |
| 14:16:46 | lyarwood | noonedeadpunk: we don't workaround this in nova | |
| 14:16:46 | lyarwood | noonedeadpunk: we don't workaround this in nova | |
| 14:16:56 | dansmith | gibi: but if that's the goal, then I guess it's okay | |
| 14:16:56 | dansmith | gibi: but if that's the goal, then I guess it's okay | |
| 14:17:26 | noonedeadpunk | lyarwood: yeah, I see that, but libvirt 6.1.0 is smth that seems needs to be built manually I guess | |
| 14:17:26 | noonedeadpunk | lyarwood: yeah, I see that, but libvirt 6.1.0 is smth that seems needs to be built manually I guess | |
| 14:17:32 | noonedeadpunk | as UCA doesn't provide that :( | |
| 14:17:32 | noonedeadpunk | as UCA doesn't provide that :( | |
| 14:17:40 | lyarwood | noonedeadpunk: then raise a bug with the UCA folks | |
| 14:17:40 | lyarwood | noonedeadpunk: then raise a bug with the UCA folks | |
| 14:18:01 | gibi | dansmith: at least it is documented in the code comment above the alias https://github.com/openstack/nova/blob/dab4ec1a534d4774e61a24cdace0f3491788e4e7/nova/objects/service.py#L206 so the intention is clear for the reader | |
| 14:18:01 | gibi | dansmith: at least it is documented in the code comment above the alias https://github.com/openstack/nova/blob/dab4ec1a534d4774e61a24cdace0f3491788e4e7/nova/objects/service.py#L206 so the intention is clear for the reader | |
| 14:18:10 | noonedeadpunk | I believe there were solid reasons not to workaround in nova? | |
| 14:18:10 | noonedeadpunk | I believe there were solid reasons not to workaround in nova? | |
| 14:18:27 | dansmith | gibi: okay :) | |
| 14:18:27 | dansmith | gibi: okay :) | |
| 14:18:29 | lyarwood | noonedeadpunk: yeah it would require us to rebase the disks at launch | |
| 14:18:29 | lyarwood | noonedeadpunk: yeah it would require us to rebase the disks at launch | |