| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-30 | |||
| 11:09:59 | ierdem | it says "Queue.declare: (404) NOT_FOUND". I can not reach the VMs. I don't know why rabbitmq queues deleted from the list, or why these queues can not recreated by service requests? Can you help me? How can I solve this queue problem and what is the reason of this? btw, this is nova-compute logs: http://paste.openstack.org/show/804880/ , Thanks | |
| 11:11:04 | ierdem | Rabbitmq and other services are working now, but problem still exist | |
| 11:11:05 | ierdem | Rabbitmq and other services are working now, but problem still exist | |
| 11:21:15 | noonedeadpunk | I'm not really sure that there's no problem with cluster now... | |
| 11:21:15 | noonedeadpunk | I'm not really sure that there's no problem with cluster now... | |
| 11:22:31 | noonedeadpunk | Does other services work properly? Or it's only nova that complain? | |
| 11:22:31 | noonedeadpunk | Does other services work properly? Or it's only nova that complain? | |
| 11:26:59 | ierdem | After restarting all services, nova-compute services seems work fine but neutron-server containers have the same error log now, "queue declare(404) not found". I could not inspect the all services for now, I am checking one by one but you probably right, there will be errors in cluster I think | |
| 11:26:59 | ierdem | After restarting all services, nova-compute services seems work fine but neutron-server containers have the same error log now, "queue declare(404) not found". I could not inspect the all services for now, I am checking one by one but you probably right, there will be errors in cluster I think | |
| 11:29:45 | noonedeadpunk | ierdem: jsut try running rabbitmq-install.yml -e rabbitmq_upgrade=true - it should fix issue with missing queues (with several mins of rabbit cluster downtime) | |
| 11:29:45 | noonedeadpunk | ierdem: jsut try running rabbitmq-install.yml -e rabbitmq_upgrade=true - it should fix issue with missing queues (with several mins of rabbit cluster downtime) | |
| 11:34:13 | ierdem | I am running now, thanks | |
| 11:34:13 | ierdem | I am running now, thanks | |
| 11:34:58 | stephenfin | lyarwood: Just a reminder to take a punt at https://review.opendev.org/c/openstack/nova/+/676209 if you could today | |
| 11:34:59 | stephenfin | lyarwood: Just a reminder to take a punt at https://review.opendev.org/c/openstack/nova/+/676209 if you could today | |
| 11:41:10 | lyarwood | stephenfin: ack still on my list, if not today it might have to wait until after PTO | |
| 11:41:10 | lyarwood | stephenfin: ack still on my list, if not today it might have to wait until after PTO | |
| 11:46:45 | stephenfin | (y) | |
| 11:46:45 | stephenfin | (y) | |
| 11:54:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Delegate OVS plug to os-vif https://review.opendev.org/c/openstack/nova/+/602432 | |
| 11:54:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Delegate OVS plug to os-vif https://review.opendev.org/c/openstack/nova/+/602432 | |
| 12:35:53 | damiandabrowski | hey! libvirt v6.0.0 fails to launch a domain when using qcow2 disks where the backing file format is not explicitly specified in the qcow2 metadata. | |
| 12:35:53 | damiandabrowski | We have merged a fix for newly created disks: https://review.opendev.org/c/openstack/nova/+/710790/ | |
| 12:35:53 | damiandabrowski | But do we have any fix for already running VMs? Because now, they fails to spawn after reboot. | |
| 12:35:53 | damiandabrowski | hey! libvirt v6.0.0 fails to launch a domain when using qcow2 disks where the backing file format is not explicitly specified in the qcow2 metadata. | |
| 12:35:53 | damiandabrowski | We have merged a fix for newly created disks: https://review.opendev.org/c/openstack/nova/+/710790/ | |
| 12:35:53 | damiandabrowski | But do we have any fix for already running VMs? Because now, they fails to spawn after reboot. | |
| 12:45:43 | lyarwood | damiandabrowski: I don't think we had to fix that as the libvirt devs eventually stopped enforcing that | |
| 12:45:44 | lyarwood | damiandabrowski: I don't think we had to fix that as the libvirt devs eventually stopped enforcing that | |
| 12:46:02 | lyarwood | damiandabrowski: which base OS is this? | |
| 12:46:03 | lyarwood | damiandabrowski: which base OS is this? | |
| 12:47:30 | damiandabrowski | <lyarwood "damiandabrowski: which base OS i"> ubuntu 18.04, libvirt 6.0.0-0ubuntu8.8~cloud0 | |
| 12:47:30 | damiandabrowski | <lyarwood "damiandabrowski: which base OS i"> ubuntu 18.04, libvirt 6.0.0-0ubuntu8.8~cloud0 | |
| 12:48:00 | lyarwood | kashyap: ^ the libvirt devs eventually removed that right? | |
| 12:48:00 | lyarwood | kashyap: ^ the libvirt devs eventually removed that right? | |
| 12:48:14 | kashyap | * kashyap reads | |
| 12:48:52 | kashyap | lyarwood: Yes, I recall; but I need to find trhe right commit | |
| 12:48:52 | kashyap | lyarwood: Yes, I recall; but I need to find trhe right commit | |
| 12:49:19 | kashyap | * kashyap digs up | |
| 13:08:52 | kashyap | damiandabrowski: lyarwood: There we go: "Allow format probing under special circumstances": https://github.com/libvirt/libvirt/commit/ae9e6c2a2b | |
| 13:08:52 | kashyap | damiandabrowski: lyarwood: There we go: "Allow format probing under special circumstances": https://github.com/libvirt/libvirt/commit/ae9e6c2a2b | |
| 13:09:02 | kashyap | damiandabrowski: lyarwood: That's the libvirt commit that relaxes the restriction | |
| 13:09:02 | kashyap | damiandabrowski: lyarwood: That's the libvirt commit that relaxes the restriction | |
| 13:09:50 | lyarwood | so you need >=6.1.0 | |
| 13:09:51 | lyarwood | so you need >=6.1.0 | |
| 13:09:53 | kashyap | damiandabrowski: That should be in 6.1.0 and above. (I checked via `git tag --contains`) | |
| 13:09:53 | kashyap | damiandabrowski: That should be in 6.1.0 and above. (I checked via `git tag --contains`) | |
| 13:09:56 | kashyap | lyarwood: Yep | |
| 13:09:56 | kashyap | lyarwood: Yep | |
| 13:10:31 | lyarwood | kashyap: I guess we could write a known issues releasenote about that | |
| 13:10:31 | lyarwood | kashyap: I guess we could write a known issues releasenote about that | |
| 13:11:06 | kashyap | lyarwood: Yeah; that's reasonable. I can volunteer to write one. | |
| 13:11:06 | kashyap | lyarwood: Yeah; that's reasonable. I can volunteer to write one. | |
| 13:13:54 | lyarwood | kashyap: awesome thanks | |
| 13:13:54 | lyarwood | kashyap: awesome thanks | |
| 13:16:31 | damiandabrowski | thanks a lot for the input! | |
| 13:16:31 | damiandabrowski | thanks a lot for the input! | |
| 13:16:40 | kashyap | I'd call it a "resolved known issue" :-) | |
| 13:16:40 | kashyap | I'd call it a "resolved known issue" :-) | |
| 14:09:39 | dansmith | gibi: but we should be checking for the *last* victoria version to determine if something is victoria, right? | |
| 14:09:39 | dansmith | gibi: but we should be checking for the *last* victoria version to determine if something is victoria, right? | |
| 14:12:59 | noonedeadpunk | hey there! | |
| 14:12:59 | noonedeadpunk | hey there! | |
| 14:13:48 | gibi | dansmith: when we says that Wallaby controller supports Victoria compute, doesn't that means it supports even first Victoria service version? | |
| 14:13:48 | gibi | dansmith: when we says that Wallaby controller supports Victoria compute, doesn't that means it supports even first Victoria service version? | |
| 14:14:30 | gibi | ~ the oldest thing we support is the oldest Victoria pieces | |
| 14:14:30 | gibi | ~ the oldest thing we support is the oldest Victoria pieces | |
| 14:15:03 | noonedeadpunk | We've faced nasty thing on bionic, regarding libvirt virStorageFileGetMetadataRecurse. While new images are created properly thanks to https://review.opendev.org/c/openstack/nova/+/710790/ I'm not sure about the way forward for existing ones | |
| 14:15:03 | noonedeadpunk | We've faced nasty thing on bionic, regarding libvirt virStorageFileGetMetadataRecurse. While new images are created properly thanks to https://review.opendev.org/c/openstack/nova/+/710790/ I'm not sure about the way forward for existing ones | |
| 14:15:20 | noonedeadpunk | as once VM stopps, it can't be started anymore | |
| 14:15:20 | noonedeadpunk | as once VM stopps, it can't be started anymore | |
| 14:16:06 | noonedeadpunk | I believe I'm not the first who comes with this, but yeah :( | |
| 14:16:06 | noonedeadpunk | I believe I'm not the first who comes with this, but yeah :( | |
| 14:16:24 | dansmith | gibi: that's not how we've ever done it for RPC, for example, but I guess it's fair.. it's just confusing though, | |
| 14:16:24 | dansmith | gibi: that's not how we've ever done it for RPC, for example, but I guess it's fair.. it's just confusing though, | |
| 14:16:38 | lyarwood | noonedeadpunk: it was discussed earlier in the channel, you need libvirt >= 6.1.0 | |
| 14:16:39 | lyarwood | noonedeadpunk: it was discussed earlier in the channel, you need libvirt >= 6.1.0 | |
| 14:16:45 | dansmith | because from the perspective of someone trying to figure out what it's looking for, the numbers don't match up | |
| 14:16:45 | dansmith | because from the perspective of someone trying to figure out what it's looking for, the numbers don't match up | |
| 14:16:46 | lyarwood | noonedeadpunk: we don't workaround this in nova | |
| 14:16:46 | lyarwood | noonedeadpunk: we don't workaround this in nova | |
| 14:16:56 | dansmith | gibi: but if that's the goal, then I guess it's okay | |
| 14:16:56 | dansmith | gibi: but if that's the goal, then I guess it's okay | |
| 14:17:26 | noonedeadpunk | lyarwood: yeah, I see that, but libvirt 6.1.0 is smth that seems needs to be built manually I guess | |
| 14:17:26 | noonedeadpunk | lyarwood: yeah, I see that, but libvirt 6.1.0 is smth that seems needs to be built manually I guess | |
| 14:17:32 | noonedeadpunk | as UCA doesn't provide that :( | |
| 14:17:32 | noonedeadpunk | as UCA doesn't provide that :( | |
| 14:17:40 | lyarwood | noonedeadpunk: then raise a bug with the UCA folks | |
| 14:17:40 | lyarwood | noonedeadpunk: then raise a bug with the UCA folks | |
| 14:18:01 | gibi | dansmith: at least it is documented in the code comment above the alias https://github.com/openstack/nova/blob/dab4ec1a534d4774e61a24cdace0f3491788e4e7/nova/objects/service.py#L206 so the intention is clear for the reader | |
| 14:18:01 | gibi | dansmith: at least it is documented in the code comment above the alias https://github.com/openstack/nova/blob/dab4ec1a534d4774e61a24cdace0f3491788e4e7/nova/objects/service.py#L206 so the intention is clear for the reader | |
| 14:18:10 | noonedeadpunk | I believe there were solid reasons not to workaround in nova? | |
| 14:18:10 | noonedeadpunk | I believe there were solid reasons not to workaround in nova? | |
| 14:18:27 | dansmith | gibi: okay :) | |
| 14:18:27 | dansmith | gibi: okay :) | |
| 14:18:29 | lyarwood | noonedeadpunk: yeah it would require us to rebase the disks at launch | |
| 14:18:29 | lyarwood | noonedeadpunk: yeah it would require us to rebase the disks at launch | |
| 14:18:38 | noonedeadpunk | Just thinking if it make sense to look into this direction or not | |
| 14:18:38 | noonedeadpunk | Just thinking if it make sense to look into this direction or not | |
| 14:18:41 | gibi | dansmith: :) | |