| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-12 | |||
| 16:33:18 | fwiesel | thanks o/ | |
| 16:35:50 | opendevreview | Merged openstack/nova master: Skip functional tests on pre-commit config update https://review.opendev.org/c/openstack/nova/+/933192 | |
| 18:27:42 | opendevreview | Artem Goncharov proposed openstack/nova master: Fix add/remove SecurityGroup action json schemas https://review.opendev.org/c/openstack/nova/+/934823 | |
| #openstack-nova - 2024-11-13 | |||
| 08:24:22 | opendevreview | Artem Goncharov proposed openstack/nova master: Fix add/remove SecurityGroup action json schemas https://review.opendev.org/c/openstack/nova/+/934823 | |
| 15:28:58 | opendevreview | Max proposed openstack/nova master: fix: leftover BDM after instance delete https://review.opendev.org/c/openstack/nova/+/934984 | |
| 15:44:49 | dansmith | sean-k-mooney: can you look at the osvif failure here? https://review.opendev.org/c/openstack/requirements/+/934176 | |
| 15:44:57 | dansmith | cc frickler ^ | |
| 15:53:42 | tkajinam | dansmith, we need https://review.opendev.org/c/openstack/os-vif/+/932436 and a new release of os-vif. | |
| 15:53:56 | tkajinam | but that change is blocked due to broken openstack-tox-functional-ovs-with-sudo | |
| 15:54:35 | dansmith | ah I didn't see a comment on that requirements patch so I didn't know | |
| 15:54:52 | dansmith | that requirements thing is holding up another nova fix, among other things | |
| 16:10:34 | s3rj1k | sean-k-mooney: Hi, was able to repro that live-migration issue with broken DNS, please see details here https://bugs.launchpad.net/nova/+bug/2086867/comments/3 | |
| 16:12:53 | opendevreview | Merged openstack/nova-specs master: Update and Re-propose "Allow Manila shares to be directly attached to an instance when using libvirt" for Epoxy https://review.opendev.org/c/openstack/nova-specs/+/930471 | |
| 16:15:24 | sean-k-mooney | s3rj1k: ok that not a nova bug :) | |
| 16:16:01 | sean-k-mooney | dansmith: i cna take a look but the the last time i looked at it a few months ago it work for me locally and i coudl not repoduce the failure | |
| 16:16:12 | sean-k-mooney | dansmith: so ill see if it repodcues localy now | |
| 16:16:45 | sean-k-mooney | s3rj1k: oh this was in relation to using the ip adress | |
| 16:17:23 | sean-k-mooney | s3rj1k: so have you defiend the two nova config option i noted to use the ip? | |
| 16:18:54 | sean-k-mooney | oh this is failing on config drive | |
| 16:19:07 | sean-k-mooney | this i shiti som old fallback cod ethat is not needed anymore i think | |
| 16:21:05 | sean-k-mooney | we used to have a workaround for old libvirt version that we have not supproted for many many years | |
| 16:21:29 | sean-k-mooney | where we copy the config drive in nova instead of leaving libvirt do it | |
| 16:22:26 | sean-k-mooney | ya this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11456-L11468 | |
| 16:22:51 | sean-k-mooney | s3rj1k: so that cod ethat we can delete now, there is a patch somewhere to fix that | |
| 16:23:23 | sean-k-mooney | as a workaround you could use vfat config drives but we shoudl just delete that code. | |
| 16:24:00 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/909122 | |
| 16:24:08 | sean-k-mooney | s3rj1k: ^ thats the fix for your issue | |
| 16:24:42 | opendevreview | Doug Szumski proposed openstack/nova master: Revert "[libvirt] Live migration fails when config_drive_format=iso9660" https://review.opendev.org/c/openstack/nova/+/909122 | |
| 16:24:58 | s3rj1k | sean-k-mooney: yea both options on both computes with corresponding IPs | |
| 16:26:45 | s3rj1k | hmm, https://review.opendev.org/c/openstack/nova/+/909122 will remove support for config-drive? | |
| 16:26:52 | sean-k-mooney | no | |
| 16:27:22 | sean-k-mooney | libvirt and qemu can automatically copy the iso file as part fo the live migration | |
| 16:27:38 | sean-k-mooney | but libvirt had a bug 8-9 years ago | |
| 16:27:46 | s3rj1k | ok, let me try that, I'll report back | |
| 16:27:52 | sean-k-mooney | and that was a tempory workaround until it could be fixed and we never reverted it | |
| 16:28:46 | sean-k-mooney | i think this stopped being a thing we needed to care about when we dropped ubuntu 12.04 or 14.04 support | |
| 16:31:21 | sean-k-mooney | s3rj1k: ah someone else already reported a similar issue to yours https://bugs.launchpad.net/nova/+bug/1939869 | |
| 16:37:35 | s3rj1k | similar yes, fails on "Host key verification failed" | |
| 16:44:11 | pas-ha[m] | hi folks, today we faced what seems another fallout of those qcow-related CVE fixes. After update to latest Antelope code, some of our previously working (windows) images no longer boot (and volume can not be created). Upon closer inspection, those happened to be qcow v1 images uploaded to glance with disk_format=qcow2. This is clearly not right, yet it was working before and I wonder who else might get bitten.. | |
| 16:46:54 | dansmith | pas-ha[m]: it was working before because glance does absolutely no inspection of images | |
| 16:47:05 | pas-ha[m] | from the comment in oslo.utils image format inspector, I get that this was intentional, yet I do not see that in release notes or smth. | |
| 16:47:21 | pas-ha[m] | yeah, I know that 🙂 | |
| 16:47:28 | dansmith | also I suspect your images are not qcow1, but qcow2v1 | |
| 16:48:04 | dansmith | pas-ha[m]: https://review.opendev.org/c/openstack/nova/+/928937 | |
| 16:48:15 | pas-ha[m] | Just wanted to highlight that may be it worth mentioning that in release notes, even retro-actively | |
| 16:48:31 | dansmith | I wasn't planning to merge that, I mostly did it just so it was available | |
| 16:49:27 | dansmith | pas-ha[m]: ack, it wasn't mentioned in the release notes because most of us had assumed people had long since moved off of unsupported formats like that.. not sure we'd go back and put in a retroactive reno like that or not | |
| 16:53:32 | pas-ha[m] | in my understanding it is really qcow == qcow v1. then came qcow v2 == qcow2 which is very different internally, and then came qcow v3 which is an extension of v2 and is backward compatible, so it is still called qcow2 file format. | |
| 16:53:32 | pas-ha[m] | this is tricky semantic 🙂 qemu-img calls it 'qcow' (not qcow2), calling 'file' shows 'QEMU QCOW Image (v1)'. | |
| 16:54:31 | pas-ha[m] | when I tried to repro such an image, this is what you get with 'qemu-img convert -O qcow' | |
| 16:55:27 | pas-ha[m] | and there was QED which is even older than QCOW | |
| 16:57:32 | dansmith | pas-ha[m]: I'm quite sure you don't have any *actual* qcowv1 files | |
| 16:58:20 | dansmith | if it really is, I think we can safely say we don't intend to support your use of those | |
| 16:58:37 | dansmith | I know some *cough* distributions of qemu disable write support for the actual v1 format | |
| 16:59:09 | pas-ha[m] | as I said, this is what 'file' and 'qemu-img info' showed once I fetched files locally. I personally think someone fat fingered -O qcow2 🙂 still, came as a surprise once it broke and stopped working | |
| 17:03:16 | dansmith | yeah, if qemu-img is showing "file format: qcow" then you're using the ancient version that uses totally different code in qemu (as opposed to 2 and 2v3) | |
| 17:04:09 | dansmith | and we have never intended to support that, so I wouldn't even say we broke it.. I'd say that fits into another category of issues where you can fool nova into booting images on libvirt we intend to disallow by lying in glance | |
| 17:04:33 | dansmith | so this is that sort of thing, you claimed it was a v2, it's not, but we sent it to the compute node anyway and didn't notice that it's actually something other than what we expected | |
| 17:05:19 | dansmith | the "unintended" inspector stuff was around breaking qcow2 (vs qcow2v3) *not* about breaking the old-old format we didn't intend to support anyway | |
| 17:20:14 | s3rj1k | sean-k-mooney: with https://review.opendev.org/c/openstack/nova/+/909122 the actual stack trace error is gone but I still see migration failure, "Live Migration failure: Cannot recv data: ssh: Could not resolve hostname" | |
| 17:23:53 | sean-k-mooney | s3rj1k: ok so the revert is not enough to fix https://bugs.launchpad.net/nova/+bug/1939869 then its just part of it | |
| 17:24:28 | sean-k-mooney | s3rj1k: you tested that with both options set to the ip yes adn the revert applied to the dest node | |
| 17:24:40 | sean-k-mooney | (it shoudl be on both but i think it actully runs on the dest node) | |
| 17:26:09 | s3rj1k | both options set yea, also checked with and without config-drive after applying patch | |
| 17:26:33 | s3rj1k | revert is on both computes | |
| 17:26:54 | sean-k-mooney | ok you got the same issue without config drive | |
| 17:27:06 | sean-k-mooney | what are you using for the live migration url | |
| 17:27:29 | sean-k-mooney | are you using the default | |
| 17:27:34 | s3rj1k | yea, I think before revert without config drive migration was working | |
| 17:27:36 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_uri | |
| 17:27:47 | s3rj1k | default migration url | |
| 17:28:09 | sean-k-mooney | ok so plain text over tcp qemu+tcp://%s/system | |
| 17:28:20 | sean-k-mooney | unless you have tls enabled | |
| 17:28:56 | s3rj1k | ok no so default then, I have live_migration_uri = qemu+ssh://stack@%s/system | |
| 17:29:22 | sean-k-mooney | ah you using ssh. its what most do when not using tls | |
| 17:29:32 | sean-k-mooney | so %s is replaced with the hostname | |
| 17:29:50 | sean-k-mooney | well | |
| 17:29:52 | sean-k-mooney | the docs say | |
| 17:29:54 | sean-k-mooney | Any included “%s” is replaced with the migration target hostname, or live_migration_inbound_addr if set. | |
| 17:30:07 | sean-k-mooney | so that shoudl use the ip | |
| 17:30:17 | s3rj1k | live_migration_inbound_addr is set to IP | |
| 17:30:23 | sean-k-mooney | do you have a trace fo the error youc an share with the revert applied | |
| 17:30:44 | sean-k-mooney | s3rj1k: basically im wondering if there is another scp somewhere | |
| 17:30:46 | s3rj1k | actually no trace where in logs, let me check again | |
| 17:30:53 | s3rj1k | actually no trace where in logs, let me check again | |
| 17:32:29 | s3rj1k | yea, see it, I'll update launchpad in a moment | |
| 17:32:30 | sean-k-mooney | ack. my gues is we just missed using the config option somewhere and hardcoded the hypervisor_hostname, the other possibleity is it is using the ip now and that is not in the ssh know hosts | |
| 17:34:13 | sean-k-mooney | oh its might be that change this https://review.opendev.org/c/openstack/nova/+/906053/4/nova/virt/libvirt/driver.py | |
| 17:34:34 | sean-k-mooney | you not tryign to migrate an isntance that uses an image that was deleted in glance are you | |
| 17:35:46 | sean-k-mooney | at the moment we are passign instance.host as the fallback host but that is likely not correct in this case | |
| 17:36:22 | s3rj1k | updated https://bugs.launchpad.net/nova/+bug/2086867/comments/4 | |
| 17:36:44 | sean-k-mooney | oh its comming form migrateToURI3 | |
| 17:37:18 | s3rj1k | using cirros for debug | |
| 17:37:29 | sean-k-mooney | that does not make sense based on what you said was set on the config | |
| 17:37:47 | sean-k-mooney | ya the guest iamge should not factor in here other then if its deleted in glance | |
| 17:38:16 | s3rj1k | it's definitely in config, let me restart services just in case and retry | |
| 17:38:22 | sean-k-mooney | we might just need to set up this evn and debug it | |
| 17:39:06 | sean-k-mooney | this is basiclly just a normal multi node devstack but instead of regestierign the host in dns, mdns or /etchost you have set the ips | |
| 17:39:25 | sean-k-mooney | which on master shoudl work | |
| 17:39:48 | sean-k-mooney | you coudl also set [DEFAULT]host=<ip> i guess | |