| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-13 | |||
| 16:47:21 | pas-ha[m] | yeah, I know that 🙂 | |
| 16:47:28 | dansmith | also I suspect your images are not qcow1, but qcow2v1 | |
| 16:48:04 | dansmith | pas-ha[m]: https://review.opendev.org/c/openstack/nova/+/928937 | |
| 16:48:15 | pas-ha[m] | Just wanted to highlight that may be it worth mentioning that in release notes, even retro-actively | |
| 16:48:31 | dansmith | I wasn't planning to merge that, I mostly did it just so it was available | |
| 16:49:27 | dansmith | pas-ha[m]: ack, it wasn't mentioned in the release notes because most of us had assumed people had long since moved off of unsupported formats like that.. not sure we'd go back and put in a retroactive reno like that or not | |
| 16:53:32 | pas-ha[m] | this is tricky semantic 🙂 qemu-img calls it 'qcow' (not qcow2), calling 'file' shows 'QEMU QCOW Image (v1)'. | |
| 16:53:32 | pas-ha[m] | in my understanding it is really qcow == qcow v1. then came qcow v2 == qcow2 which is very different internally, and then came qcow v3 which is an extension of v2 and is backward compatible, so it is still called qcow2 file format. | |
| 16:54:31 | pas-ha[m] | when I tried to repro such an image, this is what you get with 'qemu-img convert -O qcow' | |
| 16:55:27 | pas-ha[m] | and there was QED which is even older than QCOW | |
| 16:57:32 | dansmith | pas-ha[m]: I'm quite sure you don't have any *actual* qcowv1 files | |
| 16:58:20 | dansmith | if it really is, I think we can safely say we don't intend to support your use of those | |
| 16:58:37 | dansmith | I know some *cough* distributions of qemu disable write support for the actual v1 format | |
| 16:59:09 | pas-ha[m] | as I said, this is what 'file' and 'qemu-img info' showed once I fetched files locally. I personally think someone fat fingered -O qcow2 🙂 still, came as a surprise once it broke and stopped working | |
| 17:03:16 | dansmith | yeah, if qemu-img is showing "file format: qcow" then you're using the ancient version that uses totally different code in qemu (as opposed to 2 and 2v3) | |
| 17:04:09 | dansmith | and we have never intended to support that, so I wouldn't even say we broke it.. I'd say that fits into another category of issues where you can fool nova into booting images on libvirt we intend to disallow by lying in glance | |
| 17:04:33 | dansmith | so this is that sort of thing, you claimed it was a v2, it's not, but we sent it to the compute node anyway and didn't notice that it's actually something other than what we expected | |
| 17:05:19 | dansmith | the "unintended" inspector stuff was around breaking qcow2 (vs qcow2v3) *not* about breaking the old-old format we didn't intend to support anyway | |
| 17:20:14 | s3rj1k | sean-k-mooney: with https://review.opendev.org/c/openstack/nova/+/909122 the actual stack trace error is gone but I still see migration failure, "Live Migration failure: Cannot recv data: ssh: Could not resolve hostname" | |
| 17:23:53 | sean-k-mooney | s3rj1k: ok so the revert is not enough to fix https://bugs.launchpad.net/nova/+bug/1939869 then its just part of it | |
| 17:24:28 | sean-k-mooney | s3rj1k: you tested that with both options set to the ip yes adn the revert applied to the dest node | |
| 17:24:40 | sean-k-mooney | (it shoudl be on both but i think it actully runs on the dest node) | |
| 17:26:09 | s3rj1k | both options set yea, also checked with and without config-drive after applying patch | |
| 17:26:33 | s3rj1k | revert is on both computes | |
| 17:26:54 | sean-k-mooney | ok you got the same issue without config drive | |
| 17:27:06 | sean-k-mooney | what are you using for the live migration url | |
| 17:27:29 | sean-k-mooney | are you using the default | |
| 17:27:34 | s3rj1k | yea, I think before revert without config drive migration was working | |
| 17:27:36 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_uri | |
| 17:27:47 | s3rj1k | default migration url | |
| 17:28:09 | sean-k-mooney | ok so plain text over tcp qemu+tcp://%s/system | |
| 17:28:20 | sean-k-mooney | unless you have tls enabled | |
| 17:28:56 | s3rj1k | ok no so default then, I have live_migration_uri = qemu+ssh://stack@%s/system | |
| 17:29:22 | sean-k-mooney | ah you using ssh. its what most do when not using tls | |
| 17:29:32 | sean-k-mooney | so %s is replaced with the hostname | |
| 17:29:50 | sean-k-mooney | well | |
| 17:29:52 | sean-k-mooney | the docs say | |
| 17:29:54 | sean-k-mooney | Any included “%s” is replaced with the migration target hostname, or live_migration_inbound_addr if set. | |
| 17:30:07 | sean-k-mooney | so that shoudl use the ip | |
| 17:30:17 | s3rj1k | live_migration_inbound_addr is set to IP | |
| 17:30:23 | sean-k-mooney | do you have a trace fo the error youc an share with the revert applied | |
| 17:30:44 | sean-k-mooney | s3rj1k: basically im wondering if there is another scp somewhere | |
| 17:30:46 | s3rj1k | actually no trace where in logs, let me check again | |
| 17:30:53 | s3rj1k | actually no trace where in logs, let me check again | |
| 17:32:29 | s3rj1k | yea, see it, I'll update launchpad in a moment | |
| 17:32:30 | sean-k-mooney | ack. my gues is we just missed using the config option somewhere and hardcoded the hypervisor_hostname, the other possibleity is it is using the ip now and that is not in the ssh know hosts | |
| 17:34:13 | sean-k-mooney | oh its might be that change this https://review.opendev.org/c/openstack/nova/+/906053/4/nova/virt/libvirt/driver.py | |
| 17:34:34 | sean-k-mooney | you not tryign to migrate an isntance that uses an image that was deleted in glance are you | |
| 17:35:46 | sean-k-mooney | at the moment we are passign instance.host as the fallback host but that is likely not correct in this case | |
| 17:36:22 | s3rj1k | updated https://bugs.launchpad.net/nova/+bug/2086867/comments/4 | |
| 17:36:44 | sean-k-mooney | oh its comming form migrateToURI3 | |
| 17:37:18 | s3rj1k | using cirros for debug | |
| 17:37:29 | sean-k-mooney | that does not make sense based on what you said was set on the config | |
| 17:37:47 | sean-k-mooney | ya the guest iamge should not factor in here other then if its deleted in glance | |
| 17:38:16 | s3rj1k | it's definitely in config, let me restart services just in case and retry | |
| 17:38:22 | sean-k-mooney | we might just need to set up this evn and debug it | |
| 17:39:06 | sean-k-mooney | this is basiclly just a normal multi node devstack but instead of regestierign the host in dns, mdns or /etchost you have set the ips | |
| 17:39:25 | sean-k-mooney | which on master shoudl work | |
| 17:39:48 | sean-k-mooney | you coudl also set [DEFAULT]host=<ip> i guess | |
| 17:40:01 | sean-k-mooney | you should not have too | |
| 17:40:31 | sean-k-mooney | but im pretty sure that would fix this and break neutorn unless you did the same there.... | |
| 17:41:26 | sean-k-mooney | im debating how easy it would be to repoduce this in the ci | |
| 17:41:43 | sean-k-mooney | vs the hour it woudl take me to set this up by hand. that is proably quicker | |
| 17:41:50 | sean-k-mooney | but you know lazy | |
| 17:47:39 | s3rj1k | ok, so cold-migrate works fine, live-migrate is not | |
| 17:48:01 | s3rj1k | I am pretty sure that this is one-shot repro, so CI is overkill | |
| 17:49:13 | s3rj1k | idea is to have a fallback on IPs when DNS has issues, so directly using would work but UX is bad in this case | |
| 17:52:21 | sean-k-mooney | s3rj1k: so yes an no | |
| 17:52:34 | sean-k-mooney | nova does not genreally have ips to fall back too | |
| 17:52:40 | sean-k-mooney | and in this case its failing in libvirt not nova | |
| 17:53:11 | sean-k-mooney | so nova required that you either have dns for each of the compute nodes ro that you provie hostname resolution some ohter way like /etc/hosts | |
| 17:53:28 | sean-k-mooney | now if you put the ips int eh config that is intended to work | |
| 17:53:35 | sean-k-mooney | so there is a bug somewhere | |
| 17:53:55 | sean-k-mooney | but we require the opertor to make the hostnames resouceabel or provide the ips | |
| 17:57:10 | s3rj1k | so, should we start with merging revert and fixing edge after merge or we should fix-all-the-things :) ? | |
| 17:57:23 | s3rj1k | * edge case | |
| 18:29:46 | sean-k-mooney | s3rj1k: we proably should proceed with the revert if that does not regess anything then fix what is left in a follow up patch | |
| 18:30:32 | sean-k-mooney | s3rj1k: what release are you trying to adress this in by the way | |
| 18:33:21 | s3rj1k | sean-k-mooney: hopefully upcoming 2025.1 Epoxy | |
| 18:33:35 | sean-k-mooney | oh ok you dont need this in older releases as well | |
| 18:33:55 | sean-k-mooney | seperate form the revert im wondering if there was a bug in https://github.com/openstack/nova/commit/6bca37e904e9e56b250b123abde1901e951c8c9a | |
| 18:34:40 | s3rj1k | well, probably some older releases could be potentially needed, but this is another story that I hope I could avoid :) | |
| 18:34:55 | sean-k-mooney | no worries | |
| 18:36:01 | sean-k-mooney | what im condiering is if we consider it a regresion intoduced by the "libvirt-migrate-with-hostname-instead-of-ip" which was not ment to change the default behaicvor then it would be a bug and backportable | |
| 18:36:09 | s3rj1k | migration_inbound_addr is for none-live migration? if so, none-live migration worked for me when DNS is blocked | |
| 18:36:12 | sean-k-mooney | otherwise this is kind of a mini featur and only fixable on master | |
| 18:37:03 | sean-k-mooney | s3rj1k: it is but its also for some shared commdns that are used during live migration | |
| 18:37:47 | sean-k-mooney | s3rj1k: im wondering if this https://github.com/openstack/nova/commit/6bca37e904e9e56b250b123abde1901e951c8c9a#diff-486ddc37ce8e7c105a51566dd2cffbda034f28fae7eaad2e74abf470ab296352R4514-R4523 | |
| 18:37:59 | sean-k-mooney | impacted live migrtaion in a way we did not plan it too | |
| 18:38:14 | sean-k-mooney | although you have set both to an ip | |
| 18:38:18 | sean-k-mooney | so proably something else | |
| 18:38:47 | sean-k-mooney | i need to set up a multi node devstack for somethign else | |
| 18:38:53 | s3rj1k | sean-k-mooney: I can retested after revert gets merged, so no problem with that | |
| 18:38:58 | s3rj1k | hmm, "if "%s" in addr:" | |
| 18:39:00 | sean-k-mooney | ill see if i can repoduce this when i do | |
| 18:39:28 | sean-k-mooney | oh | |
| 18:39:31 | sean-k-mooney | ya that is the bug | |
| 18:39:45 | sean-k-mooney | that code is assumeing that %s is always going to repalce with the hostname | |
| 18:40:14 | sean-k-mooney | but if you set the livemigration config var to the ip we should not be using get_hostname there | |
| 18:40:38 | sean-k-mooney | well hum | |