| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-13 | |||
| 17:29:32 | sean-k-mooney | so %s is replaced with the hostname | |
| 17:29:50 | sean-k-mooney | well | |
| 17:29:52 | sean-k-mooney | the docs say | |
| 17:29:54 | sean-k-mooney | Any included “%s” is replaced with the migration target hostname, or live_migration_inbound_addr if set. | |
| 17:30:07 | sean-k-mooney | so that shoudl use the ip | |
| 17:30:17 | s3rj1k | live_migration_inbound_addr is set to IP | |
| 17:30:23 | sean-k-mooney | do you have a trace fo the error youc an share with the revert applied | |
| 17:30:44 | sean-k-mooney | s3rj1k: basically im wondering if there is another scp somewhere | |
| 17:30:46 | s3rj1k | actually no trace where in logs, let me check again | |
| 17:30:53 | s3rj1k | actually no trace where in logs, let me check again | |
| 17:32:29 | s3rj1k | yea, see it, I'll update launchpad in a moment | |
| 17:32:30 | sean-k-mooney | ack. my gues is we just missed using the config option somewhere and hardcoded the hypervisor_hostname, the other possibleity is it is using the ip now and that is not in the ssh know hosts | |
| 17:34:13 | sean-k-mooney | oh its might be that change this https://review.opendev.org/c/openstack/nova/+/906053/4/nova/virt/libvirt/driver.py | |
| 17:34:34 | sean-k-mooney | you not tryign to migrate an isntance that uses an image that was deleted in glance are you | |
| 17:35:46 | sean-k-mooney | at the moment we are passign instance.host as the fallback host but that is likely not correct in this case | |
| 17:36:22 | s3rj1k | updated https://bugs.launchpad.net/nova/+bug/2086867/comments/4 | |
| 17:36:44 | sean-k-mooney | oh its comming form migrateToURI3 | |
| 17:37:18 | s3rj1k | using cirros for debug | |
| 17:37:29 | sean-k-mooney | that does not make sense based on what you said was set on the config | |
| 17:37:47 | sean-k-mooney | ya the guest iamge should not factor in here other then if its deleted in glance | |
| 17:38:16 | s3rj1k | it's definitely in config, let me restart services just in case and retry | |
| 17:38:22 | sean-k-mooney | we might just need to set up this evn and debug it | |
| 17:39:06 | sean-k-mooney | this is basiclly just a normal multi node devstack but instead of regestierign the host in dns, mdns or /etchost you have set the ips | |
| 17:39:25 | sean-k-mooney | which on master shoudl work | |
| 17:39:48 | sean-k-mooney | you coudl also set [DEFAULT]host=<ip> i guess | |
| 17:40:01 | sean-k-mooney | you should not have too | |
| 17:40:31 | sean-k-mooney | but im pretty sure that would fix this and break neutorn unless you did the same there.... | |
| 17:41:26 | sean-k-mooney | im debating how easy it would be to repoduce this in the ci | |
| 17:41:43 | sean-k-mooney | vs the hour it woudl take me to set this up by hand. that is proably quicker | |
| 17:41:50 | sean-k-mooney | but you know lazy | |
| 17:47:39 | s3rj1k | ok, so cold-migrate works fine, live-migrate is not | |
| 17:48:01 | s3rj1k | I am pretty sure that this is one-shot repro, so CI is overkill | |
| 17:49:13 | s3rj1k | idea is to have a fallback on IPs when DNS has issues, so directly using would work but UX is bad in this case | |
| 17:52:21 | sean-k-mooney | s3rj1k: so yes an no | |
| 17:52:34 | sean-k-mooney | nova does not genreally have ips to fall back too | |
| 17:52:40 | sean-k-mooney | and in this case its failing in libvirt not nova | |
| 17:53:11 | sean-k-mooney | so nova required that you either have dns for each of the compute nodes ro that you provie hostname resolution some ohter way like /etc/hosts | |
| 17:53:28 | sean-k-mooney | now if you put the ips int eh config that is intended to work | |
| 17:53:35 | sean-k-mooney | so there is a bug somewhere | |
| 17:53:55 | sean-k-mooney | but we require the opertor to make the hostnames resouceabel or provide the ips | |
| 17:57:10 | s3rj1k | so, should we start with merging revert and fixing edge after merge or we should fix-all-the-things :) ? | |
| 17:57:23 | s3rj1k | * edge case | |
| 18:29:46 | sean-k-mooney | s3rj1k: we proably should proceed with the revert if that does not regess anything then fix what is left in a follow up patch | |
| 18:30:32 | sean-k-mooney | s3rj1k: what release are you trying to adress this in by the way | |
| 18:33:21 | s3rj1k | sean-k-mooney: hopefully upcoming 2025.1 Epoxy | |
| 18:33:35 | sean-k-mooney | oh ok you dont need this in older releases as well | |
| 18:33:55 | sean-k-mooney | seperate form the revert im wondering if there was a bug in https://github.com/openstack/nova/commit/6bca37e904e9e56b250b123abde1901e951c8c9a | |
| 18:34:40 | s3rj1k | well, probably some older releases could be potentially needed, but this is another story that I hope I could avoid :) | |
| 18:34:55 | sean-k-mooney | no worries | |
| 18:36:01 | sean-k-mooney | what im condiering is if we consider it a regresion intoduced by the "libvirt-migrate-with-hostname-instead-of-ip" which was not ment to change the default behaicvor then it would be a bug and backportable | |
| 18:36:09 | s3rj1k | migration_inbound_addr is for none-live migration? if so, none-live migration worked for me when DNS is blocked | |
| 18:36:12 | sean-k-mooney | otherwise this is kind of a mini featur and only fixable on master | |
| 18:37:03 | sean-k-mooney | s3rj1k: it is but its also for some shared commdns that are used during live migration | |
| 18:37:47 | sean-k-mooney | s3rj1k: im wondering if this https://github.com/openstack/nova/commit/6bca37e904e9e56b250b123abde1901e951c8c9a#diff-486ddc37ce8e7c105a51566dd2cffbda034f28fae7eaad2e74abf470ab296352R4514-R4523 | |
| 18:37:59 | sean-k-mooney | impacted live migrtaion in a way we did not plan it too | |
| 18:38:14 | sean-k-mooney | although you have set both to an ip | |
| 18:38:18 | sean-k-mooney | so proably something else | |
| 18:38:47 | sean-k-mooney | i need to set up a multi node devstack for somethign else | |
| 18:38:53 | s3rj1k | sean-k-mooney: I can retested after revert gets merged, so no problem with that | |
| 18:38:58 | s3rj1k | hmm, "if "%s" in addr:" | |
| 18:39:00 | sean-k-mooney | ill see if i can repoduce this when i do | |
| 18:39:28 | sean-k-mooney | oh | |
| 18:39:31 | sean-k-mooney | ya that is the bug | |
| 18:39:45 | sean-k-mooney | that code is assumeing that %s is always going to repalce with the hostname | |
| 18:40:14 | sean-k-mooney | but if you set the livemigration config var to the ip we should not be using get_hostname there | |
| 18:40:38 | sean-k-mooney | well hum | |
| 18:40:53 | s3rj1k | fix would be moving line 4520 below if I guess | |
| 18:40:53 | sean-k-mooney | i think that depned on where get_host_ip_addr is being called form at the time | |
| 18:41:38 | s3rj1k | with some check if conf is set | |
| 18:42:19 | sean-k-mooney | something along those lines maybe | |
| 18:42:52 | sean-k-mooney | could you try regerting get_host_ip_addr to just return CONF.my_ip | |
| 18:42:58 | sean-k-mooney | and see if that fixes it for you locally | |
| 18:43:19 | sean-k-mooney | it will break the other fature so that not the correct fix but it will confirm if that is the souce of the regression | |
| 18:43:26 | s3rj1k | yea, give me few minutes | |
| 18:45:09 | sean-k-mooney | although you do not have "%s" in CONF.libvirt.migration_inbound_addr do you? | |
| 18:45:27 | sean-k-mooney | you do in the migration uri but that is just an ip adress reight | |
| 18:45:30 | sean-k-mooney | *right | |
| 18:54:30 | s3rj1k | I have live_migration_uri = qemu+ssh://stack@%s/system and live_migration_inbound_addr, migration_inbound_addr set to IPs, nothing more related to migration in libvirt section | |
| 18:55:20 | s3rj1k | so I've tried to return both CONF.my_ip and CONF.libvirt.migration_inbound_addr, migration still tries to resolve by dns | |
| 19:35:10 | sean-k-mooney | s3rj1k: ok we are going to need ot look at exaclty what we are passing to the migrate too uri libvirt call | |
| 19:36:56 | s3rj1k | sean-k-mooney: do you have some specific debug changes in mind? | |
| 19:40:31 | sean-k-mooney | baiscally debugging this call https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10854-L10859 | |
| 19:41:03 | sean-k-mooney | we can do that by putting some debug logs or prints in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L591 | |
| 19:42:06 | sean-k-mooney | s3rj1k: in your config we would expect both destination and migrate_uri to have the ip | |
| 19:42:15 | sean-k-mooney | but i suspect one or both of them have the hostname | |
| 19:42:58 | sean-k-mooney | s3rj1k: the excption form libvirt is comming form this funciton call https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L647 | |
| 19:51:56 | s3rj1k | hmm, destination has hostname as expected I guess, but migrate_uri is None for some reason | |
| 19:54:26 | s3rj1k | smth wrong in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10781-L10786 ? | |
| 19:55:06 | s3rj1k | "non tunneled migration" | |
| 20:09:18 | sean-k-mooney | we do not want to use tunneled migration so i think that is fine | |
| 20:09:34 | sean-k-mooney | so the quetion is where did destinaiton get set ot the host name | |
| 20:10:04 | s3rj1k | migrate_uri is None, this is expected? | |
| 20:10:30 | sean-k-mooney | i dont recall of the top of my head but we do not adivse using tunneld migration | |
| 20:10:52 | sean-k-mooney | so that being none is proably ok | |
| 20:11:01 | sean-k-mooney | dist i think is coming form (self._live_migration_uri(dest) | |
| 20:11:40 | sean-k-mooney | ok that just ipv6 escapign the dest that past in | |
| 20:14:23 | sean-k-mooney | so thats passed in to live_migrate in the driver and that invoked over rpc via the compute manager | |
| 20:16:43 | s3rj1k | here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10733 it is already a hostname | |
| 20:17:02 | s3rj1k | (dest var) | |
| 20:17:11 | sean-k-mooney | yep | |