Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-11-13
17:30:17 s3rj1k live_migration_inbound_addr is set to IP
17:30:23 sean-k-mooney do you have a trace fo the error youc an share with the revert applied
17:30:44 sean-k-mooney s3rj1k: basically im wondering if there is another scp somewhere
17:30:46 s3rj1k actually no trace where in logs, let me check again
17:30:53 s3rj1k actually no trace where in logs, let me check again
17:32:29 s3rj1k yea, see it, I'll update launchpad in a moment
17:32:30 sean-k-mooney ack. my gues is we just missed using the config option somewhere and hardcoded the hypervisor_hostname, the other possibleity is it is using the ip now and that is not in the ssh know hosts
17:34:13 sean-k-mooney oh its might be that change this https://review.opendev.org/c/openstack/nova/+/906053/4/nova/virt/libvirt/driver.py
17:34:34 sean-k-mooney you not tryign to migrate an isntance that uses an image that was deleted in glance are you
17:35:46 sean-k-mooney at the moment we are passign instance.host as the fallback host but that is likely not correct in this case
17:36:22 s3rj1k updated https://bugs.launchpad.net/nova/+bug/2086867/comments/4
17:36:44 sean-k-mooney oh its comming form migrateToURI3
17:37:18 s3rj1k using cirros for debug
17:37:29 sean-k-mooney that does not make sense based on what you said was set on the config
17:37:47 sean-k-mooney ya the guest iamge should not factor in here other then if its deleted in glance
17:38:16 s3rj1k it's definitely in config, let me restart services just in case and retry
17:38:22 sean-k-mooney we might just need to set up this evn and debug it
17:39:06 sean-k-mooney this is basiclly just a normal multi node devstack but instead of regestierign the host in dns, mdns or /etchost you have set the ips
17:39:25 sean-k-mooney which on master shoudl work
17:39:48 sean-k-mooney you coudl also set [DEFAULT]host=<ip> i guess
17:40:01 sean-k-mooney you should not have too
17:40:31 sean-k-mooney but im pretty sure that would fix this and break neutorn unless you did the same there....
17:41:26 sean-k-mooney im debating how easy it would be to repoduce this in the ci
17:41:43 sean-k-mooney vs the hour it woudl take me to set this up by hand. that is proably quicker
17:41:50 sean-k-mooney but you know lazy
17:47:39 s3rj1k ok, so cold-migrate works fine, live-migrate is not
17:48:01 s3rj1k I am pretty sure that this is one-shot repro, so CI is overkill
17:49:13 s3rj1k idea is to have a fallback on IPs when DNS has issues, so directly using would work but UX is bad in this case
17:52:21 sean-k-mooney s3rj1k: so yes an no
17:52:34 sean-k-mooney nova does not genreally have ips to fall back too
17:52:40 sean-k-mooney and in this case its failing in libvirt not nova
17:53:11 sean-k-mooney so nova required that you either have dns for each of the compute nodes ro that you provie hostname resolution some ohter way like /etc/hosts
17:53:28 sean-k-mooney now if you put the ips int eh config that is intended to work
17:53:35 sean-k-mooney so there is a bug somewhere
17:53:55 sean-k-mooney but we require the opertor to make the hostnames resouceabel or provide the ips
17:57:10 s3rj1k so, should we start with merging revert and fixing edge after merge or we should fix-all-the-things :) ?
17:57:23 s3rj1k * edge case
18:29:46 sean-k-mooney s3rj1k: we proably should proceed with the revert if that does not regess anything then fix what is left in a follow up patch
18:30:32 sean-k-mooney s3rj1k: what release are you trying to adress this in by the way
18:33:21 s3rj1k sean-k-mooney: hopefully upcoming 2025.1 Epoxy
18:33:35 sean-k-mooney oh ok you dont need this in older releases as well
18:33:55 sean-k-mooney seperate form the revert im wondering if there was a bug in https://github.com/openstack/nova/commit/6bca37e904e9e56b250b123abde1901e951c8c9a
18:34:40 s3rj1k well, probably some older releases could be potentially needed, but this is another story that I hope I could avoid :)
18:34:55 sean-k-mooney no worries
18:36:01 sean-k-mooney what im condiering is if we consider it a regresion intoduced by the "libvirt-migrate-with-hostname-instead-of-ip" which was not ment to change the default behaicvor then it would be a bug and backportable
18:36:09 s3rj1k migration_inbound_addr is for none-live migration? if so, none-live migration worked for me when DNS is blocked
18:36:12 sean-k-mooney otherwise this is kind of a mini featur and only fixable on master
18:37:03 sean-k-mooney s3rj1k: it is but its also for some shared commdns that are used during live migration
18:37:47 sean-k-mooney s3rj1k: im wondering if this https://github.com/openstack/nova/commit/6bca37e904e9e56b250b123abde1901e951c8c9a#diff-486ddc37ce8e7c105a51566dd2cffbda034f28fae7eaad2e74abf470ab296352R4514-R4523
18:37:59 sean-k-mooney impacted live migrtaion in a way we did not plan it too
18:38:14 sean-k-mooney although you have set both to an ip
18:38:18 sean-k-mooney so proably something else
18:38:47 sean-k-mooney i need to set up a multi node devstack for somethign else
18:38:53 s3rj1k sean-k-mooney: I can retested after revert gets merged, so no problem with that
18:38:58 s3rj1k hmm, "if "%s" in addr:"
18:39:00 sean-k-mooney ill see if i can repoduce this when i do
18:39:28 sean-k-mooney oh
18:39:31 sean-k-mooney ya that is the bug
18:39:45 sean-k-mooney that code is assumeing that %s is always going to repalce with the hostname
18:40:14 sean-k-mooney but if you set the livemigration config var to the ip we should not be using get_hostname there
18:40:38 sean-k-mooney well hum
18:40:53 s3rj1k fix would be moving line 4520 below if I guess
18:40:53 sean-k-mooney i think that depned on where get_host_ip_addr is being called form at the time
18:41:38 s3rj1k with some check if conf is set
18:42:19 sean-k-mooney something along those lines maybe
18:42:52 sean-k-mooney could you try regerting get_host_ip_addr to just return CONF.my_ip
18:42:58 sean-k-mooney and see if that fixes it for you locally
18:43:19 sean-k-mooney it will break the other fature so that not the correct fix but it will confirm if that is the souce of the regression
18:43:26 s3rj1k yea, give me few minutes
18:45:09 sean-k-mooney although you do not have "%s" in CONF.libvirt.migration_inbound_addr do you?
18:45:27 sean-k-mooney you do in the migration uri but that is just an ip adress reight
18:45:30 sean-k-mooney *right
18:54:30 s3rj1k I have live_migration_uri = qemu+ssh://stack@%s/system and live_migration_inbound_addr, migration_inbound_addr set to IPs, nothing more related to migration in libvirt section
18:55:20 s3rj1k so I've tried to return both CONF.my_ip and CONF.libvirt.migration_inbound_addr, migration still tries to resolve by dns
19:35:10 sean-k-mooney s3rj1k: ok we are going to need ot look at exaclty what we are passing to the migrate too uri libvirt call
19:36:56 s3rj1k sean-k-mooney: do you have some specific debug changes in mind?
19:40:31 sean-k-mooney baiscally debugging this call https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10854-L10859
19:41:03 sean-k-mooney we can do that by putting some debug logs or prints in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L591
19:42:06 sean-k-mooney s3rj1k: in your config we would expect both destination and migrate_uri to have the ip
19:42:15 sean-k-mooney but i suspect one or both of them have the hostname
19:42:58 sean-k-mooney s3rj1k: the excption form libvirt is comming form this funciton call https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L647
19:51:56 s3rj1k hmm, destination has hostname as expected I guess, but migrate_uri is None for some reason
19:54:26 s3rj1k smth wrong in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10781-L10786 ?
19:55:06 s3rj1k "non tunneled migration"
20:09:18 sean-k-mooney we do not want to use tunneled migration so i think that is fine
20:09:34 sean-k-mooney so the quetion is where did destinaiton get set ot the host name
20:10:04 s3rj1k migrate_uri is None, this is expected?
20:10:30 sean-k-mooney i dont recall of the top of my head but we do not adivse using tunneld migration
20:10:52 sean-k-mooney so that being none is proably ok
20:11:01 sean-k-mooney dist i think is coming form (self._live_migration_uri(dest)
20:11:40 sean-k-mooney ok that just ipv6 escapign the dest that past in
20:14:23 sean-k-mooney so thats passed in to live_migrate in the driver and that invoked over rpc via the compute manager
20:16:43 s3rj1k here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10733 it is already a hostname
20:17:02 s3rj1k (dest var)
20:17:11 sean-k-mooney yep
20:17:43 sean-k-mooney so in the conduction its a destation object i think it being converted to the host name in the compute manager but im trying to find that
20:19:42 sean-k-mooney or not https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9037
20:19:56 sean-k-mooney that looks like its alrady going to be the host name on the comptue manager side of the rpc
20:21:43 sean-k-mooney s3rj1k: so https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L509-L562 is returning the host form the select_destinations call to the schduler
20:22:08 sean-k-mooney selection.service_host

Earlier   Later