| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-13 | |||
| 17:52:21 | sean-k-mooney | s3rj1k: so yes an no | |
| 17:52:34 | sean-k-mooney | nova does not genreally have ips to fall back too | |
| 17:52:40 | sean-k-mooney | and in this case its failing in libvirt not nova | |
| 17:53:11 | sean-k-mooney | so nova required that you either have dns for each of the compute nodes ro that you provie hostname resolution some ohter way like /etc/hosts | |
| 17:53:28 | sean-k-mooney | now if you put the ips int eh config that is intended to work | |
| 17:53:35 | sean-k-mooney | so there is a bug somewhere | |
| 17:53:55 | sean-k-mooney | but we require the opertor to make the hostnames resouceabel or provide the ips | |
| 17:57:10 | s3rj1k | so, should we start with merging revert and fixing edge after merge or we should fix-all-the-things :) ? | |
| 17:57:23 | s3rj1k | * edge case | |
| 18:29:46 | sean-k-mooney | s3rj1k: we proably should proceed with the revert if that does not regess anything then fix what is left in a follow up patch | |
| 18:30:32 | sean-k-mooney | s3rj1k: what release are you trying to adress this in by the way | |
| 18:33:21 | s3rj1k | sean-k-mooney: hopefully upcoming 2025.1 Epoxy | |
| 18:33:35 | sean-k-mooney | oh ok you dont need this in older releases as well | |
| 18:33:55 | sean-k-mooney | seperate form the revert im wondering if there was a bug in https://github.com/openstack/nova/commit/6bca37e904e9e56b250b123abde1901e951c8c9a | |
| 18:34:40 | s3rj1k | well, probably some older releases could be potentially needed, but this is another story that I hope I could avoid :) | |
| 18:34:55 | sean-k-mooney | no worries | |
| 18:36:01 | sean-k-mooney | what im condiering is if we consider it a regresion intoduced by the "libvirt-migrate-with-hostname-instead-of-ip" which was not ment to change the default behaicvor then it would be a bug and backportable | |
| 18:36:09 | s3rj1k | migration_inbound_addr is for none-live migration? if so, none-live migration worked for me when DNS is blocked | |
| 18:36:12 | sean-k-mooney | otherwise this is kind of a mini featur and only fixable on master | |
| 18:37:03 | sean-k-mooney | s3rj1k: it is but its also for some shared commdns that are used during live migration | |
| 18:37:47 | sean-k-mooney | s3rj1k: im wondering if this https://github.com/openstack/nova/commit/6bca37e904e9e56b250b123abde1901e951c8c9a#diff-486ddc37ce8e7c105a51566dd2cffbda034f28fae7eaad2e74abf470ab296352R4514-R4523 | |
| 18:37:59 | sean-k-mooney | impacted live migrtaion in a way we did not plan it too | |
| 18:38:14 | sean-k-mooney | although you have set both to an ip | |
| 18:38:18 | sean-k-mooney | so proably something else | |
| 18:38:47 | sean-k-mooney | i need to set up a multi node devstack for somethign else | |
| 18:38:53 | s3rj1k | sean-k-mooney: I can retested after revert gets merged, so no problem with that | |
| 18:38:58 | s3rj1k | hmm, "if "%s" in addr:" | |
| 18:39:00 | sean-k-mooney | ill see if i can repoduce this when i do | |
| 18:39:28 | sean-k-mooney | oh | |
| 18:39:31 | sean-k-mooney | ya that is the bug | |
| 18:39:45 | sean-k-mooney | that code is assumeing that %s is always going to repalce with the hostname | |
| 18:40:14 | sean-k-mooney | but if you set the livemigration config var to the ip we should not be using get_hostname there | |
| 18:40:38 | sean-k-mooney | well hum | |
| 18:40:53 | s3rj1k | fix would be moving line 4520 below if I guess | |
| 18:40:53 | sean-k-mooney | i think that depned on where get_host_ip_addr is being called form at the time | |
| 18:41:38 | s3rj1k | with some check if conf is set | |
| 18:42:19 | sean-k-mooney | something along those lines maybe | |
| 18:42:52 | sean-k-mooney | could you try regerting get_host_ip_addr to just return CONF.my_ip | |
| 18:42:58 | sean-k-mooney | and see if that fixes it for you locally | |
| 18:43:19 | sean-k-mooney | it will break the other fature so that not the correct fix but it will confirm if that is the souce of the regression | |
| 18:43:26 | s3rj1k | yea, give me few minutes | |
| 18:45:09 | sean-k-mooney | although you do not have "%s" in CONF.libvirt.migration_inbound_addr do you? | |
| 18:45:27 | sean-k-mooney | you do in the migration uri but that is just an ip adress reight | |
| 18:45:30 | sean-k-mooney | *right | |
| 18:54:30 | s3rj1k | I have live_migration_uri = qemu+ssh://stack@%s/system and live_migration_inbound_addr, migration_inbound_addr set to IPs, nothing more related to migration in libvirt section | |
| 18:55:20 | s3rj1k | so I've tried to return both CONF.my_ip and CONF.libvirt.migration_inbound_addr, migration still tries to resolve by dns | |
| 19:35:10 | sean-k-mooney | s3rj1k: ok we are going to need ot look at exaclty what we are passing to the migrate too uri libvirt call | |
| 19:36:56 | s3rj1k | sean-k-mooney: do you have some specific debug changes in mind? | |
| 19:40:31 | sean-k-mooney | baiscally debugging this call https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10854-L10859 | |
| 19:41:03 | sean-k-mooney | we can do that by putting some debug logs or prints in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L591 | |
| 19:42:06 | sean-k-mooney | s3rj1k: in your config we would expect both destination and migrate_uri to have the ip | |
| 19:42:15 | sean-k-mooney | but i suspect one or both of them have the hostname | |
| 19:42:58 | sean-k-mooney | s3rj1k: the excption form libvirt is comming form this funciton call https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L647 | |
| 19:51:56 | s3rj1k | hmm, destination has hostname as expected I guess, but migrate_uri is None for some reason | |
| 19:54:26 | s3rj1k | smth wrong in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10781-L10786 ? | |
| 19:55:06 | s3rj1k | "non tunneled migration" | |
| 20:09:18 | sean-k-mooney | we do not want to use tunneled migration so i think that is fine | |
| 20:09:34 | sean-k-mooney | so the quetion is where did destinaiton get set ot the host name | |
| 20:10:04 | s3rj1k | migrate_uri is None, this is expected? | |
| 20:10:30 | sean-k-mooney | i dont recall of the top of my head but we do not adivse using tunneld migration | |
| 20:10:52 | sean-k-mooney | so that being none is proably ok | |
| 20:11:01 | sean-k-mooney | dist i think is coming form (self._live_migration_uri(dest) | |
| 20:11:40 | sean-k-mooney | ok that just ipv6 escapign the dest that past in | |
| 20:14:23 | sean-k-mooney | so thats passed in to live_migrate in the driver and that invoked over rpc via the compute manager | |
| 20:16:43 | s3rj1k | here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10733 it is already a hostname | |
| 20:17:02 | s3rj1k | (dest var) | |
| 20:17:11 | sean-k-mooney | yep | |
| 20:17:43 | sean-k-mooney | so in the conduction its a destation object i think it being converted to the host name in the compute manager but im trying to find that | |
| 20:19:42 | sean-k-mooney | or not https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9037 | |
| 20:19:56 | sean-k-mooney | that looks like its alrady going to be the host name on the comptue manager side of the rpc | |
| 20:21:43 | sean-k-mooney | s3rj1k: so https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L509-L562 is returning the host form the select_destinations call to the schduler | |
| 20:22:08 | sean-k-mooney | selection.service_host | |
| 20:22:48 | sean-k-mooney | is what self.destination has here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L100 | |
| 20:23:30 | sean-k-mooney | which we pass here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L157 | |
| 20:24:59 | s3rj1k | I assume that we want this hotfixed near live_migration_inbound_addr handling? | |
| 20:26:12 | sean-k-mooney | so the porblem is the schduelr does not realy knwo about the ips of the comptue nodes so it just sleect the dstionation compute service to hanel the request | |
| 20:27:06 | sean-k-mooney | which means that we need to transport the information form the dest comptue node back to the source eiver via the database or rpc | |
| 20:27:25 | sean-k-mooney | to then pass it to libvirt on the souce node | |
| 20:28:01 | sean-k-mooney | in other words the source compute node need the value of live_migration_inbound_addr form the destioantion host when it calls libvirt | |
| 20:28:46 | s3rj1k | ahh, that makes sense, and makes fix more complicated :( | |
| 20:30:26 | sean-k-mooney | we shoudl proably do that by adding it to the libvirtLiveMigrationData object https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L229 | |
| 20:30:34 | sean-k-mooney | in pre live migrate | |
| 20:31:12 | sean-k-mooney | so we need to add a new field "destiantion_address" | |
| 20:31:39 | s3rj1k | target_connect_addr seems familiar | |
| 20:31:59 | s3rj1k | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10781-L10786 | |
| 20:32:04 | sean-k-mooney | it deso but im not sure if that is related to the storage network | |
| 20:32:56 | sean-k-mooney | hum | |
| 20:33:14 | sean-k-mooney | can you pritn some of that locally and see what branch we are taking | |
| 20:33:39 | s3rj1k | yea, few min | |
| 20:33:43 | sean-k-mooney | if im readign that proprly then if target_connect_addr was set by the dest node to the ip then it would use it | |
| 20:34:04 | sean-k-mooney | assuming tunneled migraiton is off which it is by default | |
| 20:35:54 | sean-k-mooney | hum it should be uncondtionally set here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11503-L11505 | |
| 20:39:29 | s3rj1k | target_connect_addr in migrate_data but value is None and (migration_flags & libvirt.VIR_MIGRATE_TUNNELLED == 0) | |
| 20:40:52 | sean-k-mooney | thats odd because its clearly bing set unconditionally on the dest so either we are loosign the value or there is a bug in how we are setting it form the config | |
| 20:47:53 | s3rj1k | sean-k-mooney: ok, so just to verify, live_migration_inbound_addr is under libvirt in /etc/nova/nova.conf for devstack right? | |
| 20:48:42 | sean-k-mooney | good question i think so | |
| 20:49:11 | sean-k-mooney | yep https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_inbound_addr | |
| 20:49:19 | s3rj1k | because I've just added log.debug in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11503-L11505 on target node and I see that value is None | |
| 20:50:17 | sean-k-mooney | when the comptue start up in debug mode it dumps the config to the log | |
| 20:50:26 | sean-k-mooney | is it set to the correct value there? | |