| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-13 | |||
| 18:40:14 | sean-k-mooney | but if you set the livemigration config var to the ip we should not be using get_hostname there | |
| 18:40:38 | sean-k-mooney | well hum | |
| 18:40:53 | s3rj1k | fix would be moving line 4520 below if I guess | |
| 18:40:53 | sean-k-mooney | i think that depned on where get_host_ip_addr is being called form at the time | |
| 18:41:38 | s3rj1k | with some check if conf is set | |
| 18:42:19 | sean-k-mooney | something along those lines maybe | |
| 18:42:52 | sean-k-mooney | could you try regerting get_host_ip_addr to just return CONF.my_ip | |
| 18:42:58 | sean-k-mooney | and see if that fixes it for you locally | |
| 18:43:19 | sean-k-mooney | it will break the other fature so that not the correct fix but it will confirm if that is the souce of the regression | |
| 18:43:26 | s3rj1k | yea, give me few minutes | |
| 18:45:09 | sean-k-mooney | although you do not have "%s" in CONF.libvirt.migration_inbound_addr do you? | |
| 18:45:27 | sean-k-mooney | you do in the migration uri but that is just an ip adress reight | |
| 18:45:30 | sean-k-mooney | *right | |
| 18:54:30 | s3rj1k | I have live_migration_uri = qemu+ssh://stack@%s/system and live_migration_inbound_addr, migration_inbound_addr set to IPs, nothing more related to migration in libvirt section | |
| 18:55:20 | s3rj1k | so I've tried to return both CONF.my_ip and CONF.libvirt.migration_inbound_addr, migration still tries to resolve by dns | |
| 19:35:10 | sean-k-mooney | s3rj1k: ok we are going to need ot look at exaclty what we are passing to the migrate too uri libvirt call | |
| 19:36:56 | s3rj1k | sean-k-mooney: do you have some specific debug changes in mind? | |
| 19:40:31 | sean-k-mooney | baiscally debugging this call https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10854-L10859 | |
| 19:41:03 | sean-k-mooney | we can do that by putting some debug logs or prints in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L591 | |
| 19:42:06 | sean-k-mooney | s3rj1k: in your config we would expect both destination and migrate_uri to have the ip | |
| 19:42:15 | sean-k-mooney | but i suspect one or both of them have the hostname | |
| 19:42:58 | sean-k-mooney | s3rj1k: the excption form libvirt is comming form this funciton call https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L647 | |
| 19:51:56 | s3rj1k | hmm, destination has hostname as expected I guess, but migrate_uri is None for some reason | |
| 19:54:26 | s3rj1k | smth wrong in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10781-L10786 ? | |
| 19:55:06 | s3rj1k | "non tunneled migration" | |
| 20:09:18 | sean-k-mooney | we do not want to use tunneled migration so i think that is fine | |
| 20:09:34 | sean-k-mooney | so the quetion is where did destinaiton get set ot the host name | |
| 20:10:04 | s3rj1k | migrate_uri is None, this is expected? | |
| 20:10:30 | sean-k-mooney | i dont recall of the top of my head but we do not adivse using tunneld migration | |
| 20:10:52 | sean-k-mooney | so that being none is proably ok | |
| 20:11:01 | sean-k-mooney | dist i think is coming form (self._live_migration_uri(dest) | |
| 20:11:40 | sean-k-mooney | ok that just ipv6 escapign the dest that past in | |
| 20:14:23 | sean-k-mooney | so thats passed in to live_migrate in the driver and that invoked over rpc via the compute manager | |
| 20:16:43 | s3rj1k | here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10733 it is already a hostname | |
| 20:17:02 | s3rj1k | (dest var) | |
| 20:17:11 | sean-k-mooney | yep | |
| 20:17:43 | sean-k-mooney | so in the conduction its a destation object i think it being converted to the host name in the compute manager but im trying to find that | |
| 20:19:42 | sean-k-mooney | or not https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9037 | |
| 20:19:56 | sean-k-mooney | that looks like its alrady going to be the host name on the comptue manager side of the rpc | |
| 20:21:43 | sean-k-mooney | s3rj1k: so https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L509-L562 is returning the host form the select_destinations call to the schduler | |
| 20:22:08 | sean-k-mooney | selection.service_host | |
| 20:22:48 | sean-k-mooney | is what self.destination has here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L100 | |
| 20:23:30 | sean-k-mooney | which we pass here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L157 | |
| 20:24:59 | s3rj1k | I assume that we want this hotfixed near live_migration_inbound_addr handling? | |
| 20:26:12 | sean-k-mooney | so the porblem is the schduelr does not realy knwo about the ips of the comptue nodes so it just sleect the dstionation compute service to hanel the request | |
| 20:27:06 | sean-k-mooney | which means that we need to transport the information form the dest comptue node back to the source eiver via the database or rpc | |
| 20:27:25 | sean-k-mooney | to then pass it to libvirt on the souce node | |
| 20:28:01 | sean-k-mooney | in other words the source compute node need the value of live_migration_inbound_addr form the destioantion host when it calls libvirt | |
| 20:28:46 | s3rj1k | ahh, that makes sense, and makes fix more complicated :( | |
| 20:30:26 | sean-k-mooney | we shoudl proably do that by adding it to the libvirtLiveMigrationData object https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L229 | |
| 20:30:34 | sean-k-mooney | in pre live migrate | |
| 20:31:12 | sean-k-mooney | so we need to add a new field "destiantion_address" | |
| 20:31:39 | s3rj1k | target_connect_addr seems familiar | |
| 20:31:59 | s3rj1k | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10781-L10786 | |
| 20:32:04 | sean-k-mooney | it deso but im not sure if that is related to the storage network | |
| 20:32:56 | sean-k-mooney | hum | |
| 20:33:14 | sean-k-mooney | can you pritn some of that locally and see what branch we are taking | |
| 20:33:39 | s3rj1k | yea, few min | |
| 20:33:43 | sean-k-mooney | if im readign that proprly then if target_connect_addr was set by the dest node to the ip then it would use it | |
| 20:34:04 | sean-k-mooney | assuming tunneled migraiton is off which it is by default | |
| 20:35:54 | sean-k-mooney | hum it should be uncondtionally set here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11503-L11505 | |
| 20:39:29 | s3rj1k | target_connect_addr in migrate_data but value is None and (migration_flags & libvirt.VIR_MIGRATE_TUNNELLED == 0) | |
| 20:40:52 | sean-k-mooney | thats odd because its clearly bing set unconditionally on the dest so either we are loosign the value or there is a bug in how we are setting it form the config | |
| 20:47:53 | s3rj1k | sean-k-mooney: ok, so just to verify, live_migration_inbound_addr is under libvirt in /etc/nova/nova.conf for devstack right? | |
| 20:48:42 | sean-k-mooney | good question i think so | |
| 20:49:11 | sean-k-mooney | yep https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_inbound_addr | |
| 20:49:19 | s3rj1k | because I've just added log.debug in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11503-L11505 on target node and I see that value is None | |
| 20:50:17 | sean-k-mooney | when the comptue start up in debug mode it dumps the config to the log | |
| 20:50:26 | sean-k-mooney | is it set to the correct value there? | |
| 20:50:50 | opendevreview | ribaudr proposed openstack/nova master: Amend ShareMappingStatus due to asynchronous call https://review.opendev.org/c/openstack/nova/+/908864 | |
| 20:50:51 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |
| 20:50:51 | opendevreview | ribaudr proposed openstack/nova master: Amend DB model add a unique constraint. https://review.opendev.org/c/openstack/nova/+/912518 | |
| 20:50:52 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 20:50:52 | opendevreview | ribaudr proposed openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | |
| 20:50:53 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 20:50:53 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 20:50:54 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 20:50:54 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 20:50:56 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 20:50:56 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 20:50:58 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 20:50:58 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 20:51:00 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 20:51:00 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | |
| 20:51:02 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 20:51:02 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 20:51:04 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 20:51:04 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 20:51:06 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 20:51:06 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 20:51:08 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 20:51:08 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 20:51:10 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 20:51:10 | opendevreview | ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088 | |
| 20:51:12 | opendevreview | ribaudr proposed openstack/nova master: Manila shares admin guide documentation https://review.opendev.org/c/openstack/nova/+/871642 | |
| 20:51:12 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 20:51:54 | sean-k-mooney | s3rj1k: its defiend as a HostDomainOpt type shich shoudl accpaate any ip, hostanem or fqdn | |
| 20:51:56 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/conf/libvirt.py#L239-L254 | |
| 20:52:40 | sean-k-mooney | https://github.com/openstack/oslo.config/blob/82552116f827030ce1e493506fe83ef71656e447/oslo_config/cfg.py#L1189 | |
| 20:54:34 | s3rj1k | sean-k-mooney: yea, I see value in logs DEBUG nova.api.openstack.wsgi_app [-] libvirt.live_migration_inbound_addr = 10.42.71.101 {{(pid=205504)... | |