| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-13 | |||
| 19:42:15 | sean-k-mooney | but i suspect one or both of them have the hostname | |
| 19:42:58 | sean-k-mooney | s3rj1k: the excption form libvirt is comming form this funciton call https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L647 | |
| 19:51:56 | s3rj1k | hmm, destination has hostname as expected I guess, but migrate_uri is None for some reason | |
| 19:54:26 | s3rj1k | smth wrong in here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10781-L10786 ? | |
| 19:55:06 | s3rj1k | "non tunneled migration" | |
| 20:09:18 | sean-k-mooney | we do not want to use tunneled migration so i think that is fine | |
| 20:09:34 | sean-k-mooney | so the quetion is where did destinaiton get set ot the host name | |
| 20:10:04 | s3rj1k | migrate_uri is None, this is expected? | |
| 20:10:30 | sean-k-mooney | i dont recall of the top of my head but we do not adivse using tunneld migration | |
| 20:10:52 | sean-k-mooney | so that being none is proably ok | |
| 20:11:01 | sean-k-mooney | dist i think is coming form (self._live_migration_uri(dest) | |
| 20:11:40 | sean-k-mooney | ok that just ipv6 escapign the dest that past in | |
| 20:14:23 | sean-k-mooney | so thats passed in to live_migrate in the driver and that invoked over rpc via the compute manager | |
| 20:16:43 | s3rj1k | here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10733 it is already a hostname | |
| 20:17:02 | s3rj1k | (dest var) | |
| 20:17:11 | sean-k-mooney | yep | |
| 20:17:43 | sean-k-mooney | so in the conduction its a destation object i think it being converted to the host name in the compute manager but im trying to find that | |
| 20:19:42 | sean-k-mooney | or not https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9037 | |
| 20:19:56 | sean-k-mooney | that looks like its alrady going to be the host name on the comptue manager side of the rpc | |
| 20:21:43 | sean-k-mooney | s3rj1k: so https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L509-L562 is returning the host form the select_destinations call to the schduler | |
| 20:22:08 | sean-k-mooney | selection.service_host | |
| 20:22:48 | sean-k-mooney | is what self.destination has here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L100 | |
| 20:23:30 | sean-k-mooney | which we pass here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L157 | |
| 20:24:59 | s3rj1k | I assume that we want this hotfixed near live_migration_inbound_addr handling? | |
| 20:26:12 | sean-k-mooney | so the porblem is the schduelr does not realy knwo about the ips of the comptue nodes so it just sleect the dstionation compute service to hanel the request | |
| 20:27:06 | sean-k-mooney | which means that we need to transport the information form the dest comptue node back to the source eiver via the database or rpc | |
| 20:27:25 | sean-k-mooney | to then pass it to libvirt on the souce node | |
| 20:28:01 | sean-k-mooney | in other words the source compute node need the value of live_migration_inbound_addr form the destioantion host when it calls libvirt | |
| 20:28:46 | s3rj1k | ahh, that makes sense, and makes fix more complicated :( | |
| 20:30:26 | sean-k-mooney | we shoudl proably do that by adding it to the libvirtLiveMigrationData object https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L229 | |
| 20:30:34 | sean-k-mooney | in pre live migrate | |
| 20:31:12 | sean-k-mooney | so we need to add a new field "destiantion_address" | |
| 20:31:39 | s3rj1k | target_connect_addr seems familiar | |
| 20:31:59 | s3rj1k | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10781-L10786 | |
| 20:32:04 | sean-k-mooney | it deso but im not sure if that is related to the storage network | |
| 20:32:56 | sean-k-mooney | hum | |
| 20:33:14 | sean-k-mooney | can you pritn some of that locally and see what branch we are taking | |
| 20:33:39 | s3rj1k | yea, few min | |
| 20:33:43 | sean-k-mooney | if im readign that proprly then if target_connect_addr was set by the dest node to the ip then it would use it | |
| 20:34:04 | sean-k-mooney | assuming tunneled migraiton is off which it is by default | |
| 20:35:54 | sean-k-mooney | hum it should be uncondtionally set here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11503-L11505 | |
| 20:39:29 | s3rj1k | target_connect_addr in migrate_data but value is None and (migration_flags & libvirt.VIR_MIGRATE_TUNNELLED == 0) | |
| 20:40:52 | sean-k-mooney | thats odd because its clearly bing set unconditionally on the dest so either we are loosign the value or there is a bug in how we are setting it form the config | |
| 20:47:53 | s3rj1k | sean-k-mooney: ok, so just to verify, live_migration_inbound_addr is under libvirt in /etc/nova/nova.conf for devstack right? | |
| 20:48:42 | sean-k-mooney | good question i think so | |
| 20:49:11 | sean-k-mooney | yep https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_inbound_addr | |
| 20:49:19 | s3rj1k | because I've just added log.debug in https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L11503-L11505 on target node and I see that value is None | |
| 20:50:17 | sean-k-mooney | when the comptue start up in debug mode it dumps the config to the log | |
| 20:50:26 | sean-k-mooney | is it set to the correct value there? | |
| 20:50:50 | opendevreview | ribaudr proposed openstack/nova master: Amend ShareMappingStatus due to asynchronous call https://review.opendev.org/c/openstack/nova/+/908864 | |
| 20:50:51 | opendevreview | ribaudr proposed openstack/nova master: Amend DB model add a unique constraint. https://review.opendev.org/c/openstack/nova/+/912518 | |
| 20:50:51 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |
| 20:50:52 | opendevreview | ribaudr proposed openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | |
| 20:50:52 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 20:50:53 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 20:50:53 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 20:50:54 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 20:50:54 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 20:50:56 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 20:50:56 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 20:50:58 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 20:50:58 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 20:51:00 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | |
| 20:51:00 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 20:51:02 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 20:51:02 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 20:51:04 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 20:51:04 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 20:51:06 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 20:51:06 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 20:51:08 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 20:51:08 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 20:51:10 | opendevreview | ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088 | |
| 20:51:10 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 20:51:12 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 20:51:12 | opendevreview | ribaudr proposed openstack/nova master: Manila shares admin guide documentation https://review.opendev.org/c/openstack/nova/+/871642 | |
| 20:51:54 | sean-k-mooney | s3rj1k: its defiend as a HostDomainOpt type shich shoudl accpaate any ip, hostanem or fqdn | |
| 20:51:56 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/conf/libvirt.py#L239-L254 | |
| 20:52:40 | sean-k-mooney | https://github.com/openstack/oslo.config/blob/82552116f827030ce1e493506fe83ef71656e447/oslo_config/cfg.py#L1189 | |
| 20:54:34 | s3rj1k | sean-k-mooney: yea, I see value in logs DEBUG nova.api.openstack.wsgi_app [-] libvirt.live_migration_inbound_addr = 10.42.71.101 {{(pid=205504)... | |
| 20:55:13 | sean-k-mooney | wait where are you definign the values ? | |
| 20:55:25 | sean-k-mooney | your putting them on the compute nodes nova.conf right | |
| 20:55:40 | sean-k-mooney | for devstack this is called /etc/nova/nova-cpu.conf | |
| 20:56:09 | s3rj1k | was putting it in nova.conf :( | |
| 20:56:18 | s3rj1k | lemme check again | |
| 20:58:39 | s3rj1k | ::facepalm | |
| 20:59:01 | s3rj1k | well, in works | |
| 20:59:14 | sean-k-mooney | with the revert rightt | |
| 20:59:27 | s3rj1k | sean-k-mooney: sorry for that, yea, with revert | |
| 20:59:39 | sean-k-mooney | so it woudl b einteresting if it fails if you drop the revert | |
| 20:59:52 | sean-k-mooney | cause if so then that is the probelm and the fix | |
| 21:00:24 | s3rj1k | hmm, lemme check | |
| 21:01:14 | sean-k-mooney | no rush can you update the bug wiht your finding either way | |
| 21:03:31 | sean-k-mooney | the revert is good to do anyway but i think im going to finish for today | |
| 21:06:31 | s3rj1k | sean-k-mooney: yep, fails without revert | |
| 21:06:52 | s3rj1k | i'll update bugreport | |
| 21:10:53 | s3rj1k | done https://bugs.launchpad.net/nova/+bug/2086867/comments/5 | |
| 21:11:03 | s3rj1k | thanks Sean! | |
| 21:12:50 | sean-k-mooney | cool then lets proceed with the revert we can follow up with others to review tomrorow | |
| 21:13:21 | s3rj1k | I'll also will revalidate that after merge on clean DevStack, just in case :) | |