| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-09-12 | |||
| 07:58:19 | bauzas | but we check two different fields for shared storage | |
| 07:58:34 | bauzas | the conditional checks whether both computes use the same path | |
| 07:58:36 | gibi | yes I spotted that but did not dig deeper tehre | |
| 07:59:05 | bauzas | while the destroy_disks arg checks whether we use block storage | |
| 07:59:38 | bauzas | I assume the reporter gets something like True for the shared path but False for shared block storage | |
| 08:01:36 | gibi | what is a shared block storage? | |
| 08:01:41 | bauzas | for mdevs and vpmems, there was a reason to call cleanup | |
| 08:01:45 | gibi | just cinder? or something more special? | |
| 08:01:55 | bauzas | we need to delete the local residues | |
| 08:02:24 | bauzas | honestly, that's the first time I'm hearing this | |
| 08:02:45 | bauzas | I'm used to the regular FS share between instance pathes | |
| 08:03:01 | bauzas | that's the regular shared storage I know | |
| 08:03:28 | gibi | yeah I think is_shared_instance_path is the case when the instance dir is on e.g. NFS | |
| 08:03:41 | bauzas | yup | |
| 08:04:24 | gibi | we need to call cleanup if power mgmt is enabled so the current fix proposed would regress that | |
| 08:04:38 | bauzas | I need to see what makes migrate_data.is_shared_block_storage | |
| 08:04:54 | gibi | yep | |
| 08:05:02 | bauzas | and it would also regress the mdev live-migration support | |
| 08:05:14 | bauzas | we need to call cleanup to delete the mdevs we reserved | |
| 08:05:32 | bauzas | hence the 'or' | |
| 08:06:00 | bauzas | I need a coffee | |
| 08:06:14 | bauzas | it's early and I'm already fried | |
| 08:06:45 | bauzas | trying to improve my local LTE connection took me too much of time | |
| 08:08:39 | gibi | https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/virt/libvirt/driver.py#L10367-L10369 | |
| 08:09:35 | bauzas | ooooh I see | |
| 08:09:42 | gibi | https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/virt/libvirt/driver.py#L10470 | |
| 08:10:25 | bauzas | yeah, some this is when you have BFV or you share the same libvirt image backend between both | |
| 08:10:33 | bauzas | that's not NFS | |
| 08:11:13 | gibi | yeah it is the RBD image backend | |
| 08:11:20 | bauzas | so | |
| 08:11:27 | bauzas | I think of something quick | |
| 08:11:50 | bauzas | previously, we were NOT calling cleanup on shared storage | |
| 08:12:02 | bauzas | right? | |
| 08:12:25 | bauzas | now we WANT to call it too, but we don't want to destroy the disks | |
| 08:12:35 | bauzas | I see a simple solution | |
| 08:13:06 | bauzas | which is to add another clause to destroy_disks | |
| 08:13:25 | bauzas | gibi: thoughts ? | |
| 08:14:09 | gibi | thinking... | |
| 08:14:49 | gibi | we did not call cleanup if is_shared_instance_path was true. And now we need it due to power mgmt. | |
| 08:14:58 | bauzas | yup | |
| 08:15:06 | bauzas | but we need to call it smartly | |
| 08:15:09 | gibi | yes | |
| 08:16:27 | gibi | would the change destroy_disks = not (migrate_data.is_shared_block_storage or migrate_data.is_shared_instance_path) work? | |
| 08:16:42 | bauzas | that's my hope | |
| 08:16:54 | bauzas | but we need to check the cleanup method | |
| 08:20:24 | gibi | https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/virt/libvirt/driver.py#L1707-L1724 this is the relevant path. I think it would work | |
| 08:20:33 | gibi | but we don't have reproduction | |
| 08:21:35 | bauzas | yeah | |
| 08:21:42 | bauzas | we need a functional test | |
| 08:22:14 | gibi | In 20 mins I need to step away for more than an hour | |
| 08:22:23 | bauzas | I can try to work on the reproducer | |
| 08:22:30 | gibi | I will summarize the above discussion in the bug | |
| 08:22:37 | bauzas | I already did in the report | |
| 08:23:13 | gibi | cool | |
| 08:23:37 | gibi | I will add a request to the reporter to try,if they can, our fix proposal | |
| 08:23:59 | bauzas | please, but I assume he's on West-US timezones | |
| 08:36:12 | gibi | done | |
| 10:28:25 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add Dalmatian prelude section https://review.opendev.org/c/openstack/nova/+/928995 | |
| 10:28:34 | opendevreview | Merged openstack/osc-placement master: Update master for stable/2024.2 https://review.opendev.org/c/openstack/osc-placement/+/928360 | |
| 11:32:56 | yusufgungor_ | hi everyone, we think hit a strange bug. We got the mtu error as below on the source compute node, when live migrating an instance with the specified scenarios. | |
| 11:32:56 | yusufgungor_ | "Target network card MTU 0 does not match source 1450: libvirt.libvirtError: unsupported configuration: Target network card MTU 0 does not match source 1450" | |
| 11:32:56 | yusufgungor_ | Also debug logs shows that mtu is null for one of the interfaces for the failed scenarios. _update_vif_xml function prints log as below on debug: | |
| 11:32:58 | yusufgungor_ | Updating guest XML with vif config: <interface type="ethernet"> | |
| 11:33:00 | yusufgungor_ | <mac address="fa:16:3e:19:a8:ba"/> | |
| 11:33:00 | yusufgungor_ | <model type="virtio"/> | |
| 11:33:02 | yusufgungor_ | <target dev="tapfb1b3816-ed"/> | |
| 11:33:02 | yusufgungor_ | </interface> | |
| 11:33:04 | yusufgungor_ | Updating guest XML with vif config: <interface type="ethernet"> | |
| 11:33:06 | yusufgungor_ | <mac address="fa:16:3e:8e:e7:b3"/> | |
| 11:33:06 | yusufgungor_ | <model type="virtio"/> | |
| 11:33:08 | yusufgungor_ | <mtu size="1500"/> | |
| 11:33:08 | yusufgungor_ | <target dev="tapd001bd30-9c"/> | |
| 11:33:10 | yusufgungor_ | </interface> | |
| 11:33:12 | yusufgungor_ | Should we create a bug report? This mtu error causes revert the migration and triggers another strange bug on the os-brick side which unmaps the incorrect iscsi device for random instance :/ It is another story. | |
| 11:33:14 | yusufgungor_ | Scenarios: | |
| 11:33:14 | yusufgungor_ | 1. When creating an instance from a vxlan tenant network, then attaching an interface from the provider network, and live migrating, it gives an mtu error! | |
| 11:33:16 | yusufgungor_ | 2. When creating an instance from a vxlan tenant network, then attaching an interface from the provider network, rebooting, and live migrating, it does not give an mtu error. | |
| 11:33:16 | yusufgungor_ | 3. When creating an instance only from the provider network, it does not give an mtu error. | |
| 11:33:18 | yusufgungor_ | 4. When creating an instance from the provider network, then attaching an interface from the vxlan tenant network, and migrating, it gives an mtu error! | |
| 11:33:18 | yusufgungor_ | 5. When creating an instance, giving an IP from both the vxlan tenant network and the provider network, and live migrating, it does not give an mtu error. | |
| 11:33:20 | yusufgungor_ | 6. When creating an instance only from the provider network, it does not give an mtu error. Then, when attaching a leg from the vxlan tenant network and migrating again, it gives an mtu error! | |
| 11:36:16 | yusufgungor_ | nova-compute --version : 26.2.1 (osp cluster is zed version) | |
| 12:04:49 | sean-k-mooney | yusufgungor_: so it shoudl be noted that nova does not supprot multi segment networks with differnt mtu | |
| 12:05:46 | sean-k-mooney | yusufgungor_: vxlan has 48? byte of encapsulation overhead so we round that up to 50 and reduce its mtu vs vlan/flat in neutorn automatically | |
| 12:07:45 | sean-k-mooney | yusufgungor_: you should file a bug with as much detail as is relevent | |
| 12:08:05 | yusufgungor_ | @sean thanks for reply. so a nova instance with both vxlan tenant network (mtu: 1450) and provider network (mtu: 1500) is not supported? Actually it works except live migration on spesific case. For example after hard reboot the instance we can live migrate too. | |
| 12:08:32 | yusufgungor_ | @sean-k-mooney thanks, then i will file a bug report to nova | |
| 12:09:42 | sean-k-mooney | no you can have instance with a mix of both | |
| 12:09:45 | sean-k-mooney | that should be fine | |
| 12:10:04 | sean-k-mooney | just not a neutron network that has a vlan segment and a vxlan segment with differnt mtus | |
| 12:10:57 | sean-k-mooney | that actully might work form a nova side as we take the mtu form the port | |
| 12:11:21 | sean-k-mooney | but it would break l2 connectivty between the segments as mtus should only chagne on l3(subnet) bondaries | |
| 12:12:19 | yusufgungor_ | @sean-k-mooney thanks. actually we have tried to set the provider network mtu as 1450 too and tested but hitting the same live migrate error | |
| 12:12:40 | yusufgungor_ | @sean-k-mooney i will test it again, and file a bug | |
| 12:12:56 | sean-k-mooney | oh this might be an old bug then | |
| 12:13:20 | sean-k-mooney | so we do not suprot changing the mtu of neutron netowrks | |
| 12:13:29 | sean-k-mooney | and by we i mean nova | |
| 12:14:12 | sean-k-mooney | the only way to do that properly is if you detach and reattach all the ports or otherwise regenerate the guest xml | |
| 12:15:21 | yusufgungor_ | @sean-k-mooney thanks 🙏 | |
| 12:55:41 | ratailor_ | sean-k-mooney, could you please provide your suggestion on this approach https://review.opendev.org/c/openstack/nova/+/928933 for https://bugs.launchpad.net/nova/+bug/2058928 | |