Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-09-12
08:14:09 gibi thinking...
08:14:49 gibi we did not call cleanup if is_shared_instance_path was true. And now we need it due to power mgmt.
08:14:58 bauzas yup
08:15:06 bauzas but we need to call it smartly
08:15:09 gibi yes
08:16:27 gibi would the change destroy_disks = not (migrate_data.is_shared_block_storage or migrate_data.is_shared_instance_path) work?
08:16:42 bauzas that's my hope
08:16:54 bauzas but we need to check the cleanup method
08:20:24 gibi https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/virt/libvirt/driver.py#L1707-L1724 this is the relevant path. I think it would work
08:20:33 gibi but we don't have reproduction
08:21:35 bauzas yeah
08:21:42 bauzas we need a functional test
08:22:14 gibi In 20 mins I need to step away for more than an hour
08:22:23 bauzas I can try to work on the reproducer
08:22:30 gibi I will summarize the above discussion in the bug
08:22:37 bauzas I already did in the report
08:23:13 gibi cool
08:23:37 gibi I will add a request to the reporter to try,if they can, our fix proposal
08:23:59 bauzas please, but I assume he's on West-US timezones
08:36:12 gibi done
10:28:25 opendevreview Sylvain Bauza proposed openstack/nova master: Add Dalmatian prelude section https://review.opendev.org/c/openstack/nova/+/928995
10:28:34 opendevreview Merged openstack/osc-placement master: Update master for stable/2024.2 https://review.opendev.org/c/openstack/osc-placement/+/928360
11:32:56 yusufgungor_ Also debug logs shows that mtu is null for one of the interfaces for the failed scenarios. _update_vif_xml function prints log as below on debug:
11:32:56 yusufgungor_ "Target network card MTU 0 does not match source 1450: libvirt.libvirtError: unsupported configuration: Target network card MTU 0 does not match source 1450"
11:32:56 yusufgungor_ hi everyone, we think hit a strange bug. We got the mtu error as below on the source compute node, when live migrating an instance with the specified scenarios.
11:32:58 yusufgungor_ Updating guest XML with vif config: <interface type="ethernet">
11:33:00 yusufgungor_ <model type="virtio"/>
11:33:00 yusufgungor_ <mac address="fa:16:3e:19:a8:ba"/>
11:33:02 yusufgungor_ </interface>
11:33:02 yusufgungor_ <target dev="tapfb1b3816-ed"/>
11:33:04 yusufgungor_ Updating guest XML with vif config: <interface type="ethernet">
11:33:06 yusufgungor_ <model type="virtio"/>
11:33:06 yusufgungor_ <mac address="fa:16:3e:8e:e7:b3"/>
11:33:08 yusufgungor_ <target dev="tapd001bd30-9c"/>
11:33:08 yusufgungor_ <mtu size="1500"/>
11:33:10 yusufgungor_ </interface>
11:33:12 yusufgungor_ Should we create a bug report? This mtu error causes revert the migration and triggers another strange bug on the os-brick side which unmaps the incorrect iscsi device for random instance :/ It is another story.
11:33:14 yusufgungor_ 1. When creating an instance from a vxlan tenant network, then attaching an interface from the provider network, and live migrating, it gives an mtu error!
11:33:14 yusufgungor_ Scenarios:
11:33:16 yusufgungor_ 3. When creating an instance only from the provider network, it does not give an mtu error.
11:33:16 yusufgungor_ 2. When creating an instance from a vxlan tenant network, then attaching an interface from the provider network, rebooting, and live migrating, it does not give an mtu error.
11:33:18 yusufgungor_ 5. When creating an instance, giving an IP from both the vxlan tenant network and the provider network, and live migrating, it does not give an mtu error.
11:33:18 yusufgungor_ 4. When creating an instance from the provider network, then attaching an interface from the vxlan tenant network, and migrating, it gives an mtu error!
11:33:20 yusufgungor_ 6. When creating an instance only from the provider network, it does not give an mtu error. Then, when attaching a leg from the vxlan tenant network and migrating again, it gives an mtu error!
11:36:16 yusufgungor_ nova-compute --version : 26.2.1 (osp cluster is zed version)
12:04:49 sean-k-mooney yusufgungor_: so it shoudl be noted that nova does not supprot multi segment networks with differnt mtu
12:05:46 sean-k-mooney yusufgungor_: vxlan has 48? byte of encapsulation overhead so we round that up to 50 and reduce its mtu vs vlan/flat in neutorn automatically
12:07:45 sean-k-mooney yusufgungor_: you should file a bug with as much detail as is relevent
12:08:05 yusufgungor_ @sean thanks for reply. so a nova instance with both vxlan tenant network (mtu: 1450) and provider network (mtu: 1500) is not supported? Actually it works except live migration on spesific case. For example after hard reboot the instance we can live migrate too.
12:08:32 yusufgungor_ @sean-k-mooney thanks, then i will file a bug report to nova
12:09:42 sean-k-mooney no you can have instance with a mix of both
12:09:45 sean-k-mooney that should be fine
12:10:04 sean-k-mooney just not a neutron network that has a vlan segment and a vxlan segment with differnt mtus
12:10:57 sean-k-mooney that actully might work form a nova side as we take the mtu form the port
12:11:21 sean-k-mooney but it would break l2 connectivty between the segments as mtus should only chagne on l3(subnet) bondaries
12:12:19 yusufgungor_ @sean-k-mooney thanks. actually we have tried to set the provider network mtu as 1450 too and tested but hitting the same live migrate error
12:12:40 yusufgungor_ @sean-k-mooney i will test it again, and file a bug
12:12:56 sean-k-mooney oh this might be an old bug then
12:13:20 sean-k-mooney so we do not suprot changing the mtu of neutron netowrks
12:13:29 sean-k-mooney and by we i mean nova
12:14:12 sean-k-mooney the only way to do that properly is if you detach and reattach all the ports or otherwise regenerate the guest xml
12:15:21 yusufgungor_ @sean-k-mooney thanks 🙏
12:55:41 ratailor_ sean-k-mooney, could you please provide your suggestion on this approach https://review.opendev.org/c/openstack/nova/+/928933 for https://bugs.launchpad.net/nova/+bug/2058928
12:56:44 ratailor_ sean-k-mooney, the finish_time field is not shown in instance action show output, that's why we need to do this before calling instance action finish method for respective APIs where we need to finish the started action.
12:57:20 ratailor_ sean-k-mooney, I will fix failing tests and add new tests wherever required, once this approach looks good.
13:00:17 ratailor_ artom, gibi please provide your suggestions as well ^^
13:01:59 opendevreview sean mooney proposed openstack/nova master: repoduce post liberty pre vicoria instance numa db issue https://review.opendev.org/c/openstack/nova/+/929025
13:02:13 sean-k-mooney dansmith: gibi bauzas ^
13:04:19 sean-k-mooney ill file a bug for that and update it when i push the fix patch
13:31:08 yusufgungor_ @sean-k-mooney i have filed a bug: https://bugs.launchpad.net/nova/+bug/2080531
14:02:31 opendevreview Dan Smith proposed openstack/nova stable/2023.1: [stable-only]: Port qcow2v2 support from oslo https://review.opendev.org/c/openstack/nova/+/928937
14:08:33 opendevreview Rajesh Tailor proposed openstack/nova master: Add support for showing finish_time https://review.opendev.org/c/openstack/nova/+/928933
16:29:59 dansmith sean-k-mooney: are you respinning the safety patch to invert the safe/unsafe logic or no?
16:30:07 dansmith your comment made it sound like yes, but...
16:31:05 sean-k-mooney no
16:31:10 sean-k-mooney i inverted the logic already
16:31:19 sean-k-mooney so it should be good as is
16:31:27 dansmith okay
16:31:35 sean-k-mooney unsafe would have been less work but i was alrey 30 mins into fixing the tests
16:31:46 sean-k-mooney so i decied to just finish it
16:32:04 dansmith yeah I just wasn't sure from your comment if you were going to respin is all
16:32:11 bauzas hum, looks like we don't have any existing functional tests checking shared storage
16:32:18 sean-k-mooney ya i proably should have dropped the comment
16:32:56 sean-k-mooney bauzas: we may under the libvirt section
16:33:18 sean-k-mooney but in generally its light if it exists
16:33:36 bauzas my grep on shared_storage or just storage shows me nothing
16:34:17 bauzas I assume I may have to mock this
16:34:46 bauzas iirc, we just detect shared storage if we can see the same instance files on both computes, right?
16:36:02 sean-k-mooney yep you can mock that function
16:36:16 sean-k-mooney i would just create a regression test in a spereate file
16:36:23 sean-k-mooney and use https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/base.py#L264 to make live migration work
16:36:39 sean-k-mooney then stub the function for the shared storage check to return true
16:36:57 sean-k-mooney https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/tests/functional/regressions/test_bug_1939545.py
16:36:59 bauzas we have a couple of functional tests for live-migration, that's not a problem
16:37:05 sean-k-mooney i think has all the ground work
16:37:16 bauzas oh cool, thanks for the digging but I was about to do it :)
17:20:08 bauzas okay, I'm out of steam and I don't have ideas on how to correctly assert we delete the local disks, except checking we call the delete
17:20:17 bauzas see you tomorrow
20:09:17 opendevreview sean mooney proposed openstack/nova master: repoduce post liberty pre vicoria instance numa db issue https://review.opendev.org/c/openstack/nova/+/929025
20:09:18 opendevreview sean mooney proposed openstack/nova master: allow upgrade of pre-victoria InstanceNUMACells https://review.opendev.org/c/openstack/nova/+/929187

Earlier   Later