| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-09-12 | |||
| 08:06:00 | bauzas | I need a coffee | |
| 08:06:14 | bauzas | it's early and I'm already fried | |
| 08:06:45 | bauzas | trying to improve my local LTE connection took me too much of time | |
| 08:08:39 | gibi | https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/virt/libvirt/driver.py#L10367-L10369 | |
| 08:09:35 | bauzas | ooooh I see | |
| 08:09:42 | gibi | https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/virt/libvirt/driver.py#L10470 | |
| 08:10:25 | bauzas | yeah, some this is when you have BFV or you share the same libvirt image backend between both | |
| 08:10:33 | bauzas | that's not NFS | |
| 08:11:13 | gibi | yeah it is the RBD image backend | |
| 08:11:20 | bauzas | so | |
| 08:11:27 | bauzas | I think of something quick | |
| 08:11:50 | bauzas | previously, we were NOT calling cleanup on shared storage | |
| 08:12:02 | bauzas | right? | |
| 08:12:25 | bauzas | now we WANT to call it too, but we don't want to destroy the disks | |
| 08:12:35 | bauzas | I see a simple solution | |
| 08:13:06 | bauzas | which is to add another clause to destroy_disks | |
| 08:13:25 | bauzas | gibi: thoughts ? | |
| 08:14:09 | gibi | thinking... | |
| 08:14:49 | gibi | we did not call cleanup if is_shared_instance_path was true. And now we need it due to power mgmt. | |
| 08:14:58 | bauzas | yup | |
| 08:15:06 | bauzas | but we need to call it smartly | |
| 08:15:09 | gibi | yes | |
| 08:16:27 | gibi | would the change destroy_disks = not (migrate_data.is_shared_block_storage or migrate_data.is_shared_instance_path) work? | |
| 08:16:42 | bauzas | that's my hope | |
| 08:16:54 | bauzas | but we need to check the cleanup method | |
| 08:20:24 | gibi | https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/virt/libvirt/driver.py#L1707-L1724 this is the relevant path. I think it would work | |
| 08:20:33 | gibi | but we don't have reproduction | |
| 08:21:35 | bauzas | yeah | |
| 08:21:42 | bauzas | we need a functional test | |
| 08:22:14 | gibi | In 20 mins I need to step away for more than an hour | |
| 08:22:23 | bauzas | I can try to work on the reproducer | |
| 08:22:30 | gibi | I will summarize the above discussion in the bug | |
| 08:22:37 | bauzas | I already did in the report | |
| 08:23:13 | gibi | cool | |
| 08:23:37 | gibi | I will add a request to the reporter to try,if they can, our fix proposal | |
| 08:23:59 | bauzas | please, but I assume he's on West-US timezones | |
| 08:36:12 | gibi | done | |
| 10:28:25 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add Dalmatian prelude section https://review.opendev.org/c/openstack/nova/+/928995 | |
| 10:28:34 | opendevreview | Merged openstack/osc-placement master: Update master for stable/2024.2 https://review.opendev.org/c/openstack/osc-placement/+/928360 | |
| 11:32:56 | yusufgungor_ | hi everyone, we think hit a strange bug. We got the mtu error as below on the source compute node, when live migrating an instance with the specified scenarios. | |
| 11:32:56 | yusufgungor_ | "Target network card MTU 0 does not match source 1450: libvirt.libvirtError: unsupported configuration: Target network card MTU 0 does not match source 1450" | |
| 11:32:56 | yusufgungor_ | Also debug logs shows that mtu is null for one of the interfaces for the failed scenarios. _update_vif_xml function prints log as below on debug: | |
| 11:32:58 | yusufgungor_ | Updating guest XML with vif config: <interface type="ethernet"> | |
| 11:33:00 | yusufgungor_ | <mac address="fa:16:3e:19:a8:ba"/> | |
| 11:33:00 | yusufgungor_ | <model type="virtio"/> | |
| 11:33:02 | yusufgungor_ | <target dev="tapfb1b3816-ed"/> | |
| 11:33:02 | yusufgungor_ | </interface> | |
| 11:33:04 | yusufgungor_ | Updating guest XML with vif config: <interface type="ethernet"> | |
| 11:33:06 | yusufgungor_ | <mac address="fa:16:3e:8e:e7:b3"/> | |
| 11:33:06 | yusufgungor_ | <model type="virtio"/> | |
| 11:33:08 | yusufgungor_ | <mtu size="1500"/> | |
| 11:33:08 | yusufgungor_ | <target dev="tapd001bd30-9c"/> | |
| 11:33:10 | yusufgungor_ | </interface> | |
| 11:33:12 | yusufgungor_ | Should we create a bug report? This mtu error causes revert the migration and triggers another strange bug on the os-brick side which unmaps the incorrect iscsi device for random instance :/ It is another story. | |
| 11:33:14 | yusufgungor_ | Scenarios: | |
| 11:33:14 | yusufgungor_ | 1. When creating an instance from a vxlan tenant network, then attaching an interface from the provider network, and live migrating, it gives an mtu error! | |
| 11:33:16 | yusufgungor_ | 2. When creating an instance from a vxlan tenant network, then attaching an interface from the provider network, rebooting, and live migrating, it does not give an mtu error. | |
| 11:33:16 | yusufgungor_ | 3. When creating an instance only from the provider network, it does not give an mtu error. | |
| 11:33:18 | yusufgungor_ | 4. When creating an instance from the provider network, then attaching an interface from the vxlan tenant network, and migrating, it gives an mtu error! | |
| 11:33:18 | yusufgungor_ | 5. When creating an instance, giving an IP from both the vxlan tenant network and the provider network, and live migrating, it does not give an mtu error. | |
| 11:33:20 | yusufgungor_ | 6. When creating an instance only from the provider network, it does not give an mtu error. Then, when attaching a leg from the vxlan tenant network and migrating again, it gives an mtu error! | |
| 11:36:16 | yusufgungor_ | nova-compute --version : 26.2.1 (osp cluster is zed version) | |
| 12:04:49 | sean-k-mooney | yusufgungor_: so it shoudl be noted that nova does not supprot multi segment networks with differnt mtu | |
| 12:05:46 | sean-k-mooney | yusufgungor_: vxlan has 48? byte of encapsulation overhead so we round that up to 50 and reduce its mtu vs vlan/flat in neutorn automatically | |
| 12:07:45 | sean-k-mooney | yusufgungor_: you should file a bug with as much detail as is relevent | |
| 12:08:05 | yusufgungor_ | @sean thanks for reply. so a nova instance with both vxlan tenant network (mtu: 1450) and provider network (mtu: 1500) is not supported? Actually it works except live migration on spesific case. For example after hard reboot the instance we can live migrate too. | |
| 12:08:32 | yusufgungor_ | @sean-k-mooney thanks, then i will file a bug report to nova | |
| 12:09:42 | sean-k-mooney | no you can have instance with a mix of both | |
| 12:09:45 | sean-k-mooney | that should be fine | |
| 12:10:04 | sean-k-mooney | just not a neutron network that has a vlan segment and a vxlan segment with differnt mtus | |
| 12:10:57 | sean-k-mooney | that actully might work form a nova side as we take the mtu form the port | |
| 12:11:21 | sean-k-mooney | but it would break l2 connectivty between the segments as mtus should only chagne on l3(subnet) bondaries | |
| 12:12:19 | yusufgungor_ | @sean-k-mooney thanks. actually we have tried to set the provider network mtu as 1450 too and tested but hitting the same live migrate error | |
| 12:12:40 | yusufgungor_ | @sean-k-mooney i will test it again, and file a bug | |
| 12:12:56 | sean-k-mooney | oh this might be an old bug then | |
| 12:13:20 | sean-k-mooney | so we do not suprot changing the mtu of neutron netowrks | |
| 12:13:29 | sean-k-mooney | and by we i mean nova | |
| 12:14:12 | sean-k-mooney | the only way to do that properly is if you detach and reattach all the ports or otherwise regenerate the guest xml | |
| 12:15:21 | yusufgungor_ | @sean-k-mooney thanks 🙏 | |
| 12:55:41 | ratailor_ | sean-k-mooney, could you please provide your suggestion on this approach https://review.opendev.org/c/openstack/nova/+/928933 for https://bugs.launchpad.net/nova/+bug/2058928 | |
| 12:56:44 | ratailor_ | sean-k-mooney, the finish_time field is not shown in instance action show output, that's why we need to do this before calling instance action finish method for respective APIs where we need to finish the started action. | |
| 12:57:20 | ratailor_ | sean-k-mooney, I will fix failing tests and add new tests wherever required, once this approach looks good. | |
| 13:00:17 | ratailor_ | artom, gibi please provide your suggestions as well ^^ | |
| 13:01:59 | opendevreview | sean mooney proposed openstack/nova master: repoduce post liberty pre vicoria instance numa db issue https://review.opendev.org/c/openstack/nova/+/929025 | |
| 13:02:13 | sean-k-mooney | dansmith: gibi bauzas ^ | |
| 13:04:19 | sean-k-mooney | ill file a bug for that and update it when i push the fix patch | |
| 13:31:08 | yusufgungor_ | @sean-k-mooney i have filed a bug: https://bugs.launchpad.net/nova/+bug/2080531 | |
| 14:02:31 | opendevreview | Dan Smith proposed openstack/nova stable/2023.1: [stable-only]: Port qcow2v2 support from oslo https://review.opendev.org/c/openstack/nova/+/928937 | |
| 14:08:33 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing finish_time https://review.opendev.org/c/openstack/nova/+/928933 | |
| 16:29:59 | dansmith | sean-k-mooney: are you respinning the safety patch to invert the safe/unsafe logic or no? | |
| 16:30:07 | dansmith | your comment made it sound like yes, but... | |
| 16:31:05 | sean-k-mooney | no | |
| 16:31:10 | sean-k-mooney | i inverted the logic already | |
| 16:31:19 | sean-k-mooney | so it should be good as is | |
| 16:31:27 | dansmith | okay | |
| 16:31:35 | sean-k-mooney | unsafe would have been less work but i was alrey 30 mins into fixing the tests | |
| 16:31:46 | sean-k-mooney | so i decied to just finish it | |
| 16:32:04 | dansmith | yeah I just wasn't sure from your comment if you were going to respin is all | |
| 16:32:11 | bauzas | hum, looks like we don't have any existing functional tests checking shared storage | |
| 16:32:18 | sean-k-mooney | ya i proably should have dropped the comment | |