| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-02-24 | |||
| 17:31:15 | sean-k-mooney | its more subtle then that but that the main part | |
| 17:32:09 | melwitt | ok, thanks | |
| 17:32:51 | pas-ha[m] | fwiw, I recently tried to implement a cold migration / resize for LVM image backend... and it even kind of worked but was ugly AF... really not worth actually doing for sure | |
| 17:34:57 | melwitt | ah, interesting | |
| 17:35:19 | sean-k-mooney | pas-ha[m]: it used to wrok a very long time ago | |
| 17:35:36 | sean-k-mooney | it broke at some point but i have no idea when | |
| 17:36:08 | sean-k-mooney | i dont see why we woudl not supprot it for lvm again eventurlly | |
| 17:36:21 | sean-k-mooney | we just need to scp/rsync the data | |
| 17:36:42 | sean-k-mooney | it very similar to raw/qcow local storage migration | |
| 17:36:51 | sean-k-mooney | we have apis to do this | |
| 17:37:04 | sean-k-mooney | via the remotefs dirvers | |
| 17:37:18 | sean-k-mooney | there is a copy image fucntion for this that lvm shoudl be implementing | |
| 17:37:25 | sean-k-mooney | or using | |
| 17:38:33 | sean-k-mooney | the lvm image backend dirver is the highest perfromign stroage option nova currently supports | |
| 17:38:42 | sean-k-mooney | espically for write intensive operations | |
| 17:38:53 | pas-ha[m] | yeah, kind of what I did. the main problem is that I did not thought of a correct way of dd | ssh | dd to skip the intermediate file, so I was dd to local file, scp to remote, on remote make a new lvm and dd local file into that.. | |
| 17:39:18 | sean-k-mooney | you do not need an a intermediate file | |
| 17:39:50 | sean-k-mooney | dd directly to the remote lvm volume shoudl be enough | |
| 17:40:23 | sean-k-mooney | that is an optimistaion howver | |
| 17:41:06 | pas-ha[m] | also, it seems the libvirt itself does support the migration with lvm, even live one, but for that this lvm must be managed by libvirt itself as a storage pool... which is to much of a rewrite for how nova manages local storage I guess.. | |
| 17:41:40 | sean-k-mooney | i think you can live migrate without the storage pool | |
| 17:41:50 | sean-k-mooney | but for cold migration it need it yes | |
| 17:42:01 | sean-k-mooney | and ya we have no plans to use libvirt sotrage pools in the future | |
| 17:44:13 | sean-k-mooney | libivrt is one of those things that is a doubel edge sword. it provides some useful abstration over qemu but at a cost of addtioanl complexity in the tech stack | |
| 17:44:26 | sean-k-mooney | for the most part its been a win | |
| 17:44:29 | sean-k-mooney | but not alwasys | |
| 17:57:36 | melwitt | hi all, I need a second reviewer on this not-yet-released-regression fix (partial revert) related to tpool.Proxy + libvirt listDevices() if anyone could please look https://review.opendev.org/q/topic:%22bug/2098892%22 | |
| 17:58:16 | sean-k-mooney | oh right i have reviewd that already so someone other then melwitt or i | |
| 17:59:42 | sean-k-mooney | bauzas: gibi this impacts gpus and pci passthough so perhaps ye could find time dansmith may also have context since we chatted about it a little when it was found downstream | |
| 18:00:11 | sean-k-mooney | melwitt: thats needed on stable too yes? | |
| 18:00:37 | sean-k-mooney | i dont think we have done a release with it yet | |
| 18:00:51 | sean-k-mooney | but we sould avoid doing one until the latests patches are included | |
| 18:00:56 | melwitt | yes, it is needed on 4 stable branches also :( | |
| 18:04:02 | gibi | sean-k-mooney: I'm out of energy today, made a note to check it tomorrow | |
| 18:22:43 | sshmlbrg | sean-k-mooney: o/ Could u plz review this mr https://review.opendev.org/c/openstack/nova/+/942561 ? | |
| 18:32:19 | dansmith | melwitt: question hre: https://review.opendev.org/c/openstack/nova/+/942334/6/nova/virt/libvirt/host.py | |
| 18:36:47 | dansmith | ah, nevermind, it's in the commit message below the fold | |
| 18:41:26 | melwitt | thanks dansmith! | |
| 18:48:33 | danfai | sean-k-mooney: thanks for the response re vif_type change. sorry only saw the message now. what I wondered was during a migration, the target hypervisor could have a different model. from what I see nova picks up the new vif_type after the migration, but the bridge_name is at that time wrong | |
| 18:50:56 | sean-k-mooney | so it used to be possibale a long time ago to live migrate between linux bridge and ovs | |
| 18:51:06 | sean-k-mooney | danfai: and i had patches up to fix the bridge name | |
| 18:51:15 | sean-k-mooney | but i dont think neuton ever merged them | |
| 18:51:21 | sean-k-mooney | ill see if i still have the fixes in gerrit | |
| 18:52:01 | danfai | thanks, I saw some comments in the code base from you and mriedem :) | |
| 18:52:13 | sean-k-mooney | https://bugs.launchpad.net/neutron/+bug/1788009 | |
| 18:52:39 | sean-k-mooney | we fixed it for ml2/ovs | |
| 18:52:53 | sean-k-mooney | but ml2/ovn didnt populated it proeprly | |
| 18:53:15 | sean-k-mooney | i think that is still a gap in tehr migration tooling but it might have got fixed in the last year or two | |
| 18:53:38 | danfai | oh, ok. I'll have a more detailed look on the neutron side then. We are still in yoga, so... | |
| 18:54:00 | sean-k-mooney | danfai: for what ite work codl migration shoudl still work i think | |
| 18:54:12 | sean-k-mooney | its been a long time since i tried to fix this | |
| 18:54:27 | mikal | Morning | |
| 18:54:34 | sean-k-mooney | danfai: if your using vlan networking with linux breige yoru move will be less painful | |
| 18:54:45 | danfai | cold migration works flawlessly indeed. also two live migrations (we only have provider networks though) | |
| 18:54:48 | sean-k-mooney | danfai: but just be aware ovn does not supprot vxlan or gre tunnes | |
| 18:54:53 | mikal | sean-k-mooney: I feel like I should recheck the spice-direct final patch now that melwitt's tempest fix has landed. Do you see any reason to not do that? | |
| 18:55:10 | sean-k-mooney | mikal:i already did | |
| 18:55:14 | sean-k-mooney | did it fail again | |
| 18:55:21 | sean-k-mooney | mikal: i have started merging your code | |
| 18:55:39 | mikal | sean-k-mooney: doh, I mismashed the refresh button because I am pre coffee. | |
| 18:55:45 | mikal | Nah, it passed. | |
| 18:56:04 | mikal | Ugh, arm passed. | |
| 18:56:12 | mikal | I will be quiet now. | |
| 18:56:22 | sean-k-mooney | mikal: its still in check i think | |
| 18:56:45 | sean-k-mooney | so it failed on the funtional jobs and weill need a recheck | |
| 18:56:58 | sean-k-mooney | but https://zuul.opendev.org/t/openstack/build/1098d7514ba843e28a5477c69892b176 the hybird plug job passed again | |
| 18:57:05 | danfai | yes, regarding vxlans, this was mentioned in the meeting we had with operators. | |
| 18:57:20 | sean-k-mooney | mikal: rechecks wont work until it repors back so lets just let it complete | |
| 18:57:29 | mikal | sean-k-mooney: will do | |
| 18:58:12 | sean-k-mooney | danfai: the main issue iwht the migration scripts that neutron provides is geneve requires a small mtu on the client or extra overhead on the underlying network then vxlan | |
| 18:59:09 | danfai | hmm, I thought there was more, like ID mappings. I'll add the MTU to the notes, thanks | |
| 18:59:44 | sshmlbrg | sean-k-mooney: Hi! Did u see my request on review? | |
| 19:00:00 | sean-k-mooney | danfai: nova does not supprot updatign the mtu on conencted ports for what its worth | |
| 19:00:11 | sean-k-mooney | danfai: the mtu change wil only take effect after a hard reboot | |
| 19:00:22 | sean-k-mooney | danfai: https://github.com/openstack/neutron/commit/baaf240ce3f7802fe1431cc13913b9d93fc7f742 that is the ovn bridge name patch | |
| 19:01:53 | sean-k-mooney | sshmlbrg: yep. we will need to add some nova functioanl tests to replciate the manual procedure you followed in the bug report | |
| 19:02:03 | danfai | Thank you very much | |
| 19:02:52 | sean-k-mooney | sshmlbrg: unit test are good to have but for operations like shelve we try to also have functional tests. and for bugs we prefer to create a functional test that show the broken behavior as an addtion patch and then have the fix as a second commit | |
| 19:03:48 | sean-k-mooney | sshmlbrg: without loading all the context im not sure that the finally block is corect | |
| 19:04:08 | sean-k-mooney | sshmlbrg: you should not be modifying the az of the instnace directly | |
| 19:04:45 | sshmlbrg | sean-k-mooney: ok, thanks. I will add functional tests as a first commit | |
| 19:04:46 | sean-k-mooney | sshmlbrg: we do not really use the az colume in the cell database to store the az of the instance anymore. | |
| 19:05:31 | sean-k-mooney | sshmlbrg: cool i would od it be creating a new file in https://github.com/openstack/nova/tree/master/nova/tests/functional/regressions | |
| 19:05:45 | sean-k-mooney | for the bug ill see if i can quickly find an exmaple for you to copy | |
| 19:06:11 | sean-k-mooney | sshmlbrg: perhaps https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_2025480.py | |
| 19:06:24 | sshmlbrg | sean-k-mooney: then will need to change the methods used and add the az argument to them | |
| 19:07:04 | sean-k-mooney | updatign the object in memory may be ok its the save that is a little suspect | |
| 19:07:24 | sean-k-mooney | we have a temporay mutation context manager | |
| 19:07:44 | sean-k-mooney | i that we may want to condier | |
| 19:07:53 | sean-k-mooney | we can dicuss that on the review | |
| 19:08:22 | sshmlbrg | sean-k-mooney: thank you for example! ok, sounds good | |
| 19:08:51 | sean-k-mooney | if you look at similar files in the same directory there shoudl be a boot form volume example | |
| 19:09:32 | sean-k-mooney | or other shelve example so it should not be too hard to create a repoducer following the steps in the bug | |
| 19:11:46 | sshmlbrg | sean-k-mooney: i got it. I am gonna return with changes tomorrow. Have a nice day or evening :) | |
| 19:12:43 | sean-k-mooney | sshmlbrg: https://github.com/openstack/nova/blob/375d95565d594182c5472f710b3f7bd53f0746d4/nova/tests/functional/regressions/test_bug_1675570.py#L26 | |
| 19:12:47 | sean-k-mooney | that a better example | |
| 19:13:10 | sean-k-mooney | sshmlbrg: o/ | |
| 19:14:19 | sean-k-mooney | sshmlbrg: that actully attaching a server to a shelve offloaded instnace | |
| 19:14:35 | sean-k-mooney | the reason its not failing is cross az attach is allowed by defult | |
| 19:14:58 | sean-k-mooney | so if you were to set that to false in yoru version i woudl eeepct the attach to fail and repoduce the bug :) | |