| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-22 | |||
| 07:37:05 | sean-k-mooney | so i dont really expect there to be any thing that need to be changed | |
| 07:37:09 | mikal | Yeah, certainly I can't see anything major in the comments unless I've missed something. | |
| 07:37:29 | mikal | Ok cool. I will do the refactor then and implement the compute manager protection for USB as well. | |
| 09:23:25 | noonedeadpunk | Hey folks! Got a question. Today realized that quite an unexpected behaviour is happening with live migrations. So on memory-intense VMs migration seems to fail on post-migration step. | |
| 09:23:57 | noonedeadpunk | The weird/enexpected thing, that as a result VM keeps shut down on both hosts and it moves to ERROR state | |
| 09:24:22 | noonedeadpunk | so decided to ask if this is smth quite well known? | |
| 09:24:44 | noonedeadpunk | As right now I think it's related to `live_migration_permit_auto_converge = true` | |
| 09:25:32 | noonedeadpunk | as issue happens only on memory intense VMs. So if you run stress inside VM - it will 99% end up in ERROR and shut down. If there's some reasonable applucation running - it's fine | |
| 09:26:17 | noonedeadpunk | So my guess is that if live_migration_permit_auto_converge kicks in, libvirt simply incapable to resume the VM on thye destination, but I'm not really sure if that is true | |
| 09:26:57 | noonedeadpunk | Thus before going to this rabbit hole wanted to ask if you might now anything about that :) | |
| 09:30:15 | frickler | noonedeadpunk: I assume you checked the nova-compute logs without success? how about the libvirt instance logs? but maybe wait for sean-k-mooney to chime in first | |
| 09:31:00 | sean-k-mooney | sorry was looking at something else reading back | |
| 09:32:09 | sean-k-mooney | noonedeadpunk: do you have post_copy migration enabeld? | |
| 09:32:44 | sean-k-mooney | auto converge just add cpu puases so that is very unlikely to be a factor | |
| 09:34:37 | sean-k-mooney | noonedeadpunk: for what its worth if there is high ram load it basically imposibel to migrate without post-copy migration. auto converge si not entough to supprot that usecase | |
| 09:34:52 | sean-k-mooney | it just slightly chagne wehre the live mgiration will fall over | |
| 09:36:00 | sean-k-mooney | auto converge is still a type of "pre-copy" migration when qemu copes all the memroy first , pauses the vm on the dest copyies the detal and resuses it on the souce | |
| 09:36:12 | noonedeadpunk | no, post_copy is disabled | |
| 09:36:16 | noonedeadpunk | only converge is | |
| 09:36:47 | sean-k-mooney | the difference betwen live_migration_permit_auto_converge = true and false is if its set to true. the qemu on the souce is allwo to micro pause the souce vm cpus before switching to the dest | |
| 09:37:10 | noonedeadpunk | yes, right, and that used to work nicely | |
| 09:37:27 | sean-k-mooney | in your case if your getting to post_live_migration that mean libvirt told nova the migration was complete | |
| 09:37:29 | noonedeadpunk | and I think regardless that should not end up with VM jsut being down on both sides? | |
| 09:37:45 | noonedeadpunk | jsut a sec, I'll re-paste logs | |
| 09:38:03 | sean-k-mooney | at which point the vm on the souce is not need any mroe and shoudl be terminate by libvirt | |
| 09:38:10 | sean-k-mooney | as you cant rool back at that point | |
| 09:38:18 | noonedeadpunk | it;s from source https://paste.openstack.org/show/bSc55xr9LSgCXiaj1uo1/ | |
| 09:38:33 | noonedeadpunk | and that's destination: https://paste.openstack.org/show/boBhuqrR8mYDoqCjYxex/ | |
| 09:39:01 | noonedeadpunk | the only very suspicious thing I've spotted, is that I also do see a libvirt reseted communication to qemu | |
| 09:40:25 | sean-k-mooney | i knwo you said this but the vm is activly runing right it not paused when you live migrated it right | |
| 09:40:30 | noonedeadpunk | so it feels that during unpause things crash unobviously | |
| 09:40:41 | noonedeadpunk | it's running, yes | |
| 09:40:46 | noonedeadpunk | I mean paused on destination | |
| 09:41:22 | sean-k-mooney | the reaons im asking is this is sort of similar to https://issues.redhat.com/browse/RHEL-48801 but that about live migrating a puased instance twice | |
| 09:41:59 | sean-k-mooney | qemu was not probly handling something in the block storage layer in that case | |
| 09:42:00 | noonedeadpunk | there's also no block migration as storage is ceph | |
| 09:42:10 | sean-k-mooney | ack that also good to know | |
| 09:43:03 | noonedeadpunk | but yeah, trying out post_copy was my next step to try | |
| 09:43:15 | noonedeadpunk | Was jsut a bit afraid of it tbh | |
| 09:43:20 | sean-k-mooney | noonedeadpunk: can you check the qemu instance logs and see if there is an error on the source or dest | |
| 09:43:36 | sean-k-mooney | for the block device one we saw a message like this | |
| 09:43:39 | sean-k-mooney | qemu-kvm: ../block.c:6979: int bdrv_inactivate_recurse(BlockDriverState *): Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed. | |
| 09:43:41 | sean-k-mooney | 2024-07-16 13:29:55.795+0000: shutting down, reason=crashed | |
| 09:44:30 | sean-k-mooney | noonedeadpunk: post-copy is more robust to activly loaded vms and is effectivly requried if your using 1G hugepages | |
| 09:45:04 | sean-k-mooney | you can kind of get vms with 2M hugepages to migate without it but 1G hugepages guests are very hard to make function | |
| 09:45:32 | sean-k-mooney | it woudl be woth trying but i suspect your hiting a qemu but and post copy might help you work around that but im not sure | |
| 09:45:32 | noonedeadpunk | I did not configure hugepages explicitly tbh | |
| 09:45:49 | sean-k-mooney | noonedeadpunk: if you didnt then its not a factor | |
| 09:45:59 | sean-k-mooney | the kernel might be using transparent hugepages | |
| 09:46:06 | sean-k-mooney | but if that the case it wont impact the migration | |
| 09:46:19 | noonedeadpunk | yeah, let me try to reproduce one more time as original VM was dropped apparently | |
| 09:46:23 | sean-k-mooney | the issue with explcit hugepages is if you dirty the page qemu has to copy the entire page again | |
| 09:46:42 | sean-k-mooney | mean a 1 byte write requrie 1G of data to be copied again | |
| 09:47:42 | noonedeadpunk | so what would be suggested value? 1G? | |
| 09:48:03 | sean-k-mooney | sorry can you say that again | |
| 09:48:27 | sean-k-mooney | are you askign what i recomemnd if you choosoe to use hugepages? | |
| 09:48:48 | noonedeadpunk | Yeah :) I';d need to read about them more, but maybe you have a suggestion from your experience | |
| 09:49:06 | sean-k-mooney | if so i recommend 2MB hugepages for vms and if you happen ot use ovs-dpdk or have some other use for hugepages in teh ssytem then use 1G hugepages for those but not for the nova vms | |
| 09:49:23 | noonedeadpunk | ok, right | |
| 09:49:31 | noonedeadpunk | I don't have anything specific | |
| 09:49:38 | opendevreview | Markus Hentsch proposed openstack/nova master: Use os-brick for volume attachment passphrases https://review.opendev.org/c/openstack/nova/+/926326 | |
| 09:49:47 | noonedeadpunk | thus - never bothered to be explicit in defining the size | |
| 09:50:17 | sean-k-mooney | noonedeadpunk: well you also need to preallcoate them on teh kernel commandlien or with a start up script and some other config. | |
| 09:50:46 | sean-k-mooney | they do improve performance but the make the vm numa affined and you cant have any memroy over subscription | |
| 09:50:51 | noonedeadpunk | yeah, see that in the doc https://docs.openstack.org/nova/latest/admin/huge-pages.html | |
| 09:51:02 | sean-k-mooney | so for high performacne workload they are greate but they are not for all usecases | |
| 09:51:10 | noonedeadpunk | we have ram allocation ratio to 1 anyway | |
| 09:51:19 | sean-k-mooney | then you should use them :) | |
| 09:51:30 | noonedeadpunk | numa affinity is more scary though | |
| 09:51:47 | sean-k-mooney | do you use cpu pinning | |
| 09:51:58 | noonedeadpunk | as I can recall some painful experience on scheduling/evacuating with numa pinning | |
| 09:52:06 | noonedeadpunk | no, not really | |
| 09:52:09 | sean-k-mooney | ack | |
| 09:52:22 | noonedeadpunk | there's a specific tier with cpu pinning though | |
| 09:53:00 | noonedeadpunk | so probably there it makes most sense to use hugepages... | |
| 09:53:02 | noonedeadpunk | anyway | |
| 09:53:06 | sean-k-mooney | all cpu pinned guests are numa afifned and unless your using file backed memory when you use cpu pinning you need to specify hw:mem_page_size | |
| 09:53:12 | sean-k-mooney | you can set it to small | |
| 09:53:17 | sean-k-mooney | so you dont need hugepages | |
| 09:53:32 | sean-k-mooney | but if you dont set it your pinned VMs will eventually get OOM killed | |
| 09:54:09 | sean-k-mooney | noonedeadpunk: lets park the wonderful world of numa for now | |
| 09:54:15 | noonedeadpunk | right | |
| 09:54:22 | noonedeadpunk | but it's super interesting as well;) | |
| 09:54:24 | sean-k-mooney | if you replicate the issue please check if there is anything in the qemu isntance logs | |
| 09:54:30 | noonedeadpunk | thanks for enlighting me | |
| 09:54:40 | sean-k-mooney | im guessing your hiting a bug at that level but if not that also valumbel to knwo | |
| 10:02:00 | sean-k-mooney | noonedeadpunk: by the way on the srouce node sie we see in the logs that the migration compelted an libvirt stop the vm based on the auti message then it proceed to post live migration. | |
| 10:02:14 | noonedeadpunk | yeah | |
| 10:02:29 | noonedeadpunk | so it's kinda nowhere to revert at this point | |
| 10:02:30 | sean-k-mooney | so if there is a failure i woudl expect it to be in the destionat qemu instance log | |
| 10:03:06 | noonedeadpunk | I was thinking it's smth related to neutron port attachment, but I saw that on both OVN and OVS deployments | |
| 10:03:18 | noonedeadpunk | on Ubuntu 22.04 and 24.04 | |
| 10:03:32 | noonedeadpunk | (Caracal and Dalmatian) | |
| 10:03:52 | sean-k-mooney | ya i guess this was 22.04 basesed on teh fact your runnning caracal on 3.10 | |
| 10:04:09 | noonedeadpunk | this one was ovs one, yeah | |
| 10:04:36 | sean-k-mooney | and i assum OSA based on the fact your installed in a venv but not the normal kolla location | |
| 10:04:44 | noonedeadpunk | sure :) | |
| 10:05:15 | noonedeadpunk | "normal kolla location" is a bit hilarious, but yeah :) | |
| 10:05:15 | sean-k-mooney | i mean that a pretty battel tested config at this point | |
| 10:05:39 | sean-k-mooney | so its interesting that this is the first tiem someone raised this specific failrue mode | |