| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-22 | |||
| 09:43:39 | sean-k-mooney | qemu-kvm: ../block.c:6979: int bdrv_inactivate_recurse(BlockDriverState *): Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed. | |
| 09:43:41 | sean-k-mooney | 2024-07-16 13:29:55.795+0000: shutting down, reason=crashed | |
| 09:44:30 | sean-k-mooney | noonedeadpunk: post-copy is more robust to activly loaded vms and is effectivly requried if your using 1G hugepages | |
| 09:45:04 | sean-k-mooney | you can kind of get vms with 2M hugepages to migate without it but 1G hugepages guests are very hard to make function | |
| 09:45:32 | noonedeadpunk | I did not configure hugepages explicitly tbh | |
| 09:45:32 | sean-k-mooney | it woudl be woth trying but i suspect your hiting a qemu but and post copy might help you work around that but im not sure | |
| 09:45:49 | sean-k-mooney | noonedeadpunk: if you didnt then its not a factor | |
| 09:45:59 | sean-k-mooney | the kernel might be using transparent hugepages | |
| 09:46:06 | sean-k-mooney | but if that the case it wont impact the migration | |
| 09:46:19 | noonedeadpunk | yeah, let me try to reproduce one more time as original VM was dropped apparently | |
| 09:46:23 | sean-k-mooney | the issue with explcit hugepages is if you dirty the page qemu has to copy the entire page again | |
| 09:46:42 | sean-k-mooney | mean a 1 byte write requrie 1G of data to be copied again | |
| 09:47:42 | noonedeadpunk | so what would be suggested value? 1G? | |
| 09:48:03 | sean-k-mooney | sorry can you say that again | |
| 09:48:27 | sean-k-mooney | are you askign what i recomemnd if you choosoe to use hugepages? | |
| 09:48:48 | noonedeadpunk | Yeah :) I';d need to read about them more, but maybe you have a suggestion from your experience | |
| 09:49:06 | sean-k-mooney | if so i recommend 2MB hugepages for vms and if you happen ot use ovs-dpdk or have some other use for hugepages in teh ssytem then use 1G hugepages for those but not for the nova vms | |
| 09:49:23 | noonedeadpunk | ok, right | |
| 09:49:31 | noonedeadpunk | I don't have anything specific | |
| 09:49:38 | opendevreview | Markus Hentsch proposed openstack/nova master: Use os-brick for volume attachment passphrases https://review.opendev.org/c/openstack/nova/+/926326 | |
| 09:49:47 | noonedeadpunk | thus - never bothered to be explicit in defining the size | |
| 09:50:17 | sean-k-mooney | noonedeadpunk: well you also need to preallcoate them on teh kernel commandlien or with a start up script and some other config. | |
| 09:50:46 | sean-k-mooney | they do improve performance but the make the vm numa affined and you cant have any memroy over subscription | |
| 09:50:51 | noonedeadpunk | yeah, see that in the doc https://docs.openstack.org/nova/latest/admin/huge-pages.html | |
| 09:51:02 | sean-k-mooney | so for high performacne workload they are greate but they are not for all usecases | |
| 09:51:10 | noonedeadpunk | we have ram allocation ratio to 1 anyway | |
| 09:51:19 | sean-k-mooney | then you should use them :) | |
| 09:51:30 | noonedeadpunk | numa affinity is more scary though | |
| 09:51:47 | sean-k-mooney | do you use cpu pinning | |
| 09:51:58 | noonedeadpunk | as I can recall some painful experience on scheduling/evacuating with numa pinning | |
| 09:52:06 | noonedeadpunk | no, not really | |
| 09:52:09 | sean-k-mooney | ack | |
| 09:52:22 | noonedeadpunk | there's a specific tier with cpu pinning though | |
| 09:53:00 | noonedeadpunk | so probably there it makes most sense to use hugepages... | |
| 09:53:02 | noonedeadpunk | anyway | |
| 09:53:06 | sean-k-mooney | all cpu pinned guests are numa afifned and unless your using file backed memory when you use cpu pinning you need to specify hw:mem_page_size | |
| 09:53:12 | sean-k-mooney | you can set it to small | |
| 09:53:17 | sean-k-mooney | so you dont need hugepages | |
| 09:53:32 | sean-k-mooney | but if you dont set it your pinned VMs will eventually get OOM killed | |
| 09:54:09 | sean-k-mooney | noonedeadpunk: lets park the wonderful world of numa for now | |
| 09:54:15 | noonedeadpunk | right | |
| 09:54:22 | noonedeadpunk | but it's super interesting as well;) | |
| 09:54:24 | sean-k-mooney | if you replicate the issue please check if there is anything in the qemu isntance logs | |
| 09:54:30 | noonedeadpunk | thanks for enlighting me | |
| 09:54:40 | sean-k-mooney | im guessing your hiting a bug at that level but if not that also valumbel to knwo | |
| 10:02:00 | sean-k-mooney | noonedeadpunk: by the way on the srouce node sie we see in the logs that the migration compelted an libvirt stop the vm based on the auti message then it proceed to post live migration. | |
| 10:02:14 | noonedeadpunk | yeah | |
| 10:02:29 | noonedeadpunk | so it's kinda nowhere to revert at this point | |
| 10:02:30 | sean-k-mooney | so if there is a failure i woudl expect it to be in the destionat qemu instance log | |
| 10:03:06 | noonedeadpunk | I was thinking it's smth related to neutron port attachment, but I saw that on both OVN and OVS deployments | |
| 10:03:18 | noonedeadpunk | on Ubuntu 22.04 and 24.04 | |
| 10:03:32 | noonedeadpunk | (Caracal and Dalmatian) | |
| 10:03:52 | sean-k-mooney | ya i guess this was 22.04 basesed on teh fact your runnning caracal on 3.10 | |
| 10:04:09 | noonedeadpunk | this one was ovs one, yeah | |
| 10:04:36 | sean-k-mooney | and i assum OSA based on the fact your installed in a venv but not the normal kolla location | |
| 10:04:44 | noonedeadpunk | sure :) | |
| 10:05:15 | sean-k-mooney | i mean that a pretty battel tested config at this point | |
| 10:05:15 | noonedeadpunk | "normal kolla location" is a bit hilarious, but yeah :) | |
| 10:05:39 | sean-k-mooney | so its interesting that this is the first tiem someone raised this specific failrue mode | |
| 10:05:40 | noonedeadpunk | as I think system packages would use same path as kolla? | |
| 10:06:04 | sean-k-mooney | kolla supprots soruce or binary installs in the contianer | |
| 10:06:26 | noonedeadpunk | that's why my question also was if it's smth known already so I don't have to dig through the rabbit hole | |
| 10:06:30 | sean-k-mooney | if it source it builds the contaienr with a venv ate /var/lib/kolla something i think | |
| 10:06:39 | noonedeadpunk | ah, ok, right | |
| 10:07:17 | sean-k-mooney | that how it used to work anyway i have not contributed to kolla in a long time although my home oepnstack s deployed with caracal using kolla :) | |
| 10:08:30 | sean-k-mooney | my current config is defintly not a supproted one however so it prehaps not the best refernce | |
| 10:08:43 | noonedeadpunk | I don't think it has anything to do with deployment method tbh, unless we've messed up with mtls certificate generation | |
| 10:08:51 | noonedeadpunk | or ssh cert | |
| 10:09:08 | noonedeadpunk | but I don't think live migrations use ssh for post anyway? | |
| 10:09:17 | sean-k-mooney | i dont thnk so either i was more wondering what the rought qemu/libvirt/kernel version woudl be in this case | |
| 10:10:04 | noonedeadpunk | qemu 6.2 libvirt 8.0.0 kernel 6.5.0 | |
| 10:10:10 | sean-k-mooney | noonedeadpunk: so form the libvirt point of view the migration was complete. and at this point knwo shoudl not need to use ssh because the only thing its copying with images_type=rbd is liek the consoel | |
| 10:10:13 | sean-k-mooney | and that happend already | |
| 10:11:07 | sean-k-mooney | s/point know/point nova/ | |
| 10:11:24 | noonedeadpunk | and for 24.04 it was qemu 8.2.2 libvirt 10.0.0 and kernel 6.8,0 | |
| 10:11:30 | sean-k-mooney | what ever happend to the vm on the dest i think is local to the dest host. | |
| 10:11:55 | sean-k-mooney | are you migrating acrsso distor verison or is the os the saem on both ends | |
| 10:12:17 | noonedeadpunk | it;'s same on both ends. just different envs I've seen same thing | |
| 10:12:41 | sean-k-mooney | goign old to new is supproted but if thise does end up beign a qemu issue thten that will be an impoarnt factor if the qemu versions differ | |
| 10:12:57 | noonedeadpunk | yeah, right, that would be completely different story | |
| 10:13:11 | noonedeadpunk | and where 24.04 is OVN deployment as well | |
| 10:13:29 | noonedeadpunk | as I had suspicions for port attachement tbh | |
| 10:13:37 | sean-k-mooney | so that almost makes me think its ceph related | |
| 10:14:06 | sean-k-mooney | seperate ceph clusters? | |
| 10:14:46 | sean-k-mooney | so regardign the ports we create the ovs port on the destiatnion in pre-live-migration | |
| 10:14:52 | noonedeadpunk | separate in terms of? different? yes, these are 2 completely independent clusters at different sites in different countries, so nothing is shared between them at all | |
| 10:15:00 | sean-k-mooney | and then we activate the port binding in neutron in post live migration | |
| 10:15:19 | sean-k-mooney | but we are not modifyign the data path just switchihng which host is the active one | |
| 10:15:31 | noonedeadpunk | yeah, so I saw on OVN some fishy logs regarding port attachement... anyway, let me reproduce the thing :) | |
| 10:15:37 | sean-k-mooney | so the port biding activation shoudl not impact teh vm as that is out side its view | |
| 10:15:51 | noonedeadpunk | ah, right... | |
| 10:15:56 | noonedeadpunk | makes sense | |
| 10:16:57 | sean-k-mooney | if they are two entrily diffent site witn two entrily diffent versio oeprating system version an qemu/libvirt that wild | |
| 10:17:14 | sean-k-mooney | and also two diffent verison of opesntack | |
| 10:17:26 | noonedeadpunk | but I'm not migrating between them | |
| 10:17:43 | noonedeadpunk | jsut 2 envs where I can redroduce failure quite reliably | |
| 10:17:48 | sean-k-mooney | so etierh it a regression in nova (but i dont knwo what it could be) that in caracal and not fixed in dalmation or some things you shoudl have a bad day | |
| 10:18:15 | noonedeadpunk | or I messed up in setup/osa... | |
| 10:28:32 | noonedeadpunk | sean-k-mooney: destination logs do not have anything enlighting: https://paste.openstack.org/show/bcocoH9OrjscQt4gPTOr/ | |
| 10:29:11 | sean-k-mooney | well it has | |