Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-05-22
09:51:30 noonedeadpunk numa affinity is more scary though
09:51:47 sean-k-mooney do you use cpu pinning
09:51:58 noonedeadpunk as I can recall some painful experience on scheduling/evacuating with numa pinning
09:52:06 noonedeadpunk no, not really
09:52:09 sean-k-mooney ack
09:52:22 noonedeadpunk there's a specific tier with cpu pinning though
09:53:00 noonedeadpunk so probably there it makes most sense to use hugepages...
09:53:02 noonedeadpunk anyway
09:53:06 sean-k-mooney all cpu pinned guests are numa afifned and unless your using file backed memory when you use cpu pinning you need to specify hw:mem_page_size
09:53:12 sean-k-mooney you can set it to small
09:53:17 sean-k-mooney so you dont need hugepages
09:53:32 sean-k-mooney but if you dont set it your pinned VMs will eventually get OOM killed
09:54:09 sean-k-mooney noonedeadpunk: lets park the wonderful world of numa for now
09:54:15 noonedeadpunk right
09:54:22 noonedeadpunk but it's super interesting as well;)
09:54:24 sean-k-mooney if you replicate the issue please check if there is anything in the qemu isntance logs
09:54:30 noonedeadpunk thanks for enlighting me
09:54:40 sean-k-mooney im guessing your hiting a bug at that level but if not that also valumbel to knwo
10:02:00 sean-k-mooney noonedeadpunk: by the way on the srouce node sie we see in the logs that the migration compelted an libvirt stop the vm based on the auti message then it proceed to post live migration.
10:02:14 noonedeadpunk yeah
10:02:29 noonedeadpunk so it's kinda nowhere to revert at this point
10:02:30 sean-k-mooney so if there is a failure i woudl expect it to be in the destionat qemu instance log
10:03:06 noonedeadpunk I was thinking it's smth related to neutron port attachment, but I saw that on both OVN and OVS deployments
10:03:18 noonedeadpunk on Ubuntu 22.04 and 24.04
10:03:32 noonedeadpunk (Caracal and Dalmatian)
10:03:52 sean-k-mooney ya i guess this was 22.04 basesed on teh fact your runnning caracal on 3.10
10:04:09 noonedeadpunk this one was ovs one, yeah
10:04:36 sean-k-mooney and i assum OSA based on the fact your installed in a venv but not the normal kolla location
10:04:44 noonedeadpunk sure :)
10:05:15 sean-k-mooney i mean that a pretty battel tested config at this point
10:05:15 noonedeadpunk "normal kolla location" is a bit hilarious, but yeah :)
10:05:39 sean-k-mooney so its interesting that this is the first tiem someone raised this specific failrue mode
10:05:40 noonedeadpunk as I think system packages would use same path as kolla?
10:06:04 sean-k-mooney kolla supprots soruce or binary installs in the contianer
10:06:26 noonedeadpunk that's why my question also was if it's smth known already so I don't have to dig through the rabbit hole
10:06:30 sean-k-mooney if it source it builds the contaienr with a venv ate /var/lib/kolla something i think
10:06:39 noonedeadpunk ah, ok, right
10:07:17 sean-k-mooney that how it used to work anyway i have not contributed to kolla in a long time although my home oepnstack s deployed with caracal using kolla :)
10:08:30 sean-k-mooney my current config is defintly not a supproted one however so it prehaps not the best refernce
10:08:43 noonedeadpunk I don't think it has anything to do with deployment method tbh, unless we've messed up with mtls certificate generation
10:08:51 noonedeadpunk or ssh cert
10:09:08 noonedeadpunk but I don't think live migrations use ssh for post anyway?
10:09:17 sean-k-mooney i dont thnk so either i was more wondering what the rought qemu/libvirt/kernel version woudl be in this case
10:10:04 noonedeadpunk qemu 6.2 libvirt 8.0.0 kernel 6.5.0
10:10:10 sean-k-mooney noonedeadpunk: so form the libvirt point of view the migration was complete. and at this point knwo shoudl not need to use ssh because the only thing its copying with images_type=rbd is liek the consoel
10:10:13 sean-k-mooney and that happend already
10:11:07 sean-k-mooney s/point know/point nova/
10:11:24 noonedeadpunk and for 24.04 it was qemu 8.2.2 libvirt 10.0.0 and kernel 6.8,0
10:11:30 sean-k-mooney what ever happend to the vm on the dest i think is local to the dest host.
10:11:55 sean-k-mooney are you migrating acrsso distor verison or is the os the saem on both ends
10:12:17 noonedeadpunk it;'s same on both ends. just different envs I've seen same thing
10:12:41 sean-k-mooney goign old to new is supproted but if thise does end up beign a qemu issue thten that will be an impoarnt factor if the qemu versions differ
10:12:57 noonedeadpunk yeah, right, that would be completely different story
10:13:11 noonedeadpunk and where 24.04 is OVN deployment as well
10:13:29 noonedeadpunk as I had suspicions for port attachement tbh
10:13:37 sean-k-mooney so that almost makes me think its ceph related
10:14:06 sean-k-mooney seperate ceph clusters?
10:14:46 sean-k-mooney so regardign the ports we create the ovs port on the destiatnion in pre-live-migration
10:14:52 noonedeadpunk separate in terms of? different? yes, these are 2 completely independent clusters at different sites in different countries, so nothing is shared between them at all
10:15:00 sean-k-mooney and then we activate the port binding in neutron in post live migration
10:15:19 sean-k-mooney but we are not modifyign the data path just switchihng which host is the active one
10:15:31 noonedeadpunk yeah, so I saw on OVN some fishy logs regarding port attachement... anyway, let me reproduce the thing :)
10:15:37 sean-k-mooney so the port biding activation shoudl not impact teh vm as that is out side its view
10:15:51 noonedeadpunk ah, right...
10:15:56 noonedeadpunk makes sense
10:16:57 sean-k-mooney if they are two entrily diffent site witn two entrily diffent versio oeprating system version an qemu/libvirt that wild
10:17:14 sean-k-mooney and also two diffent verison of opesntack
10:17:26 noonedeadpunk but I'm not migrating between them
10:17:43 noonedeadpunk jsut 2 envs where I can redroduce failure quite reliably
10:17:48 sean-k-mooney so etierh it a regression in nova (but i dont knwo what it could be) that in caracal and not fixed in dalmation or some things you shoudl have a bad day
10:18:15 noonedeadpunk or I messed up in setup/osa...
10:28:32 noonedeadpunk sean-k-mooney: destination logs do not have anything enlighting: https://paste.openstack.org/show/bcocoH9OrjscQt4gPTOr/
10:29:11 sean-k-mooney well it has
10:29:13 sean-k-mooney 2025-05-22 06:55:49.882+0000: initiating migration
10:29:15 sean-k-mooney 2025-05-22 07:18:40.025+0000: shutting down, reason=crashed
10:29:17 noonedeadpunk shutting down, reason=migrated and next thing it crashed
10:29:22 opendevreview Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for sound model. https://review.opendev.org/c/openstack/nova/+/926126
10:29:22 opendevreview Michael Still proposed openstack/nova master: Implement sound model extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/940770
10:29:23 opendevreview Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for USB controller model. https://review.opendev.org/c/openstack/nova/+/927354
10:29:23 opendevreview Michael Still proposed openstack/nova master: Implement USB controller extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/950643
10:29:51 noonedeadpunk ah, ok, I think it was first attempt on migrating
10:29:52 noonedeadpunk yeah
10:30:10 noonedeadpunk but it does not point to any reason....
10:31:33 mikal sean-k-mooney: the fourth patch lacks the compute protection, I'll do that tomorrow. So don't go mad and merge that one too early!
10:33:44 noonedeadpunk *spice-direct
10:34:39 sean-k-mooney noonedeadpunk: this all look very stnadard. not nessiarly optimised for performacne but this is a very typeical looking vm so you not enabling anyting that raises red falgs to me
10:35:58 mikal noonedeadpunk: So its sort of half done -- there are now Nova APIs to create a console of type spice-direct, with openstacksdk and python-openstackclient support. Sound and USB are still missing (that's the patches above), so if that's a thing you care about you might need to hold your breath a bit. I have patches for kolla-ansible to deploy it,
10:35:58 mikal but I am sure it has bugs like no HA -- its still a work in progress. My pain focus right now is working through load testing the SPICE native proxy so I can try and quantify if it needs tweaking and what scaling to recommend with it.
10:36:47 noonedeadpunk mikal: not specifically care about spice-direct itself, but more wanted to implement that for OSA as well
10:36:56 sean-k-mooney mikal: ack
10:37:26 mikal noonedeadpunk: so OSA would need to know how to deploy the proxy and to twiddle some Nova config options. The existing Kolla-Ansible patches demonstrate how to do that.
10:37:57 noonedeadpunk mikal: do you have link top patches somewhere nearby?
10:38:32 noonedeadpunk there's support for spice-html5 but direct looks quite different
10:38:48 mikal noonedeadpunk: this is the list of patches I am carrying against Kolla-Ansible right now -- https://github.com/shakenfist/kerbside-patches/blob/develop/kolla-ansible/ORDER. There are also some minor patches against Kolla itself at https://github.com/shakenfist/kerbside-patches/blob/develop/kolla/ORDER. I'm sorry that's not the most helpful way to
10:38:48 mikal express them but its what I have right now.
10:39:17 sean-k-mooney noonedeadpunk: it is its using a out of tree proyx that mikal is developing called kerbside that actully allows raw spice conenction form a client to the vm
10:39:21 mikal noonedeadpunk: HTML5 and spice-direct can coexist I think, as long as you don't turn on the new require_secure option. The HTML5 client doesn't support TLS.
10:39:25 sean-k-mooney the proxy makes that safe
10:39:37 sean-k-mooney at least in thory
10:39:57 mikal noonedeadpunk: the proxy is at https://github.com/shakenfist/kerbside for what its worth. Like I said, the proxy itself is still a bit of a work in progress.

Earlier   Later