| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-22 | |||
| 10:06:30 | sean-k-mooney | if it source it builds the contaienr with a venv ate /var/lib/kolla something i think | |
| 10:06:39 | noonedeadpunk | ah, ok, right | |
| 10:07:17 | sean-k-mooney | that how it used to work anyway i have not contributed to kolla in a long time although my home oepnstack s deployed with caracal using kolla :) | |
| 10:08:30 | sean-k-mooney | my current config is defintly not a supproted one however so it prehaps not the best refernce | |
| 10:08:43 | noonedeadpunk | I don't think it has anything to do with deployment method tbh, unless we've messed up with mtls certificate generation | |
| 10:08:51 | noonedeadpunk | or ssh cert | |
| 10:09:08 | noonedeadpunk | but I don't think live migrations use ssh for post anyway? | |
| 10:09:17 | sean-k-mooney | i dont thnk so either i was more wondering what the rought qemu/libvirt/kernel version woudl be in this case | |
| 10:10:04 | noonedeadpunk | qemu 6.2 libvirt 8.0.0 kernel 6.5.0 | |
| 10:10:10 | sean-k-mooney | noonedeadpunk: so form the libvirt point of view the migration was complete. and at this point knwo shoudl not need to use ssh because the only thing its copying with images_type=rbd is liek the consoel | |
| 10:10:13 | sean-k-mooney | and that happend already | |
| 10:11:07 | sean-k-mooney | s/point know/point nova/ | |
| 10:11:24 | noonedeadpunk | and for 24.04 it was qemu 8.2.2 libvirt 10.0.0 and kernel 6.8,0 | |
| 10:11:30 | sean-k-mooney | what ever happend to the vm on the dest i think is local to the dest host. | |
| 10:11:55 | sean-k-mooney | are you migrating acrsso distor verison or is the os the saem on both ends | |
| 10:12:17 | noonedeadpunk | it;'s same on both ends. just different envs I've seen same thing | |
| 10:12:41 | sean-k-mooney | goign old to new is supproted but if thise does end up beign a qemu issue thten that will be an impoarnt factor if the qemu versions differ | |
| 10:12:57 | noonedeadpunk | yeah, right, that would be completely different story | |
| 10:13:11 | noonedeadpunk | and where 24.04 is OVN deployment as well | |
| 10:13:29 | noonedeadpunk | as I had suspicions for port attachement tbh | |
| 10:13:37 | sean-k-mooney | so that almost makes me think its ceph related | |
| 10:14:06 | sean-k-mooney | seperate ceph clusters? | |
| 10:14:46 | sean-k-mooney | so regardign the ports we create the ovs port on the destiatnion in pre-live-migration | |
| 10:14:52 | noonedeadpunk | separate in terms of? different? yes, these are 2 completely independent clusters at different sites in different countries, so nothing is shared between them at all | |
| 10:15:00 | sean-k-mooney | and then we activate the port binding in neutron in post live migration | |
| 10:15:19 | sean-k-mooney | but we are not modifyign the data path just switchihng which host is the active one | |
| 10:15:31 | noonedeadpunk | yeah, so I saw on OVN some fishy logs regarding port attachement... anyway, let me reproduce the thing :) | |
| 10:15:37 | sean-k-mooney | so the port biding activation shoudl not impact teh vm as that is out side its view | |
| 10:15:51 | noonedeadpunk | ah, right... | |
| 10:15:56 | noonedeadpunk | makes sense | |
| 10:16:57 | sean-k-mooney | if they are two entrily diffent site witn two entrily diffent versio oeprating system version an qemu/libvirt that wild | |
| 10:17:14 | sean-k-mooney | and also two diffent verison of opesntack | |
| 10:17:26 | noonedeadpunk | but I'm not migrating between them | |
| 10:17:43 | noonedeadpunk | jsut 2 envs where I can redroduce failure quite reliably | |
| 10:17:48 | sean-k-mooney | so etierh it a regression in nova (but i dont knwo what it could be) that in caracal and not fixed in dalmation or some things you shoudl have a bad day | |
| 10:18:15 | noonedeadpunk | or I messed up in setup/osa... | |
| 10:28:32 | noonedeadpunk | sean-k-mooney: destination logs do not have anything enlighting: https://paste.openstack.org/show/bcocoH9OrjscQt4gPTOr/ | |
| 10:29:11 | sean-k-mooney | well it has | |
| 10:29:13 | sean-k-mooney | 2025-05-22 06:55:49.882+0000: initiating migration | |
| 10:29:15 | sean-k-mooney | 2025-05-22 07:18:40.025+0000: shutting down, reason=crashed | |
| 10:29:17 | noonedeadpunk | shutting down, reason=migrated and next thing it crashed | |
| 10:29:22 | opendevreview | Michael Still proposed openstack/nova master: Implement sound model extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/940770 | |
| 10:29:22 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for sound model. https://review.opendev.org/c/openstack/nova/+/926126 | |
| 10:29:23 | opendevreview | Michael Still proposed openstack/nova master: Implement USB controller extra spec for libvirt. https://review.opendev.org/c/openstack/nova/+/950643 | |
| 10:29:23 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add objects and notifications for USB controller model. https://review.opendev.org/c/openstack/nova/+/927354 | |
| 10:29:51 | noonedeadpunk | ah, ok, I think it was first attempt on migrating | |
| 10:29:52 | noonedeadpunk | yeah | |
| 10:30:10 | noonedeadpunk | but it does not point to any reason.... | |
| 10:31:33 | mikal | sean-k-mooney: the fourth patch lacks the compute protection, I'll do that tomorrow. So don't go mad and merge that one too early! | |
| 10:33:44 | noonedeadpunk | *spice-direct | |
| 10:34:39 | sean-k-mooney | noonedeadpunk: this all look very stnadard. not nessiarly optimised for performacne but this is a very typeical looking vm so you not enabling anyting that raises red falgs to me | |
| 10:35:58 | mikal | but I am sure it has bugs like no HA -- its still a work in progress. My pain focus right now is working through load testing the SPICE native proxy so I can try and quantify if it needs tweaking and what scaling to recommend with it. | |
| 10:35:58 | mikal | noonedeadpunk: So its sort of half done -- there are now Nova APIs to create a console of type spice-direct, with openstacksdk and python-openstackclient support. Sound and USB are still missing (that's the patches above), so if that's a thing you care about you might need to hold your breath a bit. I have patches for kolla-ansible to deploy it, | |
| 10:36:47 | noonedeadpunk | mikal: not specifically care about spice-direct itself, but more wanted to implement that for OSA as well | |
| 10:36:56 | sean-k-mooney | mikal: ack | |
| 10:37:26 | mikal | noonedeadpunk: so OSA would need to know how to deploy the proxy and to twiddle some Nova config options. The existing Kolla-Ansible patches demonstrate how to do that. | |
| 10:37:57 | noonedeadpunk | mikal: do you have link top patches somewhere nearby? | |
| 10:38:32 | noonedeadpunk | there's support for spice-html5 but direct looks quite different | |
| 10:38:48 | mikal | express them but its what I have right now. | |
| 10:38:48 | mikal | noonedeadpunk: this is the list of patches I am carrying against Kolla-Ansible right now -- https://github.com/shakenfist/kerbside-patches/blob/develop/kolla-ansible/ORDER. There are also some minor patches against Kolla itself at https://github.com/shakenfist/kerbside-patches/blob/develop/kolla/ORDER. I'm sorry that's not the most helpful way to | |
| 10:39:17 | sean-k-mooney | noonedeadpunk: it is its using a out of tree proyx that mikal is developing called kerbside that actully allows raw spice conenction form a client to the vm | |
| 10:39:21 | mikal | noonedeadpunk: HTML5 and spice-direct can coexist I think, as long as you don't turn on the new require_secure option. The HTML5 client doesn't support TLS. | |
| 10:39:25 | sean-k-mooney | the proxy makes that safe | |
| 10:39:37 | sean-k-mooney | at least in thory | |
| 10:39:57 | mikal | noonedeadpunk: the proxy is at https://github.com/shakenfist/kerbside for what its worth. Like I said, the proxy itself is still a bit of a work in progress. | |
| 10:40:39 | noonedeadpunk | mikal: ok, so proxy needs to be started on control side I assume? | |
| 10:41:02 | noonedeadpunk | and then nova should be configured to point to it? | |
| 10:42:07 | mikal | into hypervisor / port and then proxies the SPICE traffic between the client and the hypervisor. | |
| 10:42:07 | mikal | noonedeadpunk: yes. The proxy sits running all the time and needs to be network accessible to your users. Nova is configured with the public URL to the proxy, and generates console access URLs using that URL template when a user creates a spice-direct console and then requests access to it. The proxy uses the console auth token to turn that back | |
| 10:42:52 | mikal | noonedeadpunk: https://github.com/shakenfist/kerbside-patches/blob/develop/tools/test-console#L88 is a worked example of that with a terrible fake client. | |
| 10:43:03 | noonedeadpunk | so in theory, L4 balancing can be added on top for public access? | |
| 10:43:55 | mikal | noonedeadpunk: yes. I expect the proxy would be clustered and it would be fine with traffic hitting multiple proxies. I suspect you'd want to try and keep a single client limited to a single backend for failure domain reasons, but apart from that it should just work. | |
| 10:44:01 | noonedeadpunk | ok, I should try making some patches, looks quite doable so far | |
| 10:44:08 | mikal | noonedeadpunk: state is in a central mysql database much like nova does. | |
| 10:44:31 | mikal | noonedeadpunk: by far my biggest concern is if the proxy is performant enough, but that's why that's the bit I am focused on right now. | |
| 10:44:47 | mikal | sean-k-mooney should still merge my patches though ;) | |
| 10:45:27 | mikal | (oVirt does something similar to all this, but they use squid as the proxy which means the client needs to know which hypervisor and port to connect to because its basically just doing SSL CONNECTs under the hood). | |
| 10:47:03 | noonedeadpunk | mikal: um... is the config respecting only ENV vars? | |
| 10:48:48 | mikal | noonedeadpunk: so kerbside itself isn't very "openstacky". Its written more like other personal projects of mine. It expects its configuration in environment variables, which can be in a systemd envvar file if that is less messy. Kolla-Ansible likes these start scripts, which is why it is done like that for those patches. It wouldn't be too hard to | |
| 10:48:49 | mikal | support a more traditional config file as well, its all just pydantic under the hood -- https://github.com/shakenfist/kerbside/blob/develop/kerbside/config.py | |
| 10:49:43 | noonedeadpunk | systemd envvar works as well, ofc. was just asking if I'm not missing more "conventional" config :) | |
| 10:50:13 | noonedeadpunk | And then servcice itself launches gunicorn? meaning https://github.com/shakenfist/kerbside/blob/053d068b0cc5b25a6263fe39a3548ca985b972ee/kerbside/api.py#L555-L563 | |
| 10:50:30 | mikal | noonedeadpunk: you didn't miss it. Pydantic can do YAML etc as well apparently https://docs.pydantic.dev/latest/concepts/pydantic_settings/#other-settings-source | |
| 10:50:36 | mikal | noonedeadpunk: correct. | |
| 10:50:40 | noonedeadpunk | so, like, it does not make sense to try to launch it with uWSGI or smth like that? | |
| 10:51:07 | noonedeadpunk | as if I can run service as wsgi - I'd rather do that... | |
| 10:51:24 | mikal | noonedeadpunk: maybe? The gunicorn bit is only for the REST API and these Nova generated access URLs, so its not particularly high traffic or anything. I am not opposed to supporting other runtimes, it just hasn't been a priority for me yet. | |
| 10:52:04 | mikal | noonedeadpunk: like I said my current focus is working through some sort of performance testing, because I'd say there's a 50% change kerbside is going to get a rewrite in golang or C or something. | |
| 10:52:24 | noonedeadpunk | right. | |
| 10:52:40 | noonedeadpunk | not sure if golang will help a lot, but yeah | |
| 10:52:48 | mikal | So it works right now, but I am unclear if it is performant enough. The proxy is basically a proof of concept right now. | |
| 10:56:59 | noonedeadpunk | From my prespective separation of uwsgi API and worker itself would be really nice to have. Probably bercause too much got used to how openstack is written :) | |
| 10:57:55 | mikal | Ok, I'll make a note to circle back to that once I have a stronger view of if a rewrite is required or not. | |
| 10:58:02 | noonedeadpunk | As for instance we have quite established way of running wsgi apps, as well as wiring around TLS generation and deployment for it | |
| 10:58:36 | mikal | Basically the focus so far has been on getting a proxy to work at all, given the poor state of SPICE protocol documentation, and then on landing the supporting nova patches. I am only just now circling back to productionizing the proxy. | |
| 10:58:45 | noonedeadpunk | yeah, right | |
| 10:59:03 | mikal | I don't know much about uwsgi. If you have best practise recommendations I'd be quite interested in those. | |
| 10:59:07 | noonedeadpunk | it's not that I'm blaming or anything, just commenting :) | |
| 10:59:48 | mikal | Yeah its all good. I'm mildly excited someone is considering deploying this at all. Its sometimes lonely over here. | |
| 11:00:00 | noonedeadpunk | we have a uwsgi role we're actively using, which produced config file like this one: https://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/templates/uwsgi.ini.j2 | |