| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-17 | |||
| 18:59:21 | sean-k-mooney | and more recently the service user token for the cve | |
| 19:00:28 | noonedeadpunk | Yeah, I guess one "but" for service user token - is that it's not since 24.0.0 as it was during Zed in fact and then backported... But anyway - it's small detail | |
| 19:03:24 | noonedeadpunk | I also wondering how nobody has raised that thing until now... As we've seen failure in CI right after the coordinated release, but nobody had time to look into the rootcause... | |
| 19:05:40 | sean-k-mooney | right so the upgrade check is ment to help catch this before you deploy to production | |
| 19:06:06 | sean-k-mooney | what this si also telling me is we dont actullly fail on error in our grenade jobs. | |
| 19:06:31 | sean-k-mooney | that would have caught this before it merged i think | |
| 19:10:24 | dansmith | well, tbh I think we're running it after upgrade to see if you've got everything done | |
| 19:10:28 | dansmith | and thus it's passing | |
| 19:10:49 | dansmith | it's supposed to be used for that as well.. basically "am I done with my homework" | |
| 19:11:46 | sean-k-mooney | ya that makes sense i gues although this would still fial in the multi node job in that case | |
| 19:12:00 | sean-k-mooney | since we only upgrade the contoler node and not the extra compute | |
| 19:26:10 | noonedeadpunk | Sorry, I think I have one more question. Today I got quite stuid idea, but wanna confirm how stupid it is :) | |
| 19:26:52 | noonedeadpunk | How bad is to run `nova-manage cell_v2 discover_hosts --by-service` if you don't have ironic | |
| 19:27:27 | noonedeadpunk | I see there's a performance penalty of doing that in the docs... | |
| 19:27:46 | noonedeadpunk | But that would simplify some logic quite a lot on the other hand... | |
| 19:30:21 | noonedeadpunk | So wondering how bad that trade off might be from the nova prespective | |
| 19:57:59 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Add functional tests to reproduce bug #1994983 https://review.opendev.org/c/openstack/nova/+/863416 | |
| 19:59:48 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Add functional tests to reproduce bug #1994983 https://review.opendev.org/c/openstack/nova/+/863416 | |
| 19:59:56 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Log some InstanceNotFound exceptions from libvirt https://review.opendev.org/c/openstack/nova/+/863665 | |
| 20:18:24 | colby__ | Hey Guys. We are in the process of upgrading our hypervisors to Yoga. We are on centos8 stream. Once I update openstack and qemu/libvirt packages instances are not able to be live migrated to the hypervisor. Im seeing the following: | |
| 20:18:26 | colby__ | qemu-kvm: Missing section footer for 0000:00:01.3/piix4_pm#0122023-10-17T19:07:20.896349Z qemu-kvm: load of migration failed: Invalid argument | |
| 20:19:09 | colby__ | did the commands to migrate change at all with yoga release? Where can I see that in the code? Im wondering if this is a qemu update issue | |
| 20:50:14 | opendevreview | Merged openstack/nova master: Install lxml before we need it in post-run https://review.opendev.org/c/openstack/nova/+/898435 | |
| #openstack-nova - 2023-10-18 | |||
| 07:14:54 | noonedeadpunk | colby__: there was deprecation of live_migration_tunnelled in favor of native libvirtd migrations (live_migration_with_native_tls). Not sure if that is the reason of your issue, but smth to check I guess | |
| 07:15:55 | noonedeadpunk | As it seems that live_migration_tunnelled were not fully dropped as of today | |
| 07:53:11 | tobias-urdin | (proposed removing that but was not ok in bobcat, maybe will be done in caracal https://review.opendev.org/c/openstack/nova/+/879021) | |
| 08:17:13 | bauzas | sean-k-mooney: dansmith: excellent news, my mtty WIP patch works with devstack and we can create instances https://paste.opendev.org/show/bBKFG3Vg7YTQBHPz11Za/ | |
| 08:17:28 | bauzas | if we pre-create mdevs of course | |
| 08:23:18 | bauzas | I just need to fix the RP name which is because of the <parent> value in the mtty :) | |
| 08:23:38 | bauzas | or we can just meh this | |
| 12:58:23 | sean-k-mooney | bauzas: cool i was conidering extending the devstack plugin to be able to optionally precrete the mdev and generate the approate config as required but i have not had time to play with it yet | |
| 12:59:01 | bauzas | sean-k-mooney: we have some internal but report that I created for RHEL asking to fix the libvirt problem | |
| 12:59:31 | bauzas | but until this is fixed (and delivered to Ubuntu 22.04 somehow), we may need to precreate them indeed | |
| 12:59:40 | bauzas | I can do the plugin thing | |
| 12:59:54 | bauzas | if you don't mind | |
| 13:00:12 | bauzas | I'm not super expert in writing CI things, and that would improve a bit my knowledge | |
| 13:00:44 | sean-k-mooney | well it would jsut be an extion to the devstack plugin so you can dev that locally | |
| 13:03:57 | sean-k-mooney | baisclly i was going to add something like this NOVA_MDEV_TYPE=${NOVA_MDEV_TYPES:-$(ls /sys/devices/virtual/*/*/mdev_supported_types/ | grep -v ":" | xargs)} | |
| 13:04:30 | sean-k-mooney | and then loop over the NOVA_MDEV_TYPE list and allcoate all mdevs for each enabeld type | |
| 13:04:59 | sean-k-mooney | so by default we would be enabling all mdev but if you wnated to just enabel 1 or a set you could define NOVA_MDEV_TYPE in yoru local.conf | |
| 13:05:19 | sean-k-mooney | that should be NOVA_MDEV_TYPES consitnetly as it woudl be a list | |
| 13:06:21 | bauzas | okay, I can try to work on it | |
| 13:06:38 | sean-k-mooney | addtionally i was going to write the mdev uuids into a file per mdev type and create a systemd service file to create them if we wanted to test reboots in a followup | |
| 13:07:03 | sean-k-mooney | so intally jsut do it at runtime and if we wnated to test reboot later add that as a follow up | |
| 13:08:18 | sean-k-mooney | we can test reboots in ci we normally dont because it slows down the job a non trivial amount | |
| 13:09:39 | bauzas | cool | |
| 13:39:31 | bauzas | dansmith: https://paste.opendev.org/show/bBKFG3Vg7YTQBHPz11Za/ (at the bottom) is what you need to do in order to configure nova to use mtty | |
| 13:40:07 | bauzas | tl;dr: instead of specifying a pci address for the type named 'mtty-1', you just write 'mtty' | |
| 13:40:41 | bauzas | but in order to pass the pci address check we have, I hardcoded that string (plus the other module names) in https://review.opendev.org/c/openstack/nova/+/898100/2/nova/virt/libvirt/driver.py#245 | |
| 13:40:41 | sean-k-mooney | ya so i woudl be happy with sbauza-dev2_computer being sbauza-dev2_mtty | |
| 13:41:10 | bauzas | and sean-k-mooney had concerns about hardcoding that list | |
| 13:41:18 | bauzas | which I can understand | |
| 13:41:37 | sean-k-mooney | well partly becasue you dont even supprot all the modules that dan is compiling in the plugin | |
| 13:41:58 | dansmith | we can make up that list from looking in sysfs right? | |
| 13:42:04 | sean-k-mooney | so at a minium i want all of them to be valid but i woudl prefer it to be a config option in the [devices] section | |
| 13:42:14 | sean-k-mooney | dansmith: yes we can do that too | |
| 13:42:22 | dansmith | I guess if the operator might want to limit some devices to not be picked up by nova, then there's a legit reason to have a conf option | |
| 13:42:41 | bauzas | sean-k-mooney: ack I can try to change https://review.opendev.org/c/openstack/nova/+/898100/2/nova/virt/libvirt/driver.py#8065 for not using the "parent" name | |
| 13:42:43 | dansmith | I'm just a bit over having to config everything by hand that we should be able to discover | |
| 13:42:59 | bauzas | me too | |
| 13:43:08 | sean-k-mooney | dansmith: so unless we enable an mdev type form the device | |
| 13:43:09 | bauzas | so basically I let you guys decide | |
| 13:43:16 | sean-k-mooney | we shoudl not be reporting the mtty devices anyway | |
| 13:43:23 | sean-k-mooney | so im not sure we need to configure things at all | |
| 13:43:59 | sean-k-mooney | basically if you dont have | |
| 13:44:03 | sean-k-mooney | [devices] | |
| 13:44:04 | sean-k-mooney | enabled_mdev_types = mtty-1 | |
| 13:44:06 | sean-k-mooney | [mdev_mtty-1] | |
| 13:44:08 | sean-k-mooney | device_addresses = mtty | |
| 13:44:19 | sean-k-mooney | then the mtty rp shoudl not be created right | |
| 13:44:20 | bauzas | so, shall we just do an exec on lsmod and see whether mtty is loaded ? | |
| 13:44:22 | dansmith | ah right right | |
| 13:44:29 | dansmith | bauzas: don't be silly | |
| 13:44:31 | sean-k-mooney | bauzas: hell no | |
| 13:44:31 | bauzas | yeah, correct | |
| 13:44:50 | bauzas | about the fact that no RPs are created until you configure nova.confg | |
| 13:44:59 | sean-k-mooney | do me the fact device_addresses is not a pci adress shoudl be all you need | |
| 13:45:06 | dansmith | sean-k-mooney: so discover the parents but still obviously let them configure them specifically | |
| 13:45:07 | sean-k-mooney | no need for hardcoded sentineal | |
| 13:45:29 | dansmith | configure the devices specifically I mean | |
| 13:45:30 | bauzas | sean-k-mooney: if we do this, we would remove the pci check and let anyone do fatfingering | |
| 13:45:40 | bauzas | I'm not sure I like that | |
| 13:46:05 | bauzas | https://review.opendev.org/c/openstack/nova/+/898100/2/nova/virt/libvirt/driver.py#7974 is where we parse config | |
| 13:46:31 | sean-k-mooney | what i was orginally thinkgig is as follows | |
| 13:46:41 | sean-k-mooney | if device_addresses does not container a pci adress | |
| 13:46:52 | dansmith | being only able to handle pci devices seems too specific to me anyway | |
| 13:48:19 | bauzas | dansmith: so you'd be in favor of lifting this all ? | |
| 13:48:38 | sean-k-mooney | check if the value of device_addresses is in /sys/devices/virtual/<device_address> | |
| 13:48:50 | bauzas | tbc, there are some assumptions where we say that we have a PCI address | |
| 13:49:14 | bauzas | sean-k-mooney: ah, good idea | |
| 13:49:34 | sean-k-mooney | if that folder does not exsit we can raise that as a config error in init_host | |
| 13:49:44 | bauzas | that's a readonly call, we don't need to privsep call it | |
| 13:50:02 | bauzas | and that's a cheap I/O call only made at startup | |
| 13:50:09 | bauzas | wfm so | |
| 13:50:16 | sean-k-mooney | ya its listraly just a file exist check | |
| 13:50:21 | dansmith | bauzas: but is that really ideal? I mean, it's a bit of a reach, but there are other busses on other systems, and I can totally imagine this sort of virt-friendly hardware someday exposing something totally different there | |
| 13:50:44 | dansmith | (that == being so tied to pci addresses specifically) | |
| 13:50:45 | sean-k-mooney | dansmith: you are correct that mdev does not require pci device parents | |
| 13:51:00 | bauzas | dansmith: I don't disagree with you long-term, but fwiw, the only vfio-mdev devices I've seen rely on being PCI addresses | |
| 13:51:02 | sean-k-mooney | that is an atritifal limiation in novas code base today to simplfy the code | |