| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-17 | |||
| 18:46:38 | sean-k-mooney | so cn_no_service is the list of all compute nodes that have service id set to None | |
| 18:46:46 | dansmith | sean-k-mooney: it's really more applicable to the release where we start requiring it | |
| 18:47:11 | sean-k-mooney | dansmith: ya it is. im wonderign what happens for ironic in this case | |
| 18:47:59 | dansmith | idk. I have a big "if ironic: return" in my head over all that stuff :P | |
| 18:48:47 | sean-k-mooney | i think this will always fail if you have ironic | |
| 18:48:57 | sean-k-mooney | but i am also pretty sure we have not test coverage | |
| 18:49:00 | noonedeadpunk | https://bugs.launchpad.net/nova/+bug/2039597 | |
| 18:50:02 | sean-k-mooney | as in, in the compute manager i think your correect we have "if ironic: return" so i dont think we set the service id for ironic computes | |
| 18:50:35 | dansmith | sean-k-mooney: no I meant I have that conditional in my _brain_ :) | |
| 18:50:45 | sean-k-mooney | oh hehe | |
| 18:51:29 | dansmith | okay well, I have the FAIL->WARN thing queued, I can either submit that or change to a revert | |
| 18:53:11 | sean-k-mooney | given this converstaion im inclided to revert | |
| 18:53:37 | sean-k-mooney | and then when we actully start depenign on this we can decide if we shoudl add a status check | |
| 18:53:47 | sean-k-mooney | as you suggested before | |
| 18:53:58 | noonedeadpunk | I mean. I can ignore WARNs and just make them pass.... But what value of warns then... | |
| 18:54:32 | noonedeadpunk | (or depending on the source version for upgrade for 2024.1 | |
| 18:54:52 | sean-k-mooney | noonedeadpunk: no warning are ment to be "this thing is deprecated and you should fix it before upgrading again" | |
| 18:55:00 | dansmith | sean-k-mooney: well, maybe comment on that in the bug and we can get bauzas to opine tomorrow. ISTR he was pro check as well, but maybe only because you said it and it "seems right" :) | |
| 18:55:48 | noonedeadpunk | sean-k-mooney: then it's worth fixing docs to state that | |
| 18:55:52 | sean-k-mooney | if we keep the check we need to filter out ironic compute nodes too and or add the min compute version cehck | |
| 18:56:04 | sean-k-mooney | but sure ill comment on the bug | |
| 18:56:56 | sean-k-mooney | noonedeadpunk: to be honest we try to avoid requiuring the operator to do things to be able to upgrade | |
| 18:56:59 | noonedeadpunk | As right now reading the doc it sounds a bit different then it's fine now - but will be not next time | |
| 18:57:27 | sean-k-mooney | so we very really add nova-status checks becuase we try to amke sure not to design something to require them | |
| 18:58:20 | sean-k-mooney | s/really/rarely/ | |
| 18:58:21 | noonedeadpunk | and you're doing great job to have that said | |
| 18:59:08 | sean-k-mooney | if you look at the first copule they are things like you must have placment, or cells v2 | |
| 18:59:13 | noonedeadpunk | And I'm not insisting that we're doing things right - it's just we're doing them how it's written more or less... And if you say we should do things differently - I'm really fine with that, and can propose update to docs as well | |
| 18:59:21 | sean-k-mooney | and more recently the service user token for the cve | |
| 19:00:28 | noonedeadpunk | Yeah, I guess one "but" for service user token - is that it's not since 24.0.0 as it was during Zed in fact and then backported... But anyway - it's small detail | |
| 19:03:24 | noonedeadpunk | I also wondering how nobody has raised that thing until now... As we've seen failure in CI right after the coordinated release, but nobody had time to look into the rootcause... | |
| 19:05:40 | sean-k-mooney | right so the upgrade check is ment to help catch this before you deploy to production | |
| 19:06:06 | sean-k-mooney | what this si also telling me is we dont actullly fail on error in our grenade jobs. | |
| 19:06:31 | sean-k-mooney | that would have caught this before it merged i think | |
| 19:10:24 | dansmith | well, tbh I think we're running it after upgrade to see if you've got everything done | |
| 19:10:28 | dansmith | and thus it's passing | |
| 19:10:49 | dansmith | it's supposed to be used for that as well.. basically "am I done with my homework" | |
| 19:11:46 | sean-k-mooney | ya that makes sense i gues although this would still fial in the multi node job in that case | |
| 19:12:00 | sean-k-mooney | since we only upgrade the contoler node and not the extra compute | |
| 19:26:10 | noonedeadpunk | Sorry, I think I have one more question. Today I got quite stuid idea, but wanna confirm how stupid it is :) | |
| 19:26:52 | noonedeadpunk | How bad is to run `nova-manage cell_v2 discover_hosts --by-service` if you don't have ironic | |
| 19:27:27 | noonedeadpunk | I see there's a performance penalty of doing that in the docs... | |
| 19:27:46 | noonedeadpunk | But that would simplify some logic quite a lot on the other hand... | |
| 19:30:21 | noonedeadpunk | So wondering how bad that trade off might be from the nova prespective | |
| 19:57:59 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Add functional tests to reproduce bug #1994983 https://review.opendev.org/c/openstack/nova/+/863416 | |
| 19:59:48 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Add functional tests to reproduce bug #1994983 https://review.opendev.org/c/openstack/nova/+/863416 | |
| 19:59:56 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Log some InstanceNotFound exceptions from libvirt https://review.opendev.org/c/openstack/nova/+/863665 | |
| 20:18:24 | colby__ | Hey Guys. We are in the process of upgrading our hypervisors to Yoga. We are on centos8 stream. Once I update openstack and qemu/libvirt packages instances are not able to be live migrated to the hypervisor. Im seeing the following: | |
| 20:18:26 | colby__ | qemu-kvm: Missing section footer for 0000:00:01.3/piix4_pm#0122023-10-17T19:07:20.896349Z qemu-kvm: load of migration failed: Invalid argument | |
| 20:19:09 | colby__ | did the commands to migrate change at all with yoga release? Where can I see that in the code? Im wondering if this is a qemu update issue | |
| 20:50:14 | opendevreview | Merged openstack/nova master: Install lxml before we need it in post-run https://review.opendev.org/c/openstack/nova/+/898435 | |
| #openstack-nova - 2023-10-18 | |||
| 07:14:54 | noonedeadpunk | colby__: there was deprecation of live_migration_tunnelled in favor of native libvirtd migrations (live_migration_with_native_tls). Not sure if that is the reason of your issue, but smth to check I guess | |
| 07:15:55 | noonedeadpunk | As it seems that live_migration_tunnelled were not fully dropped as of today | |
| 07:53:11 | tobias-urdin | (proposed removing that but was not ok in bobcat, maybe will be done in caracal https://review.opendev.org/c/openstack/nova/+/879021) | |
| 08:17:13 | bauzas | sean-k-mooney: dansmith: excellent news, my mtty WIP patch works with devstack and we can create instances https://paste.opendev.org/show/bBKFG3Vg7YTQBHPz11Za/ | |
| 08:17:28 | bauzas | if we pre-create mdevs of course | |
| 08:23:18 | bauzas | I just need to fix the RP name which is because of the <parent> value in the mtty :) | |
| 08:23:38 | bauzas | or we can just meh this | |
| 12:58:23 | sean-k-mooney | bauzas: cool i was conidering extending the devstack plugin to be able to optionally precrete the mdev and generate the approate config as required but i have not had time to play with it yet | |
| 12:59:01 | bauzas | sean-k-mooney: we have some internal but report that I created for RHEL asking to fix the libvirt problem | |
| 12:59:31 | bauzas | but until this is fixed (and delivered to Ubuntu 22.04 somehow), we may need to precreate them indeed | |
| 12:59:40 | bauzas | I can do the plugin thing | |
| 12:59:54 | bauzas | if you don't mind | |
| 13:00:12 | bauzas | I'm not super expert in writing CI things, and that would improve a bit my knowledge | |
| 13:00:44 | sean-k-mooney | well it would jsut be an extion to the devstack plugin so you can dev that locally | |
| 13:03:57 | sean-k-mooney | baisclly i was going to add something like this NOVA_MDEV_TYPE=${NOVA_MDEV_TYPES:-$(ls /sys/devices/virtual/*/*/mdev_supported_types/ | grep -v ":" | xargs)} | |
| 13:04:30 | sean-k-mooney | and then loop over the NOVA_MDEV_TYPE list and allcoate all mdevs for each enabeld type | |
| 13:04:59 | sean-k-mooney | so by default we would be enabling all mdev but if you wnated to just enabel 1 or a set you could define NOVA_MDEV_TYPE in yoru local.conf | |
| 13:05:19 | sean-k-mooney | that should be NOVA_MDEV_TYPES consitnetly as it woudl be a list | |
| 13:06:21 | bauzas | okay, I can try to work on it | |
| 13:06:38 | sean-k-mooney | addtionally i was going to write the mdev uuids into a file per mdev type and create a systemd service file to create them if we wanted to test reboots in a followup | |
| 13:07:03 | sean-k-mooney | so intally jsut do it at runtime and if we wnated to test reboot later add that as a follow up | |
| 13:08:18 | sean-k-mooney | we can test reboots in ci we normally dont because it slows down the job a non trivial amount | |
| 13:09:39 | bauzas | cool | |
| 13:39:31 | bauzas | dansmith: https://paste.opendev.org/show/bBKFG3Vg7YTQBHPz11Za/ (at the bottom) is what you need to do in order to configure nova to use mtty | |
| 13:40:07 | bauzas | tl;dr: instead of specifying a pci address for the type named 'mtty-1', you just write 'mtty' | |
| 13:40:41 | sean-k-mooney | ya so i woudl be happy with sbauza-dev2_computer being sbauza-dev2_mtty | |
| 13:40:41 | bauzas | but in order to pass the pci address check we have, I hardcoded that string (plus the other module names) in https://review.opendev.org/c/openstack/nova/+/898100/2/nova/virt/libvirt/driver.py#245 | |
| 13:41:10 | bauzas | and sean-k-mooney had concerns about hardcoding that list | |
| 13:41:18 | bauzas | which I can understand | |
| 13:41:37 | sean-k-mooney | well partly becasue you dont even supprot all the modules that dan is compiling in the plugin | |
| 13:41:58 | dansmith | we can make up that list from looking in sysfs right? | |
| 13:42:04 | sean-k-mooney | so at a minium i want all of them to be valid but i woudl prefer it to be a config option in the [devices] section | |
| 13:42:14 | sean-k-mooney | dansmith: yes we can do that too | |
| 13:42:22 | dansmith | I guess if the operator might want to limit some devices to not be picked up by nova, then there's a legit reason to have a conf option | |
| 13:42:41 | bauzas | sean-k-mooney: ack I can try to change https://review.opendev.org/c/openstack/nova/+/898100/2/nova/virt/libvirt/driver.py#8065 for not using the "parent" name | |
| 13:42:43 | dansmith | I'm just a bit over having to config everything by hand that we should be able to discover | |
| 13:42:59 | bauzas | me too | |
| 13:43:08 | sean-k-mooney | dansmith: so unless we enable an mdev type form the device | |
| 13:43:09 | bauzas | so basically I let you guys decide | |
| 13:43:16 | sean-k-mooney | we shoudl not be reporting the mtty devices anyway | |
| 13:43:23 | sean-k-mooney | so im not sure we need to configure things at all | |
| 13:43:59 | sean-k-mooney | basically if you dont have | |
| 13:44:03 | sean-k-mooney | [devices] | |
| 13:44:04 | sean-k-mooney | enabled_mdev_types = mtty-1 | |
| 13:44:06 | sean-k-mooney | [mdev_mtty-1] | |
| 13:44:08 | sean-k-mooney | device_addresses = mtty | |
| 13:44:19 | sean-k-mooney | then the mtty rp shoudl not be created right | |
| 13:44:20 | bauzas | so, shall we just do an exec on lsmod and see whether mtty is loaded ? | |
| 13:44:22 | dansmith | ah right right | |