| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-17 | |||
| 18:38:32 | noonedeadpunk | just to state this one more time - warning will fire in 100% cases, right? | |
| 18:39:04 | sean-k-mooney | hum yes until the upgrade is compelte | |
| 18:39:09 | dansmith | yes | |
| 18:39:34 | noonedeadpunk | And what docs say - `At least one check encountered an issue and requires further investigation.` So all who do upgrades will spend some time to find out that they should proceed first when it's not inclined but the command nature? | |
| 18:39:43 | dansmith | it wouldn't make much sense to say "you pass this test with OK" and then after you restart it goes to warning or error | |
| 18:39:56 | dansmith | if we had a way to skip a test because it's irrelevant, then we'd do that, otherwise I think warning makes sense | |
| 18:39:58 | noonedeadpunk | it does on 2024.1 | |
| 18:40:05 | noonedeadpunk | if you upgrade from 2023.2 | |
| 18:40:44 | sean-k-mooney | dansmith: if we modify the check to do a min compute version check it might but then perhasp i was wrong to ask for this as its not as useful as i tought | |
| 18:40:45 | dansmith | does what? make sense? I disagree | |
| 18:41:16 | sean-k-mooney | noonedeadpunk: baislly i asked for this to catch the case where you are upgrading aging and had not started the computed i.e. to 2024.2 i guess | |
| 18:41:33 | noonedeadpunk | but then I'm really not getting what are expectations on me as operator to perform upgrade from 2023.1 to 2023.2 and what I should do when I see that warning | |
| 18:42:26 | dansmith | saying we checked a thing and everything is good when we really checked and see that you're not but we're ignoring the done-ness of it because you're in the middle of an upgrade doesn't seem to fit very well to me | |
| 18:42:36 | dansmith | like I say if we had "skipped" or "not due yet" then that'd make sense | |
| 18:42:37 | noonedeadpunk | yeah, I kind of already got what it does and why exist :) just trying to say that raise warning for expected behaviour is also... meh? | |
| 18:45:09 | sean-k-mooney | i wonder if it would be beter to remove this check. i know i asked for it orginally but im wondering if its adding benifit. im also wondering if there is another bug | |
| 18:46:07 | sean-k-mooney | we are doing " cn_no_service = main_db_api.compute_nodes_get_by_service_id(ctx, None) | |
| 18:46:12 | sean-k-mooney | " | |
| 18:46:38 | sean-k-mooney | so cn_no_service is the list of all compute nodes that have service id set to None | |
| 18:46:46 | dansmith | sean-k-mooney: it's really more applicable to the release where we start requiring it | |
| 18:47:11 | sean-k-mooney | dansmith: ya it is. im wonderign what happens for ironic in this case | |
| 18:47:59 | dansmith | idk. I have a big "if ironic: return" in my head over all that stuff :P | |
| 18:48:47 | sean-k-mooney | i think this will always fail if you have ironic | |
| 18:48:57 | sean-k-mooney | but i am also pretty sure we have not test coverage | |
| 18:49:00 | noonedeadpunk | https://bugs.launchpad.net/nova/+bug/2039597 | |
| 18:50:02 | sean-k-mooney | as in, in the compute manager i think your correect we have "if ironic: return" so i dont think we set the service id for ironic computes | |
| 18:50:35 | dansmith | sean-k-mooney: no I meant I have that conditional in my _brain_ :) | |
| 18:50:45 | sean-k-mooney | oh hehe | |
| 18:51:29 | dansmith | okay well, I have the FAIL->WARN thing queued, I can either submit that or change to a revert | |
| 18:53:11 | sean-k-mooney | given this converstaion im inclided to revert | |
| 18:53:37 | sean-k-mooney | and then when we actully start depenign on this we can decide if we shoudl add a status check | |
| 18:53:47 | sean-k-mooney | as you suggested before | |
| 18:53:58 | noonedeadpunk | I mean. I can ignore WARNs and just make them pass.... But what value of warns then... | |
| 18:54:32 | noonedeadpunk | (or depending on the source version for upgrade for 2024.1 | |
| 18:54:52 | sean-k-mooney | noonedeadpunk: no warning are ment to be "this thing is deprecated and you should fix it before upgrading again" | |
| 18:55:00 | dansmith | sean-k-mooney: well, maybe comment on that in the bug and we can get bauzas to opine tomorrow. ISTR he was pro check as well, but maybe only because you said it and it "seems right" :) | |
| 18:55:48 | noonedeadpunk | sean-k-mooney: then it's worth fixing docs to state that | |
| 18:55:52 | sean-k-mooney | if we keep the check we need to filter out ironic compute nodes too and or add the min compute version cehck | |
| 18:56:04 | sean-k-mooney | but sure ill comment on the bug | |
| 18:56:56 | sean-k-mooney | noonedeadpunk: to be honest we try to avoid requiuring the operator to do things to be able to upgrade | |
| 18:56:59 | noonedeadpunk | As right now reading the doc it sounds a bit different then it's fine now - but will be not next time | |
| 18:57:27 | sean-k-mooney | so we very really add nova-status checks becuase we try to amke sure not to design something to require them | |
| 18:58:20 | sean-k-mooney | s/really/rarely/ | |
| 18:58:21 | noonedeadpunk | and you're doing great job to have that said | |
| 18:59:08 | sean-k-mooney | if you look at the first copule they are things like you must have placment, or cells v2 | |
| 18:59:13 | noonedeadpunk | And I'm not insisting that we're doing things right - it's just we're doing them how it's written more or less... And if you say we should do things differently - I'm really fine with that, and can propose update to docs as well | |
| 18:59:21 | sean-k-mooney | and more recently the service user token for the cve | |
| 19:00:28 | noonedeadpunk | Yeah, I guess one "but" for service user token - is that it's not since 24.0.0 as it was during Zed in fact and then backported... But anyway - it's small detail | |
| 19:03:24 | noonedeadpunk | I also wondering how nobody has raised that thing until now... As we've seen failure in CI right after the coordinated release, but nobody had time to look into the rootcause... | |
| 19:05:40 | sean-k-mooney | right so the upgrade check is ment to help catch this before you deploy to production | |
| 19:06:06 | sean-k-mooney | what this si also telling me is we dont actullly fail on error in our grenade jobs. | |
| 19:06:31 | sean-k-mooney | that would have caught this before it merged i think | |
| 19:10:24 | dansmith | well, tbh I think we're running it after upgrade to see if you've got everything done | |
| 19:10:28 | dansmith | and thus it's passing | |
| 19:10:49 | dansmith | it's supposed to be used for that as well.. basically "am I done with my homework" | |
| 19:11:46 | sean-k-mooney | ya that makes sense i gues although this would still fial in the multi node job in that case | |
| 19:12:00 | sean-k-mooney | since we only upgrade the contoler node and not the extra compute | |
| 19:26:10 | noonedeadpunk | Sorry, I think I have one more question. Today I got quite stuid idea, but wanna confirm how stupid it is :) | |
| 19:26:52 | noonedeadpunk | How bad is to run `nova-manage cell_v2 discover_hosts --by-service` if you don't have ironic | |
| 19:27:27 | noonedeadpunk | I see there's a performance penalty of doing that in the docs... | |
| 19:27:46 | noonedeadpunk | But that would simplify some logic quite a lot on the other hand... | |
| 19:30:21 | noonedeadpunk | So wondering how bad that trade off might be from the nova prespective | |
| 19:57:59 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Add functional tests to reproduce bug #1994983 https://review.opendev.org/c/openstack/nova/+/863416 | |
| 19:59:48 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Add functional tests to reproduce bug #1994983 https://review.opendev.org/c/openstack/nova/+/863416 | |
| 19:59:56 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Log some InstanceNotFound exceptions from libvirt https://review.opendev.org/c/openstack/nova/+/863665 | |
| 20:18:24 | colby__ | Hey Guys. We are in the process of upgrading our hypervisors to Yoga. We are on centos8 stream. Once I update openstack and qemu/libvirt packages instances are not able to be live migrated to the hypervisor. Im seeing the following: | |
| 20:18:26 | colby__ | qemu-kvm: Missing section footer for 0000:00:01.3/piix4_pm#0122023-10-17T19:07:20.896349Z qemu-kvm: load of migration failed: Invalid argument | |
| 20:19:09 | colby__ | did the commands to migrate change at all with yoga release? Where can I see that in the code? Im wondering if this is a qemu update issue | |
| 20:50:14 | opendevreview | Merged openstack/nova master: Install lxml before we need it in post-run https://review.opendev.org/c/openstack/nova/+/898435 | |
| #openstack-nova - 2023-10-18 | |||
| 07:14:54 | noonedeadpunk | colby__: there was deprecation of live_migration_tunnelled in favor of native libvirtd migrations (live_migration_with_native_tls). Not sure if that is the reason of your issue, but smth to check I guess | |
| 07:15:55 | noonedeadpunk | As it seems that live_migration_tunnelled were not fully dropped as of today | |
| 07:53:11 | tobias-urdin | (proposed removing that but was not ok in bobcat, maybe will be done in caracal https://review.opendev.org/c/openstack/nova/+/879021) | |
| 08:17:13 | bauzas | sean-k-mooney: dansmith: excellent news, my mtty WIP patch works with devstack and we can create instances https://paste.opendev.org/show/bBKFG3Vg7YTQBHPz11Za/ | |
| 08:17:28 | bauzas | if we pre-create mdevs of course | |
| 08:23:18 | bauzas | I just need to fix the RP name which is because of the <parent> value in the mtty :) | |
| 08:23:38 | bauzas | or we can just meh this | |
| 12:58:23 | sean-k-mooney | bauzas: cool i was conidering extending the devstack plugin to be able to optionally precrete the mdev and generate the approate config as required but i have not had time to play with it yet | |
| 12:59:01 | bauzas | sean-k-mooney: we have some internal but report that I created for RHEL asking to fix the libvirt problem | |
| 12:59:31 | bauzas | but until this is fixed (and delivered to Ubuntu 22.04 somehow), we may need to precreate them indeed | |
| 12:59:40 | bauzas | I can do the plugin thing | |
| 12:59:54 | bauzas | if you don't mind | |
| 13:00:12 | bauzas | I'm not super expert in writing CI things, and that would improve a bit my knowledge | |
| 13:00:44 | sean-k-mooney | well it would jsut be an extion to the devstack plugin so you can dev that locally | |
| 13:03:57 | sean-k-mooney | baisclly i was going to add something like this NOVA_MDEV_TYPE=${NOVA_MDEV_TYPES:-$(ls /sys/devices/virtual/*/*/mdev_supported_types/ | grep -v ":" | xargs)} | |
| 13:04:30 | sean-k-mooney | and then loop over the NOVA_MDEV_TYPE list and allcoate all mdevs for each enabeld type | |
| 13:04:59 | sean-k-mooney | so by default we would be enabling all mdev but if you wnated to just enabel 1 or a set you could define NOVA_MDEV_TYPE in yoru local.conf | |
| 13:05:19 | sean-k-mooney | that should be NOVA_MDEV_TYPES consitnetly as it woudl be a list | |
| 13:06:21 | bauzas | okay, I can try to work on it | |
| 13:06:38 | sean-k-mooney | addtionally i was going to write the mdev uuids into a file per mdev type and create a systemd service file to create them if we wanted to test reboots in a followup | |
| 13:07:03 | sean-k-mooney | so intally jsut do it at runtime and if we wnated to test reboot later add that as a follow up | |
| 13:08:18 | sean-k-mooney | we can test reboots in ci we normally dont because it slows down the job a non trivial amount | |
| 13:09:39 | bauzas | cool | |
| 13:39:31 | bauzas | dansmith: https://paste.opendev.org/show/bBKFG3Vg7YTQBHPz11Za/ (at the bottom) is what you need to do in order to configure nova to use mtty | |
| 13:40:07 | bauzas | tl;dr: instead of specifying a pci address for the type named 'mtty-1', you just write 'mtty' | |
| 13:40:41 | sean-k-mooney | ya so i woudl be happy with sbauza-dev2_computer being sbauza-dev2_mtty | |
| 13:40:41 | bauzas | but in order to pass the pci address check we have, I hardcoded that string (plus the other module names) in https://review.opendev.org/c/openstack/nova/+/898100/2/nova/virt/libvirt/driver.py#245 | |
| 13:41:10 | bauzas | and sean-k-mooney had concerns about hardcoding that list | |
| 13:41:18 | bauzas | which I can understand | |
| 13:41:37 | sean-k-mooney | well partly becasue you dont even supprot all the modules that dan is compiling in the plugin | |
| 13:41:58 | dansmith | we can make up that list from looking in sysfs right? | |