| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-17 | |||
| 13:31:05 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Translate VF network capabilities to port binding https://review.opendev.org/c/openstack/nova/+/884439 | |
| 13:40:38 | opendevreview | Tobias Urdin proposed openstack/nova master: [WIP] Handle scaling of cputune.shares https://review.opendev.org/c/openstack/nova/+/898326 | |
| 13:41:22 | sean-k-mooney | tobias-urdin: thisis not something we can do in nova | |
| 13:41:54 | sean-k-mooney | tobias-urdin: we can discuss it again but we considerd that option and rejected it before | |
| 13:42:41 | sean-k-mooney | tobias-urdin: its not jsut eh cpu_shares that woudl need to be adjusted | |
| 14:00:59 | opendevreview | Tobias Urdin proposed openstack/nova master: [WIP] Handle scaling of cputune.shares https://review.opendev.org/c/openstack/nova/+/898326 | |
| 14:01:40 | tobias-urdin | sean-k-mooney: are you thinking about quota:cpu_shares flavor extra spec as well? | |
| 14:02:24 | sean-k-mooney | tobias-urdin: yes | |
| 14:02:40 | tobias-urdin | i don't understand the reasoning for "libvirt broke us let's remove the default completely" | |
| 14:02:43 | sean-k-mooney | so on master (since zed) we nolonger genreate cpu_shares implictly | |
| 14:02:52 | sean-k-mooney | tobias-urdin: it was nto libvirt | |
| 14:03:04 | sean-k-mooney | this was cause by your kernel being compiled with cgroups_v2 | |
| 14:03:24 | sean-k-mooney | it was broken by the kernel team changeing the allowed ranges between api versions | |
| 14:05:45 | sean-k-mooney | tobias-urdin: neither nova nor libvirt provided any normalistaion of values menaing the admin is responsible for selecting values that are allowed by there kernel cgroup version | |
| 14:09:22 | tobias-urdin | while that is true and libvirt never normalized the values, i don't agree, libvirt is an abstraction and should've handled it | |
| 14:09:50 | sean-k-mooney | tobias-urdin: we made that argument to the libvirt maintainer and they disagreed | |
| 14:12:14 | sean-k-mooney | tobias-urdin: https://bugs.launchpad.net/nova/+bug/1960840 is related | |
| 14:12:33 | sean-k-mooney | we considerd extneding the flavor validation https://review.opendev.org/c/openstack/nova/+/829064 | |
| 14:13:00 | sean-k-mooney | however part of the problem is the range depend on the virtdriver in use and the cgroup version in the libvirt case | |
| 14:14:36 | sean-k-mooney | in that case actully the limit is a tc one rather the cgroup but we considerd deprecting and removing these as a result | |
| 14:15:07 | sean-k-mooney | that simpler to do with the vif quotas since they generally nolonger fucntion | |
| 14:15:18 | sean-k-mooney | and have a neutron replacemnt that does | |
| 14:17:24 | tobias-urdin | as an operator i basically have two options 1) apply patch https://review.opendev.org/c/openstack/nova/+/824048 and just live with the behaviour change and larger instances does not get favored for oversubscription and i have to manually update cpu_shares or backport that patch to yoga to move to zed with live migration or 2) scale the value and | |
| 14:17:24 | tobias-urdin | set correct on all current instances and live migrate them over and make sure new instances also gets a scaled value | |
| 14:17:52 | tobias-urdin | for an operator where touching the code is an issue this might be very messy | |
| 14:18:28 | tobias-urdin | i opted for adding it as a workaround and upon nova-compute startup fix the values, exactly because of the upgrade issue | |
| 14:18:39 | sean-k-mooney | so im not against backporting the disabling of the implict cpu_share request for what its worht | |
| 14:18:59 | sean-k-mooney | tobias-urdin: we have dont this back to wallaby downstream because we considerd it a release blocker for our product | |
| 14:19:31 | sean-k-mooney | tobias-urdin: we dont have any other code that modifes a guest on startup like that and im not sure hta tis a patten we should follow in general | |
| 14:21:19 | sean-k-mooney | tobias-urdin: if we were to backport https://review.opendev.org/c/openstack/nova/+/824048 to yoga would that solve your issue | |
| 14:22:41 | tobias-urdin | it feel like kind of a limbo, if libvirt maintainers indeed informed that this will never change then the only way forward is fix applications or drop it, i'm just surpised we went for drop it for a default value that has been there since years(?) | |
| 14:23:33 | tobias-urdin | personally i would like it backported but i dont know if stable backport policy covers performance impact as well? | |
| 14:27:08 | opendevreview | Amit Uniyal proposed openstack/nova-specs master: WIP: Enforce console session timeout https://review.opendev.org/c/openstack/nova-specs/+/898553 | |
| 14:29:26 | opendevreview | Tobias Urdin proposed openstack/nova stable/yoga: libvirt: remove default cputune shares value https://review.opendev.org/c/openstack/nova/+/898554 | |
| 14:31:41 | tobias-urdin | i guess ^ and then hope that operators take the proper action to update cpu_shares to default value or live migrate to get rid of it, annoying that nothing with this is optimal | |
| 14:33:03 | tobias-urdin | a shame that libvirt didn't even try by introducing a cputune.cpu_weight, deprecating and scaling the cputune.cpu_shares value if required, like systemd did and took backward compatibility more seriously | |
| 14:33:10 | tobias-urdin | / end of rant :p | |
| 14:50:05 | bauzas | reminder : nova meeting in 1 hour and 10 mins here | |
| 15:43:25 | opendevreview | Takashi Kajinami proposed openstack/nova master: Fix python shebang https://review.opendev.org/c/openstack/nova/+/898594 | |
| 15:46:59 | opendevreview | Merged openstack/nova stable/wallaby: Accept both 1 and Y as AMD SEV KVM kernel param value https://review.opendev.org/c/openstack/nova/+/843939 | |
| 16:01:18 | bauzas | #startmeeting nova | |
| 16:01:18 | opendevmeet | Meeting started Tue Oct 17 16:01:18 2023 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:01:18 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:01:18 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:01:21 | bauzas | hey folsk | |
| 16:01:24 | dansmith | o/ | |
| 16:01:27 | elodilles | o/ | |
| 16:01:30 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:02:38 | auniyal | 0/\ | |
| 16:02:52 | auniyal | o/ | |
| 16:02:57 | bauzas | okay, let's start | |
| 16:03:02 | bauzas | #topic Bugs (stuck/critical) | |
| 16:03:04 | bauzas | #info No Critical bug | |
| 16:03:09 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 47 new untriaged bugs (+1 since the last meeting) | |
| 16:03:15 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:03:34 | Uggla | o/ | |
| 16:03:38 | bauzas | artom: you're next in the roster list, fancy trying to look at some upstream bugs ? | |
| 16:04:33 | bauzas | artom seems to be offline, let's move on and we'll see | |
| 16:04:38 | bauzas | #info bug baton is artom | |
| 16:04:40 | bauzas | #undo | |
| 16:04:41 | opendevmeet | Removing item from minutes: #info bug baton is artom | |
| 16:04:49 | artom | Eh? No I'm here | |
| 16:04:50 | bauzas | #info bug baton is tentatively artom | |
| 16:05:19 | bauzas | artom: I was just asking you whether you were happy to go looking at Launchpad | |
| 16:05:22 | artom | (to my own surprise, I should say - I've been trying a new wayland-native IRC client, and it got lost somewhere on my 9 workspaces) | |
| 16:05:47 | bauzas | trust me, this is a happy place compared to some other bug reporting tools I know | |
| 16:05:52 | artom | Yep, I'll launch all the triage pads | |
| 16:06:09 | artom | And pad all the triage launches | |
| 16:06:29 | bauzas | I give you my pad, bro | |
| 16:06:48 | bauzas | anyway, moving on | |
| 16:06:48 | artom | Pad accepted, brah | |
| 16:06:52 | bauzas | #topic Gate status | |
| 16:06:59 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:07:09 | bauzas | this was a funny week | |
| 16:07:22 | sean-k-mooney | dansmith: have you seen that grenade failure more then once | |
| 16:07:26 | bauzas | but afaict, the gate postfailure is now fixed ? | |
| 16:07:30 | dansmith | sean-k-mooney: yeah | |
| 16:07:49 | dansmith | looks like failure to ssh to the test instance though | |
| 16:07:55 | dansmith | I haven't dug in deep yet | |
| 16:08:01 | sean-k-mooney | ok so that might be another gate blocker | |
| 16:08:16 | bauzas | shit | |
| 16:08:25 | sean-k-mooney | ya it looks like an ssh issue but im not sure if its consitent or intermitant | |
| 16:08:26 | dansmith | idk, but it's blocking the fix for another blocker :) | |
| 16:08:57 | bauzas | I don't you for you folks, but I have the impression that the universe is after me | |
| 16:09:01 | sean-k-mooney | for context https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L240 is failing | |
| 16:09:40 | sean-k-mooney | but have not looked at it properly either | |
| 16:09:46 | bauzas | ack | |
| 16:10:04 | bauzas | I guess the cinder team is fully aware of the situation ? | |
| 16:10:31 | dansmith | I doubt it's a cinder problem | |
| 16:10:34 | bauzas | but since we ssh to the guest, this is our mud, right ? | |
| 16:10:43 | sean-k-mooney | if they run grenade maybe but i just saw it this morning | |
| 16:11:02 | sean-k-mooney | im logging in to opensarch to see how common it is now | |
| 16:11:08 | bauzas | and no guest console saying anything ? | |
| 16:11:35 | sean-k-mooney | again i havent debug it so im not sure | |
| 16:11:38 | dansmith | no guest console dump in grenade | |
| 16:11:41 | dansmith | that's a tempest thing | |
| 16:11:45 | dansmith | lets not debug here | |
| 16:12:27 | bauzas | I quite agree with the fact that we shouldn't debug now | |
| 16:12:31 | bauzas | moving on so | |
| 16:12:44 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |