| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-10-21 | |||
| 17:07:19 | bauzas | we'll have long days of nova sessions so today is a hard stop | |
| 17:07:29 | sean-k-mooney | we fixed that with https://review.opendev.org/c/openstack/nova/+/806943 | |
| 17:08:08 | fungi | i also linked it in the dalmation retrospective section of the ptg pad since bauzas asked | |
| 17:09:21 | sean-k-mooney | ack | |
| 17:10:33 | fungi | and yes, the proposer seemed to think he was being told 1. we don't support your use case, and 2. if you want this merged you need to set up your own ci to provide testing for it | |
| 17:10:36 | johnthetubaguy | To be clear, my motive is I know people using vGPU on Antelope, and we are about to SLURP them to Caracal, but no one can afford GPUs in their staging environments these days. So people raising the flag and saying it’s all broken make me worry, and want to know how I can help fix things. A chat in the retrospective sounds like a good idea, regardless. | |
| 17:10:54 | fungi | which i don't get when i read the review comments, so they were reading a lot more into it for unknown reasons | |
| 17:14:39 | johnthetubaguy | Ah… I had missed that crucial detail, probably a misunderstanding of “add a functional test” talking about a different tox target. | |
| 17:20:55 | fungi | yeah, as for their unpleasant experience, some people come into this expecting to have a bad time and then read between the lines in whatever way is needed to reinforce their expectation, so aside from being more clear in communicating (maybe? it was pretty clear to me already), i'm not sure what else might have helped the situation | |
| 17:30:15 | sean-k-mooney | johnthetubaguy: yep tox not third party ci | |
| 17:31:09 | sean-k-mooney | this is a valid bug to fix. but the fact we had two almost identical issues suggests our existign test coverage is lacking | |
| 17:31:41 | sean-k-mooney | the orginal fix did not fix the underlying problem just eh symtom | |
| 17:31:56 | sean-k-mooney | but thats a seperate matter. | |
| 17:32:55 | sean-k-mooney | johnthetubaguy: for what its worth we are trying to get mdev testign in first and thirdparty ci currently | |
| 17:33:21 | sean-k-mooney | johnthetubaguy: for the first party ci we have a poc of using the mtty sample modle which we hope to merge sooner rather then later into nova-next | |
| 17:33:43 | sean-k-mooney | for third party ci we are hopeing to have a limited capablity to trigger a thirdparty job using redhat resouces evenutlly | |
| 17:33:59 | sean-k-mooney | i.e. agent changes to master before they merge | |
| 17:34:28 | sean-k-mooney | its a lot of work but somehtign we are trying to line up. the bit problem as you noted is hardware aviablity but that is not the only one | |
| 17:34:53 | sean-k-mooney | we hope the mtty supprot will give a baseline of coverage for things like this | |
| 17:35:15 | sean-k-mooney | johnthetubaguy: https://review.opendev.org/q/topic:%22mtty_support%22 | |
| 17:36:13 | sean-k-mooney | we need https://review.opendev.org/c/openstack/nova/+/898100/4 first ot allow using virtual devices | |
| 17:42:12 | opendevreview | Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926 | |
| 17:42:30 | sean-k-mooney | fungi: the orgianl confugion arount "we dont support your usecase" came from the fact they fix the bug by adding the infra required to be able to detach an mdev via the alisa, without stating that they were actully trying to detach a volume and this just so happend to fail when the vm also had an mdev | |
| 17:43:26 | sean-k-mooney | they had a limited explcation of that in the bug but not in the commit message | |
| 18:15:02 | fungi | sean-k-mooney: thanks for the clarification, makes sense | |
| 18:16:00 | fungi | also worth noting, there was a chameleon cloud representative at oid-na who was talking about the possibility of making vgpu resources available for upstream testing. not sure where that will go, but something to consider | |
| 18:16:58 | sean-k-mooney | the issue is we cant do that with nested vir | |
| 18:17:08 | sean-k-mooney | i.e. we can only test it properly with baremental compute nodes | |
| 18:17:28 | sean-k-mooney | if you passhtough afully gpu to a vm you cant then use vgpus | |
| 18:17:44 | JayF | darn, too bad we don't know how to provision baremetal /s | |
| 18:18:01 | JayF | seriously though, having to provision baremetal is very tough in CI, it's part of why Ironic 3rd party CIs died out | |
| 18:18:04 | JayF | just a little high touch | |
| 18:18:25 | fungi | well, yeah they were talking about nova baremetal api to boot machines with vgpu-capable resources | |
| 18:18:31 | fungi | (i.e. ironic) | |
| 18:19:23 | fungi | but i agree with you about the whole safety challenge with providing root access to physical machines for untrusted job payloads | |
| 18:19:41 | JayF | it's not just a safety challenge, it's a challenge-challenge | |
| 18:19:58 | JayF | many real-world failure modes which needs human hands, or at least, non-automated intervention to fix | |
| 18:20:26 | sean-k-mooney | the tldr is when you passthough a fully pci device in qemu it removes all the sriov capablities because qemu did not have a way to make that work | |
| 18:20:58 | sean-k-mooney | for what its worth we are lookign to see if we can use our internal hardware in teh future to test specifi upstream changes | |
| 18:21:50 | sean-k-mooney | that woudl do a baremteally deployment of a nova compute (using ironic) with a manual triger to basicaly say "this is safe to test" | |
| 18:21:53 | sean-k-mooney | as a third party ci | |
| 18:22:52 | JayF | I think there's a lot of untapped value in that idea of on-demand testing | |
| 18:23:09 | JayF | we can *usually* pinpoint when we're getting close to an edge case that might need revalidation | |
| 18:23:20 | sean-k-mooney | but that is why we are also trying to figure out how to test without real hardware using the sample kernel dirvers for a baseline of coverage | |
| 18:24:05 | sean-k-mooney | for example we are enableing igb vif model in nova so tthat in the future we can use that to test sriov | |
| 18:24:40 | sean-k-mooney | if our providers can provide neutron ports with igb vif model in the future we can fully test sriov in the 1st party ci | |
| 18:25:16 | sean-k-mooney | unfrogually it needs a new enough nova/qemu but by 2026.1 we might be in a position to do that | |
| 18:25:32 | sean-k-mooney | *unfortunetly | |
| 19:04:37 | opendevreview | Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926 | |
| 19:54:32 | opendevreview | sean mooney proposed openstack/nova master: Add repoducer test for bug 2074219 https://review.opendev.org/c/openstack/nova/+/932930 | |
| 19:54:33 | opendevreview | sean mooney proposed openstack/nova master: Fix detaching devices by alias with mdevs https://review.opendev.org/c/openstack/nova/+/932931 | |
| 19:55:09 | sean-k-mooney | JayF: johnthetubaguy: fungi: ^ that is the actual correct fix and what i was asking for in that review | |
| 20:13:03 | fungi | awesome! | |
| #openstack-nova - 2024-10-22 | |||
| 06:10:40 | opendevreview | Aleksandr Chudinov proposed openstack/nova master: Fix instance snapshot min disk size after instance resize https://review.opendev.org/c/openstack/nova/+/931737 | |
| 07:24:01 | opendevreview | Takashi Kajinami proposed openstack/nova master: Deprecate [wsgi] secure_proxy_ssl_header https://review.opendev.org/c/openstack/nova/+/817927 | |
| 07:24:06 | opendevreview | Takashi Kajinami proposed openstack/nova master: Deprecate [wsgi] secure_proxy_ssl_header https://review.opendev.org/c/openstack/nova/+/817927 | |
| 07:25:48 | opendevreview | Takashi Kajinami proposed openstack/nova master: Deprecate [wsgi] secure_proxy_ssl_header https://review.opendev.org/c/openstack/nova/+/817927 | |
| 07:26:51 | opendevreview | Takashi Kajinami proposed openstack/nova master: Deprecate [wsgi] secure_proxy_ssl_header https://review.opendev.org/c/openstack/nova/+/817927 | |
| 07:29:06 | opendevreview | Takashi Kajinami proposed openstack/nova master: Drop unused parameter type validation method https://review.opendev.org/c/openstack/nova/+/909189 | |
| 08:09:16 | elodilles | bauzas: auniyal: Uggla: if you have some time could you please review (release liaison approve) the antelope final stable release? https://review.opendev.org/c/openstack/releases/+/932406 | |
| 08:14:01 | mikal | sean-k-mooney: I've reproposed https://review.opendev.org/c/openstack/nova-specs/+/932387 (spice-direct spec), just wondering if there's anything else I should do apart from put it in the status etherpad? | |
| 08:17:14 | sean-k-mooney[m] | mikal: you updated the history section so that all that was needed | |
| 08:21:40 | sean-k-mooney[m] | gibi: ill fix the typos in https://review.opendev.org/c/openstack/nova/+/932931/1 once i have had coffee and am actually awake. thanks for the review | |
| 08:22:11 | sean-k-mooney[m] | mikal: i assume the rest of the pataches are ready for review? | |
| 08:24:38 | sean-k-mooney[m] | im not seeing any merge conflicts or direct requests form mel that need to be addressed but also have not looked closely at the current state | |
| 08:55:23 | gibi | sean-k-mooney[m]: sure, just ping me after the respin and I will re-add my +2 | |
| 09:02:18 | gibi | bauzas: when you are up, I'm wondering how much quorum the nova room will have on wednesday during the global eventlet removal discussion? | |
| 09:02:33 | bauzas | I'm already up :) | |
| 09:02:38 | gibi | o/ :) | |
| 09:02:49 | bauzas | gibi: that's a good question | |
| 09:03:06 | gibi | I would like to join to the eventlet removal discussion at least | |
| 09:03:06 | bauzas | that discussion will trample our own sessions | |
| 09:03:24 | bauzas | I was thinking it could be on Monday | |
| 09:03:25 | gibi | but other cores might have the same plans | |
| 09:03:26 | bauzas | :( | |
| 09:03:35 | bauzas | let's discuss then the agenda today | |
| 09:03:44 | bauzas | I'm currently looking at the etherpad | |
| 09:03:49 | gibi | yeah I think the deeper dive will be on wednesday, yesterday we only covered high level things | |
| 09:03:56 | bauzas | and I'll try to provide an agenda | |
| 09:04:03 | gibi | cool, thanks | |
| 09:04:18 | gibi | I have no other limitation during the week as far as I see | |
| 09:10:57 | bauzas | gibi: my own limitation will be on Friday due to the TC sessions :( | |
| 09:11:06 | bauzas | hence the agenda proposal | |
| 09:22:11 | bauzas | gibi: I think we have a serious problem that we need to solve about Wed | |
| 09:41:54 | sean-k-mooney | bauzas: gibi: i dont know if we want to discuss this at the ptg https://bugs.launchpad.net/nova/+bug/2076089 but it would be good if other could confirm or refute my assertion that we shoudl not be abel to force boot a new instance on a disabled compute | |
| 09:42:17 | bauzas | sean-k-mooney: the agenda is now fully packed tbh | |
| 09:42:22 | sean-k-mooney | as far as i am aware we made an intentional decsion to prevent this | |
| 09:42:32 | sean-k-mooney | bauzas: that fine its a bug so we can discuss it async | |
| 09:42:34 | bauzas | the eventlet-removal session just ate all the left time | |
| 09:42:40 | sean-k-mooney | i just want to spread awareness of it | |
| 09:44:02 | sean-k-mooney | im basically asking for a sanity check to ensure im not off base when i say this is not a bug. we can put it on the next team meeting adgenda too if we like | |
| 09:44:09 | bauzas | well | |
| 09:44:22 | bauzas | the forced AZ hack was the usual way for ops to test new computes | |
| 09:44:36 | sean-k-mooney | right but not disabled hosts | |
| 09:44:45 | bauzas | no, even disabled ones | |
| 09:44:56 | sean-k-mooney | that was definetly a bug then | |
| 09:45:09 | sean-k-mooney | i.e that that ever worked | |
| 09:45:21 | bauzas | yeah, for me it looks to me a regression | |
| 09:45:39 | sean-k-mooney | im saying we should not allow that to happen and that i dont condier this a regression | |
| 09:46:01 | sean-k-mooney | if the az hack worked in the past tha that was a bug IMO | |