Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-10-21
16:34:11 sean-k-mooney they do supprot timouts on the connect method and the "cil command" wrappers
16:34:20 sean-k-mooney https://docs.ceph.com/en/reef/rados/api/python/#rados.Rados.connect https://docs.ceph.com/en/reef/rados/api/python/#rados.Rados.mon_command
16:37:31 bauzas gouthamr: hey, what exactly do you want to discuss about virtiofs during nova sessions ? https://etherpad.opendev.org/p/nova-2025.1-ptg#L231
16:37:42 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:37:43 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
16:39:15 gouthamr bauzas: i'd like to know the current status of the development / focus of the team going into Epoxy
16:39:31 bauzas okay, noted
16:39:37 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:39:38 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
16:39:49 bauzas I'll provide a raw agenday tomorrow morning European time
16:56:01 johnthetubaguy bauzas: I believe its MIG vGPU not working with detach volume any more, Caracal onwards? Came up in the scientific SIG
16:56:50 bauzas johnthetubaguy: I may have said that MIG isn't officially supported in Nova
16:57:05 bauzas because that's nvidia specific and can't be documented upstream
16:58:37 johnthetubaguy bauzas: could you link me to where we say that? Its in the PCI docs I guess?
16:59:32 johnthetubaguy (It might be vGPU not MIG directly, but I could well be confused)
17:01:29 sean-k-mooney the gap when is the write/read commends to add data to the volume
17:01:47 sean-k-mooney context of ^ is our current abllity to timeout ceph commands
17:02:23 sean-k-mooney johnthetubaguy: i see i tought we fixed that?
17:02:37 johnthetubaguy @bauzas https://review.opendev.org/c/openstack/nova/+/925037 Maybe it got fixed another way?
17:02:57 sean-k-mooney johnthetubaguy: there was a breakage casue by the alisa feature i think, i dont recall the full details i woudl have to find the review
17:03:07 bauzas johnthetubaguy: we don't exactly mention that nvidia bits are forbidden, but vgpu upstream docs somehow show it up https://docs.openstack.org/nova/latest/admin/virtual-gpu.html
17:04:17 bauzas oh ok, now I get the picture
17:04:33 sean-k-mooney johnthetubaguy: you dont happen to have the review do you. ill see if i can find it
17:05:15 sean-k-mooney johnthetubaguy: if i recall correctly we asked them to repoduce the failure in a fucntional test and then demonstrate there patch fixed it
17:05:21 sean-k-mooney but i dont think we ask for thid party ci
17:06:22 sean-k-mooney johnthetubaguy: was it related to https://review.opendev.org/c/openstack/nova/+/925037 https://bugs.launchpad.net/nova/+bug/1942345
17:06:55 bauzas guys, I need to drop for my sanity
17:07:19 bauzas we'll have long days of nova sessions so today is a hard stop
17:07:29 sean-k-mooney we fixed that with https://review.opendev.org/c/openstack/nova/+/806943
17:08:08 fungi i also linked it in the dalmation retrospective section of the ptg pad since bauzas asked
17:09:21 sean-k-mooney ack
17:10:33 fungi and yes, the proposer seemed to think he was being told 1. we don't support your use case, and 2. if you want this merged you need to set up your own ci to provide testing for it
17:10:36 johnthetubaguy To be clear, my motive is I know people using vGPU on Antelope, and we are about to SLURP them to Caracal, but no one can afford GPUs in their staging environments these days. So people raising the flag and saying it’s all broken make me worry, and want to know how I can help fix things. A chat in the retrospective sounds like a good idea, regardless.
17:10:54 fungi which i don't get when i read the review comments, so they were reading a lot more into it for unknown reasons
17:14:39 johnthetubaguy Ah… I had missed that crucial detail, probably a misunderstanding of “add a functional test” talking about a different tox target.
17:20:55 fungi yeah, as for their unpleasant experience, some people come into this expecting to have a bad time and then read between the lines in whatever way is needed to reinforce their expectation, so aside from being more clear in communicating (maybe? it was pretty clear to me already), i'm not sure what else might have helped the situation
17:30:15 sean-k-mooney johnthetubaguy: yep tox not third party ci
17:31:09 sean-k-mooney this is a valid bug to fix. but the fact we had two almost identical issues suggests our existign test coverage is lacking
17:31:41 sean-k-mooney the orginal fix did not fix the underlying problem just eh symtom
17:31:56 sean-k-mooney but thats a seperate matter.
17:32:55 sean-k-mooney johnthetubaguy: for what its worth we are trying to get mdev testign in first and thirdparty ci currently
17:33:21 sean-k-mooney johnthetubaguy: for the first party ci we have a poc of using the mtty sample modle which we hope to merge sooner rather then later into nova-next
17:33:43 sean-k-mooney for third party ci we are hopeing to have a limited capablity to trigger a thirdparty job using redhat resouces evenutlly
17:33:59 sean-k-mooney i.e. agent changes to master before they merge
17:34:28 sean-k-mooney its a lot of work but somehtign we are trying to line up. the bit problem as you noted is hardware aviablity but that is not the only one
17:34:53 sean-k-mooney we hope the mtty supprot will give a baseline of coverage for things like this
17:35:15 sean-k-mooney johnthetubaguy: https://review.opendev.org/q/topic:%22mtty_support%22
17:36:13 sean-k-mooney we need https://review.opendev.org/c/openstack/nova/+/898100/4 first ot allow using virtual devices
17:42:12 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
17:42:30 sean-k-mooney fungi: the orgianl confugion arount "we dont support your usecase" came from the fact they fix the bug by adding the infra required to be able to detach an mdev via the alisa, without stating that they were actully trying to detach a volume and this just so happend to fail when the vm also had an mdev
17:43:26 sean-k-mooney they had a limited explcation of that in the bug but not in the commit message
18:15:02 fungi sean-k-mooney: thanks for the clarification, makes sense
18:16:00 fungi also worth noting, there was a chameleon cloud representative at oid-na who was talking about the possibility of making vgpu resources available for upstream testing. not sure where that will go, but something to consider
18:16:58 sean-k-mooney the issue is we cant do that with nested vir
18:17:08 sean-k-mooney i.e. we can only test it properly with baremental compute nodes
18:17:28 sean-k-mooney if you passhtough afully gpu to a vm you cant then use vgpus
18:17:44 JayF darn, too bad we don't know how to provision baremetal /s
18:18:01 JayF seriously though, having to provision baremetal is very tough in CI, it's part of why Ironic 3rd party CIs died out
18:18:04 JayF just a little high touch
18:18:25 fungi well, yeah they were talking about nova baremetal api to boot machines with vgpu-capable resources
18:18:31 fungi (i.e. ironic)
18:19:23 fungi but i agree with you about the whole safety challenge with providing root access to physical machines for untrusted job payloads
18:19:41 JayF it's not just a safety challenge, it's a challenge-challenge
18:19:58 JayF many real-world failure modes which needs human hands, or at least, non-automated intervention to fix
18:20:26 sean-k-mooney the tldr is when you passthough a fully pci device in qemu it removes all the sriov capablities because qemu did not have a way to make that work
18:20:58 sean-k-mooney for what its worth we are lookign to see if we can use our internal hardware in teh future to test specifi upstream changes
18:21:50 sean-k-mooney that woudl do a baremteally deployment of a nova compute (using ironic) with a manual triger to basicaly say "this is safe to test"
18:21:53 sean-k-mooney as a third party ci
18:22:52 JayF I think there's a lot of untapped value in that idea of on-demand testing
18:23:09 JayF we can *usually* pinpoint when we're getting close to an edge case that might need revalidation
18:23:20 sean-k-mooney but that is why we are also trying to figure out how to test without real hardware using the sample kernel dirvers for a baseline of coverage
18:24:05 sean-k-mooney for example we are enableing igb vif model in nova so tthat in the future we can use that to test sriov
18:24:40 sean-k-mooney if our providers can provide neutron ports with igb vif model in the future we can fully test sriov in the 1st party ci
18:25:16 sean-k-mooney unfrogually it needs a new enough nova/qemu but by 2026.1 we might be in a position to do that
18:25:32 sean-k-mooney *unfortunetly
19:04:37 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
19:54:32 opendevreview sean mooney proposed openstack/nova master: Add repoducer test for bug 2074219 https://review.opendev.org/c/openstack/nova/+/932930
19:54:33 opendevreview sean mooney proposed openstack/nova master: Fix detaching devices by alias with mdevs https://review.opendev.org/c/openstack/nova/+/932931
19:55:09 sean-k-mooney JayF: johnthetubaguy: fungi: ^ that is the actual correct fix and what i was asking for in that review
20:13:03 fungi awesome!
#openstack-nova - 2024-10-22
06:10:40 opendevreview Aleksandr Chudinov proposed openstack/nova master: Fix instance snapshot min disk size after instance resize https://review.opendev.org/c/openstack/nova/+/931737
07:24:01 opendevreview Takashi Kajinami proposed openstack/nova master: Deprecate [wsgi] secure_proxy_ssl_header https://review.opendev.org/c/openstack/nova/+/817927
07:24:06 opendevreview Takashi Kajinami proposed openstack/nova master: Deprecate [wsgi] secure_proxy_ssl_header https://review.opendev.org/c/openstack/nova/+/817927
07:25:48 opendevreview Takashi Kajinami proposed openstack/nova master: Deprecate [wsgi] secure_proxy_ssl_header https://review.opendev.org/c/openstack/nova/+/817927
07:26:51 opendevreview Takashi Kajinami proposed openstack/nova master: Deprecate [wsgi] secure_proxy_ssl_header https://review.opendev.org/c/openstack/nova/+/817927
07:29:06 opendevreview Takashi Kajinami proposed openstack/nova master: Drop unused parameter type validation method https://review.opendev.org/c/openstack/nova/+/909189
08:09:16 elodilles bauzas: auniyal: Uggla: if you have some time could you please review (release liaison approve) the antelope final stable release? https://review.opendev.org/c/openstack/releases/+/932406
08:14:01 mikal sean-k-mooney: I've reproposed https://review.opendev.org/c/openstack/nova-specs/+/932387 (spice-direct spec), just wondering if there's anything else I should do apart from put it in the status etherpad?
08:17:14 sean-k-mooney[m] mikal: you updated the history section so that all that was needed
08:21:40 sean-k-mooney[m] gibi: ill fix the typos in https://review.opendev.org/c/openstack/nova/+/932931/1 once i have had coffee and am actually awake. thanks for the review
08:22:11 sean-k-mooney[m] mikal: i assume the rest of the pataches are ready for review?
08:24:38 sean-k-mooney[m] im not seeing any merge conflicts or direct requests form mel that need to be addressed but also have not looked closely at the current state
08:55:23 gibi sean-k-mooney[m]: sure, just ping me after the respin and I will re-add my +2
09:02:18 gibi bauzas: when you are up, I'm wondering how much quorum the nova room will have on wednesday during the global eventlet removal discussion?
09:02:33 bauzas I'm already up :)
09:02:38 gibi o/ :)
09:02:49 bauzas gibi: that's a good question
09:03:06 gibi I would like to join to the eventlet removal discussion at least
09:03:06 bauzas that discussion will trample our own sessions
09:03:24 bauzas I was thinking it could be on Monday

Earlier   Later