Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-10-21
16:14:34 johnthetubaguy Ah… got you.
16:14:36 dansmith sean-k-mooney: eight
16:14:44 dansmith sean-k-mooney: *right
16:14:51 sean-k-mooney process can be killd but that not reaonble ot do in the api
16:15:39 gibi sean-k-mooney: sure, if all the blocking call can be made non blocking with a timeout then we can just push down our timeout handling to the bottom of the call stack
16:15:39 sean-k-mooney ok so i think ill pause lookign at other "timeout methods" that work for arbitry funcitons
16:15:57 sean-k-mooney and insted look at how we can pass timeouts to existing libs directly i.e. sqlachmey
16:16:31 gibi yeah, I think we should not chase the "timeut for arbitrary function" direction with native threads. But accept that we need to per task type based solution
16:16:44 dansmith agree
16:17:15 sean-k-mooney that is fine with me.
16:17:35 sean-k-mooney i was hoping to limit the scope fo the refactors but lets expore that instead then
16:18:22 sean-k-mooney to be clear changing form eventlet.timeout to native timeouts in sqlachemy shoudl work with or without eventlet monkypatching
16:18:43 dansmith yeah
16:18:54 sean-k-mooney dansmith: gibi: would you condier this in or out of scope for this cycle. perhasps we shoudl dicsuss in ptg session
16:19:10 sean-k-mooney im not sure if that is high or low risk
16:19:14 gibi sean-k-mooney: do we have a timeout today both on the gather greenthread and on the task greenthreads doing the work?
16:19:15 dansmith I think we could get an eventlet.Timeout or a sqla.timeout at any point so we should already be handling those
16:20:13 gibi sean-k-mooney: I consider the work in scope for E, landing it for E might not happen but if I want to learn the devils in the details I need to work on an example, and this is a good example.
16:20:24 sean-k-mooney ack
16:20:26 dansmith also, AFAIK eventlet will let the timeout happen to unblock the waiter, but the worker thread actually blocked remains there to be GC'd until it frees up,
16:20:44 dansmith so using real timeouts should be better anyway to unblock and return threads to the pool quicker
16:21:18 sean-k-mooney when the timeout fire i belvie the eventlet calls greenthread.throw on the worker greentrhead
16:21:35 dansmith right but not in the actual worker thread which is blocked on a C call
16:21:43 sean-k-mooney but to dans point it may very well block for ever or untill the os kills the tcp socket
16:21:46 dansmith there's no way to generically interrupt the latter
16:22:08 dansmith the greenthread.throw is only handled as an exception when the blocking call returns
16:22:15 sean-k-mooney right
16:22:35 sean-k-mooney so until the os returns an interupt that resuble the worker thread it wont die
16:23:04 dansmith right
16:23:21 dansmith that's why moving to real library timeouts will benefit real threads and the green threads approache
16:25:41 sean-k-mooney ack for now the syncronos ceph lib does not supprot that but i think there async (not asynio) function that return futres can, still need to confirm that
16:29:20 dansmith ack
16:34:11 sean-k-mooney they do supprot timouts on the connect method and the "cil command" wrappers
16:34:20 sean-k-mooney https://docs.ceph.com/en/reef/rados/api/python/#rados.Rados.connect https://docs.ceph.com/en/reef/rados/api/python/#rados.Rados.mon_command
16:37:31 bauzas gouthamr: hey, what exactly do you want to discuss about virtiofs during nova sessions ? https://etherpad.opendev.org/p/nova-2025.1-ptg#L231
16:37:42 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:37:43 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
16:39:15 gouthamr bauzas: i'd like to know the current status of the development / focus of the team going into Epoxy
16:39:31 bauzas okay, noted
16:39:37 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:39:38 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
16:39:49 bauzas I'll provide a raw agenday tomorrow morning European time
16:56:01 johnthetubaguy bauzas: I believe its MIG vGPU not working with detach volume any more, Caracal onwards? Came up in the scientific SIG
16:56:50 bauzas johnthetubaguy: I may have said that MIG isn't officially supported in Nova
16:57:05 bauzas because that's nvidia specific and can't be documented upstream
16:58:37 johnthetubaguy bauzas: could you link me to where we say that? Its in the PCI docs I guess?
16:59:32 johnthetubaguy (It might be vGPU not MIG directly, but I could well be confused)
17:01:29 sean-k-mooney the gap when is the write/read commends to add data to the volume
17:01:47 sean-k-mooney context of ^ is our current abllity to timeout ceph commands
17:02:23 sean-k-mooney johnthetubaguy: i see i tought we fixed that?
17:02:37 johnthetubaguy @bauzas https://review.opendev.org/c/openstack/nova/+/925037 Maybe it got fixed another way?
17:02:57 sean-k-mooney johnthetubaguy: there was a breakage casue by the alisa feature i think, i dont recall the full details i woudl have to find the review
17:03:07 bauzas johnthetubaguy: we don't exactly mention that nvidia bits are forbidden, but vgpu upstream docs somehow show it up https://docs.openstack.org/nova/latest/admin/virtual-gpu.html
17:04:17 bauzas oh ok, now I get the picture
17:04:33 sean-k-mooney johnthetubaguy: you dont happen to have the review do you. ill see if i can find it
17:05:15 sean-k-mooney johnthetubaguy: if i recall correctly we asked them to repoduce the failure in a fucntional test and then demonstrate there patch fixed it
17:05:21 sean-k-mooney but i dont think we ask for thid party ci
17:06:22 sean-k-mooney johnthetubaguy: was it related to https://review.opendev.org/c/openstack/nova/+/925037 https://bugs.launchpad.net/nova/+bug/1942345
17:06:55 bauzas guys, I need to drop for my sanity
17:07:19 bauzas we'll have long days of nova sessions so today is a hard stop
17:07:29 sean-k-mooney we fixed that with https://review.opendev.org/c/openstack/nova/+/806943
17:08:08 fungi i also linked it in the dalmation retrospective section of the ptg pad since bauzas asked
17:09:21 sean-k-mooney ack
17:10:33 fungi and yes, the proposer seemed to think he was being told 1. we don't support your use case, and 2. if you want this merged you need to set up your own ci to provide testing for it
17:10:36 johnthetubaguy To be clear, my motive is I know people using vGPU on Antelope, and we are about to SLURP them to Caracal, but no one can afford GPUs in their staging environments these days. So people raising the flag and saying it’s all broken make me worry, and want to know how I can help fix things. A chat in the retrospective sounds like a good idea, regardless.
17:10:54 fungi which i don't get when i read the review comments, so they were reading a lot more into it for unknown reasons
17:14:39 johnthetubaguy Ah… I had missed that crucial detail, probably a misunderstanding of “add a functional test” talking about a different tox target.
17:20:55 fungi yeah, as for their unpleasant experience, some people come into this expecting to have a bad time and then read between the lines in whatever way is needed to reinforce their expectation, so aside from being more clear in communicating (maybe? it was pretty clear to me already), i'm not sure what else might have helped the situation
17:30:15 sean-k-mooney johnthetubaguy: yep tox not third party ci
17:31:09 sean-k-mooney this is a valid bug to fix. but the fact we had two almost identical issues suggests our existign test coverage is lacking
17:31:41 sean-k-mooney the orginal fix did not fix the underlying problem just eh symtom
17:31:56 sean-k-mooney but thats a seperate matter.
17:32:55 sean-k-mooney johnthetubaguy: for what its worth we are trying to get mdev testign in first and thirdparty ci currently
17:33:21 sean-k-mooney johnthetubaguy: for the first party ci we have a poc of using the mtty sample modle which we hope to merge sooner rather then later into nova-next
17:33:43 sean-k-mooney for third party ci we are hopeing to have a limited capablity to trigger a thirdparty job using redhat resouces evenutlly
17:33:59 sean-k-mooney i.e. agent changes to master before they merge
17:34:28 sean-k-mooney its a lot of work but somehtign we are trying to line up. the bit problem as you noted is hardware aviablity but that is not the only one
17:34:53 sean-k-mooney we hope the mtty supprot will give a baseline of coverage for things like this
17:35:15 sean-k-mooney johnthetubaguy: https://review.opendev.org/q/topic:%22mtty_support%22
17:36:13 sean-k-mooney we need https://review.opendev.org/c/openstack/nova/+/898100/4 first ot allow using virtual devices
17:42:12 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
17:42:30 sean-k-mooney fungi: the orgianl confugion arount "we dont support your usecase" came from the fact they fix the bug by adding the infra required to be able to detach an mdev via the alisa, without stating that they were actully trying to detach a volume and this just so happend to fail when the vm also had an mdev
17:43:26 sean-k-mooney they had a limited explcation of that in the bug but not in the commit message
18:15:02 fungi sean-k-mooney: thanks for the clarification, makes sense
18:16:00 fungi also worth noting, there was a chameleon cloud representative at oid-na who was talking about the possibility of making vgpu resources available for upstream testing. not sure where that will go, but something to consider
18:16:58 sean-k-mooney the issue is we cant do that with nested vir
18:17:08 sean-k-mooney i.e. we can only test it properly with baremental compute nodes
18:17:28 sean-k-mooney if you passhtough afully gpu to a vm you cant then use vgpus
18:17:44 JayF darn, too bad we don't know how to provision baremetal /s
18:18:01 JayF seriously though, having to provision baremetal is very tough in CI, it's part of why Ironic 3rd party CIs died out
18:18:04 JayF just a little high touch
18:18:25 fungi well, yeah they were talking about nova baremetal api to boot machines with vgpu-capable resources
18:18:31 fungi (i.e. ironic)
18:19:23 fungi but i agree with you about the whole safety challenge with providing root access to physical machines for untrusted job payloads
18:19:41 JayF it's not just a safety challenge, it's a challenge-challenge
18:19:58 JayF many real-world failure modes which needs human hands, or at least, non-automated intervention to fix
18:20:26 sean-k-mooney the tldr is when you passthough a fully pci device in qemu it removes all the sriov capablities because qemu did not have a way to make that work
18:20:58 sean-k-mooney for what its worth we are lookign to see if we can use our internal hardware in teh future to test specifi upstream changes
18:21:50 sean-k-mooney that woudl do a baremteally deployment of a nova compute (using ironic) with a manual triger to basicaly say "this is safe to test"
18:21:53 sean-k-mooney as a third party ci

Earlier   Later