Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-10-21
12:46:25 tkajinam ok the logic for packed_ring is specifically implemented in get_packed_virtqueue_constraint
12:46:30 sean-k-mooney i.e. pass in bool or oslo.utils.str_to_bool whatever taht is called
12:47:48 sean-k-mooney ya so the issue is proably here https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1985
12:47:57 sean-k-mooney in that path we are not converting it to the correct type
12:48:16 sean-k-mooney we convert the flavor to the correct type here https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1990-L1992
12:48:28 sean-k-mooney but in the dict case the image_value is just a string
12:49:15 sean-k-mooney so we need to call strutils.bool_from_string here https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1985
13:54:13 jlejeune hello, I have a question regarding host_mapping, is it "normal" that when we delete a nova-compute service, the 'mapped' field in nova_cell1.compute_nodes table stays at 1 instead of 0 while nova_api.host_mappings related entry is removed ?
13:55:23 jlejeune by letting that 'mapped' field to 1 prevent nova-manage cell_v2 discover_hosts command to create the missing mapping
13:56:37 gibi jlejeune: it sounds like bug #1 in scenario 1 in https://bugs.launchpad.net/nova/+bug/2077070/comments/7
13:57:17 gibi jlejeune: could you check if adding --by-service to the discover_hosts command helps you or not
13:57:21 gibi ?
13:57:41 jlejeune I've already test it and indeed, it "fix" the issue by creating the missing mapping
13:58:14 gibi OK. Then I think it is the same issue.
13:58:32 gibi so yeah, nova should set mapped to 0 when the host mapping is deleted.
13:58:40 jlejeune but not sure I want to do it by service, given the number of computes we have in production
13:59:01 jlejeune agree
13:59:35 gibi if you have time to propose a fix then I'm happy to review it. If not then eventually I get to it myself to propose the fix.
14:01:46 jlejeune I will have a look !
14:02:26 gibi jlejeune: thanks!
14:27:27 jlejeune gibi: maybe we should split the bug in two, one for pci devices and another one for the mapping issue ?
14:31:56 gibi jlejeune: I don't think there is anythin else in the pci side, but if you want a separate bug to fix the mapping issue then sure go ahead and file one. I agree that not mentioning pci devices in the separate bug will help the reviewers not bog down on the pci stuff.
14:34:00 jlejeune yes exactly :)
14:35:43 gibi ping me with the separate bug and I will quickly do the triage of it
14:37:21 jlejeune ok
14:55:02 jlejeune gibi: https://bugs.launchpad.net/nova/+bug/2085135
15:09:46 gibi jlejeune: thank
15:44:06 johnthetubaguy dansmith: are you thinking threads for each task being better for what nova-compute is generally doing? I hadn’t really thought about that. I could see that limiting the code changes, I suppose?
15:44:29 dansmith johnthetubaguy: I want us to at least consider that yeah
15:44:35 bauzas johnthetubaguy: that was something we were considering from the last PTG
15:45:29 dansmith johnthetubaguy: I suspect we have some places where we think we're threadsafe, but aren't
15:45:38 dansmith so we'll need to fix that up, which we should have fixed anyway
15:46:14 dansmith but I think much of nova (potentially conductor as well) could just move straight to real threads, ops can control how much parallelism they want (which they've asked for before) to scale performance, and then we can move on
15:47:17 johnthetubaguy Yeah, I had not thought of it like that. Its a good point.
15:49:06 clarkb just as a point of info zuul runs largely with real threads (I think the websocket lib we use may involve asyncio but I can't remember)
15:49:32 JayF ^ is what dtantsur has been pitching for IPA, although we've written no code in that direction
15:49:38 clarkb it works reasonably well and in theory will work better when we can drop the gil and sort out all of the places we missed extra lockign (but we already coordinate multiple processes with zookeeper so don't really need that scaling approach with dropping gil)
15:50:25 dtantsur Yep. Not all of our project needs to handle an extremely high concurrency.
15:50:40 dtantsur Especially Ironic, which has "I'm busy, come back later" as an inherent part of its API :)
15:51:20 johnthetubaguy Honestly, most of the bugs I find are where we attempt high concurrency badly, hard blocking on some of the API calls and tasks would totally make me operator self happier :)
15:51:25 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:05:12 sean-k-mooney gibi: im not actully sold on usign ayncio for the timout stuff by the way
16:05:38 sean-k-mooney gibi: it was just going to be my next poc, warp the existing fucntion in awaitlet and dispatch ot asyncio on a seperat ethead
16:05:56 sean-k-mooney specificlay for the nova-api sacather gather usecause
16:06:08 dansmith I'm quite sure that doing that will not be a magic bullet unless we're actually running the asyncio event loop
16:06:32 sean-k-mooney right that woudl be idea run the asyncio eventloop in a sperate thread nad dispatch the async task into it
16:06:58 sean-k-mooney again this is just one of the many altertinitve i was condiering
16:07:10 sean-k-mooney im not conviced it is a good approch but its what was suggested in teh wiki
16:07:28 dansmith yeah, but unless we're the parent, I think that the thing that spawned us may receive the SIGALRM and not deliver it to our event loop as a result meaning it doesn't really work like we think if it's blocked in a C call
16:08:06 dansmith anyway, we should dive into a poc and confirm
16:08:13 sean-k-mooney https://wiki.openstack.org/wiki/Eventlet-removal#3._Timeout_and_Deadline_Management_2
16:09:06 gibi sean-k-mooney: for me more moving pieces sounds worse, e.g. awaitlet + asyncio + a worker thread running the eventloop is just too much complexity.
16:09:17 sean-k-mooney i did a poc of this before https://review.opendev.org/c/openstack/nova/+/904425/10/nova/context.py#422
16:09:28 sean-k-mooney with a diffent approch
16:09:35 sean-k-mooney gibi: ack which is valid
16:10:17 sean-k-mooney gibi: i dont think the event approch you suggest really works as we cant assume we can generially check the event within the fucntio we are dispatching
16:10:22 dansmith gibi: ++
16:10:30 sean-k-mooney gibi: sqlachmen does supprot a statement_timeout
16:10:43 sean-k-mooney which i think is going to be the answer for db queies
16:10:54 sean-k-mooney i think this is currently set per session or conenction
16:11:05 gibi sean-k-mooney: my point that we need to make assumption about the functions we run as a task or we need to prepare to abandon them from the gather thread
16:11:07 dansmith sean-k-mooney: "To handle signals the event loop must be run in the main thread."
16:11:25 sean-k-mooney dansmith: sure but we cant use sinals for this
16:11:40 dansmith I think that unless you're the main thread, you can't delegate things to executors and expect things like timeouts to preempt
16:11:45 sean-k-mooney when run under mod_wsig some of those are uncondtionally handeled by apache
16:12:05 sean-k-mooney so i dont think we can rely on sig_arlarm in general
16:12:36 dansmith right, but I think that's how asyncio handles preemptable timeouts right?
16:12:47 sean-k-mooney i do not think so
16:12:58 sean-k-mooney asycio also works on windows where it cant
16:13:18 dansmith sure, but it doesn't have to implement the primitives in the same way
16:13:26 johnthetubaguy What is the argument against brute forcing it with extra threads (for the scatter/gather) apache just says no? Or just worried about the system resources/latency of the extra thread?
16:13:57 sean-k-mooney johnthetubaguy: its not really the scather gahter that the concern
16:13:58 dansmith I've just run into some weirdness when trying to run asyncio compatible stuff from deep inside "regular" code and it seems like it was related to some of the caveats around the event loop being in the main thread
16:14:13 sean-k-mooney we coudl remove that but we currently use eventlet to timeout queiry that done return
16:14:14 gibi johnthetubaguy: threads are not cancellable so we cannot easily port the today's timeout handling
16:14:29 dansmith right ^
16:14:30 sean-k-mooney so i think the reall fix for that is the sql alchemey statement timeout
16:14:34 johnthetubaguy Ah… got you.
16:14:36 dansmith sean-k-mooney: eight
16:14:44 dansmith sean-k-mooney: *right
16:14:51 sean-k-mooney process can be killd but that not reaonble ot do in the api
16:15:39 sean-k-mooney ok so i think ill pause lookign at other "timeout methods" that work for arbitry funcitons
16:15:39 gibi sean-k-mooney: sure, if all the blocking call can be made non blocking with a timeout then we can just push down our timeout handling to the bottom of the call stack
16:15:57 sean-k-mooney and insted look at how we can pass timeouts to existing libs directly i.e. sqlachmey
16:16:31 gibi yeah, I think we should not chase the "timeut for arbitrary function" direction with native threads. But accept that we need to per task type based solution
16:16:44 dansmith agree
16:17:15 sean-k-mooney that is fine with me.
16:17:35 sean-k-mooney i was hoping to limit the scope fo the refactors but lets expore that instead then
16:18:22 sean-k-mooney to be clear changing form eventlet.timeout to native timeouts in sqlachemy shoudl work with or without eventlet monkypatching
16:18:43 dansmith yeah
16:18:54 sean-k-mooney dansmith: gibi: would you condier this in or out of scope for this cycle. perhasps we shoudl dicsuss in ptg session
16:19:10 sean-k-mooney im not sure if that is high or low risk
16:19:14 gibi sean-k-mooney: do we have a timeout today both on the gather greenthread and on the task greenthreads doing the work?
16:19:15 dansmith I think we could get an eventlet.Timeout or a sqla.timeout at any point so we should already be handling those
16:20:13 gibi sean-k-mooney: I consider the work in scope for E, landing it for E might not happen but if I want to learn the devils in the details I need to work on an example, and this is a good example.
16:20:24 sean-k-mooney ack
16:20:26 dansmith also, AFAIK eventlet will let the timeout happen to unblock the waiter, but the worker thread actually blocked remains there to be GC'd until it frees up,
16:20:44 dansmith so using real timeouts should be better anyway to unblock and return threads to the pool quicker
16:21:18 sean-k-mooney when the timeout fire i belvie the eventlet calls greenthread.throw on the worker greentrhead
16:21:35 dansmith right but not in the actual worker thread which is blocked on a C call

Earlier   Later