Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-05-24
16:20:23 dansmith okay, because nova-api already runs with a combination of native and green threads, so if it was just api, then that could be why,
16:20:45 dansmith but if scheduler and conductor see it as well, I'm not sure why no other service would be affected
16:22:53 melwitt yeah, I'm guessing it's because it becomes very rare when the only things using eventlet are the periodic tasks, timers/retries, and some bits of scatter/gather. the error shows up under high load and coexists with other various connection errors to the database
16:23:21 melwitt well, maybe "very rare" is not a good way to put it, "more rare"
16:25:43 melwitt I have searched for some way that nova does something different than any other service wrt to database access and found nothing
16:26:15 melwitt the only lead we have so far is that nova uses eventlet more than any other services do (afaik so far)
16:26:17 sean-k-mooney melwitt: i assume we have not had any downstream sqlalcamy updtes in rhel 7
16:27:21 sean-k-mooney does this happen in modren openstack that was for 13 so queens
16:27:48 melwitt sean-k-mooney: what do you mean, like version changes? I don't think so but zzzeek would have covered that
16:28:12 sean-k-mooney yep or perhaps a backport that could have caused it
16:28:59 melwitt sean-k-mooney: to your other question, I could find no bug reports for this for newer than 13
16:29:39 melwitt that could either mean it went away or that it happens rarely enough that no one has bothered reporting it. not sure what to think
16:30:09 dansmith melwitt: but nothing changed about conductor and scheduler that would account for them not showing the issue in later versions
16:30:37 dansmith meaning api going from evenlet to real wsgi is a change that could affect this, but that doesn't impact the other services
16:30:53 melwitt zzzeek anyway strongly recommended we stop using eventlet for its known issues with the mysql connectors
16:31:10 dansmith I've read the bug now, and I see that he has,
16:31:20 dansmith but it's not quite as simple as just turning it off
16:31:51 melwitt dansmith: yeah, I appreciate that. but it's hard to know if this is just not been reported or if it's really not there anymore
16:31:57 sean-k-mooney i wonder is this wsgi related
16:32:07 dansmith so much of what goes on changes if you don't have those yield points patched in
16:32:29 dansmith sean-k-mooney: melwitt says she has reports of it from scheduler and conductor, which is what really puzzles me
16:32:47 melwitt yeah, I know it's not simple as turning it off but afaict we could switch everything to native threads. I've started looking into it (just by having to explain to the customer where/when it's workaroundable and when it's not)
16:32:53 sean-k-mooney ok i was wondering if it was related to the issue with enabling multiple tread in the wsgi process
16:33:07 sean-k-mooney but if its in the schduler and conductor its not that
16:33:16 dansmith melwitt: native threads mean a ton of code can race that can't race now
16:33:42 sean-k-mooney dansmith: well im not sure about "cant race now" but i agree it would be more
16:33:43 melwitt dansmith: yeah, those logs are attached to the case and accessible on supportshell if you're interested in looking
16:34:08 sean-k-mooney the GIL will save use somewhat but not entirely
16:34:24 dansmith sean-k-mooney: not all code, a ton of code.. there is a whole class of stuff that can't race because it's single threaded and can't overlap except at schedule points.. all of that stuff will suddenly be actually paralell
16:34:41 dansmith sean-k-mooney: it will save individual accesses to data structures, but not multiple statements making changes
16:34:55 melwitt maybe at the very least we could use futurist and make it configurable whether to use eventlet or native threading, and default to eventlet so as not to change existing behavior for those it works ok for
16:35:19 sean-k-mooney ya thats true
16:35:41 melwitt and let people like these customers try out the native threading and let us know if it works well in a real deployment or not
16:35:44 dansmith well, any change would have to be gradual like that I think.. meaning a switch to flip that we keep around for a while
16:35:58 dansmith otherwise we're going to flip the switch and not find out if we broke everyone for 18 months :)
16:36:23 sean-k-mooney we neeed "osapi_compute_workers" to be 1 though right and scale that via the process.
16:36:25 melwitt yeah, a good point
16:37:33 dansmith sean-k-mooney: that's a different concern I think
16:37:55 sean-k-mooney maybe the reinit issues for that have been fixed? but we used to have issue with the pultiple interperters running in the same wsgi process because of how it reloaded
16:37:56 dansmith not sure we need _workers at the point where we're actually natively threaded
16:38:16 melwitt sean-k-mooney: osapi_compute_workers is actually the number of processes but the wsgi.default_pool_size defaults to 1000 and represents the number of green threads for the nova-api eventlet based wsgi server
16:38:25 sean-k-mooney melwitt: ah ok
16:39:48 sean-k-mooney its https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.osapi_compute_workers
16:41:12 sean-k-mooney im still not sure it makes sense to sue that when running under a wsgi service
16:41:24 dansmith right,
16:41:27 dansmith that's unrelated I think
16:41:44 dansmith we'll never spawn our own worker processes when under uwsgi, AFAIK, we'll only spawn (green)threads
16:43:12 melwitt yeah I think with uwsgi or mod_wsgi the number of processes is configured by their respective configs
16:43:39 melwitt the osap_compute_workers is for other services or the old eventlet wsgi server we had provided back then https://github.com/openstack/nova/blob/stable/queens/nova/wsgi.py#L75
16:44:19 sean-k-mooney apparently its never used directly in the nova code
16:44:23 melwitt er sorry, osapi_compute_workers was only for nova-api. the other services have their own "workers" settings which map to the oslo.service workers
16:44:26 dansmith right, it's for when we spawn our own master and sub processes and listen on the socket ourselves
16:45:13 sean-k-mooney https://codesearch.opendev.org/?q=osapi_compute_workers&i=nope&files=&excludeFiles=&repos=openstack/nova
16:46:00 melwitt sean-k-mooney: it was here https://github.com/openstack/nova/blob/stable/queens/nova/service.py#L364
16:46:47 sean-k-mooney im wondering if it still used since it does not appear to be
16:48:00 sean-k-mooney anyway its proably unrelated to the db error
16:48:04 melwitt dansmith: I was thinking one of the reasons nova sees this more is because we use the eventlet executor for oslo.messaging any maybe other projects don't. that opens up a lot more chances to hit the error, I think
16:48:16 melwitt s/any/and/
16:48:33 sean-k-mooney melwitt: instent that the default executor
16:48:35 dansmith melwitt: as opposed to what? synchronous waiting?
16:48:46 melwitt they have a native threads executor
16:49:02 dansmith melwitt: but if eventlet is monkeypatching then they're the same I think
16:49:11 dansmith I mean, effectively the same
16:49:42 melwitt iiuc with the eventlet one, any rpc call coming into a service is in a green thread, so if it collides with a periodic task or scatter/gather, that's a chance for it to happen
16:50:33 melwitt dansmith: yeah, that is true but if other projects use the native threads executor and don't monkey patch that might be why they don't see it. afaik nova is the only project that monkey patches
16:50:38 dansmith melwitt: but if python's own threading library gets patched, the "native" one will be spawning gtreen threads too
16:50:49 dansmith melwitt: really?
16:51:02 sean-k-mooney i dont think they do
16:51:09 melwitt dansmith: yeah, I know, the configurable thing would only monkey patch if configured for eventlet, right?
16:51:24 melwitt sean-k-mooney: you don't think they monkey patch?
16:51:27 melwitt or you think they do
16:51:40 dansmith melwitt: I don't parse the configurable question
16:51:53 dansmith I'm not sure what the point of using eventlet without monkey patching is
16:52:13 dansmith otherwise you're just fully synchronous, you just have "threads" that run to completion all the time, AFAIK
16:52:22 melwitt dansmith: sorry, I guess I'm confused. I thought you were pointing out that making it configurable in nova would result in still having things be green threads
16:53:07 dansmith melwitt: I'm saying that if you were to ask for native threading in oslo.messaging, but you were monkeypatching python's thread library, then you're going to get greenthreads from your "native" o.msg threading module
16:53:14 melwitt yeah, I'm not sure if anyone else uses eventlet on purpose or if it's only indirectly through oslo.service or oslo.messaging
16:53:41 dansmith glance certainly does use it
16:53:56 dansmith they spawn background threads for import tasks
16:54:06 melwitt dansmith: ah, ok. yeah
16:54:10 melwitt hm, ok
16:54:13 dansmith and they do monkeypatch
16:54:24 dansmith because otherwise it would be kinda pointless
16:54:46 dansmith cinder monkeypatches too
16:55:20 dansmith and they call eventlet operations directly, in a looot of places
16:55:38 dansmith lots of direct threadpool and eventlet.sleep() interaction
16:55:41 melwitt ok, so when I began looking at this, I was assuming that nova does something different than anyone else and that's why we hit the error, but I could not find what that could be
16:55:43 dansmith so I think they're intentionally using eventlet
16:56:05 melwitt meaning, that we are not the only ones using eventlet, yet we're the only ones who hit this
16:56:46 dansmith well, that's what I'm trying to zero in on.. if we're the only ones.. why
16:57:24 melwitt so far, the only thing I can think of is we just having a lot more green threads flying around. either that, or there is something different about the way we do database interactions through sqla
16:58:21 melwitt I couldn't find anything when I looked, but obviously I could have missed something
17:02:09 sean-k-mooney is the reentrent call is happing as part of the rollback
17:02:37 sean-k-mooney i wonder if this could be releated to how we hanedl exction with teh scater gater implemation
17:03:41 melwitt sean-k-mooney: looks like if you don't choose an executor, it will detect whether you're monkey patched and if you are, it will use eventlet executor, else it will use native threading https://github.com/openstack/oslo.messaging/blob/5aa645b38b4c1cf08b00e687eb6c7c4b8a0211fc/oslo_messaging/_utils.py#L70
17:04:26 melwitt sean-k-mooney: yeah, something dies in the middle of the rollback and then the connection is left in a bad state and then when it's accessed again it raises that error
17:05:09 melwitt sean-k-mooney: that was one of the earliest theories but there are also bug report for this same thing in OSP10 when we didn't have scatter gather
17:05:15 sean-k-mooney welll what i was wondering is dont we have slighly odd exctpion handeling in the scater gater wehere we return the excptions instead of raisign them
17:05:19 sean-k-mooney or am i imagining things
17:05:20 melwitt and I have looked at those traces too

Earlier   Later