Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-05-24
16:34:24 dansmith sean-k-mooney: not all code, a ton of code.. there is a whole class of stuff that can't race because it's single threaded and can't overlap except at schedule points.. all of that stuff will suddenly be actually paralell
16:34:41 dansmith sean-k-mooney: it will save individual accesses to data structures, but not multiple statements making changes
16:34:55 melwitt maybe at the very least we could use futurist and make it configurable whether to use eventlet or native threading, and default to eventlet so as not to change existing behavior for those it works ok for
16:35:19 sean-k-mooney ya thats true
16:35:41 melwitt and let people like these customers try out the native threading and let us know if it works well in a real deployment or not
16:35:44 dansmith well, any change would have to be gradual like that I think.. meaning a switch to flip that we keep around for a while
16:35:58 dansmith otherwise we're going to flip the switch and not find out if we broke everyone for 18 months :)
16:36:23 sean-k-mooney we neeed "osapi_compute_workers" to be 1 though right and scale that via the process.
16:36:25 melwitt yeah, a good point
16:37:33 dansmith sean-k-mooney: that's a different concern I think
16:37:55 sean-k-mooney maybe the reinit issues for that have been fixed? but we used to have issue with the pultiple interperters running in the same wsgi process because of how it reloaded
16:37:56 dansmith not sure we need _workers at the point where we're actually natively threaded
16:38:16 melwitt sean-k-mooney: osapi_compute_workers is actually the number of processes but the wsgi.default_pool_size defaults to 1000 and represents the number of green threads for the nova-api eventlet based wsgi server
16:38:25 sean-k-mooney melwitt: ah ok
16:39:48 sean-k-mooney its https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.osapi_compute_workers
16:41:12 sean-k-mooney im still not sure it makes sense to sue that when running under a wsgi service
16:41:24 dansmith right,
16:41:27 dansmith that's unrelated I think
16:41:44 dansmith we'll never spawn our own worker processes when under uwsgi, AFAIK, we'll only spawn (green)threads
16:43:12 melwitt yeah I think with uwsgi or mod_wsgi the number of processes is configured by their respective configs
16:43:39 melwitt the osap_compute_workers is for other services or the old eventlet wsgi server we had provided back then https://github.com/openstack/nova/blob/stable/queens/nova/wsgi.py#L75
16:44:19 sean-k-mooney apparently its never used directly in the nova code
16:44:23 melwitt er sorry, osapi_compute_workers was only for nova-api. the other services have their own "workers" settings which map to the oslo.service workers
16:44:26 dansmith right, it's for when we spawn our own master and sub processes and listen on the socket ourselves
16:45:13 sean-k-mooney https://codesearch.opendev.org/?q=osapi_compute_workers&i=nope&files=&excludeFiles=&repos=openstack/nova
16:46:00 melwitt sean-k-mooney: it was here https://github.com/openstack/nova/blob/stable/queens/nova/service.py#L364
16:46:47 sean-k-mooney im wondering if it still used since it does not appear to be
16:48:00 sean-k-mooney anyway its proably unrelated to the db error
16:48:04 melwitt dansmith: I was thinking one of the reasons nova sees this more is because we use the eventlet executor for oslo.messaging any maybe other projects don't. that opens up a lot more chances to hit the error, I think
16:48:16 melwitt s/any/and/
16:48:33 sean-k-mooney melwitt: instent that the default executor
16:48:35 dansmith melwitt: as opposed to what? synchronous waiting?
16:48:46 melwitt they have a native threads executor
16:49:02 dansmith melwitt: but if eventlet is monkeypatching then they're the same I think
16:49:11 dansmith I mean, effectively the same
16:49:42 melwitt iiuc with the eventlet one, any rpc call coming into a service is in a green thread, so if it collides with a periodic task or scatter/gather, that's a chance for it to happen
16:50:33 melwitt dansmith: yeah, that is true but if other projects use the native threads executor and don't monkey patch that might be why they don't see it. afaik nova is the only project that monkey patches
16:50:38 dansmith melwitt: but if python's own threading library gets patched, the "native" one will be spawning gtreen threads too
16:50:49 dansmith melwitt: really?
16:51:02 sean-k-mooney i dont think they do
16:51:09 melwitt dansmith: yeah, I know, the configurable thing would only monkey patch if configured for eventlet, right?
16:51:24 melwitt sean-k-mooney: you don't think they monkey patch?
16:51:27 melwitt or you think they do
16:51:40 dansmith melwitt: I don't parse the configurable question
16:51:53 dansmith I'm not sure what the point of using eventlet without monkey patching is
16:52:13 dansmith otherwise you're just fully synchronous, you just have "threads" that run to completion all the time, AFAIK
16:52:22 melwitt dansmith: sorry, I guess I'm confused. I thought you were pointing out that making it configurable in nova would result in still having things be green threads
16:53:07 dansmith melwitt: I'm saying that if you were to ask for native threading in oslo.messaging, but you were monkeypatching python's thread library, then you're going to get greenthreads from your "native" o.msg threading module
16:53:14 melwitt yeah, I'm not sure if anyone else uses eventlet on purpose or if it's only indirectly through oslo.service or oslo.messaging
16:53:41 dansmith glance certainly does use it
16:53:56 dansmith they spawn background threads for import tasks
16:54:06 melwitt dansmith: ah, ok. yeah
16:54:10 melwitt hm, ok
16:54:13 dansmith and they do monkeypatch
16:54:24 dansmith because otherwise it would be kinda pointless
16:54:46 dansmith cinder monkeypatches too
16:55:20 dansmith and they call eventlet operations directly, in a looot of places
16:55:38 dansmith lots of direct threadpool and eventlet.sleep() interaction
16:55:41 melwitt ok, so when I began looking at this, I was assuming that nova does something different than anyone else and that's why we hit the error, but I could not find what that could be
16:55:43 dansmith so I think they're intentionally using eventlet
16:56:05 melwitt meaning, that we are not the only ones using eventlet, yet we're the only ones who hit this
16:56:46 dansmith well, that's what I'm trying to zero in on.. if we're the only ones.. why
16:57:24 melwitt so far, the only thing I can think of is we just having a lot more green threads flying around. either that, or there is something different about the way we do database interactions through sqla
16:58:21 melwitt I couldn't find anything when I looked, but obviously I could have missed something
17:02:09 sean-k-mooney is the reentrent call is happing as part of the rollback
17:02:37 sean-k-mooney i wonder if this could be releated to how we hanedl exction with teh scater gater implemation
17:03:41 melwitt sean-k-mooney: looks like if you don't choose an executor, it will detect whether you're monkey patched and if you are, it will use eventlet executor, else it will use native threading https://github.com/openstack/oslo.messaging/blob/5aa645b38b4c1cf08b00e687eb6c7c4b8a0211fc/oslo_messaging/_utils.py#L70
17:04:26 melwitt sean-k-mooney: yeah, something dies in the middle of the rollback and then the connection is left in a bad state and then when it's accessed again it raises that error
17:05:09 melwitt sean-k-mooney: that was one of the earliest theories but there are also bug report for this same thing in OSP10 when we didn't have scatter gather
17:05:15 sean-k-mooney welll what i was wondering is dont we have slighly odd exctpion handeling in the scater gater wehere we return the excptions instead of raisign them
17:05:19 sean-k-mooney or am i imagining things
17:05:20 melwitt and I have looked at those traces too
17:05:26 sean-k-mooney never mind then
17:06:12 dansmith melwitt: is it always during a scatter/gather?
17:06:39 melwitt dansmith: no, it happened in OSP10 too when we didn't have scatter/gather
17:07:21 melwitt and I have seen traces where it was raised from a "get quotas" call, from service_update I have seen a lot
17:07:26 dansmith oh right, I read that
17:07:31 lyarwood https://bugs.launchpad.net/nova/+bug/1929446 - This is the issue I was highlighting earlier if anyone has time to help narrow this down a little.
17:07:32 openstack Launchpad bug 1929446 in OpenStack Compute (nova) "check_can_live_migrate_source taking > 60 seconds in CI" [Undecided,New]
17:07:52 sean-k-mooney i am not sure this is evently related
17:07:58 sean-k-mooney it might be
17:08:08 sean-k-mooney but the nova api was not alwasy monky patched
17:08:22 melwitt there was a window when it wasn't
17:08:35 sean-k-mooney if you ran it under uwisgi before scater gatter was added it was not monkey patched
17:08:44 melwitt but it was prior to uwsgi/mod_wsgi being a way to run nova-api
17:08:45 sean-k-mooney the comman line nova-api alwasy was
17:08:48 melwitt right
17:09:08 melwitt *but it was monkey patched
17:09:18 sean-k-mooney no we had a perfiod of time when uwsgi was supported but we did not monkey patch
17:09:30 melwitt I know
17:09:31 sean-k-mooney athough we may not have relased that way downstream
17:09:40 melwitt I'm saying that prior to uwsgi it was always monkey patched
17:09:51 sean-k-mooney ah yes it was
17:10:20 sean-k-mooney osp 10 is what newton i think we just used the nova-api command directly at that point
17:10:27 melwitt but even if we stop monkey patching in nova-api, we will still see this in nova-scheduler and nova-conductor at least
17:11:00 sean-k-mooney is there a cler writeup of how the error happens
17:11:02 melwitt yeah. and in the sosreports I've looked at for 13 it's also the nova-api command in these cases, afaict from the ps output
17:11:49 sean-k-mooney i think ooo avoid using apache initally due to concens of memory overhead
17:11:49 melwitt sean-k-mooney: yeah but not in the context of openstack. this is a private bug but here https://bugzilla.redhat.com/show_bug.cgi?id=1927994#c45 and the links are https://github.com/PyMySQL/PyMySQL/issues/234 https://github.com/sqlalchemy/sqlalchemy/issues/3258 https://github.com/PyMySQL/PyMySQL/issues/260
17:11:50 openstack melwitt: Error: Error getting bugzilla.redhat.com bug #1927994: NotPermitted

Earlier   Later