Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-05-24
13:51:45 lyarwood okay weird
13:51:52 lyarwood I thought this worked previously
13:51:56 stephenfin because the name-based search is done against the instances in the project
13:52:19 stephenfin I doubt it. A couple of command have an '--all-projects' option just for this
13:52:51 stephenfin * stephenfin double checks to make sure he didn't change anything recently, just in case
13:53:20 lyarwood ah right I was likely using the UUID in the past
13:53:57 stephenfin Nope, I added missing options but it's otherwise unchanged, save for some docs, since it was added in 2017
13:54:07 stephenfin I'd say so
13:56:17 lyarwood Yeah sorry, I've used UUIDs everywhere in the request-id presentation and that just threw me
14:35:49 lyarwood test_volume_backed_live_migration keeps failing on master at the moment btw
14:49:44 lyarwood Ah I see why, slow nodes and pre_live_migration is timing out
15:35:57 melwitt stephenfin: were you planning to work on trying to remove eventlet or would you mind if I took a try at it? the ptg discussion on that occurred earlier than I had come online that day
15:36:57 stephenfin melwitt: go for it. I am hoping to do something on it but it's a potential minefield that would benefit from many eyes
15:41:06 melwitt stephenfin: cool, thanks. I've gone quite down the rabbit hole related to eventlet on some downstream bugs, so I've some ideas now (fortunately or unfortunately)
16:06:51 dansmith melwitt: this is remove eventlet from api right?
16:07:44 melwitt dansmith: and potentially everything else too
16:09:12 dansmith melwitt: nothing else is threadsafe so I have a hard time imagining that being a thing
16:10:31 melwitt tl;dr is there's a bad interaction and failure mode between gevent/eventlet and pymysql and mysqlconnector wherein if a green thread is killed before a connection is cleaned up, it leaves it in an inconsistent state and the next attempt to use a connection blows up
16:11:47 melwitt I've spoken at length with zzzeek about this and my understanding is this can't be worked around or handled and that replacing our usage of eventlet with native threading or similar is the only way to avoid it
16:12:33 dansmith is there a pointer to something to read about it?
16:12:51 melwitt yeah, sec
16:15:01 dansmith api and conductor going to native threads are doable I think without too much crazy,
16:15:23 dansmith but I think compute will be a nightmare, but it also doesn't use the DB driver at all, so it should be immune
16:15:30 melwitt dansmith: this comment contains the relevant references https://bugzilla.redhat.com/show_bug.cgi?id=1927994#c45 the rest of that bug has a lot of comments, most of which are private because I'm not sure they help bring any clarity. it's been a long discussion on there
16:16:01 melwitt oh, the entire bug looks to be private /facepalm
16:16:14 melwitt https://github.com/PyMySQL/PyMySQL/issues/234
16:16:19 melwitt https://github.com/sqlalchemy/sqlalchemy/issues/3258
16:16:26 melwitt https://github.com/PyMySQL/PyMySQL/issues/260
16:16:33 melwitt those are the references ^
16:17:01 dansmith thanks, I can read it at least
16:17:16 dansmith so the assertion is that all openstack projects will have to undertake this conversion?
16:17:28 dansmith that's a pretty big deal
16:18:11 melwitt for whatever reason nova is the only one that seems affected by this, this error has not been found in any other service's logs in the same deployments that see it in nova often
16:19:16 dansmith and only nova-api?
16:19:17 melwitt and I think most of the appearance of the error is from back when we had the eventlet-based wsgi server for nova-api. my guess is that since moving away from that, our usage is much reduced and reduces the chances of hitting this
16:19:45 melwitt no I have seen traces in nova-scheduler and nova-conductor as well
16:20:23 dansmith okay, because nova-api already runs with a combination of native and green threads, so if it was just api, then that could be why,
16:20:45 dansmith but if scheduler and conductor see it as well, I'm not sure why no other service would be affected
16:22:53 melwitt yeah, I'm guessing it's because it becomes very rare when the only things using eventlet are the periodic tasks, timers/retries, and some bits of scatter/gather. the error shows up under high load and coexists with other various connection errors to the database
16:23:21 melwitt well, maybe "very rare" is not a good way to put it, "more rare"
16:25:43 melwitt I have searched for some way that nova does something different than any other service wrt to database access and found nothing
16:26:15 melwitt the only lead we have so far is that nova uses eventlet more than any other services do (afaik so far)
16:26:17 sean-k-mooney melwitt: i assume we have not had any downstream sqlalcamy updtes in rhel 7
16:27:21 sean-k-mooney does this happen in modren openstack that was for 13 so queens
16:27:48 melwitt sean-k-mooney: what do you mean, like version changes? I don't think so but zzzeek would have covered that
16:28:12 sean-k-mooney yep or perhaps a backport that could have caused it
16:28:59 melwitt sean-k-mooney: to your other question, I could find no bug reports for this for newer than 13
16:29:39 melwitt that could either mean it went away or that it happens rarely enough that no one has bothered reporting it. not sure what to think
16:30:09 dansmith melwitt: but nothing changed about conductor and scheduler that would account for them not showing the issue in later versions
16:30:37 dansmith meaning api going from evenlet to real wsgi is a change that could affect this, but that doesn't impact the other services
16:30:53 melwitt zzzeek anyway strongly recommended we stop using eventlet for its known issues with the mysql connectors
16:31:10 dansmith I've read the bug now, and I see that he has,
16:31:20 dansmith but it's not quite as simple as just turning it off
16:31:51 melwitt dansmith: yeah, I appreciate that. but it's hard to know if this is just not been reported or if it's really not there anymore
16:31:57 sean-k-mooney i wonder is this wsgi related
16:32:07 dansmith so much of what goes on changes if you don't have those yield points patched in
16:32:29 dansmith sean-k-mooney: melwitt says she has reports of it from scheduler and conductor, which is what really puzzles me
16:32:47 melwitt yeah, I know it's not simple as turning it off but afaict we could switch everything to native threads. I've started looking into it (just by having to explain to the customer where/when it's workaroundable and when it's not)
16:32:53 sean-k-mooney ok i was wondering if it was related to the issue with enabling multiple tread in the wsgi process
16:33:07 sean-k-mooney but if its in the schduler and conductor its not that
16:33:16 dansmith melwitt: native threads mean a ton of code can race that can't race now
16:33:42 sean-k-mooney dansmith: well im not sure about "cant race now" but i agree it would be more
16:33:43 melwitt dansmith: yeah, those logs are attached to the case and accessible on supportshell if you're interested in looking
16:34:08 sean-k-mooney the GIL will save use somewhat but not entirely
16:34:24 dansmith sean-k-mooney: not all code, a ton of code.. there is a whole class of stuff that can't race because it's single threaded and can't overlap except at schedule points.. all of that stuff will suddenly be actually paralell
16:34:41 dansmith sean-k-mooney: it will save individual accesses to data structures, but not multiple statements making changes
16:34:55 melwitt maybe at the very least we could use futurist and make it configurable whether to use eventlet or native threading, and default to eventlet so as not to change existing behavior for those it works ok for
16:35:19 sean-k-mooney ya thats true
16:35:41 melwitt and let people like these customers try out the native threading and let us know if it works well in a real deployment or not
16:35:44 dansmith well, any change would have to be gradual like that I think.. meaning a switch to flip that we keep around for a while
16:35:58 dansmith otherwise we're going to flip the switch and not find out if we broke everyone for 18 months :)
16:36:23 sean-k-mooney we neeed "osapi_compute_workers" to be 1 though right and scale that via the process.
16:36:25 melwitt yeah, a good point
16:37:33 dansmith sean-k-mooney: that's a different concern I think
16:37:55 sean-k-mooney maybe the reinit issues for that have been fixed? but we used to have issue with the pultiple interperters running in the same wsgi process because of how it reloaded
16:37:56 dansmith not sure we need _workers at the point where we're actually natively threaded
16:38:16 melwitt sean-k-mooney: osapi_compute_workers is actually the number of processes but the wsgi.default_pool_size defaults to 1000 and represents the number of green threads for the nova-api eventlet based wsgi server
16:38:25 sean-k-mooney melwitt: ah ok
16:39:48 sean-k-mooney its https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.osapi_compute_workers
16:41:12 sean-k-mooney im still not sure it makes sense to sue that when running under a wsgi service
16:41:24 dansmith right,
16:41:27 dansmith that's unrelated I think
16:41:44 dansmith we'll never spawn our own worker processes when under uwsgi, AFAIK, we'll only spawn (green)threads
16:43:12 melwitt yeah I think with uwsgi or mod_wsgi the number of processes is configured by their respective configs
16:43:39 melwitt the osap_compute_workers is for other services or the old eventlet wsgi server we had provided back then https://github.com/openstack/nova/blob/stable/queens/nova/wsgi.py#L75
16:44:19 sean-k-mooney apparently its never used directly in the nova code
16:44:23 melwitt er sorry, osapi_compute_workers was only for nova-api. the other services have their own "workers" settings which map to the oslo.service workers
16:44:26 dansmith right, it's for when we spawn our own master and sub processes and listen on the socket ourselves
16:45:13 sean-k-mooney https://codesearch.opendev.org/?q=osapi_compute_workers&i=nope&files=&excludeFiles=&repos=openstack/nova
16:46:00 melwitt sean-k-mooney: it was here https://github.com/openstack/nova/blob/stable/queens/nova/service.py#L364
16:46:47 sean-k-mooney im wondering if it still used since it does not appear to be
16:48:00 sean-k-mooney anyway its proably unrelated to the db error
16:48:04 melwitt dansmith: I was thinking one of the reasons nova sees this more is because we use the eventlet executor for oslo.messaging any maybe other projects don't. that opens up a lot more chances to hit the error, I think
16:48:16 melwitt s/any/and/
16:48:33 sean-k-mooney melwitt: instent that the default executor
16:48:35 dansmith melwitt: as opposed to what? synchronous waiting?
16:48:46 melwitt they have a native threads executor
16:49:02 dansmith melwitt: but if eventlet is monkeypatching then they're the same I think
16:49:11 dansmith I mean, effectively the same
16:49:42 melwitt iiuc with the eventlet one, any rpc call coming into a service is in a green thread, so if it collides with a periodic task or scatter/gather, that's a chance for it to happen

Earlier   Later