Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-25
15:46:01 mriedem melwitt: ah it was the per-project thing that was busted, maybe that's what i talked about with corvus before
15:46:05 mriedem thanks for fixing that
15:50:45 lajoskatona mriedem: Hi, regardig https://review.opendev.org/640600 & https://review.opendev.org/640601 (tempest schema validation for nova v2.70 & 2.71) if you need help to push them I am happy to help you out
15:54:35 mriedem lyarwood: fyi https://graphite.opendev.org/render/?width=1231&height=593&_salt=1556207646.49&from=-8weeks&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.opendev_org.openstack_nova.master.job.tempest-full-py3.FAILURE&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.opendev_org.openstack_nova.master.job.devstack-plugin-ceph-tempest.FAILURE
15:54:39 mriedem you can get the graph from graphite
15:55:06 mriedem lajoskatona: go ahead, i'm not working on those
15:55:42 mriedem melwitt: nice catch on the tempest-full-py3 vs devstack-plugin-ceph-tempest (not py3) in the nova case, we should probably change nova to run against the devstack-plugin-ceph-tempest-py3 job instead
15:55:56 lajoskatona mriedem: ok, do I need any special thing to test in devstack?
15:56:21 melwitt mriedem: yeah, I was wondering that too
15:56:56 mriedem melwitt: that will throw the graph data off, but we could switch now, make your graph change depend on that and then revisit in a couple of weeks to compare stability
15:57:08 mriedem or just do that in a follow up, whatever
15:57:14 mriedem lajoskatona: i'm not sure i understand the question
16:00:23 lajoskatona mriedem: never mind, I try to to push, and come back if I have questions :-)
16:00:42 mriedem lajoskatona: ack and thanks
16:01:48 lajoskatona mriedem: :-)
16:11:11 openstackgerrit Dan Smith proposed openstack/nova master: Add vmware driver image type capabilities https://review.opendev.org/655730
16:11:11 openstackgerrit Dan Smith proposed openstack/nova master: Add ironic driver image type capabilities https://review.opendev.org/655729
16:11:12 openstackgerrit Dan Smith proposed openstack/nova master: Add zvm driver image type capabilities https://review.opendev.org/655732
16:11:12 openstackgerrit Dan Smith proposed openstack/nova master: Add xenapi driver image type capabilities https://review.opendev.org/655731
16:12:05 dansmith I dunno who to tag on the zvm one for review
16:12:12 dansmith but I'm pretty sure they only support raw
16:13:17 melwitt mriedem: I dunno if you saw, they did hit the eventlet monkey patching issue in ubuntu, last comment https://review.opendev.org/647310
16:16:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Always pass HostAPI to get_availability_zones https://review.opendev.org/655583
16:35:35 mriedem melwitt: yeah i saw
16:35:44 mriedem coreycb didn't +1 it though...
16:35:57 mriedem dansmith: i suppose add jichenjc
16:38:50 melwitt I guess I took his comment on the review to be in support. but yeah, coreycb if you could +1 https://review.opendev.org/647310 if you are in support of backporting it
16:40:38 imacdonn somewhat of a tangent, I suppose ... but I think we need to figure out what to do about the eventlet monkey_patching vs. WSGI issue ... it's blocking me from upgrading to Stein
16:42:54 melwitt afaik there's not a clear thing we should do to address the problem, or did I miss it? eventlet monkey patching is not something I understand well
16:44:35 imacdonn same here
16:45:19 imacdonn the nova docs rather strongly suggest moving to WSGI, but maybe my only option is to go back to running nova-api the old way :/
16:46:22 imacdonn alternative interim may be to hack the monkey-patching out, but I don't understand the implications of that well enough ...
16:46:42 dansmith yeah I don't think you can just remove it
16:47:00 imacdonn I don't have it in my Rocky deployments, though
16:47:20 dansmith don't have what? the problem or the monkeypatching?
16:47:25 imacdonn both ;)
16:48:11 dansmith hrm, I'm trying to think about what the api service does that might continue to work without being patched
16:48:36 dansmith if we use all the regular primitives, then it might just work with regular threading, but I expect there are some subtle behavior assumptions
16:48:38 imacdonn it was added here: https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae
16:48:40 dansmith locks we don't hold
16:48:45 melwitt I have a patch series that is going on the route to removing eventlet dependency for nova-api https://review.opendev.org/650172 but it needs a follow up patch that makes the executor type configurable (or outright changes it away from green threads)
16:49:32 dansmith imacdonn: right, but it was added because we didn't realize that we weren't patching when the wsgi stuff was split out from service originally, IIRC
16:49:55 melwitt I'm not sure if there are other reasons aside from scatter-gather that eventlet is a dependency in nova-api
16:50:20 imacdonn dansmith: right, and presumably someone decided that it was, infact, needed, with WSGI
16:50:48 imacdonn dansmith: there's mention of multiple cells (which I don't have (other than cell0)) ... but dunno if there are other cases where it's "needed"
16:50:56 dansmith melwitt: that'd be the big one, but any other spawn we do in some layer that can be called from api and other services
16:51:01 dansmith imacdonn: right but even with cell0 you're affected by that
16:51:21 coreycb melwitt: mriedem: added my +1
16:52:04 dansmith I've been a bit disconnected.. is the problem that seems to be introduced the amq heartbeat thing?
16:53:04 mriedem imacdonn: i thought https://review.opendev.org/#/c/647310/ was in response to https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae in a way, but you said that fix doesn't help you right?
16:54:32 imacdonn mriedem: that just changes where monkey-patching gets inserted ... I have my problem either way
16:57:01 mriedem i guess you could try exporting OS_NOVA_DISABLE_EVENTLET_PATCHING=1 but that's likely not something you want to run with
16:59:35 imacdonn yeah
16:59:39 mriedem dansmith: there are only 4 places we use the long_rpc_timeout heartbeat stuff and they'd all be calls from either conductor or compute neither of which are wsgi so i'm not sure how that would be involved, unless it was the moving of the monkey patch location in https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae
17:00:25 dansmith mriedem: I think the long rpc stuff all uses regular threading primitives so it should work regardless of if you're patched or not right?
17:00:45 mriedem idk, i've never looked at the oslo.messaging implementation
17:00:53 dansmith mriedem: do you have reason to believe the the long rpc stuff is related, or are you just looking for related things?
17:01:06 mriedem imacdonn: have you tried putting melwitt's series https://review.opendev.org/#/q/topic:cell-scatter-gather-futurist+(status:open+OR+status:merged) into rocky and see if that fixes your issue?
17:01:31 dansmith mriedem: that patch doesn't actually change anything yet, IIUC
17:01:34 mriedem dansmith: just in response to "(11:52:08 AM) dansmith: I've been a bit disconnected.. is the problem that seems to be introduced the amq heartbeat thing?"
17:01:40 dansmith it just wraps use of the eventlet stuff so we can do it differentyly
17:01:47 melwitt that's still using eventlet, you'd have to change the futurist executor to the native one
17:01:58 dansmith yeah ^
17:02:29 dansmith mriedem: I think you're confusing heartbeats no?
17:02:58 dansmith mriedem: the heartbeat that long_rpc added is layer 7, unrelated to AMQ heartbeats, which I thought was the subject of that thread
17:03:18 mriedem oh i guess i am then
17:03:28 mriedem i'll step out of this mess (this is why i'm not involved)
17:04:07 dansmith mriedem: yeah, re-reading, they're getting a connection timeout because AMQP heartbeats aren't being sent (a different thing)
17:04:20 dansmith the theory is that eventlet patching prevents some thread from running that sends those
17:04:25 dansmith totes unrelated to long_rpc
17:04:40 imacdonn yes, that's how it appears
17:13:26 openstackgerrit melanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.opendev.org/507486
17:21:04 openstackgerrit Adam Spiers proposed openstack/os-traits master: Update SEV trait docs to avoid misleading people https://review.opendev.org/655671
17:38:57 dansmith melwitt: the Future object makes it sound like cancel() might not always be supported.. have you confirmed?
17:39:53 melwitt dansmith: yeah, was just writing that on the review. I think it only does something for eventlet GreenThreadPoolExecutor. been looking around in the source to confirm because it doesn't seem like the docs are explicit about which types support cancel()
17:39:55 dansmith melwitt: "If the call is currently being executed and cannot be cancelled then the method will return"
17:40:31 dansmith I think since it's really just a wrapper around a queue, it's just going to cancel if the thing hasn't been dequeued yet
17:40:37 melwitt yeah, cancel() returns True/False and for native threads I expect it just always returns False
17:40:37 dansmith might not even kill a thread in process, I dunno
17:40:40 melwitt right
17:42:04 melwitt yeah, I don't see futurist doing anything different for cancel() other than passing it through to the primitives
17:43:02 dansmith yeah
17:43:09 melwitt so it would be a matter of what does the Future primitive do for cancel() for the different types of executors
17:43:20 melwitt not really seeing it documented on https://docs.python.org/3/library/concurrent.futures.html#future-objects
17:43:25 dansmith melwitt: just to be clear, when I say the primitives, I'm talking about the threading library not the concurrent stuff
17:43:32 melwitt oh, ok. my bad
17:43:37 dansmith the latter is the high-level implementation, which may very well not be in py2
17:44:09 dansmith concurrent (and c.futures) just use the low-level primitives in threading, I imagine
17:44:17 melwitt yeah, it's not but it's all included in a separate futures package containing backports https://pypi.org/project/futures/
17:44:19 dansmith which is why I said what I did
17:44:25 dansmith aye
17:44:43 melwitt yeah, I imagine the same, using same low-level primitives underneath
17:51:18 openstackgerrit Artom Lifshitz proposed openstack/nova master: Revert "Wait for network-vif-plugged on resize revert" https://review.opendev.org/639396
17:51:18 openstackgerrit Artom Lifshitz proposed openstack/nova master: Run revert resize tests in nova-live-migration https://review.opendev.org/653498
17:51:19 openstackgerrit Artom Lifshitz proposed openstack/nova master: Revert resize: wait for external events in compute manager https://review.opendev.org/644881
17:51:59 openstackgerrit Merged openstack/os-vif master: Remove IP proxy methods https://review.opendev.org/655695
17:57:27 mriedem the cancel() thing came up in kevin's changes for canceling queued live migrations,
17:57:33 mriedem there is probably something in the compute manager code around that
17:58:54 mriedem oh right cancel() just returns false if it's already executing and you can't stop it
17:59:04 melwitt yeah, so far I'm not seeing in the futurist source that it event does a green thread kill() when cancel() is called with the GreenThreadPoolExecutor

Earlier   Later