Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-31
20:29:24 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Fix python3 compatibility of rbd get_fsid https://review.opendev.org/664513
20:31:07 mriedem amorin: are you still working on https://review.opendev.org/#/c/667294/ ?
20:45:14 openstackgerrit Merged openstack/nova master: neutron: log something more useful in _get_instance_nw_info https://review.opendev.org/673694
21:43:21 efried mriedem: RequestContext question for ya
21:43:56 efried I've been tracking where nova-compute talks to Placement to do things like creating the compute node RP.
21:44:42 efried IIUC those interactions use a RequestContext that gets created when the service starts up, because they're done in a periodic, not based on a request coming from e.g. conductor or whatever.
21:45:00 efried which would be fine, except...
21:45:13 efried it doesn't seem to be the same context that's being used for logging.
21:45:25 mriedem b/c the request id changes etc
21:45:34 mriedem which is because of a thread local store for the context way up in oslo.context
21:45:36 mriedem and it's mega confusing
21:45:53 efried rightright, but it's a *different* context?
21:46:02 mriedem as mentioned https://bugs.launchpad.net/nova/+bug/1835958/comments/7
21:46:02 openstack Launchpad bug 1835958 in OpenStack Compute (nova) "Nova sync power state on large clusters causes poor performance" [Undecided,New]
21:46:21 efried viz: http://logs.openstack.org/86/672986/2/check/nova-live-migration/0171fa9/logs/screen-n-cpu.txt.gz#_Jul_30_22_09_35_334853
21:46:22 efried here ^ you can see that the log line is showing one request_id, but the context we're using for the placement request has a different request_id.
21:47:58 mriedem req-428963d8-bc71-4823-8e35-97db13886690 is just some admin context created when the compute service starts up right?
21:48:20 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1321
21:48:33 mriedem so you're using a random context there with a random request_id
21:48:50 efried req-42... is the placement-side local request_id, just ignore that one.
21:48:51 mriedem because https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L273
21:49:20 mriedem i was looking at that one because
21:49:20 mriedem Jul 30 22:09:35.334349 ubuntu-bionic-rax-ord-0009499845 nova-compute[28986]: INFO nova.scheduler.client.report [None req-fb42d5e6-1c0d-41d0-88c5-a4f1ac127a41 None None] [req-428963d8-bc71-4823-8e35-97db13886690] Created resource provider record via placement API for resource provider with UUID 5dcb9522-0513-440a-80db-d985b5d45704 and name ubuntu-bionic-rax-ord-0009499845.
21:49:35 efried req-fb42d5e6-1c0d-41d0-88c5-a4f1ac127a41 is the oslo.context one that oslo.log is paying attention to
21:49:35 efried req-8a7e2cc5-3870-4118-b55f-4533d7a4de80 is the one in the RequestContext being used for the placement communication.
21:50:07 efried yeah, I know, I had to track that one down as well. I started this whole thing because I was wondering why tf we needed to print that reqid at all.
21:50:37 efried it's the one being sent *back* to us in the placement response headers, which turns out to be the (not-global!) request_id on the placement side
21:50:51 efried and because of this whole snafu, it's currently the only way for us to correlate the requests on the nova & placement sides
21:51:04 efried but that flies in the face of the entire global_request_id concept
21:51:38 efried if we get global_request_id working, we don't need that at all because we'll be able to correlate with global_request_id, which is the whole friggin point.
21:51:48 efried so my question is this
21:52:21 mriedem idk what oslo.log is doing really but i know things get wonky with this https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L300
21:52:38 efried what about if, when we create the RequestContext on compute startup, we a) set its global_request_id, and b) pass overwrite=True so oslo.context replaces it in the threadlocal
21:53:15 efried yeah, b) would call update_store and replace that.
21:54:08 efried I think having talked through this I've got enough to do that ^ in a WIP and see what shakes out.
21:55:55 mriedem ....ok
21:56:12 openstackgerrit Merged openstack/nova master: Move adding vlans to interfaces to privsep. https://review.opendev.org/635436
21:56:30 mriedem i know melwitt https://review.opendev.org/#/c/524306/ and gibi https://review.opendev.org/#/c/542891/ have muddled in this and it's a mess
21:56:36 mriedem so i guess welcome to the tarpits
21:57:37 efried mriedem: I see that get_context() and get_admin_context() explicitly set overwrite=False. The former even says it's to not mess with logging. Why??
21:57:56 mriedem oh eric, i could tell you but that would be too easy
21:58:47 mriedem if you can hunt down vishy from 2012 maybe he can answer https://review.opendev.org/#/c/2923/
21:59:51 mriedem or i guess alaski from 2016 https://review.opendev.org/#/c/378941/
22:01:01 mriedem the admin context stuff is also a bit legacy from when db apis were explicitly blocked unless you had an admin context, which alex_xu removed a long time ago
22:01:09 mriedem but still affects project_id filtering in the model_query
22:01:15 mriedem as melwitt can probably go into detail about
22:01:25 mriedem as for the logging and thread local stuff, ....idk
22:05:51 openstackgerrit Eric Fried proposed openstack/nova master: Store a RequestContext on service to reuse for periodic tasks https://review.opendev.org/524306
22:05:52 openstackgerrit Eric Fried proposed openstack/nova master: Poison context usage in periodic tasks https://review.opendev.org/542891
22:07:35 efried melwitt, gibi_off: FYI, rebased & resolved merge conflicts ^. I'm going to throw one on top that overwrites the threadlocal context and creates a global_request_id so our logging might be sane. We'll see what explodes.
22:07:39 efried mriedem: fyi ^
22:10:53 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Generate and log global_request_id properly https://review.opendev.org/673924
22:13:57 openstackgerrit Eric Fried proposed openstack/nova master: Move iptables rule fetching and setting to privsep. https://review.opendev.org/636508
22:13:58 openstackgerrit Eric Fried proposed openstack/nova master: Move dnsmasq restarts to privsep. https://review.opendev.org/639280
22:13:58 openstackgerrit Eric Fried proposed openstack/nova master: Move router advertisement daemon restarts to privsep. https://review.opendev.org/639281
22:13:59 openstackgerrit Eric Fried proposed openstack/nova master: Move calls to ovs-vsctl to privsep. https://review.opendev.org/639282
22:13:59 openstackgerrit Eric Fried proposed openstack/nova master: Move setting of device trust to privsep. https://review.opendev.org/639283
22:14:00 openstackgerrit Eric Fried proposed openstack/nova master: Move final bridge commands to privsep. https://review.opendev.org/639580
22:14:00 openstackgerrit Eric Fried proposed openstack/nova master: Cleanup the _execute shim in nova/network. https://review.opendev.org/639581
22:14:01 openstackgerrit Eric Fried proposed openstack/nova master: We no longer need rootwrap. https://review.opendev.org/554438
22:14:01 openstackgerrit Eric Fried proposed openstack/nova master: Cleanup no longer required filters and add a release note. https://review.opendev.org/639826
22:24:29 openstackgerrit Merged openstack/nova master: Remove Request Spec Migration upgrade status check https://review.opendev.org/673580
22:31:23 efried o/
#openstack-nova - 2019-08-02
02:53:25 openstackgerrit Merged openstack/nova master: Follow-up for I2936ce8cb293dc80e1a426094fdae6e675461470 https://review.opendev.org/672669
02:53:35 openstackgerrit Merged openstack/nova master: nit: fix the test case of migration obj_make_compatible https://review.opendev.org/673961
02:58:37 openstackgerrit pengyuesheng proposed openstack/os-resource-classes master: Sync Sphinx requirement https://review.opendev.org/674164
03:00:25 openstackgerrit Eric Fried proposed openstack/nova master: Correct global_request_id sent to Placement https://review.opendev.org/674129
05:34:22 openstackgerrit Merged openstack/nova stable/rocky: Fix python3 compatibility of rbd get_fsid https://review.opendev.org/664513
07:51:46 aspiers o]
07:52:08 aspiers wow, that was an unintentional dance move, let's try again
07:52:10 aspiers o/
07:52:34 kashyap Hehe
07:52:59 kashyap Morning, /me is especially feeling high-spirited today
07:53:14 kashyap Maybe it's the accidental electric shock I got last night while plugging in a bad cable :D
07:55:26 cdent one more shock and you will become the superhero you were born to be
07:55:58 kashyap Hehe. That really shook me
08:42:13 aspiers X-D
09:13:14 openstackgerrit Huachang Wang proposed openstack/nova master: doc: correct the information of 'cpu_map' https://review.opendev.org/673272
09:34:01 openstackgerrit Huachang Wang proposed openstack/nova master: Add the support of CPU feature 'AVX512-VNNI' https://review.opendev.org/673480
09:39:37 openstackgerrit Brin Zhang proposed openstack/nova master: WIP: Add user_id and project_id colume to Migration https://review.opendev.org/673990
09:39:37 openstackgerrit Brin Zhang proposed openstack/nova master: WIP: Filter migrations by user_id/project_id https://review.opendev.org/674243
11:58:39 openstackgerrit Merged openstack/os-vif master: Bump the openstackdocstheme extension to 1.20 https://review.opendev.org/672857
12:29:35 openstackgerrit Huachang Wang proposed openstack/nova master: doc: correct the information of 'cpu_map' https://review.opendev.org/673272
13:05:00 openstackgerrit Merged openstack/nova master: (Re-)enable vnc console tests in nova-multi-cell job https://review.opendev.org/667388
14:12:49 aspiers is there an easy way to get encrypted ephemeral disks with devstack?
14:16:26 melwitt that would be a lyarwood question ^
14:20:14 aspiers melwitt: thanks!
14:20:22 aspiers it seems that it requires lvm
14:20:48 aspiers so just adding to [[post-config|$NOVA_CONF]] isn't enough... I think
14:21:34 openstackgerrit Stephen Finucane proposed openstack/nova master: Integrate 'pre-commit' https://review.opendev.org/665518
14:21:35 openstackgerrit Stephen Finucane proposed openstack/nova master: Keep pre-commit inline with hacking and fix whitespace https://review.opendev.org/674057
14:29:17 lyarwood aspiers: yeah there's only LVM support for that at the moment
14:29:23 aspiers lyarwood: that's fine
14:29:35 aspiers enable_plugin barbican
14:29:37 aspiers NOVA_BACKEND=LVM
14:29:47 aspiers is that enough, combined with the extra config options?
14:29:53 openstackgerrit Merged openstack/nova stable/stein: Fix AttributeError in RT._update_usage_from_migration https://review.opendev.org/673856
14:29:58 aspiers [[post-config|$NOVA_CONF]]
14:30:00 aspiers [ephemeral_storage_encryption]
14:30:02 aspiers key_size = 256

Earlier   Later