| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-06 | |||
| 14:07:26 | gibi | so in summary If your approach works then I'm OK to abbandon the old series | |
| 14:09:21 | efried | gibi: I'm sure there's still more to do. This is not a simple thing. | |
| 14:09:57 | efried | glancing at the n-api log, my initial thought is that the global_request_id is being persisted too long. | |
| 14:10:12 | efried | I think what we probably want to do is change it any time we get a new incoming API request. | |
| 14:11:00 | efried | As yet, I have no idea where in the code would be a good place to do that... | |
| 14:15:36 | gibi | efried: does this in connection with the fact that periodic tasks also needs a context / global_request_id but they are not triggered by a specific API request? | |
| 14:19:55 | efried | gibi: Yup, that's an issue as well. | |
| 14:20:19 | efried | I'm not sure whether periodics ought to have one global_request_id forever, or if they should get a new one every time they trigger. | |
| 14:21:01 | efried | but I'm *pretty* sure they shouldn't share a global_request_id with any actual operations from an API request | |
| 14:23:24 | cdent | efried: new one each job would be most useful when grepping (or otherwise searching) | |
| 14:23:42 | efried | cdent: "job", are you talking about periodics or API requests? | |
| 14:23:48 | cdent | periodics | |
| 14:24:25 | efried | Okay. Meanwhile cdent I could probably use your wsgi skills here... | |
| 14:24:44 | efried | can you help me find where nova does that middleware thing with the request_id? | |
| 14:24:55 | cdent | yeah, one sec | |
| 14:26:09 | efried | actually, I don't think that's what I want. | |
| 14:26:14 | efried | Cause that part is working | |
| 14:26:43 | efried | problem right now is, that req ID is persisting way too long in the API thread. | |
| 14:26:55 | cdent | efried: nova/api/compute_req_id.py has a subclass of oslo_middleware.request_id.RequestId | |
| 14:27:14 | efried | Yeah, that's setting the header inbound & outbound, which is working. | |
| 14:27:22 | efried | sorry, inbound only | |
| 14:27:25 | cdent | efried: is it possible that tempest is _sending_ it | |
| 14:28:52 | efried | cdent: looks like not. | |
| 14:29:00 | efried | which is actually fine for now | |
| 14:29:13 | cdent | efried: i haven't got enough context loaded in at the moment | |
| 14:29:25 | efried | actually if tempest *were* sending it, I wouldn't be seeing the behavior I'm seeing | |
| 14:29:36 | efried | cdent: So what I'm looking for is this: | |
| 14:29:45 | efried | Presumably the API server sits there waiting for requests | |
| 14:29:56 | efried | when a request comes in, it... spawns? reuses? a thread to handle it | |
| 14:30:21 | efried | Whatever RequestContext that thread gets needs to get a fresh global_request_id | |
| 14:30:31 | efried | early as possible | |
| 14:30:59 | efried | so - I need to grab hold of the code that spins off that thread | |
| 14:31:39 | efried | (btw, ^ is for API requests obvi. Periodics will be a separate thing entirely.) | |
| 14:33:01 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/auth.py#L57 is what creates the RequestContext in the compute API | |
| 14:35:21 | efried | mriedem: yup https://review.opendev.org/#/c/674138/4/nova/api/openstack/auth.py | |
| 14:35:41 | efried | So that's working now | |
| 14:35:57 | efried | the thing I'm trying to tackle now is, I want each API request to get a new global_request_id. | |
| 14:36:16 | efried | because right now it's getting reused for multiple operations | |
| 14:36:33 | efried | Actually, cdent, I think perhaps that compute_req_id might be the place to do it. | |
| 14:36:59 | efried | The superclass is pulling in a request ID from headers if available... but what I can do is *set* a new one if there wasn't one in the header. | |
| 14:43:20 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Always set a global_request_id in RequestContext https://review.opendev.org/674138 | |
| 14:43:22 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Correct global_request_id sent to Placement https://review.opendev.org/674129 | |
| 14:43:36 | efried | cdent, mriedem: let's see how that shakes out ^ | |
| 14:46:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Replace non-nova server fault message https://review.opendev.org/674821 | |
| 14:46:34 | mriedem | dansmith: ^ | |
| 14:46:47 | fungi | i'm watching these go up too, thanks mriedem! | |
| 14:47:07 | dansmith | mriedem: ack | |
| 14:55:47 | cdent | sorry efried was drawn away into another converation | |
| 14:56:07 | cdent | I had thought that the existing one was supposed to set a new one | |
| 14:56:08 | cdent | (if not present) | |
| 14:58:16 | efried | cdent: If you look at the parent impl, it doesn't. | |
| 14:58:46 | efried | Previous PS of my change did it further down, but too late, and sucked up whatever was in thread local already if one wasn't set. | |
| 14:59:22 | efried | It's possibly the guts of my ensure_global_id never gets hit now, which would possibly be a good thing. | |
| 14:59:44 | efried | mriedem: rebuild... gets a new image or no? | |
| 15:00:01 | openstackgerrit | Brin Zhang proposed openstack/nova master: Change the response to migrations of Show Migration Details API https://review.opendev.org/674825 | |
| 15:00:09 | efried | okay, yeah, looks like it. | |
| 15:01:21 | mriedem | efried: optional | |
| 15:01:49 | mriedem | https://docs.openstack.org/api-ref/compute/#rebuild-server-rebuild-action | |
| 15:01:59 | mriedem | rebuild of a volume-backed server is never a new image - you get a 400 if you try | |
| 15:01:59 | efried | mriedem: ack. I was looking at https://review.opendev.org/#/c/673407/ and requesting a func test for rebuild as well as spawn, but that wouldn't have made any sense if you couldn't specify a new image for rebuild. | |
| 15:02:04 | cdent | efried: i'm guessing that middleware is making assumptions about clients, which is unfortunate | |
| 15:02:19 | efried | cdent: whadayamean? | |
| 15:02:39 | cdent | it assumes that an e.g., nova client will set a request id | |
| 15:02:55 | cdent | (the INBOUND_HEADER) | |
| 15:02:57 | efried | oh, you mean oslo's middleware is assuming things about clients | |
| 15:03:11 | mriedem | efried: this compression container_format thing seems to be flying under the radar but is a feature/blueprint | |
| 15:03:44 | efried | mriedem: yes, the above patch just adds an early fail for it. No intention of allowing them to attempt to add support for compressed images in Train. | |
| 15:04:27 | efried | mriedem: above patch is goodness because otherwise we happily create the instance and make it ACTIVE per nova, but the OS doesn't come up so it's nonsense. | |
| 15:04:42 | openstackgerrit | Brin Zhang proposed openstack/nova master: Filter migrations by user_id/project_id https://review.opendev.org/674243 | |
| 15:05:15 | efried | mriedem: suspect one of us will get to help them write their func test tho :) | |
| 15:05:16 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Replace non-nova server fault message https://review.opendev.org/674828 | |
| 15:06:33 | KeithMnemonic | does anyone know if there are plans to put this on Rocky as well https://review.opendev.org/#/c/649372/ | |
| 15:07:15 | KeithMnemonic | i tried doing a simple cherry pick but there are some merge conflicts so was not sure the level of effort or if there are too many dependencies. | |
| 15:10:04 | efried | mnaser, melwitt: ^ ? | |
| 15:10:15 | mriedem | KeithMnemonic: i don't think there are any plans no | |
| 15:10:42 | mriedem | the comment from melwitt on the master branch change was, "Yeah, I agree a reno would be helpful here. Looks like this breakage/change in noVNC behavior was released on Apr 9 [3], which was during Stein, so we could potentially backport to Stein. I don't know about earlier though... I don't know if we should reasonably expect anyone on < Stein would possibly install noVNC 1.1.0?" | |
| 15:11:13 | mriedem | KeithMnemonic: are you using a distro with rocky that has novnc 1.1.0 or are you building packages from source? | |
| 15:11:19 | KeithMnemonic | the problem is without this vnc consoles to esx hypervisors are broken in rocky | |
| 15:11:36 | KeithMnemonic | so i kind of need rocky to support 1.1.0 | |
| 15:11:49 | mriedem | but why/how are you getting/using novnc 1.1.0? | |
| 15:11:55 | mriedem | distro or rolling your own packages from source? | |
| 15:12:19 | KeithMnemonic | i pulled from github to test outside of nova | |
| 15:12:32 | KeithMnemonic | the issue is reproducible just using novnc 1.0 | |
| 15:12:37 | KeithMnemonic | and is fixed in 1.1 | |
| 15:12:44 | sean-k-mooney | KeithMnemonic: is there a feature in 1.1.0 that you need | |
| 15:12:45 | KeithMnemonic | so we could try and patch 1.0 | |
| 15:12:59 | KeithMnemonic | yeah " the it works" feature | |
| 15:13:19 | sean-k-mooney | no i mean we used older versions in rocky | |
| 15:13:20 | mriedem | the commit message on that change from mnaser says that 1.1.0 is the breaking novnc change, i can't comment on 1.0 | |
| 15:14:20 | fungi | this is basically the "maintaining your own distro" problem. if you were a stable server linux distro packaging novnc and considered esx important, you would work out how to backport the esx fix to older novnc you were already shipping | |
| 15:14:26 | KeithMnemonic | so i am just talking about novnc for a minute. the 1.0.0 release do not work with vnc console provide for instances running on esx | |
| 15:14:32 | KeithMnemonic | the 1.1 version does work | |
| 15:14:47 | KeithMnemonic | fungi we can do that | |
| 15:15:07 | KeithMnemonic | we (SUSE) is a stable server linux distro | |
| 15:15:08 | sean-k-mooney | ok so esx is the issue | |
| 15:15:27 | sean-k-mooney | or rather that while we test with 1.0.0 | |
| 15:15:29 | fungi | KeithMnemonic: aha, got it, so yes you know this route well | |
| 15:15:29 | sean-k-mooney | https://github.com/openstack/devstack/blob/1f6bea1768c2066f3dd5604aee569ad633f45fbc/stackrc#L605-L606 | |
| 15:15:37 | sean-k-mooney | we dont test with esx | |
| 15:15:43 | KeithMnemonic | but instead of hacking in patches to noVNC, was wondering if there were plans to support 1.1 in rocky since this effects anyone using rocky and novnc 1.0 | |