| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-07 | |||
| 18:00:00 | efried | eandersson: all from the same compute host? | |
| 18:00:07 | mriedem | and we don't put the instances into ERROR state in all cases there | |
| 18:00:09 | eandersson | different hosts | |
| 18:00:42 | efried | okay cool. Trying to assess the probability of the "slow startup" thing noted by cdent and gibi. Seems like it would be pretty low. | |
| 18:00:48 | ricolin | mriedem, thanks | |
| 18:01:34 | eandersson | I was trying to figure out what happened, but couldn't really find anything. | |
| 18:01:59 | eandersson | The VM was in placement, but instance info was not yet updated. | |
| 18:02:23 | eandersson | btw also a lot of VMs stuck in deleting | |
| 18:02:26 | mriedem | ricolin: tbc those jobs were going to fail anyway because of http://status.openstack.org/elastic-recheck/#1844929 but definitely need to get that unboundlocalerror fixed | |
| 18:02:31 | eandersson | but looks like a different issue | |
| 18:02:44 | eandersson | because when VMs are stuck deleting, restarting nova-compute works everytime. | |
| 18:03:47 | mriedem | because the restart code checks for those | |
| 18:04:10 | eandersson | Been unable to figure out why they got stuck in the first place. | |
| 18:04:36 | eandersson | They spiked the hours after a RabbitMQ cluster failed, but seen it before that as well. | |
| 18:04:39 | mriedem | something blow up between the time the api marks the instance as deleting and then rpc casts to the compute? | |
| 18:04:58 | eandersson | Not that I could find. | |
| 18:06:10 | eandersson | I see RabbitMQ connection errors hours, days before that | |
| 18:06:32 | eandersson | but I feel like it shouldn't be the cause, unless RabbitMQ connections just hang dead | |
| 18:06:38 | eandersson | Maybe due to something like https://github.com/openstack/oslo.messaging/commit/26fccea84371a3690a4d7fe970f8d6c5ba9c154b | |
| 18:07:50 | eandersson | I'll try to dig a bit deeper next time it happens. | |
| 18:09:03 | efried | eandersson, mriedem, gibi: +A that fix, but I think we're going to miss the boat on VirtDriverNotReady more frequently than anticipated in ironic cases. | |
| 18:12:41 | openstackgerrit | Rico Lin proposed openstack/nova master: [Gate fix] Avoid use cell_uuid before assignment https://review.opendev.org/686996 | |
| 18:13:14 | ricolin | mriedem, just add bug link to https://review.opendev.org/#/c/686996/ . | |
| 18:14:26 | mriedem | ricolin: yeah i'm working on fixing the test that was masking the regression though | |
| 18:14:35 | ricolin | mriedem, feel free to update that patch to update the test/ whatever needed | |
| 18:15:04 | mriedem | efried: just depends on order of operations - devstack ironic jobs start ironic-api after nova-compute | |
| 18:15:08 | mriedem | b/c of the devstack plugin structure | |
| 18:15:15 | efried | yes | |
| 18:15:15 | ricolin | it's pretty late for me, so will check on this when I awake tomorrow | |
| 18:15:28 | mriedem | but in a real world deployment doohicky ironic might be up before nova | |
| 18:15:34 | mriedem | ricolin: sure np | |
| 18:15:36 | efried | *might* be | |
| 18:16:05 | efried | I guess we'll see. | |
| 18:27:32 | mriedem | melwitt: dansmith: am i overcomplicating this, or should the try/except here https://github.com/openstack/nova/blob/2c6542948f1f57c1a8a8faec1ba61800eaddff2a/nova/context.py#L442 be within the while loop? | |
| 18:27:48 | mriedem | e.g. the first result is a celltimeout but the 2nd yields results | |
| 18:28:41 | dansmith | mriedem: it won't be in the queue until it has timed out so I don't think it matters | |
| 18:29:01 | mriedem | ok so maybe just something i'm seeing in a unit test that is a problem | |
| 18:29:38 | dansmith | although I'm not sure where cell_uuid is coming from in that case | |
| 18:29:39 | mriedem | the order here https://github.com/openstack/nova/blob/1eec451b1ba254ec4c43c77f79b0d2809d6cb9a5/nova/tests/unit/test_context.py#L383 masks the regression being fixed here https://review.opendev.org/#/c/686996/ | |
| 18:30:01 | mriedem | because we get a result from cell0 so cell_uuid is in scope | |
| 18:30:30 | dansmith | well, that's a terrible reason of course :) | |
| 18:30:31 | mriedem | i've reversed the order of that side_effect and now https://github.com/openstack/nova/blob/1eec451b1ba254ec4c43c77f79b0d2809d6cb9a5/nova/tests/unit/test_context.py#L390 isn't in the results because of the try/except being outside the while loop | |
| 18:30:51 | dansmith | spawn as syncrhonous will make that very unrealistic | |
| 18:31:18 | mriedem | right so i'm just wondering if i chalk this up to fragile unit test and not try to have coverage for the regression case | |
| 18:31:36 | mriedem | or maybe i can just write a new test where the only thing we hit is a failure | |
| 18:31:46 | mriedem | that's probably easier to recreate the unboundlocalerror | |
| 18:41:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [Gate fix] Avoid use cell_uuid before assignment https://review.opendev.org/686996 | |
| 18:42:46 | mriedem | ok ^ should be good | |
| 18:46:43 | dansmith | mriedem: my 202 API returns nothing, which seems to trip up novaclient | |
| 18:47:20 | dansmith | it's trying to turn the result into an aggregate I think, but it makes no sense for me to return an aggregate | |
| 18:47:47 | dansmith | I think I can do "give me the raw response" in novaclient, but I'm not sure that's the right thing to do | |
| 18:48:25 | dansmith | I'm going to trawl through some other 202 APIs to see if I can find one that doesn't return anything, but figured I'd poke in case you can just tell me what to do :) | |
| 18:49:29 | sean-k-mooney | dansmith: really i thought 202 ment accepted (generally for async calls) and it was normal for the body to be empty. | |
| 18:49:38 | dansmith | sean-k-mooney: me too | |
| 18:50:10 | dansmith | sean-k-mooney: although for a POST->202 I can see expecting to get back the thing you POST'd, maybe with validations, a uuid assigned, or something so I dunno | |
| 18:51:06 | sean-k-mooney | right i dont think haveing a body is wrong | |
| 18:51:12 | sean-k-mooney | i just would think it optional | |
| 18:51:17 | dansmith | yeah | |
| 18:51:23 | sean-k-mooney | i can totally see geting back a handel to the async task | |
| 18:51:24 | dansmith | might just be opinionated novaclient code, I dunno | |
| 18:53:06 | sean-k-mooney | assuming you dont find and relax the restiction in novaclient im sure you could just return {} | |
| 18:53:09 | sean-k-mooney | to make it happy | |
| 18:53:34 | dansmith | well, not if it's trying to turn the result into an aggregate, which is what I think it's doing | |
| 18:53:43 | dansmith | it chokes on the fact that it's None, but that's what it's trying to do | |
| 18:53:57 | mriedem | dansmith: my guess is the novaclient Aggregate class is just biased | |
| 18:54:06 | mriedem | "might just be opinionated novaclient code" yes this | |
| 18:54:17 | sean-k-mooney | ah right this is for your aggreate image endpoint | |
| 18:54:28 | dansmith | mriedem: ack, so, is it legit to just say raw_response=True so I can get it to chillax? | |
| 18:54:33 | efried | mriedem: unbound local, RC2 yah? | |
| 18:54:33 | mriedem | i think so | |
| 18:54:39 | mriedem | efried: no, it's not in train | |
| 18:54:42 | efried | owait, we didn't put it in train. | |
| 18:54:45 | efried | sorry | |
| 18:55:58 | mriedem | dansmith: you'll also have to pass an object_class | |
| 18:56:04 | mriedem | i think | |
| 18:56:13 | mriedem | otherwise it's going to try and shove it into an Aggregate object | |
| 18:56:15 | dansmith | not for return_raw=True, by my reading | |
| 18:56:27 | mriedem | if obj_class is None: | |
| 18:56:27 | mriedem | obj_class = self.resource_class | |
| 18:56:30 | mriedem | maybe not | |
| 18:56:38 | mriedem | but if there is no body... | |
| 18:56:40 | dansmith | return_raw has returned by then | |
| 18:56:47 | dansmith | ah yeah, guh | |
| 18:56:48 | mriedem | oh right | |
| 18:56:49 | mriedem | yeah | |
| 18:56:53 | mriedem | er no | |
| 18:56:54 | mriedem | :) | |
| 18:57:01 | mriedem | i literally don't trust my eyes right now | |
| 18:57:21 | dansmith | can I just call self.api.client.post myself? | |
| 18:58:24 | sean-k-mooney | i think we do that in a few places | |
| 18:58:45 | sean-k-mooney | althout i have only done that in func tests | |
| 18:58:49 | mriedem | it looks like if you don't, _create expects a body in the response | |
| 18:59:01 | mriedem | sean-k-mooney: you know we're talking about python-novaclient here? | |
| 18:59:25 | mriedem | anyway i have to jump on a call | |
| 18:59:41 | dansmith | mriedem: ack, I'll just do that and wait for reviewers to tell me no | |
| 19:00:54 | sean-k-mooney | mriedem: right fair point i was thinking about usage in nova. | |
| 19:03:31 | dansmith | the good news, at least, | |
| 19:03:51 | dansmith | is that this is all just duck-alignment and that the whole thing works and pre-caches the image as expected | |
| 19:04:16 | sean-k-mooney | :) | |
| 19:04:51 | sean-k-mooney | have you implmented the trotteling code or just the basic poc | |