| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-07 | |||
| 16:47:10 | efried | PS11 9/18 | |
| 16:49:56 | efried | 0_0 | |
| 16:50:35 | mriedem | https://data.whicdn.com/images/98587406/superthumb.jpg?t=1390970016 | |
| 16:51:19 | mriedem | gonna get lunch and check back in a bit | |
| 16:51:51 | cdent | efried: left a +1 | |
| 16:51:58 | efried | thanks cdent | |
| 16:55:29 | eandersson | That bug sure looks like the issue. The only thing that concerns me is that we saw this hours after compute restarts, but might just have been delayed reports. | |
| 17:06:24 | efried | eandersson: I was +2 at some point in the past on that fix, and mriedem was just waiting for cdent to ack (I think), so we should be able to merge it fairly soon I would think. Will have to wait until next week to start merging backports, though. | |
| 17:07:02 | eandersson | Sounds good - we do large rolling deployments of computes | |
| 17:07:38 | eandersson | takes forever | |
| 17:08:15 | eandersson | So knowing that it's safe to do so helps a lot. Otherwise we need to disable the apis | |
| 17:25:34 | donnyd | sean-k-mooney: Yea that makes sense to me | |
| 17:42:00 | efried | stephenfin: I really appreciate the cleanup of oldy moldy change sets, but would you mind leaving a one-line reason when you abandon please? | |
| 17:47:10 | mriedem | efried: i'm +2 on gibi's interrupted build fix now https://review.opendev.org/#/c/666857/ | |
| 17:47:28 | efried | mriedem: cool, thanks, I'll get back to that in the next few minutes. | |
| 17:57:51 | efried | eandersson: How many instances are you seeing stuck, max? | |
| 17:58:07 | efried | just curious, because it seems like it would be pretty hard to have more than a small number. | |
| 17:58:50 | eandersson | 10+ | |
| 17:59:29 | mriedem | ricolin: your fix looks fine but i want to check what tests could be updated to make sure that's not regressed again | |
| 17:59:40 | eandersson | We had a RabbitMQ outage that could have caused some of them as well. | |
| 17:59:58 | mriedem | eandersson: i know you also reported issues with rescheduling failures in conductor recently | |
| 18:00:00 | efried | eandersson: all from the same compute host? | |
| 18:00:07 | mriedem | and we don't put the instances into ERROR state in all cases there | |
| 18:00:09 | eandersson | different hosts | |
| 18:00:42 | efried | okay cool. Trying to assess the probability of the "slow startup" thing noted by cdent and gibi. Seems like it would be pretty low. | |
| 18:00:48 | ricolin | mriedem, thanks | |
| 18:01:34 | eandersson | I was trying to figure out what happened, but couldn't really find anything. | |
| 18:01:59 | eandersson | The VM was in placement, but instance info was not yet updated. | |
| 18:02:23 | eandersson | btw also a lot of VMs stuck in deleting | |
| 18:02:26 | mriedem | ricolin: tbc those jobs were going to fail anyway because of http://status.openstack.org/elastic-recheck/#1844929 but definitely need to get that unboundlocalerror fixed | |
| 18:02:31 | eandersson | but looks like a different issue | |
| 18:02:44 | eandersson | because when VMs are stuck deleting, restarting nova-compute works everytime. | |
| 18:03:47 | mriedem | because the restart code checks for those | |
| 18:04:10 | eandersson | Been unable to figure out why they got stuck in the first place. | |
| 18:04:36 | eandersson | They spiked the hours after a RabbitMQ cluster failed, but seen it before that as well. | |
| 18:04:39 | mriedem | something blow up between the time the api marks the instance as deleting and then rpc casts to the compute? | |
| 18:04:58 | eandersson | Not that I could find. | |
| 18:06:10 | eandersson | I see RabbitMQ connection errors hours, days before that | |
| 18:06:32 | eandersson | but I feel like it shouldn't be the cause, unless RabbitMQ connections just hang dead | |
| 18:06:38 | eandersson | Maybe due to something like https://github.com/openstack/oslo.messaging/commit/26fccea84371a3690a4d7fe970f8d6c5ba9c154b | |
| 18:07:50 | eandersson | I'll try to dig a bit deeper next time it happens. | |
| 18:09:03 | efried | eandersson, mriedem, gibi: +A that fix, but I think we're going to miss the boat on VirtDriverNotReady more frequently than anticipated in ironic cases. | |
| 18:12:41 | openstackgerrit | Rico Lin proposed openstack/nova master: [Gate fix] Avoid use cell_uuid before assignment https://review.opendev.org/686996 | |
| 18:13:14 | ricolin | mriedem, just add bug link to https://review.opendev.org/#/c/686996/ . | |
| 18:14:26 | mriedem | ricolin: yeah i'm working on fixing the test that was masking the regression though | |
| 18:14:35 | ricolin | mriedem, feel free to update that patch to update the test/ whatever needed | |
| 18:15:04 | mriedem | efried: just depends on order of operations - devstack ironic jobs start ironic-api after nova-compute | |
| 18:15:08 | mriedem | b/c of the devstack plugin structure | |
| 18:15:15 | ricolin | it's pretty late for me, so will check on this when I awake tomorrow | |
| 18:15:15 | efried | yes | |
| 18:15:28 | mriedem | but in a real world deployment doohicky ironic might be up before nova | |
| 18:15:34 | mriedem | ricolin: sure np | |
| 18:15:36 | efried | *might* be | |
| 18:16:05 | efried | I guess we'll see. | |
| 18:27:32 | mriedem | melwitt: dansmith: am i overcomplicating this, or should the try/except here https://github.com/openstack/nova/blob/2c6542948f1f57c1a8a8faec1ba61800eaddff2a/nova/context.py#L442 be within the while loop? | |
| 18:27:48 | mriedem | e.g. the first result is a celltimeout but the 2nd yields results | |
| 18:28:41 | dansmith | mriedem: it won't be in the queue until it has timed out so I don't think it matters | |
| 18:29:01 | mriedem | ok so maybe just something i'm seeing in a unit test that is a problem | |
| 18:29:38 | dansmith | although I'm not sure where cell_uuid is coming from in that case | |
| 18:29:39 | mriedem | the order here https://github.com/openstack/nova/blob/1eec451b1ba254ec4c43c77f79b0d2809d6cb9a5/nova/tests/unit/test_context.py#L383 masks the regression being fixed here https://review.opendev.org/#/c/686996/ | |
| 18:30:01 | mriedem | because we get a result from cell0 so cell_uuid is in scope | |
| 18:30:30 | dansmith | well, that's a terrible reason of course :) | |
| 18:30:31 | mriedem | i've reversed the order of that side_effect and now https://github.com/openstack/nova/blob/1eec451b1ba254ec4c43c77f79b0d2809d6cb9a5/nova/tests/unit/test_context.py#L390 isn't in the results because of the try/except being outside the while loop | |
| 18:30:51 | dansmith | spawn as syncrhonous will make that very unrealistic | |
| 18:31:18 | mriedem | right so i'm just wondering if i chalk this up to fragile unit test and not try to have coverage for the regression case | |
| 18:31:36 | mriedem | or maybe i can just write a new test where the only thing we hit is a failure | |
| 18:31:46 | mriedem | that's probably easier to recreate the unboundlocalerror | |
| 18:41:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [Gate fix] Avoid use cell_uuid before assignment https://review.opendev.org/686996 | |
| 18:42:46 | mriedem | ok ^ should be good | |
| 18:46:43 | dansmith | mriedem: my 202 API returns nothing, which seems to trip up novaclient | |
| 18:47:20 | dansmith | it's trying to turn the result into an aggregate I think, but it makes no sense for me to return an aggregate | |
| 18:47:47 | dansmith | I think I can do "give me the raw response" in novaclient, but I'm not sure that's the right thing to do | |
| 18:48:25 | dansmith | I'm going to trawl through some other 202 APIs to see if I can find one that doesn't return anything, but figured I'd poke in case you can just tell me what to do :) | |
| 18:49:29 | sean-k-mooney | dansmith: really i thought 202 ment accepted (generally for async calls) and it was normal for the body to be empty. | |
| 18:49:38 | dansmith | sean-k-mooney: me too | |
| 18:50:10 | dansmith | sean-k-mooney: although for a POST->202 I can see expecting to get back the thing you POST'd, maybe with validations, a uuid assigned, or something so I dunno | |
| 18:51:06 | sean-k-mooney | right i dont think haveing a body is wrong | |
| 18:51:12 | sean-k-mooney | i just would think it optional | |
| 18:51:17 | dansmith | yeah | |
| 18:51:23 | sean-k-mooney | i can totally see geting back a handel to the async task | |
| 18:51:24 | dansmith | might just be opinionated novaclient code, I dunno | |
| 18:53:06 | sean-k-mooney | assuming you dont find and relax the restiction in novaclient im sure you could just return {} | |
| 18:53:09 | sean-k-mooney | to make it happy | |
| 18:53:34 | dansmith | well, not if it's trying to turn the result into an aggregate, which is what I think it's doing | |
| 18:53:43 | dansmith | it chokes on the fact that it's None, but that's what it's trying to do | |
| 18:53:57 | mriedem | dansmith: my guess is the novaclient Aggregate class is just biased | |
| 18:54:06 | mriedem | "might just be opinionated novaclient code" yes this | |
| 18:54:17 | sean-k-mooney | ah right this is for your aggreate image endpoint | |
| 18:54:28 | dansmith | mriedem: ack, so, is it legit to just say raw_response=True so I can get it to chillax? | |
| 18:54:33 | mriedem | i think so | |
| 18:54:33 | efried | mriedem: unbound local, RC2 yah? | |
| 18:54:39 | mriedem | efried: no, it's not in train | |
| 18:54:42 | efried | owait, we didn't put it in train. | |
| 18:54:45 | efried | sorry | |
| 18:55:58 | mriedem | dansmith: you'll also have to pass an object_class | |
| 18:56:04 | mriedem | i think | |
| 18:56:13 | mriedem | otherwise it's going to try and shove it into an Aggregate object | |
| 18:56:15 | dansmith | not for return_raw=True, by my reading | |
| 18:56:27 | mriedem | obj_class = self.resource_class | |
| 18:56:27 | mriedem | if obj_class is None: | |