Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-23
15:50:00 mriedem in the non-cell0 case, we log an error and map it to the cell we just scheduled it to
15:50:05 mriedem which would not be cell0
15:50:32 dansmith mriedem: oh I see, you created a helper but only call it in the non-cell0 case
15:50:33 mriedem i think those are both letting the non-cell0 ones win
15:50:37 mriedem yes
15:50:45 dansmith okay
15:50:45 dansmith I assumed you were calling it from both places
15:50:48 mriedem b/c i didn't want to have to mock 100 LOC just to test 1 line change in schedule_and_build_instances
15:50:59 dansmith sounds like cheating
15:51:39 mriedem schedule_and_build_instances is a monster
15:52:18 mriedem i also noted that if we drop that InstanceMappingNotFound pre-ocata compat in _bury_in_cell0 the _bury_in_cell0 method can just call the new _map_instance_to_cell method
15:52:44 mriedem we have a bunch of pre-ocata pre-cells v2 compat handlers all over the conductor task manager code which it'd be nice to remove
15:52:49 mriedem since they shouldn't be possible anymore
15:53:58 dansmith meh, I love piles of compat code
15:54:01 dansmith makes me feel nostalgic
15:54:24 mriedem all of the TODO(alaski)s to feel like an old sweater
15:54:25 mriedem i'll admit
15:54:32 mriedem s/to/do/
15:57:25 dansmith lol
16:03:06 stephenfin I'm almost certain we've discussed this before, but why do we drop support for old microversion in novaclient?
16:03:16 stephenfin Referring to the API part, rather than the CLI
16:03:24 stephenfin Doesn't osc use that?
16:07:28 mriedem gibi: a few comments in https://review.opendev.org/#/c/683947/ but not worth holding it up,
16:07:36 mriedem gibi: but we might want to reconsider the fault message that the user could see
16:08:01 mriedem "oh hi your cloud provider is doing an upgrade and you shouldn't be trying to resize your instance right now bob!"
16:08:43 mriedem stephenfin: where have we dropped support for old microversions in novaclient?
16:08:52 gibi mriedem: the funny thing is that I think we cannot hit that error after the fix
16:09:04 gibi mriedem: so we could even remove it
16:09:21 stephenfin mriedem: e.g. 01fb16533bf562f39fe822bc12b9cc34b8580359
16:09:30 mriedem gibi: well, technically your fix is in conductor checking it's config but the computes involved in the resize/cold migrate could have different config with different pins
16:09:53 mriedem stephenfin: that one broke osc
16:09:55 gibi mriedem: can we pin our computes differently? how will they talk to each other?
16:10:08 mriedem and they had to fix to avoid using novaclient and hit the api directly
16:10:26 mriedem gibi: well i meant if your conductor was unpinned but the computes were, or not restarted yet or something
16:10:28 mriedem after unpinning them
16:10:32 mriedem i'm nto saying that's normal
16:10:47 mriedem but i'd probably leave the error checking in place until we have a compute rpc api major version bump to 6.0
16:10:53 gibi mriedem: OK. I so it is a sort transinet. I can accept that
16:10:57 stephenfin Yup, I recall that coming up now. Any reason we opted to do that instead of reverting the changes to 'novaclient/v2/client.py'?
16:11:04 stephenfin that = call the API directly
16:11:22 mriedem gibi: so if you wanted, you could write a separate patch which (1) logs the upgrade stuff but doesn't include it in the PortUpdateFailed message, (2) adds a TODO to that PortUpdateFailed code to say we can drop it after compute RPC API is bumped to 6.0
16:11:50 gibi mriedem: sure. I can do that
16:11:51 mriedem stephenfin: it was released late in novaclient, end of the milestone before FF and i think osc found the problem too late
16:12:18 mriedem stephenfin: so a mixture of releasing breaking / major version things in novaclient too late in the release to catch them and (2) osc historically not doing functional testing
16:12:30 mriedem well, not doing good enough coverage with functional testing
16:13:40 mriedem and (3) me not realizing osc was going to be broken
16:14:06 stephenfin that makes sense
16:14:10 mriedem that was probably before i cared more about osc and just figured, "this has been deprecated in novaclient since newton, we can surely drop it and no one will care"
16:14:35 mriedem so having been burned before, and burning myself and others, that's why i'm less cavalier about just removing old stuff
16:15:40 stephenfin Yeah, certainly true for something like this
16:15:50 mriedem speaking of an upgrade-related burning sensation https://review.opendev.org/#/q/topic:bug/1843090+(status:open+OR+status:merged)
16:16:36 stephenfin I was holding off on that til some people had looked at the oslo.messaging bug
16:16:44 stephenfin though maybe that doesn't make sense
16:17:28 mriedem you mean gibi's RequestSpecImageSerializationFixture in the functoinal test?
16:17:33 mriedem it's an existing pattern / known issue
16:18:28 mriedem could also summon dansmith to review those since it's rpc pin related
16:18:33 mriedem and that's dan's middle name
16:18:50 mriedem dan rpc-pin-and-sometimes-evacuate smith
16:19:15 stephenfin If dansmith doesn't get to it by tomorrow, I will
16:29:48 mriedem gibi: i think we could drop the new and redundant unit test in https://review.opendev.org/#/c/683948/
16:29:52 mriedem the functional test covers it
17:24:01 openstackgerrit Merged openstack/nova master: Add note about needing noVNC >= v1.1.0 with using ESX https://review.opendev.org/682946
17:30:21 bbobrov_ hi! In https://opendev.org/openstack/nova/src/branch/master/nova/servicegroup/drivers/db.py#L99 messaging.MessagingTimeout is getting caught. My understanding is that service_ref.save() only interacts with the database. How can MessagingTimeout happen?
17:38:48 sean-k-mooney bbobrov: if this is executing on the compute node then the save call would invoke the db update via rpc as the comptue services do not have database access
17:41:09 sean-k-mooney bbobrov: so on the concoctor,api and schduler this shoudl not raise a messaging timeout but it can in the compute agents
17:41:32 bbobrov sean-k-mooney: understood, thanks
17:45:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Rename Claims resources to compute_node https://review.opendev.org/679470
17:50:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Add a prelude for the Train release https://review.opendev.org/683327
17:53:47 bbobrov #2 then. there is a `periodic_enable` option, and in nova codebase it seems to affect only https://opendev.org/openstack/nova/src/branch/master/nova/service.py#L205 . if true, self.periodic_tasks is scheduled to be run by add_dynamic_timer
17:54:08 bbobrov But there is no implementation of periodic_tasks or run_periodic_tasks anywhere in Nova. The method in the base scheduler class does `pass`. Are there any implementation of run_periodic_tasks in Nova that i miss or they can come from somewhere else?
17:56:58 sean-k-mooney we use the one form oslo service https://github.com/openstack/oslo.service/blob/master/oslo_service/periodic_task.py
18:07:56 mriedem melwitt: do you know if tripleo has any kind of tooling that pokes a cell mq/db on upgrade to make sure it's ok before saying everything is good to go in the api?
18:22:38 bbobrov sean-k-mooney: thanks, it makes sense now
18:32:57 lyarwood mriedem: we are only just introducing full support for multi cell deployments now in 16 but I can ask around about that in the morning.
18:33:48 lyarwood owalsh: ^ unless you're around and know?
18:34:52 mriedem lyarwood: yeah in grenade upstream jobs we don't have multiple cells, just cell0 and cell1
18:34:57 mriedem trying to debug https://bugs.launchpad.net/nova/+bug/1844929
18:34:58 openstack Launchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed]
18:35:35 mriedem it looks like we upgrade code to train, restart scheduler, it starts up, even hits the cell1 database to pull compute nodes and instances for it's in-memory cache, and then on the first scheduling attempt after that we hang
18:37:45 dansmith mriedem: scheduler doesn't need rabbit to do that, so the rabbit errors in your bug report would be unrelated I'd think
18:39:41 mriedem true, didn't think of that
18:39:42 mriedem obviously
18:39:49 mriedem i just left a comment with a bunch of log links,
18:40:17 mriedem but it looks like we start up ok, hit the cell1 db to pull compute nodes and instances, then 4 minutes later is the first scheduling attempt after the upgrade and at that point we're timing out
18:41:24 sean-k-mooney dansmith: we do eventurlly need to adress the rabbit mq heartbeat issue even if it is unrelated to this issue
18:41:32 mriedem it also looks like this goes back to 9/14 which is a few days before i originally thought
18:41:35 dansmith mriedem: that timeout is related to the scatter
18:42:04 dansmith mriedem: that's not new information, I'm just saying
18:42:29 dansmith mriedem: I was expecting to see a trace, but I think the scatter squashes that
18:43:15 sean-k-mooney i belive if any of the request raise an excpeiton we catch it and return the exception instead of raising it or something like that
18:43:20 mriedem it just logs a warning without a trace
18:43:31 dansmith mriedem: right
18:43:35 sean-k-mooney so i dont think we actuly log the traceback the way we normally would
18:43:52 dansmith sean-k-mooney: didn't I just say that?
18:44:34 sean-k-mooney yes i was halfway through typing it when you did so i finished it and hit enter
18:44:34 mriedem what i'm concerned about is if something in https://review.opendev.org/#/c/641907/ which merged and was released late is causing some side effect with the restart
18:44:48 dansmith mriedem: you said it was able to prime its cache on startup right? so this is it getting all nodes from all cells on the schedule run?
18:45:11 mriedem we're getting through this https://github.com/openstack/nova/blob/597b34cd87ac349c0f3702a872630f3c830b1483/nova/scheduler/host_manager.py#L413
18:45:18 mriedem and then 4 minutes later the first scheduling request comes,
18:45:25 mriedem and it's going back to the cell to pull compute nodes by uuid

Earlier   Later