| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-23 | |||
| 16:17:28 | mriedem | you mean gibi's RequestSpecImageSerializationFixture in the functoinal test? | |
| 16:17:33 | mriedem | it's an existing pattern / known issue | |
| 16:18:28 | mriedem | could also summon dansmith to review those since it's rpc pin related | |
| 16:18:33 | mriedem | and that's dan's middle name | |
| 16:18:50 | mriedem | dan rpc-pin-and-sometimes-evacuate smith | |
| 16:19:15 | stephenfin | If dansmith doesn't get to it by tomorrow, I will | |
| 16:29:48 | mriedem | gibi: i think we could drop the new and redundant unit test in https://review.opendev.org/#/c/683948/ | |
| 16:29:52 | mriedem | the functional test covers it | |
| 17:24:01 | openstackgerrit | Merged openstack/nova master: Add note about needing noVNC >= v1.1.0 with using ESX https://review.opendev.org/682946 | |
| 17:30:21 | bbobrov_ | hi! In https://opendev.org/openstack/nova/src/branch/master/nova/servicegroup/drivers/db.py#L99 messaging.MessagingTimeout is getting caught. My understanding is that service_ref.save() only interacts with the database. How can MessagingTimeout happen? | |
| 17:38:48 | sean-k-mooney | bbobrov: if this is executing on the compute node then the save call would invoke the db update via rpc as the comptue services do not have database access | |
| 17:41:09 | sean-k-mooney | bbobrov: so on the concoctor,api and schduler this shoudl not raise a messaging timeout but it can in the compute agents | |
| 17:41:32 | bbobrov | sean-k-mooney: understood, thanks | |
| 17:45:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Rename Claims resources to compute_node https://review.opendev.org/679470 | |
| 17:50:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add a prelude for the Train release https://review.opendev.org/683327 | |
| 17:53:47 | bbobrov | #2 then. there is a `periodic_enable` option, and in nova codebase it seems to affect only https://opendev.org/openstack/nova/src/branch/master/nova/service.py#L205 . if true, self.periodic_tasks is scheduled to be run by add_dynamic_timer | |
| 17:54:08 | bbobrov | But there is no implementation of periodic_tasks or run_periodic_tasks anywhere in Nova. The method in the base scheduler class does `pass`. Are there any implementation of run_periodic_tasks in Nova that i miss or they can come from somewhere else? | |
| 17:56:58 | sean-k-mooney | we use the one form oslo service https://github.com/openstack/oslo.service/blob/master/oslo_service/periodic_task.py | |
| 18:07:56 | mriedem | melwitt: do you know if tripleo has any kind of tooling that pokes a cell mq/db on upgrade to make sure it's ok before saying everything is good to go in the api? | |
| 18:22:38 | bbobrov | sean-k-mooney: thanks, it makes sense now | |
| 18:32:57 | lyarwood | mriedem: we are only just introducing full support for multi cell deployments now in 16 but I can ask around about that in the morning. | |
| 18:33:48 | lyarwood | owalsh: ^ unless you're around and know? | |
| 18:34:52 | mriedem | lyarwood: yeah in grenade upstream jobs we don't have multiple cells, just cell0 and cell1 | |
| 18:34:57 | mriedem | trying to debug https://bugs.launchpad.net/nova/+bug/1844929 | |
| 18:34:58 | openstack | Launchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] | |
| 18:35:35 | mriedem | it looks like we upgrade code to train, restart scheduler, it starts up, even hits the cell1 database to pull compute nodes and instances for it's in-memory cache, and then on the first scheduling attempt after that we hang | |
| 18:37:45 | dansmith | mriedem: scheduler doesn't need rabbit to do that, so the rabbit errors in your bug report would be unrelated I'd think | |
| 18:39:41 | mriedem | true, didn't think of that | |
| 18:39:42 | mriedem | obviously | |
| 18:39:49 | mriedem | i just left a comment with a bunch of log links, | |
| 18:40:17 | mriedem | but it looks like we start up ok, hit the cell1 db to pull compute nodes and instances, then 4 minutes later is the first scheduling attempt after the upgrade and at that point we're timing out | |
| 18:41:24 | sean-k-mooney | dansmith: we do eventurlly need to adress the rabbit mq heartbeat issue even if it is unrelated to this issue | |
| 18:41:32 | mriedem | it also looks like this goes back to 9/14 which is a few days before i originally thought | |
| 18:41:35 | dansmith | mriedem: that timeout is related to the scatter | |
| 18:42:04 | dansmith | mriedem: that's not new information, I'm just saying | |
| 18:42:29 | dansmith | mriedem: I was expecting to see a trace, but I think the scatter squashes that | |
| 18:43:15 | sean-k-mooney | i belive if any of the request raise an excpeiton we catch it and return the exception instead of raising it or something like that | |
| 18:43:20 | mriedem | it just logs a warning without a trace | |
| 18:43:31 | dansmith | mriedem: right | |
| 18:43:35 | sean-k-mooney | so i dont think we actuly log the traceback the way we normally would | |
| 18:43:52 | dansmith | sean-k-mooney: didn't I just say that? | |
| 18:44:34 | sean-k-mooney | yes i was halfway through typing it when you did so i finished it and hit enter | |
| 18:44:34 | mriedem | what i'm concerned about is if something in https://review.opendev.org/#/c/641907/ which merged and was released late is causing some side effect with the restart | |
| 18:44:48 | dansmith | mriedem: you said it was able to prime its cache on startup right? so this is it getting all nodes from all cells on the schedule run? | |
| 18:45:11 | mriedem | we're getting through this https://github.com/openstack/nova/blob/597b34cd87ac349c0f3702a872630f3c830b1483/nova/scheduler/host_manager.py#L413 | |
| 18:45:18 | mriedem | and then 4 minutes later the first scheduling request comes, | |
| 18:45:25 | mriedem | and it's going back to the cell to pull compute nodes by uuid | |
| 18:45:27 | mriedem | and times out | |
| 18:45:31 | dansmith | mriedem: I guess I thought we were stopping with systemd in that case, so no sighup involed yeah? | |
| 18:47:40 | sean-k-mooney | https://github.com/openstack/grenade/blob/master/projects/60_nova/shutdown.sh#L21 | |
| 18:47:57 | sean-k-mooney | i have not checked devstack but yes | |
| 18:48:16 | sean-k-mooney | i think tha tdoes systemctl stop devstack@n=* effectivly | |
| 18:48:45 | dansmith | right | |
| 18:49:03 | sean-k-mooney | ya it does https://github.com/openstack/devstack/blob/master/lib/nova#L1047-L1059 | |
| 18:49:49 | mriedem | Sep 22 00:37:32.126839 ubuntu-bionic-ovh-gra1-0011664420 systemd[1]: Stopped Devstack devstack@n-sch.service. | |
| 18:49:53 | mriedem | Sep 22 00:45:55.359862 ubuntu-bionic-ovh-gra1-0011664420 systemd[1]: Started Devstack devstack@n-sch.service. | |
| 18:50:42 | dansmith | mriedem: do we even stop/start mysql in grenade during the upgrade? | |
| 18:50:44 | mriedem | Sep 22 00:37:27.786606 ubuntu-bionic-ovh-gra1-0011664420 nova-scheduler[25563]: INFO oslo_service.service [None req-91e88f0d-9b5c-4cb7-a5e9-e7309f922832 None None] Caught SIGTERM, stopping children | |
| 18:50:46 | mriedem | yeah not a HUP | |
| 18:50:54 | mriedem | dansmith: pretty sure we don't | |
| 18:51:21 | dansmith | mriedem: yeah, so, pretty weird that it connects and works once and then times out later, because it's not like we restart mysql after that point or something | |
| 18:51:35 | dansmith | mriedem: so I wonder if we're actually doing something other than what we think | |
| 18:51:48 | dansmith | and since we don't log a trace there we don't know where it's actually timing out | |
| 18:52:46 | openstackgerrit | Dustin Cowles proposed openstack/nova-specs master: Spec: Provider config YAML file https://review.opendev.org/680471 | |
| 18:59:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log CellTimeout traceback in scatter_gather_cells https://review.opendev.org/684118 | |
| 18:59:18 | mriedem | dansmith: thinking like this? ^ | |
| 19:00:13 | dansmith | mriedem: yeah, I mean we kinda specifically decided not to explode there so we're tolerant of transient failures | |
| 19:00:26 | dansmith | I wonder if we should log debug with exc_info=True and warning without or something | |
| 19:00:31 | dansmith | but yeah, something | |
| 19:01:01 | mriedem | was thinking about passing down a kwarg to scatter_gather_cells to tell it what to do, but that's probably icky | |
| 19:01:12 | mriedem | trace_on_timeout = kwargs.pop('trace_on_timeout', False) | |
| 19:01:27 | dansmith | yeah, I don't like that | |
| 19:02:03 | dansmith | I love **kwargs behavior in python, but hate kwargs.pop('arg', None) abuses of it | |
| 19:02:09 | dansmith | which is why we can't have nice things in other languages :) | |
| 19:05:20 | dansmith | mriedem: so you just need to recheck that a few times to get a repro on it yeah? | |
| 19:06:26 | mriedem | maybe if we get lucky http://status.openstack.org/elastic-recheck/#1844929 | |
| 19:06:53 | mriedem | it mostly only shows up on certain node providers | |
| 19:07:00 | mriedem | so if i hit a rax node i likely won't see it | |
| 19:07:10 | mriedem | and only grenade jobs for some reason | |
| 19:07:15 | mriedem | something getting f'ed up in restarts | |
| 19:07:23 | sean-k-mooney | mriedem: which node providers? | |
| 19:07:31 | mriedem | fort nebula and ovh | |
| 19:07:54 | dansmith | mriedem: you should put that in the bug | |
| 19:07:56 | sean-k-mooney | we can target fort nebula using a specific node pool lable | |
| 19:08:00 | dansmith | so people can know and don't have to ask | |
| 19:08:04 | sean-k-mooney | im not sure about ovh | |
| 19:08:28 | mriedem | dansmith: from the bug, "It also appears to only show up on fortnebula and OVH nodes, primarily fortnebula." | |
| 19:08:34 | sean-k-mooney | but we coudl try and repoducice it via FN if you thing that was useful | |
| 19:08:38 | dansmith | mriedem: I know :) | |
| 19:08:44 | mriedem | i see what you did there | |
| 19:09:00 | dansmith | haha | |
| 19:11:20 | hemna | so I still can't seem to run tox -epy27 on anything less than stable/rocky | |
| 19:11:24 | hemna | queens and pike both fail | |
| 19:11:35 | hemna | http://paste.openstack.org/show/778953/ | |
| 19:11:56 | hemna | 6444 failures, all the same type of failure. sqlalchemy.exc.NoSuchTableError: migration_tmp | |
| 19:12:11 | dansmith | hemna: clean your local directory | |
| 19:12:16 | hemna | I did | |
| 19:12:21 | donnyd | Is this job CPU bound? | |
| 19:12:22 | hemna | nuked all pyc files and .tox | |
| 19:12:27 | dansmith | er, well, maybe not with that _tmp prefix | |