| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-28 | |||
| 19:00:34 | openstackgerrit | Merged openstack/nova stable/stein: Fix type error on call to mount device https://review.opendev.org/667921 | |
| 19:07:26 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove needs:* todo from deprecated APIs api-ref https://review.opendev.org/668234 | |
| 19:08:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add StopWatch profiling to SchedulerManager.select_destinations https://review.opendev.org/668235 | |
| 19:09:36 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove needs:* todo from deprecated APIs api-ref https://review.opendev.org/668234 | |
| 19:15:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add StopWatch profiling to SchedulerManager.select_destinations https://review.opendev.org/668235 | |
| 19:16:53 | openstackgerrit | Merged openstack/nova stable/stein: Change InstanceFaultRollback handling in _error_out_instance_on_exception https://review.opendev.org/666638 | |
| 19:17:00 | openstackgerrit | Merged openstack/nova stable/stein: Raise InstanceFaultRollback for UnableToMigrateToSelf from _prep_resize https://review.opendev.org/666639 | |
| 19:17:06 | openstackgerrit | Merged openstack/nova stable/stein: Avoid unnecessary joins in InstanceGroup.get_hosts https://review.opendev.org/664271 | |
| 19:17:13 | openstackgerrit | Merged openstack/nova stable/stein: Handle PortLimitExceeded in POST /servers/{server_id}/os-interface https://review.opendev.org/665513 | |
| 19:17:20 | openstackgerrit | Merged openstack/nova stable/rocky: Share snapshot image membership with instance owner https://review.opendev.org/643853 | |
| 19:20:40 | mriedem | fried_rolls: gah -f value -c ID fails me again http://logs.openstack.org/94/667994/2/check/nova-next/1519821/logs/devstack-gate-post_test_hook.txt.gz#_2019-06-28_01_50_02_145 | |
| 19:29:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add integration testing for heal_allocations https://review.opendev.org/667994 | |
| 19:45:09 | mriedem | fried_rolls: what are your thoughts on doing an os-traits release next week? trying to workaround not having https://review.opendev.org/#/c/667209/ released is becoming a pain. | |
| 19:49:58 | efried | mriedem: There's a release proposed right now, which includes ^. smcginnis is waiting for cdent (PTL) ack, but we could probably convince him to push it now if it's going to help you. | |
| 19:50:14 | efried | mriedem: https://review.opendev.org/667111 | |
| 19:50:53 | smcginnis | If it's holding up something, we can push it through. Even though it's a Friday and we generally try to avoid that kind of thing right before the weekend. | |
| 19:51:01 | smcginnis | Just depends how time critical it is. | |
| 19:51:10 | efried | this one's pretty benign, release-risk-wise | |
| 19:51:18 | efried | like, really really benign | |
| 19:51:33 | efried | nearly betegn | |
| 19:51:35 | smcginnis | Yeah, I figured something like that would be low risk. | |
| 19:52:38 | mriedem | cool thanks | |
| 19:53:23 | efried | mriedem: you're still a placement core. Add your +1 and let's rip it. cdent will forgive us. | |
| 19:54:01 | mriedem | done | |
| 19:54:11 | efried | nice | |
| 19:54:14 | efried | smcginnis: ^ | |
| 19:54:34 | smcginnis | OK, I'll send 'er through. And watch out for the wrath of cdent. :D | |
| 19:55:55 | efried | thank you sir | |
| 19:56:25 | efried | mriedem: You'll need requirements bump before you can use it anyway. | |
| 19:58:32 | mriedem | oh i know | |
| 19:58:39 | mriedem | but that's automatic | |
| 19:58:44 | mriedem | the proposal anyway | |
| 20:09:02 | mriedem | AHA https://bugs.launchpad.net/nova/+bug/1834691 | |
| 20:09:02 | openstack | Launchpad bug 1834691 in OpenStack Compute (nova) "AttributeError: 'HostState' object has no attribute 'failed_builds'" [Undecided,New] | |
| 20:19:51 | efried | mriedem: takes a while though. | |
| 20:20:28 | efried | (requirements bot) | |
| 20:21:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Init HostState.failed_builds https://review.opendev.org/668243 | |
| 20:21:43 | mriedem | what doesn't take awhile is +2ing this trivial bug fix ^ melwitt dansmith efried | |
| 20:38:50 | efried | mriedem: done. tbc the other thing is a separate issue? | |
| 20:38:54 | mriedem | yup | |
| 20:38:57 | mriedem | patch forthcoming | |
| 20:39:27 | efried | mriedem: requirements bump https://review.opendev.org/#/c/668245/ | |
| 20:39:52 | efried | smcginnis: https://review.opendev.org/#/c/668245/ do you have the power? | |
| 20:40:29 | smcginnis | efried: Just waiting for zuul to report results on that. | |
| 20:40:43 | efried | slave to process you are, sheesh | |
| 20:40:46 | smcginnis | I can +2, but don't want to ninja that one through. | |
| 20:40:47 | smcginnis | :D | |
| 20:42:44 | Sundar | efried, cdent: Is there a master topic/blueprint for all nested magic stuff? I am looking for better query than https://review.opendev.org/#/q/project:openstack/placement+nested | |
| 20:43:06 | efried | Sundar: It's all in a single spec | |
| 20:43:20 | efried | Sundar: https://docs.openstack.org/placement/latest/specs/train/approved/2005575-nested-magic-1.html | |
| 20:43:54 | efried | unless you mean a doc for all currently implemented features. That would be https://docs.openstack.org/placement/latest/usage/provider-tree.html | |
| 20:44:19 | efried | plus the api-reference https://developer.openstack.org/api-ref/placement/ | |
| 20:44:55 | Sundar | We've also had: https://review.opendev.org/657419. Some of your specs, since abandoned (like https://review.opendev.org/658510) were also useful to read | |
| 20:45:26 | Sundar | It would be useful to have a single topic or blueprint for all of these. What do you think? | |
| 20:46:23 | efried | Sundar: That patch was for the arbitrary suffix feature, which is included in the above spec. | |
| 20:46:34 | efried | If you want to track all the patches, you can look at the story | |
| 20:46:40 | efried | https://storyboard.openstack.org/#!/story/2005575 | |
| 20:47:29 | efried | agree that topic branch story/$story-$task isn't as useful as just story/$story for searching purposes. | |
| 20:47:50 | efried | but too late to modify topic for merged patches, sorry. | |
| 20:49:15 | Sundar | NP, got it. Good to know that the spec and storyboard is all there is. Is there a spec/patch on Nova side for how it all comes together -- from flavor to Placement query? | |
| 20:49:43 | Sundar | i.e. 'all there is to read for somebody to understand the whole shebang | |
| 20:55:40 | Sundar | efried: ^ | |
| 21:02:13 | efried | Sundar: There's not likely to be something that high level. And there's as yet no blueprint/spec I'm aware of on the nova side for exploiting these features. | |
| 21:02:41 | melwitt | mriedem: the failed builds thing, my fave | |
| 21:14:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update HostState.*_allocation_ratio earlier https://review.opendev.org/668252 | |
| 21:53:05 | mriedem | wow some fun debugging here | |
| 21:53:24 | mriedem | on initial start of a comptue service, we'll create the compute node, then later try to update it here https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/resource_tracker.py#L1010 | |
| 21:53:54 | mriedem | if that fails, we've changed old_resources and then a subsequent run will return false from _resource_change and we'll go on to create the resource provider in placement and report inventory | |
| 21:53:58 | mriedem | but the compute node is botched | |
| 21:54:12 | mriedem | and then things in the scheduler blow up if we try to use that node | |
| 22:00:04 | openstackgerrit | Merged openstack/nova stable/queens: libvirt: Do not reraise DiskNotFound exceptions during resize https://review.opendev.org/660363 | |
| 22:20:48 | mriedem | efried: melwitt: would you think that on start of the compute service, if we hit a DBConnectionError here https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/manager.py#L8129 and startup=true, we should bail and not let the service start? | |
| 22:21:03 | mriedem | i guess it might correct itself... | |
| 22:21:30 | mriedem | maybe that's why we don't bail out | |
| 22:21:30 | efried | mriedem: it might. But in the meantime, everything is effed. Should at least disable the service at that point. | |
| 22:21:47 | mriedem | heh, if you can....need a db connection to do that :) | |
| 22:22:07 | melwitt | I know that in tripleo-land they don't want it to bail out because of puppet starting up services in different orders etc | |
| 22:22:27 | melwitt | they want it to keep retrying the db (for at least a limited time) before bailing out | |
| 22:23:12 | mriedem | yeah, i wonder if https://docs.openstack.org/nova/latest/configuration/config.html#database.use_db_reconnect would help since it's disabled by default i've never tried it | |
| 22:23:32 | melwitt | I've dealt multiple times with tickets getting opened for not services not retrying [temporarily] upon startup | |
| 22:24:58 | melwitt | mriedem: are you saying it is or isn't bailing out today? sorry I don't remember how it behaves at this point. some of our services retry db connection underneath because oslo.service does it. but for some reason that I don't remember, it didn't work on compute | |
| 22:25:15 | mriedem | use_db_reconnect wouldn't help nova since it only tells a decorator in oslo.db it's ok to retry (safe_for_db_retry) which we don't use | |
| 22:26:15 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#database.max_retries indicates that we'd have retries by default on connection failures | |
| 22:26:50 | mriedem | with 10 seconds between retries by default (according to retry_interval) | |
| 22:26:55 | melwitt | yeah. last I knew, compute does not retry and I can't remember if I figured out why it doesn't. because from all evidence, it should, the way all our other services do | |
| 22:27:07 | melwitt | let me dig up that old moldy BZ | |
| 22:27:21 | mriedem | it'd be whatever conductor the compute is talking to | |
| 22:28:00 | mriedem | anyway, the connectoin failure doesn't kill the compute on startup so we can get into a weird state with what nova has in the compute nodes table and what is in placement for the provider | |
| 22:28:12 | mriedem | i've debugged one issue as a result https://bugs.launchpad.net/nova/+bug/1834712 | |
| 22:28:12 | openstack | Launchpad bug 1834712 in OpenStack Compute (nova) "ResourceTracker._update should restore previous old_resources value if ComputeNode.save fails" [Medium,Triaged] | |
| 22:28:31 | efried | we need some kind of decorator to... like... @safely_connect | |
| 22:28:37 | mriedem | heh | |
| 22:29:01 | mriedem | i mean, i have a very unrealistic overloaded host situation here | |
| 22:29:02 | melwitt | that bug report makes my head hurt | |
| 22:29:07 | efried | and with that lightning jolt of sarcasm, I'm outta here. See y'all Monday. | |
| 22:29:44 | melwitt | I think that situation would be realistic for our RDO cloud. I've seen so much weird crap happen with it | |
| 22:30:24 | mriedem | yeah this is a critical section it turns out https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/resource_tracker.py#L1005-L1012 | |
| 22:30:25 | melwitt | like orphaned build request records etc because of spurious db errors | |
| 22:30:50 | melwitt | instance records that live in both cell0 and cell1 (I still don't get how on earth that is possible to happen) | |
| 22:30:51 | mriedem | and _resource_change mutates a thing in the RT that is checked on a later periodic run | |