| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-11-01 | |||
| 18:42:18 | dansmith | http://logs.openstack.org/09/517009/2/check/legacy-tempest-dsvm-neutron-multinode-full/cf39ab4/logs/screen-n-cpu.txt.gz#_Nov_01_16_51_15_194726 | |
| 18:44:25 | mriedem | dansmith: ah on ^ that's the same thing we get with the ComputeHostNotFound_Remote warning on startup since ocata | |
| 18:44:35 | mriedem | i think that's a regression in the log handler stuff in oslo, i was never able to track it down | |
| 18:44:49 | dansmith | okay but exc_info=False will squelch it right? | |
| 18:44:52 | mriedem | i think there was a change in oslo.log such that it auto-traces if you're in an exception context | |
| 18:44:57 | mriedem | dansmith: idk honestly | |
| 18:44:59 | mriedem | it's worth a try | |
| 18:45:33 | mriedem | dansmith: as for the actual failure behavior, resize fails with a 400 (should be a 500) | |
| 18:45:48 | mriedem | because remember at the point in conductor when we are swapping the allocations, we haven't cast off to the compute yet, | |
| 18:45:52 | mriedem | so we're still returning a response from the api | |
| 18:46:03 | dansmith | okay so this was probably a silent leak before | |
| 18:46:05 | mriedem | yup | |
| 18:46:06 | mriedem | definitely | |
| 18:46:11 | mriedem | silent but deadly | |
| 18:46:12 | dansmith | le suck | |
| 18:46:17 | openstackgerrit | Dan Smith proposed openstack/nova master: Avoid deleting allocations for instances being built https://review.openstack.org/517009 | |
| 18:46:51 | dansmith | jaypipes: did you see this? | |
| 18:51:08 | openstackgerrit | Merged openstack/nova master: Pass requested_destination in filter_properties https://review.openstack.org/481116 | |
| 18:51:42 | rabel | now that i got a +2 from jaypipes, can we finally merge https://review.openstack.org/#/c/494169/ ? :) | |
| 18:51:49 | jaypipes | dansmith: did I see https://review.openstack.org/#/c/517009/? | |
| 18:52:16 | dansmith | jaypipes: yeah, just making sure you saw it | |
| 18:52:25 | dansmith | since we're tweaking that logic | |
| 18:52:32 | jaypipes | k | |
| 19:11:07 | efried | jaypipes Any further thoughts on shared-vs-separate RequestGroup classes? (Not sure if you read the backscroll this morning, but mriedem doesn't feel strongly one way or the other.) | |
| 19:11:23 | jaypipes | efried: what does dansmith think? | |
| 19:11:39 | jaypipes | efried: and no, been heads down on the shared/no-shared refactoring thing. | |
| 19:11:45 | efried | Didn't ask - he seemed kinda swamped today | |
| 19:38:10 | mriedem | stephenfin: do you want me to make the os-vif core team changes from your email now? | |
| 19:40:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove compatibility method from FlavorPayload https://review.openstack.org/514943 | |
| 19:40:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Flavor.description attribute https://review.openstack.org/514371 | |
| 19:40:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FlavorPayload.description for versioned notifications https://review.openstack.org/514782 | |
| 19:40:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add microversion to allow setting flavor description https://review.openstack.org/515213 | |
| 20:15:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't log exception trace with ComputeHostNotFound warnings on startup https://review.openstack.org/517102 | |
| 20:40:45 | mriedem | jaypipes: think you need to drop the +2 on this https://review.openstack.org/#/c/517009/3 | |
| 20:40:57 | mriedem | the warning is showing up a ton in CI runs | |
| 20:41:19 | mriedem | http://logs.openstack.org/09/517009/3/check/legacy-tempest-dsvm-neutron-multinode-full/fda26ca/logs/screen-n-cpu.txt.gz | |
| 20:41:28 | mriedem | http://logs.openstack.org/09/517009/3/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/19d800c/logs/screen-n-cpu.txt.gz#_Nov_01_19_34_57_360853 | |
| 20:43:12 | jaypipes | mriedem: I would expect that warning to show up... | |
| 20:43:28 | mriedem | 101 times? | |
| 20:43:29 | jaypipes | mriedem: it's before the check for whether the instance.deleted is True... | |
| 20:43:58 | jaypipes | mriedem: if it was after that check, it likely wouldn't come up at all, right? | |
| 20:44:46 | mriedem | the only way we should see this warning is if we're racing between the time the scheduler creates allocations for an instance and when the instance is created in the cell db, | |
| 20:44:58 | mriedem | unless something else is hitting this code path besides the update_available_resource periodic task | |
| 20:46:45 | jaypipes | mriedem: I don't see anything else calling _remove_deleted_instances_allocations() other than that place in update_available_resource... | |
| 20:47:12 | mriedem | yeah and the only thing calling update_available_resource is the compute manager in the periodic and in post_live_migration | |
| 20:47:37 | mriedem | the periodic does run every minute... | |
| 20:47:56 | mriedem | but still, we'd have to be hitting that same window between claim in scheduler and creation in cell db | |
| 20:48:02 | mriedem | unless that window is pretty large | |
| 20:48:33 | jaypipes | mriedem: seems fairly unlikely to me. | |
| 20:48:40 | jaypipes | hitting the window that often... | |
| 20:48:50 | mriedem | let me see how often the debug message before this change shows up | |
| 20:50:59 | openstackgerrit | Merged openstack/nova master: Make put_allocations() retry on concurrent update https://review.openstack.org/516708 | |
| 20:51:07 | mriedem | seeing the old message show up in logs in another patch, but far less (3, 4 and 6 hits in 3 different jobs) | |
| 21:04:13 | mriedem | not really sure what's up with this, unless it's just totally random based on how slow the node is in the CI job | |
| 21:06:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Time how long select_destinations() takes in conductor https://review.openstack.org/517108 | |
| 21:13:39 | mriedem | looking at some logs for an ocata patch i don't see "Deleting allocations that remained for this" show up at all, so definitely prompte by the race introduced by doing the allocation creation in the scheduler in pike | |
| 21:22:52 | mriedem | edleafe: _schedule in the FilterScheduler should be returning a list of lists of selected hosts right? | |
| 21:23:21 | mriedem | i don't think _legacy_find_hosts does that if include_alternates is False | |
| 21:23:36 | mriedem | which means we'd get an IndexError in select_destinations | |
| 21:26:26 | mriedem | https://bugs.launchpad.net/nova/+bug/1729445 | |
| 21:26:27 | openstack | Launchpad bug 1729445 in OpenStack Compute (nova) "Potential IndexError if using the CachingScheduler and not getting alternates" [Medium,Triaged] | |
| 21:28:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: see if finding alternate hosts takes a long time https://review.openstack.org/517111 | |
| 21:52:02 | efried | jaypipes Still around? | |
| 21:52:18 | jaypipes | efried: yes. still working on sharing providers + traits | |
| 21:53:12 | efried | jaypipes I'm trying to ascertain whether there's any reason to keep the group_by at https://review.openstack.org/#/c/516778/3/nova/objects/resource_provider.py@2513 | |
| 21:53:47 | efried | Upon reading it, I thought it should actually be `distinct` instead - but I can't see any way the query would produce the same RP ID more than once in the result set anyway. | |
| 21:54:14 | jaypipes | efried: correct, it wouldn't. | |
| 21:54:19 | efried | k | |
| 21:54:21 | efried | thx | |
| 21:54:32 | jaypipes | efried: but I'm actively working on this series.. just FYI. | |
| 21:54:51 | efried | jaypipes Yeah, I'm just making notes, wasn't planning to publish my sandbox (yet). | |
| 21:55:01 | jaypipes | danke | |
| 21:55:17 | efried | jaypipes I'm reviewing this perhaps more thoroughly than necessary, as an educational exercise. | |
| 21:55:24 | jaypipes | ack | |
| 21:55:32 | jaypipes | I appreciate it very muc | |
| 21:55:34 | jaypipes | much | |
| 21:59:03 | edleafe | mriedem: yes | |
| 21:59:07 | efried | jaypipes Okay, dropped some comments on PS3 - perhaps you want to hit them in your current edit. | |
| 22:14:15 | openstackgerrit | Jay Pipes proposed openstack/nova master: required traits for no sharing providers https://review.openstack.org/517027 | |
| 22:14:15 | openstackgerrit | Jay Pipes proposed openstack/nova master: WIP: handle traits with sharing providers https://review.openstack.org/517119 | |
| 22:14:45 | jaypipes | efried: gonna take a dinner break. see ^ for just how complex the sharing providers with required traits code becomes... | |
| 22:14:56 | efried | jaypipes Ack | |
| 22:15:13 | jaypipes | efried: will come back to this later. let me know if you add/modify stuff and I'll re-pull. | |
| 22:15:27 | efried | jaypipes rgr, not planning to touch anything else tonight. | |
| 22:16:02 | jaypipes | k | |
| 23:04:30 | openstackgerrit | Merged openstack/nova master: Remove compatibility method from FlavorPayload https://review.openstack.org/514943 | |
| 23:12:08 | openstackgerrit | Moshe Levi proposed openstack/nova master: Don't overwrite binding-profile https://review.openstack.org/505613 | |
| 23:25:39 | openstackgerrit | Moshe Levi proposed openstack/nova master: Don't overwrite binding-profile https://review.openstack.org/505613 | |
| 23:29:34 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge flavor extensions controller code https://review.openstack.org/516104 | |
| 23:32:30 | openstackgerrit | Ed Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver https://review.openstack.org/495854 | |
| 23:32:30 | openstackgerrit | Ed Leafe proposed openstack/nova master: Modify select_destinations() to return objects and alts https://review.openstack.org/510159 | |
| 23:32:31 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707 | |
| 23:32:31 | openstackgerrit | Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357 | |
| 23:32:32 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 23:34:08 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge flavor extensions controller code https://review.openstack.org/516104 | |
| 23:37:02 | openstackgerrit | Moshe Levi proposed openstack/nova master: Don't overwrite binding-profile https://review.openstack.org/505613 | |
| 23:41:51 | efried | jaypipes I hit my brain-fry limit. See you tomorrow. | |
| 23:50:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix return type in FilterScheduler._legacy_find_hosts https://review.openstack.org/517134 | |
| 23:50:54 | mriedem | edleafe: ^ | |
| #openstack-nova - 2017-11-02 | |||
| 00:30:15 | mriedem | oops guess i have unit test failures | |