| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-11-01 | |||
| 16:49:01 | dansmith | there'll need to be some tool and procedure for patching that up prior to a cutover | |
| 16:49:28 | jaypipes | dansmith: by "tool" do you mean "delete the chance scheduler"? if so, then yes ++ | |
| 16:49:50 | dansmith | sorry, | |
| 16:49:53 | dansmith | I meant the caching | |
| 16:50:13 | dansmith | although I guess the same goes for chance | |
| 16:50:29 | mriedem | yeah true | |
| 16:54:07 | edleafe | dansmith: yeah, caching scheduler rules out claiming | |
| 16:54:39 | dansmith | it doesn't have to, we just have said we don't want to make it do it right? | |
| 16:54:56 | edleafe | for simple compute nodes it's possible, but not for any that involve shared/nested providers | |
| 16:55:30 | edleafe | well, the only way to claim complex rps is to get the allocation_candidates, which means no caching | |
| 17:05:50 | stephenfin | mriedem: Done | |
| 17:17:09 | mriedem | stephenfin: thanks | |
| 17:30:21 | mriedem | dansmith: tssurya: melwitt: no cells meeting today right? | |
| 17:32:36 | melwitt | +1 to cancel today | |
| 17:53:00 | dansmith | mriedem: ah yep | |
| 18:39:38 | openstackgerrit | Merged openstack/nova master: Functional regression test for evacuate with a target https://review.openstack.org/490997 | |
| 18:39:39 | dansmith | mriedem: so, this might be happening more often than we think.. I hit it on one of the runs on the fix patch | |
| 18:39:46 | dansmith | mriedem: what actual failure behavior did this cause? | |
| 18:40:46 | dansmith | also I guess I have to pass exc_info=False when I log.warning in an exception handler? | |
| 18:42:09 | dansmith | not really sure why | |
| 18:42:18 | dansmith | http://logs.openstack.org/09/517009/2/check/legacy-tempest-dsvm-neutron-multinode-full/cf39ab4/logs/screen-n-cpu.txt.gz#_Nov_01_16_51_15_194726 | |
| 18:44:25 | mriedem | dansmith: ah on ^ that's the same thing we get with the ComputeHostNotFound_Remote warning on startup since ocata | |
| 18:44:35 | mriedem | i think that's a regression in the log handler stuff in oslo, i was never able to track it down | |
| 18:44:49 | dansmith | okay but exc_info=False will squelch it right? | |
| 18:44:52 | mriedem | i think there was a change in oslo.log such that it auto-traces if you're in an exception context | |
| 18:44:57 | mriedem | dansmith: idk honestly | |
| 18:44:59 | mriedem | it's worth a try | |
| 18:45:33 | mriedem | dansmith: as for the actual failure behavior, resize fails with a 400 (should be a 500) | |
| 18:45:48 | mriedem | because remember at the point in conductor when we are swapping the allocations, we haven't cast off to the compute yet, | |
| 18:45:52 | mriedem | so we're still returning a response from the api | |
| 18:46:03 | dansmith | okay so this was probably a silent leak before | |
| 18:46:05 | mriedem | yup | |
| 18:46:06 | mriedem | definitely | |
| 18:46:11 | mriedem | silent but deadly | |
| 18:46:12 | dansmith | le suck | |
| 18:46:17 | openstackgerrit | Dan Smith proposed openstack/nova master: Avoid deleting allocations for instances being built https://review.openstack.org/517009 | |
| 18:46:51 | dansmith | jaypipes: did you see this? | |
| 18:51:08 | openstackgerrit | Merged openstack/nova master: Pass requested_destination in filter_properties https://review.openstack.org/481116 | |
| 18:51:42 | rabel | now that i got a +2 from jaypipes, can we finally merge https://review.openstack.org/#/c/494169/ ? :) | |
| 18:51:49 | jaypipes | dansmith: did I see https://review.openstack.org/#/c/517009/? | |
| 18:52:16 | dansmith | jaypipes: yeah, just making sure you saw it | |
| 18:52:25 | dansmith | since we're tweaking that logic | |
| 18:52:32 | jaypipes | k | |
| 19:11:07 | efried | jaypipes Any further thoughts on shared-vs-separate RequestGroup classes? (Not sure if you read the backscroll this morning, but mriedem doesn't feel strongly one way or the other.) | |
| 19:11:23 | jaypipes | efried: what does dansmith think? | |
| 19:11:39 | jaypipes | efried: and no, been heads down on the shared/no-shared refactoring thing. | |
| 19:11:45 | efried | Didn't ask - he seemed kinda swamped today | |
| 19:38:10 | mriedem | stephenfin: do you want me to make the os-vif core team changes from your email now? | |
| 19:40:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove compatibility method from FlavorPayload https://review.openstack.org/514943 | |
| 19:40:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Flavor.description attribute https://review.openstack.org/514371 | |
| 19:40:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FlavorPayload.description for versioned notifications https://review.openstack.org/514782 | |
| 19:40:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add microversion to allow setting flavor description https://review.openstack.org/515213 | |
| 20:15:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't log exception trace with ComputeHostNotFound warnings on startup https://review.openstack.org/517102 | |
| 20:40:45 | mriedem | jaypipes: think you need to drop the +2 on this https://review.openstack.org/#/c/517009/3 | |
| 20:40:57 | mriedem | the warning is showing up a ton in CI runs | |
| 20:41:19 | mriedem | http://logs.openstack.org/09/517009/3/check/legacy-tempest-dsvm-neutron-multinode-full/fda26ca/logs/screen-n-cpu.txt.gz | |
| 20:41:28 | mriedem | http://logs.openstack.org/09/517009/3/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/19d800c/logs/screen-n-cpu.txt.gz#_Nov_01_19_34_57_360853 | |
| 20:43:12 | jaypipes | mriedem: I would expect that warning to show up... | |
| 20:43:28 | mriedem | 101 times? | |
| 20:43:29 | jaypipes | mriedem: it's before the check for whether the instance.deleted is True... | |
| 20:43:58 | jaypipes | mriedem: if it was after that check, it likely wouldn't come up at all, right? | |
| 20:44:46 | mriedem | the only way we should see this warning is if we're racing between the time the scheduler creates allocations for an instance and when the instance is created in the cell db, | |
| 20:44:58 | mriedem | unless something else is hitting this code path besides the update_available_resource periodic task | |
| 20:46:45 | jaypipes | mriedem: I don't see anything else calling _remove_deleted_instances_allocations() other than that place in update_available_resource... | |
| 20:47:12 | mriedem | yeah and the only thing calling update_available_resource is the compute manager in the periodic and in post_live_migration | |
| 20:47:37 | mriedem | the periodic does run every minute... | |
| 20:47:56 | mriedem | but still, we'd have to be hitting that same window between claim in scheduler and creation in cell db | |
| 20:48:02 | mriedem | unless that window is pretty large | |
| 20:48:33 | jaypipes | mriedem: seems fairly unlikely to me. | |
| 20:48:40 | jaypipes | hitting the window that often... | |
| 20:48:50 | mriedem | let me see how often the debug message before this change shows up | |
| 20:50:59 | openstackgerrit | Merged openstack/nova master: Make put_allocations() retry on concurrent update https://review.openstack.org/516708 | |
| 20:51:07 | mriedem | seeing the old message show up in logs in another patch, but far less (3, 4 and 6 hits in 3 different jobs) | |
| 21:04:13 | mriedem | not really sure what's up with this, unless it's just totally random based on how slow the node is in the CI job | |
| 21:06:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Time how long select_destinations() takes in conductor https://review.openstack.org/517108 | |
| 21:13:39 | mriedem | looking at some logs for an ocata patch i don't see "Deleting allocations that remained for this" show up at all, so definitely prompte by the race introduced by doing the allocation creation in the scheduler in pike | |
| 21:22:52 | mriedem | edleafe: _schedule in the FilterScheduler should be returning a list of lists of selected hosts right? | |
| 21:23:21 | mriedem | i don't think _legacy_find_hosts does that if include_alternates is False | |
| 21:23:36 | mriedem | which means we'd get an IndexError in select_destinations | |
| 21:26:26 | mriedem | https://bugs.launchpad.net/nova/+bug/1729445 | |
| 21:26:27 | openstack | Launchpad bug 1729445 in OpenStack Compute (nova) "Potential IndexError if using the CachingScheduler and not getting alternates" [Medium,Triaged] | |
| 21:28:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: see if finding alternate hosts takes a long time https://review.openstack.org/517111 | |
| 21:52:02 | efried | jaypipes Still around? | |
| 21:52:18 | jaypipes | efried: yes. still working on sharing providers + traits | |
| 21:53:12 | efried | jaypipes I'm trying to ascertain whether there's any reason to keep the group_by at https://review.openstack.org/#/c/516778/3/nova/objects/resource_provider.py@2513 | |
| 21:53:47 | efried | Upon reading it, I thought it should actually be `distinct` instead - but I can't see any way the query would produce the same RP ID more than once in the result set anyway. | |
| 21:54:14 | jaypipes | efried: correct, it wouldn't. | |
| 21:54:19 | efried | k | |
| 21:54:21 | efried | thx | |
| 21:54:32 | jaypipes | efried: but I'm actively working on this series.. just FYI. | |
| 21:54:51 | efried | jaypipes Yeah, I'm just making notes, wasn't planning to publish my sandbox (yet). | |
| 21:55:01 | jaypipes | danke | |
| 21:55:17 | efried | jaypipes I'm reviewing this perhaps more thoroughly than necessary, as an educational exercise. | |
| 21:55:24 | jaypipes | ack | |
| 21:55:32 | jaypipes | I appreciate it very muc | |
| 21:55:34 | jaypipes | much | |
| 21:59:03 | edleafe | mriedem: yes | |
| 21:59:07 | efried | jaypipes Okay, dropped some comments on PS3 - perhaps you want to hit them in your current edit. | |
| 22:14:15 | openstackgerrit | Jay Pipes proposed openstack/nova master: required traits for no sharing providers https://review.openstack.org/517027 | |
| 22:14:15 | openstackgerrit | Jay Pipes proposed openstack/nova master: WIP: handle traits with sharing providers https://review.openstack.org/517119 | |