Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-06
20:55:05 mriedem god _reschedule_resize_or_reraise in the compute manager is about as convoluted as it gets
20:55:21 mriedem it's not a bdm turducken, but it's close
20:55:36 superdan are you just trying to start trouble on a friday afternoon?
20:56:15 mriedem no, i've had a patch up for this bug since it came up in early november, but realized i needed a functional test for it which stalled it out for awhile, and then earlier in the week i triaged a duplicate of the same bug, so started working on writing that functional test,
20:56:24 mriedem which now that i have the functional test written, it doesn't fail as expected...
20:57:33 mriedem i pulled the reqspec out of the api db and the 'retry' field isn't saved, so i'm trying to figure out wth the reqspec gets saved during a reschedule, during build
20:58:04 superdan yeah, never go trawling through your backlog on a friday
20:58:09 superdan that's like tuesday stuff
20:58:17 mriedem the tab has been open since thursday, if that helps
20:58:21 superdan hah
20:59:51 mriedem far as i can tell, when we're still at the top during build, we populate the filter_properties dict with the initial host we're going to try for the build here https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155
21:00:23 mriedem but that never gets persisted back into the request_spec
21:01:08 superdan yeah, but it shouldn't right?
21:01:17 superdan because otherwise next time we went to do something we might think they requested that host specifically?
21:01:38 mriedem requesting the host specifically is the forced_hosts stuff,
21:01:45 mriedem that gets reset before all move operations
21:01:54 mriedem and yeah, i don't see a good reason to persist the retry hosts either
21:02:08 mriedem but this bug is asserting that they are/were and that causes problems with later move operations
21:02:47 mriedem time to get back to the irc logs from last september
21:04:58 mriedem oh i already found the convo http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-09-20.log.html#t2017-09-20T15:25:23
21:05:28 superdan doesn't seem to be loading for me
21:05:39 superdan my browser might have fridayitus
21:05:46 superdan it's a real plague
21:07:00 superdan lol, I just read over the dazed and confused references in that log
21:07:02 superdan good times
21:07:47 mriedem heh
21:07:59 mriedem the creepy guy that now drives a lincoln everywhere
21:08:03 superdan lol
21:11:04 mriedem this is his original build request spec http://paste.openstack.org/show/621557/
21:14:32 mriedem so, i'm wondering if either we used to save reqspec changes during reschedules back in mitaka?
21:14:38 mriedem otherwise, he might have special sauce
21:18:28 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771
21:18:29 mriedem ^ is the attempt at a recreate
21:19:09 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771
21:22:13 imacdonn fried_rice mriedem For https://review.openstack.org/#/c/558089/ , do we need to do anything further around unit tests? I can do a new PS with the requested comment changes, but wanted to see if anything else fell out of the discussion...
21:22:58 mriedem imacdonn: i have to go back and look at replies, in the middle of debugging something
21:23:53 fried_rice imacdonn: If mriedem is satisfied with my response at https://review.openstack.org/#/c/558089/1/nova/tests/unit/compute/test_compute_mgr.py@3730 then I think the remaining question is https://review.openstack.org/#/c/558089/1/nova/tests/unit/test_test.py@45 -- have you tried that out locally to see if it works?
21:25:11 mriedem looking back at mitaka, the api creates the request spec, but doesn't send it to conductor.build_instances, that just builds it's own fake request spec to pass to the scheduler; conductor passes that fake reqspec down to compute, where it's ignored
21:26:15 mriedem there are no request_spec.save() calls in mitaka, but there is that one in newton for a resize
21:26:28 imacdonn fried_rice: I haven't ... because I don't really know what that test is intended to accomplish
21:27:11 mriedem so i wonder if this is what happened: built the server on host1, resized - failed on host2, rescheduled to host3, then tried to live migrate to host2 and that was kicked out because the request_spec.save() during the resize reschedule was persiste
21:27:14 mriedem *persiste
21:27:16 mriedem gdi
21:27:25 mriedem this thing https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155
21:27:44 mriedem wrong link, this https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L349
21:27:54 fried_rice imacdonn: Does stuff break if you take out that particular PlacementFixture instantiation?
21:28:29 imacdonn fried_rice: yes, I had to add the PlacementFixture to get the test to not fail
21:29:59 cfriesen_ mriedem: seems plausible
21:30:06 mriedem cfriesen_: where the hell did you come from
21:30:37 fried_rice imacdonn: I think mriedem is correct that start_service('compute') would accomplish the same thing (just looking at what start_service('compute') now does, as amended by you). But then as you say, I'm not sure what we're gaining by that test in the first place, since there's any number of other tests that start_service('compute') already. If it were up to me, I'd say nix the test. So yeah, I guess let's wait for mrie
21:31:36 cfriesen_ mriedem: lurking all day, got bored with trying to solve internal race conditions
21:32:52 imacdonn fried_rice: ack
21:34:33 mriedem imacdonn: fried_rice: replied
21:34:36 fried_rice thx
21:35:44 imacdonn mriedem fried_rice Thanks .. I'll start wordsmithing comments then ;)
21:40:11 openstackgerrit Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597
21:43:22 imacdonn # configuration. If the request fails in any other way, it should be
21:43:22 imacdonn # MissingAuthPlugin results, it's likely due to lack of placement
21:43:22 imacdonn # requesting a bogus resource_provider (to minimise cost). If a
21:43:22 imacdonn # purposes, even if the scheduler doesn't require it. Verify this by
21:43:22 imacdonn # compute should always be configured to use placement, for reporting
21:43:23 imacdonn # obvious from the exception.
21:43:36 imacdonn mriedem fried_rice any objections to that? or edits?
21:44:14 fried_rice imacdonn: lgtm, but mriedem is the ocd one
21:44:17 fried_rice :P
21:44:20 imacdonn :)
21:46:51 mriedem that's the pot calling the kettle black if i've ever heard it
21:56:11 openstackgerrit iain MacDonnell proposed openstack/nova master: Update check to ensure compute is using placement https://review.openstack.org/558089
21:56:23 openstackgerrit Chris Dent proposed openstack/nova master: Move test_report_client out of placement namespace https://review.openstack.org/558911
22:23:47 fried_rice ö/
22:45:01 mriedem dansmith: i know you want to know this, but i figured out the bug
22:49:53 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - Add a microversion for consumer generation https://review.openstack.org/558958
22:55:46 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
22:55:46 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
23:35:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771
23:37:32 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771
#openstack-nova - 2018-04-07
00:40:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't persist RequestSpec.retry https://review.openstack.org/559447
01:29:29 openstackgerrit Merged openstack/nova master: DRY up test_rollback_live_migration_set_migration_status https://review.openstack.org/555489
02:34:55 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
02:34:55 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
02:52:14 openstackgerrit Merged openstack/nova master: Avoid unnecessary port update during live migration https://review.openstack.org/556332
05:27:12 openstackgerrit Tetsuro Nakamura proposed openstack/nova-specs master: Return all resources in provider_summaries spec https://review.openstack.org/559466
05:58:12 openstackgerrit Tetsuro Nakamura proposed openstack/nova-specs master: Enable NUMA Features for Libvirt/QEMU Driver https://review.openstack.org/533077
06:20:37 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return all resources in provider_summaries https://review.openstack.org/558045
06:51:57 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/548772
09:19:28 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Support shared and nested allocation candidates https://review.openstack.org/556514
09:19:28 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Consider nested RPs in get_all_with_shared https://review.openstack.org/556450
09:19:29 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480
12:43:44 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Add request_id to instance action notifications https://review.openstack.org/553288
13:38:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't persist RequestSpec.retry https://review.openstack.org/559447
13:38:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771
19:33:09 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
19:33:10 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
22:12:46 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
23:37:33 openstackgerrit Eric Fried proposed openstack/nova master: add lower-constraints job https://review.openstack.org/555961
23:44:44 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
#openstack-nova - 2018-04-08
00:52:07 openstackgerrit Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757
00:53:18 openstackgerrit Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757
01:10:23 openstackgerrit Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757
01:18:30 openstackgerrit Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757

Earlier   Later