Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-06
19:41:01 sean-k-mooney anyway this time im actully going to leave since its 8:40 on a friday. enjoy your weekends o/
19:43:40 cdent o/
20:45:19 mriedem for the life of me i can't figure out why this isn't a problem anymore https://bugs.launchpad.net/nova/+bug/1718512
20:45:19 openstack Launchpad bug 1718512 in OpenStack Compute (nova) "migration fails if instance build failed on destination host" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
20:46:01 mriedem or, how it was a problem to begin with - since i don't know where we save 'retry' changes to a request spec during the initial build
20:49:44 mriedem superdan: i think i see a fun little weirdness, we update the requestspec with a new flavor during resize before we've actually confirmed that the resize happened https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L349
20:49:54 mriedem b/c at that point we've rpc cast to the dest compute to start the resize
20:50:13 mriedem if it fails on the compute, the request spec now says that the instance is using a new flavor
20:51:02 superdan mriedem: and that matters because why? if you later evacuate maybe?
20:51:12 mriedem it might not matter right now,
20:51:16 mriedem i was thinking,
20:51:33 mriedem if resize does a reschedule to the cell conductor, we could try saving changes to the request spec, because of the retry stuff
20:51:36 superdan actually, I think evac is probably fine since it likely uses the instance
20:51:39 mriedem which would be an upcall to the api db
20:52:14 superdan okay, but it shouldn't need to change the flavor
20:52:17 superdan on reschedule
20:52:22 superdan (from resize)
20:52:42 mriedem right the flavor in the requestspec wouldn't change on a reschedule during resize,
20:52:49 mriedem the RequestSpec.retry field might
20:53:44 mriedem although,
20:53:57 mriedem it looks like on a reschedule for resize, the compute doesn't send the request spec back up to the cell conductor
20:54:06 mriedem which is dumb, but likely the only thing saving this
20:54:22 superdan you mean "that's by design" right? :D
20:54:49 mriedem f no
20:55:05 mriedem god _reschedule_resize_or_reraise in the compute manager is about as convoluted as it gets
20:55:21 mriedem it's not a bdm turducken, but it's close
20:55:36 superdan are you just trying to start trouble on a friday afternoon?
20:56:15 mriedem no, i've had a patch up for this bug since it came up in early november, but realized i needed a functional test for it which stalled it out for awhile, and then earlier in the week i triaged a duplicate of the same bug, so started working on writing that functional test,
20:56:24 mriedem which now that i have the functional test written, it doesn't fail as expected...
20:57:33 mriedem i pulled the reqspec out of the api db and the 'retry' field isn't saved, so i'm trying to figure out wth the reqspec gets saved during a reschedule, during build
20:58:04 superdan yeah, never go trawling through your backlog on a friday
20:58:09 superdan that's like tuesday stuff
20:58:17 mriedem the tab has been open since thursday, if that helps
20:58:21 superdan hah
20:59:51 mriedem far as i can tell, when we're still at the top during build, we populate the filter_properties dict with the initial host we're going to try for the build here https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155
21:00:23 mriedem but that never gets persisted back into the request_spec
21:01:08 superdan yeah, but it shouldn't right?
21:01:17 superdan because otherwise next time we went to do something we might think they requested that host specifically?
21:01:38 mriedem requesting the host specifically is the forced_hosts stuff,
21:01:45 mriedem that gets reset before all move operations
21:01:54 mriedem and yeah, i don't see a good reason to persist the retry hosts either
21:02:08 mriedem but this bug is asserting that they are/were and that causes problems with later move operations
21:02:47 mriedem time to get back to the irc logs from last september
21:04:58 mriedem oh i already found the convo http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-09-20.log.html#t2017-09-20T15:25:23
21:05:28 superdan doesn't seem to be loading for me
21:05:39 superdan my browser might have fridayitus
21:05:46 superdan it's a real plague
21:07:00 superdan lol, I just read over the dazed and confused references in that log
21:07:02 superdan good times
21:07:47 mriedem heh
21:07:59 mriedem the creepy guy that now drives a lincoln everywhere
21:08:03 superdan lol
21:11:04 mriedem this is his original build request spec http://paste.openstack.org/show/621557/
21:14:32 mriedem so, i'm wondering if either we used to save reqspec changes during reschedules back in mitaka?
21:14:38 mriedem otherwise, he might have special sauce
21:18:28 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771
21:18:29 mriedem ^ is the attempt at a recreate
21:19:09 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771
21:22:13 imacdonn fried_rice mriedem For https://review.openstack.org/#/c/558089/ , do we need to do anything further around unit tests? I can do a new PS with the requested comment changes, but wanted to see if anything else fell out of the discussion...
21:22:58 mriedem imacdonn: i have to go back and look at replies, in the middle of debugging something
21:23:53 fried_rice imacdonn: If mriedem is satisfied with my response at https://review.openstack.org/#/c/558089/1/nova/tests/unit/compute/test_compute_mgr.py@3730 then I think the remaining question is https://review.openstack.org/#/c/558089/1/nova/tests/unit/test_test.py@45 -- have you tried that out locally to see if it works?
21:25:11 mriedem looking back at mitaka, the api creates the request spec, but doesn't send it to conductor.build_instances, that just builds it's own fake request spec to pass to the scheduler; conductor passes that fake reqspec down to compute, where it's ignored
21:26:15 mriedem there are no request_spec.save() calls in mitaka, but there is that one in newton for a resize
21:26:28 imacdonn fried_rice: I haven't ... because I don't really know what that test is intended to accomplish
21:27:11 mriedem so i wonder if this is what happened: built the server on host1, resized - failed on host2, rescheduled to host3, then tried to live migrate to host2 and that was kicked out because the request_spec.save() during the resize reschedule was persiste
21:27:14 mriedem *persiste
21:27:16 mriedem gdi
21:27:25 mriedem this thing https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155
21:27:44 mriedem wrong link, this https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L349
21:27:54 fried_rice imacdonn: Does stuff break if you take out that particular PlacementFixture instantiation?
21:28:29 imacdonn fried_rice: yes, I had to add the PlacementFixture to get the test to not fail
21:29:59 cfriesen_ mriedem: seems plausible
21:30:06 mriedem cfriesen_: where the hell did you come from
21:30:37 fried_rice imacdonn: I think mriedem is correct that start_service('compute') would accomplish the same thing (just looking at what start_service('compute') now does, as amended by you). But then as you say, I'm not sure what we're gaining by that test in the first place, since there's any number of other tests that start_service('compute') already. If it were up to me, I'd say nix the test. So yeah, I guess let's wait for mrie
21:31:36 cfriesen_ mriedem: lurking all day, got bored with trying to solve internal race conditions
21:32:52 imacdonn fried_rice: ack
21:34:33 mriedem imacdonn: fried_rice: replied
21:34:36 fried_rice thx
21:35:44 imacdonn mriedem fried_rice Thanks .. I'll start wordsmithing comments then ;)
21:40:11 openstackgerrit Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597
21:43:22 imacdonn # compute should always be configured to use placement, for reporting
21:43:22 imacdonn # purposes, even if the scheduler doesn't require it. Verify this by
21:43:22 imacdonn # requesting a bogus resource_provider (to minimise cost). If a
21:43:22 imacdonn # MissingAuthPlugin results, it's likely due to lack of placement
21:43:22 imacdonn # configuration. If the request fails in any other way, it should be
21:43:23 imacdonn # obvious from the exception.
21:43:36 imacdonn mriedem fried_rice any objections to that? or edits?
21:44:14 fried_rice imacdonn: lgtm, but mriedem is the ocd one
21:44:17 fried_rice :P
21:44:20 imacdonn :)
21:46:51 mriedem that's the pot calling the kettle black if i've ever heard it
21:56:11 openstackgerrit iain MacDonnell proposed openstack/nova master: Update check to ensure compute is using placement https://review.openstack.org/558089
21:56:23 openstackgerrit Chris Dent proposed openstack/nova master: Move test_report_client out of placement namespace https://review.openstack.org/558911
22:23:47 fried_rice ö/
22:45:01 mriedem dansmith: i know you want to know this, but i figured out the bug
22:49:53 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - Add a microversion for consumer generation https://review.openstack.org/558958
22:55:46 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
22:55:46 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
23:35:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771
23:37:32 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771

Earlier   Later