| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-06 | |||
| 20:51:33 | mriedem | if resize does a reschedule to the cell conductor, we could try saving changes to the request spec, because of the retry stuff | |
| 20:51:36 | superdan | actually, I think evac is probably fine since it likely uses the instance | |
| 20:51:39 | mriedem | which would be an upcall to the api db | |
| 20:52:14 | superdan | okay, but it shouldn't need to change the flavor | |
| 20:52:17 | superdan | on reschedule | |
| 20:52:22 | superdan | (from resize) | |
| 20:52:42 | mriedem | right the flavor in the requestspec wouldn't change on a reschedule during resize, | |
| 20:52:49 | mriedem | the RequestSpec.retry field might | |
| 20:53:44 | mriedem | although, | |
| 20:53:57 | mriedem | it looks like on a reschedule for resize, the compute doesn't send the request spec back up to the cell conductor | |
| 20:54:06 | mriedem | which is dumb, but likely the only thing saving this | |
| 20:54:22 | superdan | you mean "that's by design" right? :D | |
| 20:54:49 | mriedem | f no | |
| 20:55:05 | mriedem | god _reschedule_resize_or_reraise in the compute manager is about as convoluted as it gets | |
| 20:55:21 | mriedem | it's not a bdm turducken, but it's close | |
| 20:55:36 | superdan | are you just trying to start trouble on a friday afternoon? | |
| 20:56:15 | mriedem | no, i've had a patch up for this bug since it came up in early november, but realized i needed a functional test for it which stalled it out for awhile, and then earlier in the week i triaged a duplicate of the same bug, so started working on writing that functional test, | |
| 20:56:24 | mriedem | which now that i have the functional test written, it doesn't fail as expected... | |
| 20:57:33 | mriedem | i pulled the reqspec out of the api db and the 'retry' field isn't saved, so i'm trying to figure out wth the reqspec gets saved during a reschedule, during build | |
| 20:58:04 | superdan | yeah, never go trawling through your backlog on a friday | |
| 20:58:09 | superdan | that's like tuesday stuff | |
| 20:58:17 | mriedem | the tab has been open since thursday, if that helps | |
| 20:58:21 | superdan | hah | |
| 20:59:51 | mriedem | far as i can tell, when we're still at the top during build, we populate the filter_properties dict with the initial host we're going to try for the build here https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155 | |
| 21:00:23 | mriedem | but that never gets persisted back into the request_spec | |
| 21:01:08 | superdan | yeah, but it shouldn't right? | |
| 21:01:17 | superdan | because otherwise next time we went to do something we might think they requested that host specifically? | |
| 21:01:38 | mriedem | requesting the host specifically is the forced_hosts stuff, | |
| 21:01:45 | mriedem | that gets reset before all move operations | |
| 21:01:54 | mriedem | and yeah, i don't see a good reason to persist the retry hosts either | |
| 21:02:08 | mriedem | but this bug is asserting that they are/were and that causes problems with later move operations | |
| 21:02:47 | mriedem | time to get back to the irc logs from last september | |
| 21:04:58 | mriedem | oh i already found the convo http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-09-20.log.html#t2017-09-20T15:25:23 | |
| 21:05:28 | superdan | doesn't seem to be loading for me | |
| 21:05:39 | superdan | my browser might have fridayitus | |
| 21:05:46 | superdan | it's a real plague | |
| 21:07:00 | superdan | lol, I just read over the dazed and confused references in that log | |
| 21:07:02 | superdan | good times | |
| 21:07:47 | mriedem | heh | |
| 21:07:59 | mriedem | the creepy guy that now drives a lincoln everywhere | |
| 21:08:03 | superdan | lol | |
| 21:11:04 | mriedem | this is his original build request spec http://paste.openstack.org/show/621557/ | |
| 21:14:32 | mriedem | so, i'm wondering if either we used to save reqspec changes during reschedules back in mitaka? | |
| 21:14:38 | mriedem | otherwise, he might have special sauce | |
| 21:18:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771 | |
| 21:18:29 | mriedem | ^ is the attempt at a recreate | |
| 21:19:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771 | |
| 21:22:13 | imacdonn | fried_rice mriedem For https://review.openstack.org/#/c/558089/ , do we need to do anything further around unit tests? I can do a new PS with the requested comment changes, but wanted to see if anything else fell out of the discussion... | |
| 21:22:58 | mriedem | imacdonn: i have to go back and look at replies, in the middle of debugging something | |
| 21:23:53 | fried_rice | imacdonn: If mriedem is satisfied with my response at https://review.openstack.org/#/c/558089/1/nova/tests/unit/compute/test_compute_mgr.py@3730 then I think the remaining question is https://review.openstack.org/#/c/558089/1/nova/tests/unit/test_test.py@45 -- have you tried that out locally to see if it works? | |
| 21:25:11 | mriedem | looking back at mitaka, the api creates the request spec, but doesn't send it to conductor.build_instances, that just builds it's own fake request spec to pass to the scheduler; conductor passes that fake reqspec down to compute, where it's ignored | |
| 21:26:15 | mriedem | there are no request_spec.save() calls in mitaka, but there is that one in newton for a resize | |
| 21:26:28 | imacdonn | fried_rice: I haven't ... because I don't really know what that test is intended to accomplish | |
| 21:27:11 | mriedem | so i wonder if this is what happened: built the server on host1, resized - failed on host2, rescheduled to host3, then tried to live migrate to host2 and that was kicked out because the request_spec.save() during the resize reschedule was persiste | |
| 21:27:14 | mriedem | *persiste | |
| 21:27:16 | mriedem | gdi | |
| 21:27:25 | mriedem | this thing https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155 | |
| 21:27:44 | mriedem | wrong link, this https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L349 | |
| 21:27:54 | fried_rice | imacdonn: Does stuff break if you take out that particular PlacementFixture instantiation? | |
| 21:28:29 | imacdonn | fried_rice: yes, I had to add the PlacementFixture to get the test to not fail | |
| 21:29:59 | cfriesen_ | mriedem: seems plausible | |
| 21:30:06 | mriedem | cfriesen_: where the hell did you come from | |
| 21:30:37 | fried_rice | imacdonn: I think mriedem is correct that start_service('compute') would accomplish the same thing (just looking at what start_service('compute') now does, as amended by you). But then as you say, I'm not sure what we're gaining by that test in the first place, since there's any number of other tests that start_service('compute') already. If it were up to me, I'd say nix the test. So yeah, I guess let's wait for mrie | |
| 21:31:36 | cfriesen_ | mriedem: lurking all day, got bored with trying to solve internal race conditions | |
| 21:32:52 | imacdonn | fried_rice: ack | |
| 21:34:33 | mriedem | imacdonn: fried_rice: replied | |
| 21:34:36 | fried_rice | thx | |
| 21:35:44 | imacdonn | mriedem fried_rice Thanks .. I'll start wordsmithing comments then ;) | |
| 21:40:11 | openstackgerrit | Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597 | |
| 21:43:22 | imacdonn | # configuration. If the request fails in any other way, it should be | |
| 21:43:22 | imacdonn | # MissingAuthPlugin results, it's likely due to lack of placement | |
| 21:43:22 | imacdonn | # requesting a bogus resource_provider (to minimise cost). If a | |
| 21:43:22 | imacdonn | # purposes, even if the scheduler doesn't require it. Verify this by | |
| 21:43:22 | imacdonn | # compute should always be configured to use placement, for reporting | |
| 21:43:23 | imacdonn | # obvious from the exception. | |
| 21:43:36 | imacdonn | mriedem fried_rice any objections to that? or edits? | |
| 21:44:14 | fried_rice | imacdonn: lgtm, but mriedem is the ocd one | |
| 21:44:17 | fried_rice | :P | |
| 21:44:20 | imacdonn | :) | |
| 21:46:51 | mriedem | that's the pot calling the kettle black if i've ever heard it | |
| 21:56:11 | openstackgerrit | iain MacDonnell proposed openstack/nova master: Update check to ensure compute is using placement https://review.openstack.org/558089 | |
| 21:56:23 | openstackgerrit | Chris Dent proposed openstack/nova master: Move test_report_client out of placement namespace https://review.openstack.org/558911 | |
| 22:23:47 | fried_rice | ö/ | |
| 22:45:01 | mriedem | dansmith: i know you want to know this, but i figured out the bug | |
| 22:49:53 | openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - Add a microversion for consumer generation https://review.openstack.org/558958 | |
| 22:55:46 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 22:55:46 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 23:35:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771 | |
| 23:37:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771 | |
| #openstack-nova - 2018-04-07 | |||
| 00:40:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't persist RequestSpec.retry https://review.openstack.org/559447 | |
| 01:29:29 | openstackgerrit | Merged openstack/nova master: DRY up test_rollback_live_migration_set_migration_status https://review.openstack.org/555489 | |
| 02:34:55 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 02:34:55 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 02:52:14 | openstackgerrit | Merged openstack/nova master: Avoid unnecessary port update during live migration https://review.openstack.org/556332 | |
| 05:27:12 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova-specs master: Return all resources in provider_summaries spec https://review.openstack.org/559466 | |
| 05:58:12 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova-specs master: Enable NUMA Features for Libvirt/QEMU Driver https://review.openstack.org/533077 | |
| 06:20:37 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Return all resources in provider_summaries https://review.openstack.org/558045 | |
| 06:51:57 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/548772 | |
| 09:19:28 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Support shared and nested allocation candidates https://review.openstack.org/556514 | |
| 09:19:28 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Consider nested RPs in get_all_with_shared https://review.openstack.org/556450 | |