| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-06 | |||
| 19:01:49 | cdent | fried_rice: since you're not one with the twitters, you may not be aware of the placedock (or the blog posts associated with it) stuff, if you go to the placedock repo there's loads of links to context | |
| 19:01:57 | cdent | fried_rice: pretty much | |
| 19:02:15 | cdent | depending on the variables you pass into the container | |
| 19:02:26 | cdent | (and I recently found a bug with that) | |
| 19:03:07 | cdent | and in terms of production, you could, except that the container builds from non master code | |
| 19:07:09 | openstackgerrit | sean mooney proposed openstack/os-vif master: make legacy-tempest-dsvm-nova-os-vif voteing https://review.openstack.org/559401 | |
| 19:31:25 | sean-k-mooney | cdent: i have been following your tweets on it on and off its a cool idea | |
| 19:33:48 | sean-k-mooney | cdent: have you considerd useing any of this work as part of functional testing for placemnet | |
| 19:34:24 | cdent | sean-k-mooney: not realy: because of the way the existing gabbi tests work with wsgi-intercept, it's not really needed | |
| 19:34:32 | cdent | using it for some kind of integration tests might be useful thought | |
| 19:34:39 | sean-k-mooney | cdent: when placement has its own repo this could be an interesting way for folks to do placement dev/testing | |
| 19:34:46 | cdent | yes | |
| 19:34:56 | sean-k-mooney | cdent: well i was thinking it would be kind of like neutron fullstack tests | |
| 19:35:13 | sean-k-mooney | they are more then fuctional tests but not quite intergration tests | |
| 19:35:30 | cdent | sean-k-mooney: what do you see as the difference between using the container and what the current wsgi-intercept driven tests do? | |
| 19:36:51 | sean-k-mooney | cdent: noting really other then the fact that you are using sqlite right so i dont need to setup a db first so it would mean a smaller testing requiremtes set | |
| 19:38:19 | cdent | the container can use any database, depending on environment variables | |
| 19:38:31 | cdent | the functional tests use sqlite, yeah? | |
| 19:39:00 | sean-k-mooney | am im not sure what they use but were you not tweeting about using sqlight | |
| 19:39:43 | sean-k-mooney | i could be getting confused between what can work and what that suff is using today | |
| 19:41:01 | sean-k-mooney | anyway this time im actully going to leave since its 8:40 on a friday. enjoy your weekends o/ | |
| 19:43:40 | cdent | o/ | |
| 20:45:19 | mriedem | for the life of me i can't figure out why this isn't a problem anymore https://bugs.launchpad.net/nova/+bug/1718512 | |
| 20:45:19 | openstack | Launchpad bug 1718512 in OpenStack Compute (nova) "migration fails if instance build failed on destination host" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 20:46:01 | mriedem | or, how it was a problem to begin with - since i don't know where we save 'retry' changes to a request spec during the initial build | |
| 20:49:44 | mriedem | superdan: i think i see a fun little weirdness, we update the requestspec with a new flavor during resize before we've actually confirmed that the resize happened https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L349 | |
| 20:49:54 | mriedem | b/c at that point we've rpc cast to the dest compute to start the resize | |
| 20:50:13 | mriedem | if it fails on the compute, the request spec now says that the instance is using a new flavor | |
| 20:51:02 | superdan | mriedem: and that matters because why? if you later evacuate maybe? | |
| 20:51:12 | mriedem | it might not matter right now, | |
| 20:51:16 | mriedem | i was thinking, | |
| 20:51:33 | mriedem | if resize does a reschedule to the cell conductor, we could try saving changes to the request spec, because of the retry stuff | |
| 20:51:36 | superdan | actually, I think evac is probably fine since it likely uses the instance | |
| 20:51:39 | mriedem | which would be an upcall to the api db | |
| 20:52:14 | superdan | okay, but it shouldn't need to change the flavor | |
| 20:52:17 | superdan | on reschedule | |
| 20:52:22 | superdan | (from resize) | |
| 20:52:42 | mriedem | right the flavor in the requestspec wouldn't change on a reschedule during resize, | |
| 20:52:49 | mriedem | the RequestSpec.retry field might | |
| 20:53:44 | mriedem | although, | |
| 20:53:57 | mriedem | it looks like on a reschedule for resize, the compute doesn't send the request spec back up to the cell conductor | |
| 20:54:06 | mriedem | which is dumb, but likely the only thing saving this | |
| 20:54:22 | superdan | you mean "that's by design" right? :D | |
| 20:54:49 | mriedem | f no | |
| 20:55:05 | mriedem | god _reschedule_resize_or_reraise in the compute manager is about as convoluted as it gets | |
| 20:55:21 | mriedem | it's not a bdm turducken, but it's close | |
| 20:55:36 | superdan | are you just trying to start trouble on a friday afternoon? | |
| 20:56:15 | mriedem | no, i've had a patch up for this bug since it came up in early november, but realized i needed a functional test for it which stalled it out for awhile, and then earlier in the week i triaged a duplicate of the same bug, so started working on writing that functional test, | |
| 20:56:24 | mriedem | which now that i have the functional test written, it doesn't fail as expected... | |
| 20:57:33 | mriedem | i pulled the reqspec out of the api db and the 'retry' field isn't saved, so i'm trying to figure out wth the reqspec gets saved during a reschedule, during build | |
| 20:58:04 | superdan | yeah, never go trawling through your backlog on a friday | |
| 20:58:09 | superdan | that's like tuesday stuff | |
| 20:58:17 | mriedem | the tab has been open since thursday, if that helps | |
| 20:58:21 | superdan | hah | |
| 20:59:51 | mriedem | far as i can tell, when we're still at the top during build, we populate the filter_properties dict with the initial host we're going to try for the build here https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155 | |
| 21:00:23 | mriedem | but that never gets persisted back into the request_spec | |
| 21:01:08 | superdan | yeah, but it shouldn't right? | |
| 21:01:17 | superdan | because otherwise next time we went to do something we might think they requested that host specifically? | |
| 21:01:38 | mriedem | requesting the host specifically is the forced_hosts stuff, | |
| 21:01:45 | mriedem | that gets reset before all move operations | |
| 21:01:54 | mriedem | and yeah, i don't see a good reason to persist the retry hosts either | |
| 21:02:08 | mriedem | but this bug is asserting that they are/were and that causes problems with later move operations | |
| 21:02:47 | mriedem | time to get back to the irc logs from last september | |
| 21:04:58 | mriedem | oh i already found the convo http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-09-20.log.html#t2017-09-20T15:25:23 | |
| 21:05:28 | superdan | doesn't seem to be loading for me | |
| 21:05:39 | superdan | my browser might have fridayitus | |
| 21:05:46 | superdan | it's a real plague | |
| 21:07:00 | superdan | lol, I just read over the dazed and confused references in that log | |
| 21:07:02 | superdan | good times | |
| 21:07:47 | mriedem | heh | |
| 21:07:59 | mriedem | the creepy guy that now drives a lincoln everywhere | |
| 21:08:03 | superdan | lol | |
| 21:11:04 | mriedem | this is his original build request spec http://paste.openstack.org/show/621557/ | |
| 21:14:32 | mriedem | so, i'm wondering if either we used to save reqspec changes during reschedules back in mitaka? | |
| 21:14:38 | mriedem | otherwise, he might have special sauce | |
| 21:18:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771 | |
| 21:18:29 | mriedem | ^ is the attempt at a recreate | |
| 21:19:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771 | |
| 21:22:13 | imacdonn | fried_rice mriedem For https://review.openstack.org/#/c/558089/ , do we need to do anything further around unit tests? I can do a new PS with the requested comment changes, but wanted to see if anything else fell out of the discussion... | |
| 21:22:58 | mriedem | imacdonn: i have to go back and look at replies, in the middle of debugging something | |
| 21:23:53 | fried_rice | imacdonn: If mriedem is satisfied with my response at https://review.openstack.org/#/c/558089/1/nova/tests/unit/compute/test_compute_mgr.py@3730 then I think the remaining question is https://review.openstack.org/#/c/558089/1/nova/tests/unit/test_test.py@45 -- have you tried that out locally to see if it works? | |
| 21:25:11 | mriedem | looking back at mitaka, the api creates the request spec, but doesn't send it to conductor.build_instances, that just builds it's own fake request spec to pass to the scheduler; conductor passes that fake reqspec down to compute, where it's ignored | |
| 21:26:15 | mriedem | there are no request_spec.save() calls in mitaka, but there is that one in newton for a resize | |
| 21:26:28 | imacdonn | fried_rice: I haven't ... because I don't really know what that test is intended to accomplish | |
| 21:27:11 | mriedem | so i wonder if this is what happened: built the server on host1, resized - failed on host2, rescheduled to host3, then tried to live migrate to host2 and that was kicked out because the request_spec.save() during the resize reschedule was persiste | |
| 21:27:14 | mriedem | *persiste | |
| 21:27:16 | mriedem | gdi | |
| 21:27:25 | mriedem | this thing https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155 | |
| 21:27:44 | mriedem | wrong link, this https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L349 | |
| 21:27:54 | fried_rice | imacdonn: Does stuff break if you take out that particular PlacementFixture instantiation? | |
| 21:28:29 | imacdonn | fried_rice: yes, I had to add the PlacementFixture to get the test to not fail | |
| 21:29:59 | cfriesen_ | mriedem: seems plausible | |
| 21:30:06 | mriedem | cfriesen_: where the hell did you come from | |
| 21:30:37 | fried_rice | imacdonn: I think mriedem is correct that start_service('compute') would accomplish the same thing (just looking at what start_service('compute') now does, as amended by you). But then as you say, I'm not sure what we're gaining by that test in the first place, since there's any number of other tests that start_service('compute') already. If it were up to me, I'd say nix the test. So yeah, I guess let's wait for mrie | |
| 21:31:36 | cfriesen_ | mriedem: lurking all day, got bored with trying to solve internal race conditions | |
| 21:32:52 | imacdonn | fried_rice: ack | |
| 21:34:33 | mriedem | imacdonn: fried_rice: replied | |
| 21:34:36 | fried_rice | thx | |
| 21:35:44 | imacdonn | mriedem fried_rice Thanks .. I'll start wordsmithing comments then ;) | |
| 21:40:11 | openstackgerrit | Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597 | |