| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-02 | |||
| 18:42:02 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/migrate.py#L291 | |
| 18:42:11 | eandersson | > legacy_props.pop('context', None) | |
| 18:42:41 | eandersson | maybe we need to do the same on both? | |
| 18:42:55 | mriedem | legacy_spec should be a dict form of the RequestSpec object w/o a context in it | |
| 18:43:06 | mriedem | the dance done there is b/c of datetime fields afaict | |
| 18:43:13 | mriedem | er per the comment from bauzas | |
| 18:43:24 | mriedem | i also recently investigated this b/c i wanted to remove that... | |
| 18:44:37 | mriedem | yeah https://review.opendev.org/#/c/582417/ | |
| 18:46:00 | eandersson | Would that be possible to backport? | |
| 18:46:10 | mriedem | no it has an rpc change | |
| 18:46:17 | mriedem | but i'm not sure it would help you anyway | |
| 18:46:43 | mriedem | note that we're still serializing/deserializing the ImageMeta object https://review.opendev.org/#/c/582417/7/nova/compute/rpcapi.py@741 | |
| 18:47:38 | mriedem | i think, but don't know why, that is only in here for unit/functional tests that rely on the oslo.messaging rpc fake driver, but it doesn't make much sense to me because our functional tests pass things with datetimes over rpc all the time, | |
| 18:47:48 | mriedem | like passing RequestSpec to the scheduler's select_destinations call | |
| 18:49:01 | mriedem | do you have any out of tree changes to RequestSpec? | |
| 18:49:16 | eandersson | Not that I am aware of | |
| 18:49:29 | mriedem | is this an isolated incident? | |
| 18:49:29 | eandersson | The only changes we do have are customer scheduler filters and weighers | |
| 18:49:43 | eandersson | isolated so far yea | |
| 18:49:51 | eandersson | but we haven't done a lot of migrations | |
| 18:50:16 | eandersson | and this happened during like test run 3 | |
| 18:50:34 | mriedem | hmm, ok so this is a cold migratoin (or resize) that is failing on one host and trying to reschedule to another and then failing when trying to serialize the legacy request spec b/c rpc casting to prep_resize on the alternative host | |
| 18:50:37 | eandersson | *customer not customer | |
| 18:50:43 | mriedem | the difference is probably in the reschedule | |
| 18:50:50 | eandersson | *custom | |
| 18:50:51 | eandersson | man | |
| 18:53:07 | mriedem | i'll poke through that flow, we don't really hit migration reschedules in tempest integration runs unless it's in a multinode job | |
| 18:53:12 | mriedem | and not intentionally | |
| 18:53:30 | eandersson | Does it use placement for scheduling migrations? | |
| 18:54:17 | eandersson | Due to how this fleet is set up it is a little too likely to hit Rescheduling errors at the moment in general. | |
| 18:55:15 | eandersson | We heavily favor stacking VMs, plus Placement isn't properly configured, as we still rely on Aggregates for over-commit etc. | |
| 18:56:01 | eandersson | One of the downsides with doing a Mitaka to Rocky jump is that you miss things in releasenotes. :p | |
| 18:56:19 | mriedem | placement is used for scheduling, | |
| 18:56:49 | mriedem | so yes, the scheduler is going to ask placement for candidate hosts and the scheduler will pick a primary host after weighing and some alternates | |
| 18:57:01 | mriedem | and those alternates are what get passed between conductor and compute during reschedules | |
| 18:57:21 | mriedem | so after the initial scheduling request placement isn't really in the picture except for cleaning up allocations once the resize is confirmed/reverted | |
| 18:57:43 | mriedem | i mean, i don't know what is triggering your reschedule - that should be clear from the compute logs, | |
| 18:57:50 | eandersson | I have a good idea. | |
| 18:57:59 | mriedem | but the valueerror thing is from something with the RequestSpec which doesn't really have anything to do with placement | |
| 19:00:17 | mriedem | we have functional tests that trigger resize reschedules but they use fake rpc drivers and serialization fixtures for the context so those might be masking something, | |
| 19:00:30 | mriedem | i'd have to stand up a 2 node devstack and hack it up to recreate | |
| 19:01:07 | mriedem | if you could recreate, it would be good to dump that thing that gets serialized before it blows up | |
| 19:01:17 | eandersson | Yep - I'll see if we can catch it | |
| 19:01:22 | eandersson | Maybe even in the lab. | |
| 19:01:44 | mriedem | i'm not sure if i can get something setup to recreate in the next hour | |
| 19:02:19 | eandersson | I am also gonna re-run the tests in a env with a more optimal placement config. | |
| 19:03:17 | mriedem | are there any special image metadata properties on the server being migrated? | |
| 19:03:58 | mriedem | numa_topology, pci_requests? | |
| 19:05:32 | eandersson | numa, nothing sepcial with pci | |
| 19:06:02 | eandersson | I don't think we have anything special in image metadata, will need to double check | |
| 19:06:08 | mriedem | hmm, the numa_topology object gets serialized in that thing as well, and that's likely not something we have in-tree functional tests for (instance + numa + resize reschedule) | |
| 19:09:00 | mriedem | looking at the rocky code, on the first host attempt, conductor will serialize the request spec and pass it to compute, but compute won't pass it back to conductor on reschedule, so conductor has to build a new one https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L304 | |
| 19:09:20 | mriedem | which will again get serialized and deserialized to dict form when calling prep_resize for the alternate host | |
| 19:09:58 | mriedem | but the point being in rocky the request used for the first host isn't necessarily the same as the one for the alternate which could mean....something | |
| 19:25:17 | donnyd | is max_concurernt_builds per hypervisor? | |
| 19:28:17 | donnyd | for instance if i want to have only 2 instances per hypervisor schedule at any one point in time I would set that in nova-compute under max_concurernt_builds correct? | |
| 19:40:13 | eandersson | donnyd, yes | |
| 19:40:26 | eandersson | https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L610 | |
| 19:43:41 | donnyd | thats what i thought, just confirming | |
| 19:43:45 | donnyd | thanks eandersson | |
| 19:48:23 | mriedem | cdent: looking at osprofiler data payed off https://bugs.launchpad.net/nova/+bug/1838817 | |
| 19:48:23 | openstack | Launchpad bug 1838817 in OpenStack Compute (nova) "neutron: refreshing vif model for a server gets the same network multiple times" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem) | |
| 20:04:12 | eandersson | mriedem, we had to set heal_instance_info_cache high because it was killing our control plane | |
| 20:04:15 | eandersson | Is that why? :D | |
| 20:04:45 | mriedem | that's not the only thing, | |
| 20:05:07 | mriedem | but yeah it's expensive b/c building the instance nw info model means calling neutron separately for ports, networks, ips and subnets | |
| 20:05:17 | mriedem | since there isn't a batch api to get all of that information | |
| 20:05:34 | eandersson | It was getting real heavy on large sites with 1k nodes | |
| 20:06:10 | mriedem | yeah b/c each node is hitting that refresh every minute | |
| 20:06:20 | eandersson | We also ended up adding a variance | |
| 20:06:39 | mriedem | a doc | |
| 20:07:01 | eandersson | Yea - let me know we have a lot of custom settings | |
| 20:07:12 | mriedem | as does cern and others | |
| 20:07:21 | mriedem | this kind of stuff comes up and we don't have a place to dump it really in the docs | |
| 20:07:26 | eandersson | Yep | |
| 20:07:29 | mriedem | similar with the long_rpc_timeout option | |
| 20:07:38 | eandersson | Yea - we have it set to 300 :D | |
| 20:07:43 | eandersson | 1k for dhcp agent lol | |
| 20:08:22 | eandersson | (due to this bug https://github.com/openstack/neutron/commit/f9f6ae9c98a9ff9d3c0807b9fd51fb2ffe25fd5d) | |
| 20:09:41 | mriedem | which thing do you have set to 300? | |
| 20:09:45 | mriedem | heal instance info cache interval? | |
| 20:10:01 | eandersson | 240 + variance on heal instance infop | |
| 20:10:17 | eandersson | rcp_timeout is set to 300, but that could have been a mitaka thing | |
| 20:11:23 | mriedem | rpc_response_timeout is the old one that defaults to 60 | |
| 20:11:33 | mriedem | rpc_long_timeout is new as of i think rocky and is only used for certain long rpc calls | |
| 20:11:37 | mriedem | like during live migratoin | |
| 20:11:43 | eandersson | ah I see - didn't know about that one at all | |
| 20:11:50 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.long_rpc_timeout | |
| 20:12:02 | eandersson | Probably another thing we missed when jumping to so many versions. | |
| 20:12:16 | eandersson | ah yea that fixes our problem | |
| 20:12:22 | eandersson | We set it to 300 due to scheduling | |
| 20:12:43 | mriedem | yup, that's using long_rpc_timeout now (select_destinations) | |
| 20:13:07 | mriedem | long_rpc_timeout heartbeats the service every $rpc_response_timeout but the overall call timeout is now long_rpc_timeout | |
| 20:13:15 | mriedem | i opened https://bugs.launchpad.net/nova/+bug/1838819 if you want to dump some stuff in there | |
| 20:13:15 | openstack | Launchpad bug 1838819 in OpenStack Compute (nova) "Docs needed for tunables at large scale" [Undecided,New] | |
| 20:13:48 | eandersson | It would probably be worth to get together during the next summit as well and talk about this (or maybe just a irc meeting) | |
| 20:14:32 | eandersson | I tend to reach out to CERN a lot since they are running similar scale | |
| 20:15:21 | mriedem | even dumping on the ML - lots of people won't be in shanghai | |
| 20:15:31 | eandersson | Yea - including myself | |
| 20:15:32 | mriedem | i'll throw that in hte ML quick | |