| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-02 | |||
| 18:01:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 18:01:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 18:01:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | |
| 18:01:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add proxy fields to NUMATopology object https://review.opendev.org/674104 | |
| 18:01:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove unnecessary argument https://review.opendev.org/674396 | |
| 18:09:58 | mriedem | efried: looks like we need a warnings filter to avoid blowing up the console logs in functional test runs https://logs.opendev.org/61/673961/1/gate/nova-tox-functional-py36/a4cb2af/job-output.txt.gz#_2019-08-01_18_05_40_322077 | |
| 18:10:15 | mriedem | b'/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/openstack/service_description.py:224: UnsupportedServiceVersion: Service placement has no discoverable version. The resulting Proxy object will only have direct passthrough REST capabilities.' | |
| 18:10:22 | mriedem | getting all sorts of warnings from openstacksdk | |
| 18:10:46 | mriedem | that or we actually have something to fix | |
| 18:13:55 | mriedem | i'll push a patch to the sdk to fix this one https://github.com/openstack/openstacksdk/blob/master/openstack/config/cloud_region.py#L173 | |
| 18:18:53 | openstack | Launchpad bug 1838807 in OpenStack Compute (nova) "UnsupportedServiceVersion: Service placement has no discoverable version. The resulting Proxy object will only have direct passthrough REST capabilities." [Medium,Confirmed] | |
| 18:18:53 | mriedem | https://bugs.launchpad.net/nova/+bug/1838807 | |
| 18:37:47 | eandersson | mriedem, you seen this before? http://paste.openstack.org/show/755342/ | |
| 18:39:31 | mriedem | hmm, i've seen something like that in the past related to versioned notification payloads but that was fixed a long time ago | |
| 18:40:23 | mriedem | https://github.com/openstack/nova/commit/3bf177a59cfd0b4e74dba256c3466ba2ea9bfbf7#diff-f009347572c3f7457d296a3b652002b5 | |
| 18:40:59 | eandersson | We for sure have that patch - this is stable/rocky (~5 commits behind) | |
| 18:40:59 | mriedem | so i wonder if there is a context in whatever you're seeing get serialized and that's blowing things up | |
| 18:41:18 | mriedem | actually hmm | |
| 18:41:19 | mriedem | legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec)) | |
| 18:41:44 | mriedem | i think there is a comment on that line about needing to do that to avoid a valuerror | |
| 18:42:02 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/migrate.py#L291 | |
| 18:42:11 | eandersson | > legacy_props.pop('context', None) | |
| 18:42:41 | eandersson | maybe we need to do the same on both? | |
| 18:42:55 | mriedem | legacy_spec should be a dict form of the RequestSpec object w/o a context in it | |
| 18:43:06 | mriedem | the dance done there is b/c of datetime fields afaict | |
| 18:43:13 | mriedem | er per the comment from bauzas | |
| 18:43:24 | mriedem | i also recently investigated this b/c i wanted to remove that... | |
| 18:44:37 | mriedem | yeah https://review.opendev.org/#/c/582417/ | |
| 18:46:00 | eandersson | Would that be possible to backport? | |
| 18:46:10 | mriedem | no it has an rpc change | |
| 18:46:17 | mriedem | but i'm not sure it would help you anyway | |
| 18:46:43 | mriedem | note that we're still serializing/deserializing the ImageMeta object https://review.opendev.org/#/c/582417/7/nova/compute/rpcapi.py@741 | |
| 18:47:38 | mriedem | i think, but don't know why, that is only in here for unit/functional tests that rely on the oslo.messaging rpc fake driver, but it doesn't make much sense to me because our functional tests pass things with datetimes over rpc all the time, | |
| 18:47:48 | mriedem | like passing RequestSpec to the scheduler's select_destinations call | |
| 18:49:01 | mriedem | do you have any out of tree changes to RequestSpec? | |
| 18:49:16 | eandersson | Not that I am aware of | |
| 18:49:29 | eandersson | The only changes we do have are customer scheduler filters and weighers | |
| 18:49:29 | mriedem | is this an isolated incident? | |
| 18:49:43 | eandersson | isolated so far yea | |
| 18:49:51 | eandersson | but we haven't done a lot of migrations | |
| 18:50:16 | eandersson | and this happened during like test run 3 | |
| 18:50:34 | mriedem | hmm, ok so this is a cold migratoin (or resize) that is failing on one host and trying to reschedule to another and then failing when trying to serialize the legacy request spec b/c rpc casting to prep_resize on the alternative host | |
| 18:50:37 | eandersson | *customer not customer | |
| 18:50:43 | mriedem | the difference is probably in the reschedule | |
| 18:50:50 | eandersson | *custom | |
| 18:50:51 | eandersson | man | |
| 18:53:07 | mriedem | i'll poke through that flow, we don't really hit migration reschedules in tempest integration runs unless it's in a multinode job | |
| 18:53:12 | mriedem | and not intentionally | |
| 18:53:30 | eandersson | Does it use placement for scheduling migrations? | |
| 18:54:17 | eandersson | Due to how this fleet is set up it is a little too likely to hit Rescheduling errors at the moment in general. | |
| 18:55:15 | eandersson | We heavily favor stacking VMs, plus Placement isn't properly configured, as we still rely on Aggregates for over-commit etc. | |
| 18:56:01 | eandersson | One of the downsides with doing a Mitaka to Rocky jump is that you miss things in releasenotes. :p | |
| 18:56:19 | mriedem | placement is used for scheduling, | |
| 18:56:49 | mriedem | so yes, the scheduler is going to ask placement for candidate hosts and the scheduler will pick a primary host after weighing and some alternates | |
| 18:57:01 | mriedem | and those alternates are what get passed between conductor and compute during reschedules | |
| 18:57:21 | mriedem | so after the initial scheduling request placement isn't really in the picture except for cleaning up allocations once the resize is confirmed/reverted | |
| 18:57:43 | mriedem | i mean, i don't know what is triggering your reschedule - that should be clear from the compute logs, | |
| 18:57:50 | eandersson | I have a good idea. | |
| 18:57:59 | mriedem | but the valueerror thing is from something with the RequestSpec which doesn't really have anything to do with placement | |
| 19:00:17 | mriedem | we have functional tests that trigger resize reschedules but they use fake rpc drivers and serialization fixtures for the context so those might be masking something, | |
| 19:00:30 | mriedem | i'd have to stand up a 2 node devstack and hack it up to recreate | |
| 19:01:07 | mriedem | if you could recreate, it would be good to dump that thing that gets serialized before it blows up | |
| 19:01:17 | eandersson | Yep - I'll see if we can catch it | |
| 19:01:22 | eandersson | Maybe even in the lab. | |
| 19:01:44 | mriedem | i'm not sure if i can get something setup to recreate in the next hour | |
| 19:02:19 | eandersson | I am also gonna re-run the tests in a env with a more optimal placement config. | |
| 19:03:17 | mriedem | are there any special image metadata properties on the server being migrated? | |
| 19:03:58 | mriedem | numa_topology, pci_requests? | |
| 19:05:32 | eandersson | numa, nothing sepcial with pci | |
| 19:06:02 | eandersson | I don't think we have anything special in image metadata, will need to double check | |
| 19:06:08 | mriedem | hmm, the numa_topology object gets serialized in that thing as well, and that's likely not something we have in-tree functional tests for (instance + numa + resize reschedule) | |
| 19:09:00 | mriedem | looking at the rocky code, on the first host attempt, conductor will serialize the request spec and pass it to compute, but compute won't pass it back to conductor on reschedule, so conductor has to build a new one https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L304 | |
| 19:09:20 | mriedem | which will again get serialized and deserialized to dict form when calling prep_resize for the alternate host | |
| 19:09:58 | mriedem | but the point being in rocky the request used for the first host isn't necessarily the same as the one for the alternate which could mean....something | |
| 19:25:17 | donnyd | is max_concurernt_builds per hypervisor? | |
| 19:28:17 | donnyd | for instance if i want to have only 2 instances per hypervisor schedule at any one point in time I would set that in nova-compute under max_concurernt_builds correct? | |
| 19:40:13 | eandersson | donnyd, yes | |
| 19:40:26 | eandersson | https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L610 | |
| 19:43:41 | donnyd | thats what i thought, just confirming | |
| 19:43:45 | donnyd | thanks eandersson | |
| 19:48:23 | openstack | Launchpad bug 1838817 in OpenStack Compute (nova) "neutron: refreshing vif model for a server gets the same network multiple times" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem) | |
| 19:48:23 | mriedem | cdent: looking at osprofiler data payed off https://bugs.launchpad.net/nova/+bug/1838817 | |
| 20:04:12 | eandersson | mriedem, we had to set heal_instance_info_cache high because it was killing our control plane | |
| 20:04:15 | eandersson | Is that why? :D | |
| 20:04:45 | mriedem | that's not the only thing, | |
| 20:05:07 | mriedem | but yeah it's expensive b/c building the instance nw info model means calling neutron separately for ports, networks, ips and subnets | |
| 20:05:17 | mriedem | since there isn't a batch api to get all of that information | |
| 20:05:34 | eandersson | It was getting real heavy on large sites with 1k nodes | |
| 20:06:10 | mriedem | yeah b/c each node is hitting that refresh every minute | |
| 20:06:20 | eandersson | We also ended up adding a variance | |
| 20:06:39 | mriedem | a doc | |
| 20:07:01 | eandersson | Yea - let me know we have a lot of custom settings | |
| 20:07:12 | mriedem | as does cern and others | |
| 20:07:21 | mriedem | this kind of stuff comes up and we don't have a place to dump it really in the docs | |
| 20:07:26 | eandersson | Yep | |
| 20:07:29 | mriedem | similar with the long_rpc_timeout option | |
| 20:07:38 | eandersson | Yea - we have it set to 300 :D | |
| 20:07:43 | eandersson | 1k for dhcp agent lol | |
| 20:08:22 | eandersson | (due to this bug https://github.com/openstack/neutron/commit/f9f6ae9c98a9ff9d3c0807b9fd51fb2ffe25fd5d) | |
| 20:09:41 | mriedem | which thing do you have set to 300? | |