| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-02 | |||
| 18:54:17 | eandersson | Due to how this fleet is set up it is a little too likely to hit Rescheduling errors at the moment in general. | |
| 18:55:15 | eandersson | We heavily favor stacking VMs, plus Placement isn't properly configured, as we still rely on Aggregates for over-commit etc. | |
| 18:56:01 | eandersson | One of the downsides with doing a Mitaka to Rocky jump is that you miss things in releasenotes. :p | |
| 18:56:19 | mriedem | placement is used for scheduling, | |
| 18:56:49 | mriedem | so yes, the scheduler is going to ask placement for candidate hosts and the scheduler will pick a primary host after weighing and some alternates | |
| 18:57:01 | mriedem | and those alternates are what get passed between conductor and compute during reschedules | |
| 18:57:21 | mriedem | so after the initial scheduling request placement isn't really in the picture except for cleaning up allocations once the resize is confirmed/reverted | |
| 18:57:43 | mriedem | i mean, i don't know what is triggering your reschedule - that should be clear from the compute logs, | |
| 18:57:50 | eandersson | I have a good idea. | |
| 18:57:59 | mriedem | but the valueerror thing is from something with the RequestSpec which doesn't really have anything to do with placement | |
| 19:00:17 | mriedem | we have functional tests that trigger resize reschedules but they use fake rpc drivers and serialization fixtures for the context so those might be masking something, | |
| 19:00:30 | mriedem | i'd have to stand up a 2 node devstack and hack it up to recreate | |
| 19:01:07 | mriedem | if you could recreate, it would be good to dump that thing that gets serialized before it blows up | |
| 19:01:17 | eandersson | Yep - I'll see if we can catch it | |
| 19:01:22 | eandersson | Maybe even in the lab. | |
| 19:01:44 | mriedem | i'm not sure if i can get something setup to recreate in the next hour | |
| 19:02:19 | eandersson | I am also gonna re-run the tests in a env with a more optimal placement config. | |
| 19:03:17 | mriedem | are there any special image metadata properties on the server being migrated? | |
| 19:03:58 | mriedem | numa_topology, pci_requests? | |
| 19:05:32 | eandersson | numa, nothing sepcial with pci | |
| 19:06:02 | eandersson | I don't think we have anything special in image metadata, will need to double check | |
| 19:06:08 | mriedem | hmm, the numa_topology object gets serialized in that thing as well, and that's likely not something we have in-tree functional tests for (instance + numa + resize reschedule) | |
| 19:09:00 | mriedem | looking at the rocky code, on the first host attempt, conductor will serialize the request spec and pass it to compute, but compute won't pass it back to conductor on reschedule, so conductor has to build a new one https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L304 | |
| 19:09:20 | mriedem | which will again get serialized and deserialized to dict form when calling prep_resize for the alternate host | |
| 19:09:58 | mriedem | but the point being in rocky the request used for the first host isn't necessarily the same as the one for the alternate which could mean....something | |
| 19:25:17 | donnyd | is max_concurernt_builds per hypervisor? | |
| 19:28:17 | donnyd | for instance if i want to have only 2 instances per hypervisor schedule at any one point in time I would set that in nova-compute under max_concurernt_builds correct? | |
| 19:40:13 | eandersson | donnyd, yes | |
| 19:40:26 | eandersson | https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L610 | |
| 19:43:41 | donnyd | thats what i thought, just confirming | |
| 19:43:45 | donnyd | thanks eandersson | |
| 19:48:23 | mriedem | cdent: looking at osprofiler data payed off https://bugs.launchpad.net/nova/+bug/1838817 | |
| 19:48:23 | openstack | Launchpad bug 1838817 in OpenStack Compute (nova) "neutron: refreshing vif model for a server gets the same network multiple times" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem) | |
| 20:04:12 | eandersson | mriedem, we had to set heal_instance_info_cache high because it was killing our control plane | |
| 20:04:15 | eandersson | Is that why? :D | |
| 20:04:45 | mriedem | that's not the only thing, | |
| 20:05:07 | mriedem | but yeah it's expensive b/c building the instance nw info model means calling neutron separately for ports, networks, ips and subnets | |
| 20:05:17 | mriedem | since there isn't a batch api to get all of that information | |
| 20:05:34 | eandersson | It was getting real heavy on large sites with 1k nodes | |
| 20:06:10 | mriedem | yeah b/c each node is hitting that refresh every minute | |
| 20:06:20 | eandersson | We also ended up adding a variance | |
| 20:06:39 | mriedem | a doc | |
| 20:07:01 | eandersson | Yea - let me know we have a lot of custom settings | |
| 20:07:12 | mriedem | as does cern and others | |
| 20:07:21 | mriedem | this kind of stuff comes up and we don't have a place to dump it really in the docs | |
| 20:07:26 | eandersson | Yep | |
| 20:07:29 | mriedem | similar with the long_rpc_timeout option | |
| 20:07:38 | eandersson | Yea - we have it set to 300 :D | |
| 20:07:43 | eandersson | 1k for dhcp agent lol | |
| 20:08:22 | eandersson | (due to this bug https://github.com/openstack/neutron/commit/f9f6ae9c98a9ff9d3c0807b9fd51fb2ffe25fd5d) | |
| 20:09:41 | mriedem | which thing do you have set to 300? | |
| 20:09:45 | mriedem | heal instance info cache interval? | |
| 20:10:01 | eandersson | 240 + variance on heal instance infop | |
| 20:10:17 | eandersson | rcp_timeout is set to 300, but that could have been a mitaka thing | |
| 20:11:23 | mriedem | rpc_response_timeout is the old one that defaults to 60 | |
| 20:11:33 | mriedem | rpc_long_timeout is new as of i think rocky and is only used for certain long rpc calls | |
| 20:11:37 | mriedem | like during live migratoin | |
| 20:11:43 | eandersson | ah I see - didn't know about that one at all | |
| 20:11:50 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.long_rpc_timeout | |
| 20:12:02 | eandersson | Probably another thing we missed when jumping to so many versions. | |
| 20:12:16 | eandersson | ah yea that fixes our problem | |
| 20:12:22 | eandersson | We set it to 300 due to scheduling | |
| 20:12:43 | mriedem | yup, that's using long_rpc_timeout now (select_destinations) | |
| 20:13:07 | mriedem | long_rpc_timeout heartbeats the service every $rpc_response_timeout but the overall call timeout is now long_rpc_timeout | |
| 20:13:15 | mriedem | i opened https://bugs.launchpad.net/nova/+bug/1838819 if you want to dump some stuff in there | |
| 20:13:15 | openstack | Launchpad bug 1838819 in OpenStack Compute (nova) "Docs needed for tunables at large scale" [Undecided,New] | |
| 20:13:48 | eandersson | It would probably be worth to get together during the next summit as well and talk about this (or maybe just a irc meeting) | |
| 20:14:32 | eandersson | I tend to reach out to CERN a lot since they are running similar scale | |
| 20:15:21 | mriedem | even dumping on the ML - lots of people won't be in shanghai | |
| 20:15:31 | eandersson | Yea - including myself | |
| 20:15:32 | mriedem | i'll throw that in hte ML quick | |
| 20:15:47 | eandersson | Not sure why but I don't like emails | |
| 20:15:59 | eandersson | Maybe I'll just join with my private email | |
| 20:16:00 | mriedem | fire and forget... | |
| 20:16:15 | mriedem | i'm ramping down to leave town for the weekend so i'm trying to fire things off | |
| 20:17:13 | eandersson | Same | |
| 20:17:26 | eandersson | It's why I am deep diving into some of the issues. | |
| 20:32:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: neutron: avoid redundant show_network calls in _get_physnet_tunneled_info https://review.opendev.org/674422 | |
| 23:21:46 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Always set a global_request_id in RequestContext https://review.opendev.org/674138 | |
| 23:22:10 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Always set a global_request_id in RequestContext https://review.opendev.org/674138 | |
| 23:27:34 | openstackgerrit | Merged openstack/os-resource-classes master: Sync Sphinx requirement https://review.opendev.org/674164 | |
| #openstack-nova - 2019-08-03 | |||
| 01:07:12 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider config file schema and loader https://review.opendev.org/673341 | |
| 08:49:18 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Filter migrations by user_id/project_id https://review.opendev.org/674243 | |
| 08:50:43 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Add user_id and project_id colume to Migration https://review.opendev.org/673990 | |
| 09:00:56 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Add user_id and project_id colume to Migration https://review.opendev.org/673990 | |
| 09:59:28 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Filter migrations by user_id/project_id https://review.opendev.org/674243 | |
| 10:12:01 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Filter migrations by user_id/project_id https://review.opendev.org/674243 | |
| 10:17:08 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Filter migrations by user_id/project_id https://review.opendev.org/674243 | |
| 12:46:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova-multi-cell job voting and gating https://review.opendev.org/674453 | |
| 14:43:07 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Add user_id and project_id colume to Migration https://review.opendev.org/673990 | |
| #openstack-nova - 2019-08-04 | |||
| 04:06:10 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Add user_id and project_id colume to Migration https://review.opendev.org/673990 | |
| 05:56:44 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Filter migrations by user_id/project_id https://review.opendev.org/674243 | |
| 06:09:20 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Filter migrations by user_id/project_id https://review.opendev.org/674243 | |
| 14:05:25 | zhouyao | @sean-k-mooney May i ask if you could check this patch? https://review.opendev.org/#/c/621646 | |
| 20:22:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | |
| 20:22:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove unnecessary argument https://review.opendev.org/674396 | |
| 20:22:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 20:22:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 20:22:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 20:22:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add proxy fields to NUMATopology object https://review.opendev.org/674104 | |