| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-17 | |||
| 16:26:38 | mriedem | vif_details and profile in a neutron port are like the connection_info in a volume, | |
| 16:26:44 | mriedem | totally backend dependent and unversioned | |
| 16:31:06 | sean-k-mooney | mriedem: well not quite. the standard keys are defiend in the port_bindings api extention. addtion backend keys can be added as vendor extentions and it is unversioned | |
| 16:31:35 | efried | I just don't know what "boat" is doing in that description. | |
| 16:31:57 | sean-k-mooney | mriedem: the binding profile can be set by the end user so it can have anything in it that fits in the db filed. | |
| 16:32:12 | mriedem | efried: they are in the same "random dict of terrible" boat | |
| 16:32:17 | mriedem | it's a boat called "random dict of terrible" | |
| 16:32:26 | mriedem | and profile is a passenger alongside vif_details | |
| 16:32:32 | efried | ahhh, I see now. | |
| 16:32:36 | mriedem | heh | |
| 16:33:08 | mriedem | sean-k-mooney: gross | |
| 16:33:09 | efried | would be clearer as "...in the same random-dict-of-terrible boat..." | |
| 16:33:57 | sean-k-mooney | efried: part of creating os-vif was to sink that boat and have neutron return serialised vif object in the future. | |
| 16:34:15 | sean-k-mooney | may next release. or the one after | |
| 16:35:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove support for /os-floating-ip-dns REST API https://review.openstack.org/582943 | |
| 16:35:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove "DEPRECATED" tag from Obsolete APIs https://review.openstack.org/583092 | |
| 16:36:53 | mriedem | once that dns one is approved ^ we can probably close out that bp for rocky | |
| 16:37:06 | mriedem | stephenfin: thanks for hitting those when the call for help was raised | |
| 16:37:37 | stephenfin | mriedem: np | |
| 16:43:42 | s10 | Can anyone confirm, that this is a bug on the nova side? https://bugs.launchpad.net/nova/+bug/1781880 | |
| 16:43:43 | openstack | Launchpad bug 1781880 in OpenStack Compute (nova) "nova service-list for new compute service causes TypeError with servicegroup_driver mc " [Undecided,Confirmed] | |
| 16:46:56 | s10 | We currently have to execute "update nova.services set updated_at = NOW() where updated_at is NULL" after every new node with nova-compute addition, which looks strange to do. We didn't have such issue before Pike. | |
| 16:47:11 | mriedem | s10: your analysis on the problem looks correct | |
| 16:47:26 | mriedem | we don't have much rigorous testing with the enable_new_services config option | |
| 16:47:30 | mriedem | or the MC driver for that matter | |
| 16:48:15 | mriedem | looks like the logic should be: | |
| 16:48:15 | mriedem | if updated_time_in_mc: | |
| 16:48:17 | mriedem | oops | |
| 16:48:24 | mriedem | if updated_time_in_mc and updated_time_in_db: | |
| 16:51:01 | mriedem | we could probably easily recreate this with a test that uses the oslo.cache.dict backend | |
| 16:51:10 | mriedem | *oslo_cache.dict | |
| 16:51:27 | mriedem | set the config options appropriately, start a compute service and see it blow up | |
| 17:13:07 | sean-k-mooney | efried: mriedem stephenfin mind taking a look at https://review.openstack.org/#/c/559398/ before ye head home today | |
| 17:13:56 | efried | sean-k-mooney: Did you see my comment there? | |
| 17:14:03 | efried | I guess maybe I'm looking for stephenfin to answer it. | |
| 17:14:04 | openstackgerrit | Merged openstack/nova master: Fix a newly introduced UUID warning in the unit test https://review.openstack.org/583205 | |
| 17:14:20 | sean-k-mooney | efried: just saw it now | |
| 17:14:45 | sean-k-mooney | am yes but then agin if we dont we are not testing if the lower contraitns are vaild for the doc requiremetns | |
| 17:15:13 | efried | I get that, I get that. But... do we care? | |
| 17:15:40 | sean-k-mooney | do we care if the unit tests run. technically we could just see if everything installs | |
| 17:16:49 | sean-k-mooney | if we are going to check lower-constratins for docs then we should build the docs. if not then we should not | |
| 17:17:13 | sean-k-mooney | we shoudl proably do the same for nova and the rest but that leaves the question do we care | |
| 17:20:14 | efried | meh, okay, +2 sean-k-mooney | |
| 17:38:11 | openstackgerrit | Merged openstack/nova master: Escalate UUID validation warning to error in test https://review.openstack.org/540386 | |
| 18:00:11 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 18:32:15 | openstackgerrit | Eric Fried proposed openstack/nova master: perform reshaper operations in single transaction https://review.openstack.org/582383 | |
| 18:32:16 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 18:32:17 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 18:38:30 | openstackgerrit | sean mooney proposed openstack/os-vif master: convert os-vif docs to follow PTI https://review.openstack.org/559398 | |
| 18:47:17 | mriedem | new theory on http://status.openstack.org/elastic-recheck/#1781710 | |
| 18:47:28 | mriedem | members = set(spec_obj.instance_group.members) is empty by the time the filter runs | |
| 18:48:20 | openstackgerrit | sean mooney proposed openstack/nova master: update tox venv env to install all requirements https://review.openstack.org/583336 | |
| 18:50:14 | sean-k-mooney | mriedem: since you are still looking at that im assuming the fixed released status of the bug is incorrect? | |
| 18:50:22 | mriedem | correct | |
| 18:50:29 | mriedem | although i can't recreate it locally with a functional test | |
| 18:52:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add regression test for bug 1781710 https://review.openstack.org/583339 | |
| 18:53:00 | openstack | bug 1781710 in OpenStack Compute (nova) "ServersOnMultiNodesTest.test_create_server_with_scheduler_hint_group_anti_affinity failing with "Servers are on the same host"" [High,Fix released] https://launchpad.net/bugs/1781710 - Assigned to Matt Riedemann (mriedem) | |
| 18:53:00 | mriedem | ^ | |
| 18:56:34 | mriedem | i think this might be causing problems https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L98 | |
| 18:57:22 | mriedem | actually it's something else | |
| 18:57:29 | mriedem | reqspec.instance_group.members is 0 | |
| 18:58:29 | sean-k-mooney | on line 103 | |
| 18:58:39 | mriedem | yes | |
| 18:58:48 | mriedem | but then i'm not sure why my functional regression test would be passing | |
| 18:58:58 | mriedem | b/c i've added a custom weigher which should put each instance on it's own host | |
| 19:00:04 | sean-k-mooney | well its anti afinity so is that not the desired result? | |
| 19:00:52 | sean-k-mooney | im just reading over the regression test code. i dont dig into the functional tests often | |
| 19:01:48 | mriedem | actually the thing in the api doesn't matter, because prior to scheduling, conductor calls setup_instance_group which gets the current group information from the db which should have the current members in it | |
| 19:01:52 | mriedem | before they get to the filter | |
| 19:04:25 | sean-k-mooney | my knoladge of multi create is a little less clear then it should be. all instace in the multicreate are part of the same api request. are all instance schduled indepently or does the schduler process the batch of instance as one request. | |
| 19:05:13 | mriedem | the scheduler processes the batch as one request, | |
| 19:05:21 | mriedem | and as a host is selected per instance, | |
| 19:05:46 | mriedem | the scheduler 'consumers' resources for the in-memory representation of that host, | |
| 19:05:51 | mriedem | then filters the hosts for the next instance in the list | |
| 19:06:44 | mriedem | which is why in my change here https://review.openstack.org/#/c/582976/ i'm mapping the instance to the selected host in its HostState.instances dict | |
| 19:07:00 | mriedem | which is used in the anti-affinity filter to determine which members of the group on a given host | |
| 19:07:04 | sean-k-mooney | right so for the second time true the anti afinity filter the instances = set(host_state.instances.keys()) line retrives the updated inmemory copy of the instance state not the current state | |
| 19:07:08 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L101 | |
| 19:07:23 | mriedem | that's the idea | |
| 19:10:01 | sean-k-mooney | hum ya reading the code of that change it makes sense in that context. | |
| 19:14:22 | mriedem | i think this is the killer https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L98 | |
| 19:15:46 | mriedem | when we're doing multi-create, we have 1 request spec per instance, so in this failing test we have 2 | |
| 19:15:47 | sean-k-mooney | checking if the instance uuid is already in the current host. | |
| 19:16:01 | mriedem | but the conductor just randomly picks the first request spec from the list when passing it to the scheduler, | |
| 19:16:07 | mriedem | and that request_spec has a random instance_uuid on it | |
| 19:16:33 | mriedem | so if i've got hosts A and B and instances X and Y, and using reqspec with instance X, | |
| 19:16:44 | mriedem | then go through host filtering and select host A for instance X, | |
| 19:17:03 | mriedem | when i go through host filtering for instance Y and host A, it will say: if spec_obj.instance_uuid in host_state.instances.keys(): | |
| 19:17:05 | sean-k-mooney | oh realy should we not be looping over the requests specs in the multicreate request | |
| 19:17:15 | mriedem | if X in A.instances(): return True | |
| 19:17:35 | mriedem | the problem is when this code was added https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L98 it assumes resize, which is a single instance, | |
| 19:17:57 | mriedem | and it worked because before https://review.openstack.org/#/c/571166/ the filter didn't look at host_state.instances within a multi-create create | |
| 19:18:08 | mriedem | s/look at/depend on/ | |
| 19:19:02 | mriedem | i'm going to see if i can make this a predictable order through the scheduler to tickle the failure | |
| 19:23:46 | sean-k-mooney | ya so that looks wrong https://github.com/openstack/nova/blob/039f7e055ee2b47e96be4e86d6f48d9ce469c123/nova/conductor/manager.py#L1191-L1194 | |
| 19:24:05 | sean-k-mooney | its using the request spec from the first instance only | |
| 19:24:22 | mriedem | well, it's not wrong, it's just very brittle assumptions in the anti-affinity filter that didn't used to be a problem | |
| 19:24:38 | sean-k-mooney | fair point :) | |
| 19:27:47 | sean-k-mooney | i guess you do want to use only 1 request spec so that state is preseved across instances | |
| 19:28:22 | mriedem | well, things have just sort of been hacked in over time | |