| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-17 | |||
| 16:11:29 | mordred | cdent: sweet | |
| 16:11:40 | bauzas | jaypipes: anyway, it's not super critical, but something is : your patch is getting -1 because http://logs.openstack.org/64/483564/2/gate/gate-nova-tox-functional-ubuntu-xenial/0f4534d/console.html#_2017-07-17_15_20_35_769898 | |
| 16:11:44 | cdent | it always sends both back in the response too | |
| 16:12:35 | jaypipes | bauzas: I can't reproduce that locally :( | |
| 16:12:57 | openstackgerrit | zhang zhenzhong proposed openstack/nova master: fix rp registration failed while placement endpoint created after compute start https://review.openstack.org/483460 | |
| 16:13:15 | bauzas | jaypipes: I was just about to run it :) | |
| 16:13:38 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%2C%20in%20test_boot_servers_with_soft_anti_affinity_one_available_host%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 16:13:54 | bauzas | my child at home is killing my work because of netflix eating all the poor internet bandwidth FWIW | |
| 16:14:05 | bauzas | plex FTW | |
| 16:14:45 | bauzas | git pull is taking more than 1 min | |
| 16:14:53 | mriedem | fwiw, that test also failed in https://review.openstack.org/#/c/482288/ | |
| 16:14:57 | mriedem | which would be totally unrelated | |
| 16:15:43 | bauzas | jaypipes: did you run the whole servergroup functional test group, or just the one failing? | |
| 16:15:51 | jaypipes | bauzas: both | |
| 16:15:53 | bauzas | k | |
| 16:16:14 | bauzas | last time I had such problem, it was because of the fixture | |
| 16:16:30 | bauzas | but I was able to reproduce it | |
| 16:19:39 | mriedem | so the test creates a soft-anti-affinity server group and kills the 2nd compute service so there should only be 1 | |
| 16:19:51 | mriedem | since the 'soft' means it will go to the same host if there is only 1 available | |
| 16:19:56 | sdague | mriedem: you have an opinion on this - https://review.openstack.org/#/c/215385/21/nova/virt/ironic/client_wrapper.py - I thought we wanted N-1 compat | |
| 16:21:09 | mriedem | sdague: that's basically how it's always worked with nova + ironic as far as i know; yes it's not what we do with anything else | |
| 16:21:28 | sdague | mriedem: ok, well we definitely need upgrade stanza in the release notes then | |
| 16:22:06 | mriedem | i raised a similar concern in another nova ironic change, | |
| 16:22:36 | mriedem | https://review.openstack.org/#/c/419975/11/releasenotes/notes/add-interface-attach-detach-support-in-ironic-cb2bf11f3875350a.yaml@4 | |
| 16:23:01 | mriedem | basically they implemented something in a driver and then started claiming support for hot plugging interfaces in nova | |
| 16:23:25 | mriedem | my argument was, what if my ironic is from ocata and i upgrade nova, and was told, that doesn't happen because everyone knows to upgrade ironic first | |
| 16:23:56 | sdague | mriedem: so... I'm extremely suspicious of "everybody knows" | |
| 16:24:10 | sdague | I'm fine with it being policy, but we still need upgrade notes | |
| 16:24:41 | mriedem | yeah they added one later here https://review.openstack.org/#/c/419975/17/releasenotes/notes/add-interface-attach-detach-support-in-ironic-cb2bf11f3875350a.yaml | |
| 16:25:16 | sdague | I'd be more happy with an N-1 policy, but we do need to communicate it every time it would trip people up | |
| 16:26:55 | mriedem | i don't disagree | |
| 16:29:56 | mriedem | jaypipes: bauzas: i have an idea on that failure | |
| 16:29:59 | mriedem | the functional test failure that is | |
| 16:30:07 | jaypipes | yeah? | |
| 16:30:09 | bauzas | ? | |
| 16:30:18 | mriedem | the test setup creates 2 compute services/nodes/hosts | |
| 16:30:25 | mriedem | you can see the 2nd host CN created here http://logs.openstack.org/64/483564/2/gate/gate-nova-tox-functional-ubuntu-xenial/0f4534d/console.html#_2017-07-17_15_20_35_768000 | |
| 16:30:31 | mriedem | with uuid a945d18d-e6b6-4fd5-8b3b-e50c973a1ff9 | |
| 16:30:35 | mriedem | and it's posted into placement | |
| 16:30:57 | mriedem | now the purpose of the test is to restrict to 1 host | |
| 16:30:58 | mriedem | so | |
| 16:30:59 | mriedem | self.compute2.kill() | |
| 16:31:02 | mriedem | HOWEVER | |
| 16:31:13 | mriedem | filter scheduler is getting it's hosties from placement now | |
| 16:31:19 | mriedem | so that RP is still floating around in placement | |
| 16:31:55 | mriedem | you can see the scheduler do it's thing here | |
| 16:31:55 | mriedem | GET /allocation_candidates?resources=DISK_GB%3A20%2CMEMORY_MB%3A2048%2CVCPU%3A1 HTTP/1.1" status: 200 len: 992 time: 0.0521650 | |
| 16:32:02 | mriedem | and then | |
| 16:32:02 | mriedem | No compute service record found for host host2 | |
| 16:32:10 | mriedem | because the test killed it | |
| 16:32:15 | mriedem | Filter RetryFilter returned 0 hosts | |
| 16:32:29 | mriedem | what i don't understand is why it doesn't try the other compute host | |
| 16:34:38 | mriedem | so one easy hack is deleting the host2 RP from test_boot_servers_with_soft_anti_affinity_one_available_host - but that's pretty hacky, it seems odd that the filtering wouldn't check the other host | |
| 16:35:34 | jaypipes | mriedem: it's also odd that this test fails randomly. | |
| 16:39:11 | mriedem | yeah so we get this https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L674 | |
| 16:39:26 | mriedem | so _get_host_states should return 1 host | |
| 16:45:20 | mriedem | my only guess is for whatever reason the retry is disabled when the test runs | |
| 16:46:10 | mriedem | nvm that doesn't make sense | |
| 16:46:37 | jaypipes | just failed that test again in the isolate get_sorted_hosts() patch :( | |
| 16:46:53 | mriedem | jaypipes: yeah that's the one bauzas was talking about | |
| 16:46:56 | mriedem | it failed in the gate | |
| 16:48:27 | mriedem | man you really need debug output when this fails | |
| 16:49:24 | mriedem | and why wouldn't we see this in the logs? https://github.com/openstack/nova/blob/master/nova/scheduler/filters/retry_filter.py#L47 | |
| 16:49:29 | jaypipes | mriedem: anything in the scheduler needs DEBUG output because of the policy of only doing DEBUG level logging :( | |
| 16:49:38 | mriedem | jaypipes: heh yeah i know | |
| 16:49:49 | jaypipes | mriedem: because it was never passed any hosts. | |
| 16:50:04 | jaypipes | mriedem: if you look at the log failures, you see REtryFilter(0 hosts) | |
| 16:50:24 | mriedem | RetryFilter: (start: 0, end: 0) ? | |
| 16:50:29 | jaypipes | yeah | |
| 16:51:45 | mriedem | ok, so the first compute node never came back from GET /allocation_candidates probably | |
| 16:53:03 | mriedem | http://logs.openstack.org/64/483564/2/gate/gate-nova-tox-functional-ubuntu-xenial/0f4534d/console.html#_2017-07-17_15_20_35_765791 | |
| 16:53:08 | mriedem | that's when the first compute host starts up | |
| 16:53:46 | mriedem | and then "Compute_service record created for compute:fake-mini" | |
| 16:54:05 | mriedem | Another thread already created a resource provider with the UUID 3694a10a-e435-4400-83aa-7d6ade1fc866. Grabbing that record from the placement API. | |
| 16:54:35 | mriedem | i don't know what 3694a10a-e435-4400-83aa-7d6ade1fc866 is | |
| 16:54:46 | mriedem | i thought the first compute RP should be e9e74898-4332-4e50-a4c6-1ad75dd295e5 | |
| 16:55:18 | mriedem | hmm wtf GET /resource_providers/3694a10a-e435-4400-83aa-7d6ade1fc866" status: 404 | |
| 16:55:29 | mriedem | Unable to refresh my resource provider record | |
| 16:56:09 | mriedem | so whatever happened, it never posted inventory for the first compute host, | |
| 16:56:18 | mriedem | which is why it didn't come back in GET /allocation_candidates | |
| 16:56:30 | mriedem | seems like another case of the wsgi eventlet weirdness | |
| 16:56:33 | mriedem | crossing streams | |
| 16:58:18 | mriedem | on a side note, it's silly that we go through all the work of calling down to get_filtered_objects when the original list we got back from GET /allocation_candidates is empty to begin with | |
| 16:59:19 | mriedem | ah yes jay's todo: "Really, we should raise NoValidHosts here" | |
| 16:59:40 | jaypipes | tada. | |
| 16:59:53 | mriedem | so i think the failure is totally unrelated | |
| 17:03:44 | mriedem | i rechecked and left a big ass summary of the failure | |
| 17:10:20 | mriedem | aha 10 hours later i think i figured out the ValueError: Cannot convert NUMATopologyLimits(cpu_allocation_ratio=16.0,ram_allocation_ratio=1.5) to primitive | |
| 17:10:26 | mriedem | and is a real bug it looks like | |
| 17:13:18 | mriedem | oooo and one mr jay introduced :) | |
| 17:13:30 | mriedem | i guess our numa 3rd party ci isn't running? | |
| 17:13:42 | jaypipes | mriedem: in, like, how long? :) | |
| 17:13:47 | mriedem | https://review.openstack.org/#/c/478291/1/nova/scheduler/manager.py | |
| 17:13:50 | mriedem | L48 | |
| 17:13:52 | mriedem | that's not a primitive | |
| 17:14:04 | mriedem | so when passing to jsonutils.to_primitive | |
| 17:14:07 | mriedem | kablammo | |
| 17:14:11 | mriedem | in the latest oslo.serialization | |
| 17:14:18 | mriedem | http://logs.openstack.org/87/482287/6/check/gate-cross-nova-func-ubuntu-xenial/a645ce8/console.html#_2017-07-16_14_02_02_649384 | |
| 17:14:22 | mriedem | but not to worry, | |