| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-10 | |||
| 19:16:41 | TheJulia | yeah | |
| 19:18:53 | mriedem | dtrainor: paste your nova.conf for the scheduler | |
| 19:19:40 | mriedem | dtrainor: in addition, are the cpu/ram/disk values on the flavor overridden to report 0? | |
| 19:19:54 | mriedem | 1 | |
| 19:20:01 | mriedem | see, "Another set of flavor properties must be used to disable scheduling based on standard properties for a bare metal flavor:" from https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html | |
| 19:24:41 | dtrainor | my scheduler section is pretty boring by default with only one change http://paste.openstack.org/show/736924/ reading... | |
| 19:25:15 | dtrainor | yes, the values are set to 0, as this is a default https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/extraconfig/post_deploy/undercloud_post.sh#L124 | |
| 19:27:05 | cdent | mriedem or yikun_ : https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L213 has trouble when the compute node is 'deleted' and the context hasn't diddled read_deleted | |
| 19:27:36 | mriedem | where are we reading deleting compute_nodes? | |
| 19:27:40 | mriedem | *deleted | |
| 19:28:02 | dtrainor | so by default, a node's resource class is not uppercase, which sounds like it's required per https://docs.openstack.org/ironic/queens/install/configure-nova-flavors.html . I just changed the resource class of a node from 'baremetal' to 'BAREMETAL', I'll try a deployment with that. | |
| 19:28:03 | cdent | this is showing up in the status test that I'm moving for placement fixture: line 265 here https://review.openstack.org/#/c/617941/28/nova/tests/functional/test_nova_status.py | |
| 19:28:34 | cdent | cn2.create() is of a node that is deleted (so that it can be "not seen" during the test) | |
| 19:28:42 | dtrainor | I think the thing that bothers me the most is that this deployment succeeds on other sets of hardware, i'm starting to suspect a bug on the platform on which this is deployed but i can't prove it | |
| 19:28:46 | cdent | I can diddle the context, but wanted to be sure that was the right things to do | |
| 19:28:48 | mriedem | cdent: yeah you didn't copy part of that change from yikun | |
| 19:29:19 | mriedem | cdent: https://review.openstack.org/#/c/613499/16/nova/tests/unit/cmd/test_status.py | |
| 19:29:27 | cdent | thanks | |
| 19:29:51 | mriedem | dtrainor: hmm i'm not sure how you have the RamFilter enabled then | |
| 19:30:11 | cdent | latency is the everything killer | |
| 19:30:57 | mriedem | dtrainor: especially since we removed that from the default enabled filters list in pike https://review.openstack.org/#/c/491854/ | |
| 19:31:04 | mriedem | although i see you're using the ironic_host_manager, | |
| 19:31:12 | mriedem | which is also deprecated and you shouldn't need | |
| 19:31:36 | dtrainor | like i said earier, I'm using a downstream release of queens, so i think that may be adding to the confusion | |
| 19:31:42 | dtrainor | right | |
| 19:32:18 | dtrainor | feels like queens was a weird transition state to deprecate some scheduling bits in place of some more modern techniques, which may very well lead to me doing something Wrong(TM) | |
| 19:32:24 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/conf/scheduler.py#L309 | |
| 19:32:35 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/conf/scheduler.py#L341 | |
| 19:32:54 | dtrainor | very good. thanks for the links. | |
| 19:32:57 | mriedem | the default on use_baremetal_filters is False so you shouldn't be using those | |
| 19:33:48 | mriedem | anyway, i'd change host_manager=host_manager in nova.conf and see if that helps | |
| 19:34:17 | mriedem | as i said, the deprecations should have happened earlier, but we likely didn't catch this stuff until tripleo failed by using the old stuff | |
| 19:34:23 | mriedem | and we don't deprecate things in stable branches | |
| 19:34:50 | dtrainor | understood. | |
| 19:34:55 | mriedem | there is a mention of this in https://docs.openstack.org/releasenotes/nova/queens.html | |
| 19:34:56 | openstackgerrit | Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941 | |
| 19:36:52 | dtrainor | interesting. again thanks for the links! | |
| 19:36:56 | dtrainor | i'll do some homework. | |
| 19:38:50 | openstackgerrit | Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941 | |
| 19:38:50 | openstackgerrit | Chris Dent proposed openstack/nova master: Delete the placement code https://review.openstack.org/618215 | |
| 19:43:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: add debug logging for bug 1807723 https://review.openstack.org/624181 | |
| 19:43:58 | openstack | bug 1807723 in OpenStack Compute (nova) "swap multiattach volume intermittently fails when servers are on different hosts" [Medium,Confirmed] https://launchpad.net/bugs/1807723 | |
| 20:04:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix target used in nova.policy.check_is_admin https://review.openstack.org/624185 | |
| 20:13:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix target used in nova.policy.check_is_admin https://review.openstack.org/624185 | |
| 20:15:33 | mriedem | gibi: there is a runway slot open https://etherpad.openstack.org/p/nova-runways-stein so i'll throw https://blueprints.launchpad.net/nova/+spec/bandwidth-resource-provider in there bug you've got two separate entries queued up so I assume you meant to keep those separate wrt runways slots | |
| 20:15:39 | mriedem | *but you've got | |
| 20:31:42 | mriedem | dansmith: melwitt: i think we can move forward with moving nova-cells-v1 to the experimental queue https://review.openstack.org/#/c/623538 | |
| 20:32:14 | dansmith | mriedem: I figured you'd drop the +W when ready, but.. got it | |
| 20:32:32 | mriedem | don't want to +W my own change | |
| 20:33:47 | sean-k-mooney | mriedem: did you mention the intel nfv ci was broken recently | |
| 20:34:30 | mriedem | yes | |
| 20:34:32 | mriedem | well, | |
| 20:34:34 | mriedem | it just skips | |
| 20:34:42 | mriedem | so not sure why it's litsening on nova changes | |
| 20:34:45 | mriedem | *listening | |
| 20:34:45 | dansmith | mriedem: well, we left enough +2s on there, but ... as you wish | |
| 20:35:25 | sean-k-mooney | im currently setting up a replacement for it at home | |
| 20:36:56 | sean-k-mooney | im going to dedicate one of my dev servers too it. assumeing the noise stays low ill leave it that way for at least the time being. im going to see if i can use other resource i have acess too also. | |
| 20:38:20 | sean-k-mooney | my plan is to add the nova core team to the list of people that can leave a comment to have it run and try to have it run on all chagne to a subset of file. | |
| 20:38:42 | sean-k-mooney | e.g. the libvirt dirver and neutron code | |
| 20:40:38 | dtrainor | mriedem, setting host_manager=host_manager didn't seen to make a difference. The deployment still fails with the same errors. | |
| 20:41:45 | dtrainor | blows my mind. this exact sample deployment with the exact same config on other hardware works just fine. | |
| 20:41:55 | dtrainor | using the exact same defaults, no less | |
| 20:42:12 | sean-k-mooney | dtrainor: whats the error you get? | |
| 20:45:16 | dtrainor | initially "No valid host was found. There are not enough hosts available., Code: 500", digging deeper in, scheduler is filtering all the hosts out http://paste.openstack.org/show/736932/ | |
| 20:46:03 | sean-k-mooney | looks like the ramfilter is filtering out the final hosts | |
| 20:46:14 | dtrainor | yep | |
| 20:46:17 | sean-k-mooney | is this an ironic deployment or something else | |
| 20:47:03 | dtrainor | initially it was, but per mriedem's suggestion, i tried a host_manager deployment with the same results. i don't much understand the difference between an ironic deployment and a host deployment though i suppose | |
| 20:47:13 | dtrainor | also i want to point out that this is a downstream queens | |
| 20:47:22 | dtrainor | i know that factors in to this to a degree | |
| 20:48:04 | sean-k-mooney | ya i was looking at mast an saw the entrypoint was gone | |
| 20:48:11 | sean-k-mooney | in queens you still have https://github.com/openstack/nova/blob/stable/queens/setup.cfg#L84-L87 | |
| 20:48:42 | sean-k-mooney | dtrainor: what do you see if you look at the hyperviors api? | |
| 20:49:34 | sean-k-mooney | e.g. do you have ram listed for the nodes | |
| 20:50:21 | dtrainor | that's what i'm looking at right now, so far no. i see 0 for all disk vcpus and memory, they are all of type 'ironic' | |
| 20:50:42 | sean-k-mooney | dtrainor: that would be expected in rocky | |
| 20:50:53 | sean-k-mooney | in rocky we disable the ram filter | |
| 20:50:58 | dtrainor | right, that's what i was reading | |
| 20:51:05 | dtrainor | in lie of resource matching | |
| 20:51:27 | sean-k-mooney | i wonder did a backport of the reporting change get lanned into osp 13 that could be cause the issue | |
| 20:52:40 | sean-k-mooney | i think there is an "openstack hypervior stats" command you can run but if the hypervior is listed as 0 ram there the scuderl would appear to be doing the right thing | |
| 20:53:58 | dtrainor | 'openstack hypervisor stats show' lists all zeroes | |
| 20:54:32 | sean-k-mooney | and the current hyperviors are ironic nodes or libvirt? | |
| 20:54:48 | dtrainor | they all have a hypervisor_type of 'ironic' | |
| 20:55:18 | sean-k-mooney | ok well from rocky on that would be expected but for queens not so much | |
| 20:56:26 | dtrainor | gotcha | |
| 21:05:54 | mriedem | the RamFilter should not be running | |
| 21:05:59 | mriedem | so something is messed up with your configuration | |
| 21:06:08 | sean-k-mooney | in queens? | |
| 21:06:35 | mriedem | correct | |
| 21:06:40 | sean-k-mooney | mriedem: the cpu disk and ram filster should be disabled in rocky definetly but i tought we used them in queens | |
| 21:06:46 | mriedem | no | |
| 21:06:52 | sean-k-mooney | oh if that change happend in queen then ya | |
| 21:06:57 | mriedem | haven't needed them since pike | |
| 21:07:07 | sean-k-mooney | ah ok | |
| 21:07:30 | sean-k-mooney | it should be an easy fix then to just disable them | |
| 21:07:58 | sean-k-mooney | not sure why triplo whould have enabled them in that case however | |
| 21:09:06 | sean-k-mooney | oh i think i messed that we removed the caching schduler recently https://github.com/openstack/nova/blob/master/setup.cfg#L86-L88 | |
| 21:09:20 | sean-k-mooney | did that happen before the summit? | |