Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-10
19:30:57 mriedem dtrainor: especially since we removed that from the default enabled filters list in pike https://review.openstack.org/#/c/491854/
19:31:04 mriedem although i see you're using the ironic_host_manager,
19:31:12 mriedem which is also deprecated and you shouldn't need
19:31:36 dtrainor like i said earier, I'm using a downstream release of queens, so i think that may be adding to the confusion
19:31:42 dtrainor right
19:32:18 dtrainor feels like queens was a weird transition state to deprecate some scheduling bits in place of some more modern techniques, which may very well lead to me doing something Wrong(TM)
19:32:24 mriedem https://github.com/openstack/nova/blob/stable/queens/nova/conf/scheduler.py#L309
19:32:35 mriedem https://github.com/openstack/nova/blob/stable/queens/nova/conf/scheduler.py#L341
19:32:54 dtrainor very good. thanks for the links.
19:32:57 mriedem the default on use_baremetal_filters is False so you shouldn't be using those
19:33:48 mriedem anyway, i'd change host_manager=host_manager in nova.conf and see if that helps
19:34:17 mriedem as i said, the deprecations should have happened earlier, but we likely didn't catch this stuff until tripleo failed by using the old stuff
19:34:23 mriedem and we don't deprecate things in stable branches
19:34:50 dtrainor understood.
19:34:55 mriedem there is a mention of this in https://docs.openstack.org/releasenotes/nova/queens.html
19:34:56 openstackgerrit Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941
19:36:52 dtrainor interesting. again thanks for the links!
19:36:56 dtrainor i'll do some homework.
19:38:50 openstackgerrit Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941
19:38:50 openstackgerrit Chris Dent proposed openstack/nova master: Delete the placement code https://review.openstack.org/618215
19:43:58 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: add debug logging for bug 1807723 https://review.openstack.org/624181
19:43:58 openstack bug 1807723 in OpenStack Compute (nova) "swap multiattach volume intermittently fails when servers are on different hosts" [Medium,Confirmed] https://launchpad.net/bugs/1807723
20:04:04 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix target used in nova.policy.check_is_admin https://review.openstack.org/624185
20:13:04 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix target used in nova.policy.check_is_admin https://review.openstack.org/624185
20:15:33 mriedem gibi: there is a runway slot open https://etherpad.openstack.org/p/nova-runways-stein so i'll throw https://blueprints.launchpad.net/nova/+spec/bandwidth-resource-provider in there bug you've got two separate entries queued up so I assume you meant to keep those separate wrt runways slots
20:15:39 mriedem *but you've got
20:31:42 mriedem dansmith: melwitt: i think we can move forward with moving nova-cells-v1 to the experimental queue https://review.openstack.org/#/c/623538
20:32:14 dansmith mriedem: I figured you'd drop the +W when ready, but.. got it
20:32:32 mriedem don't want to +W my own change
20:33:47 sean-k-mooney mriedem: did you mention the intel nfv ci was broken recently
20:34:30 mriedem yes
20:34:32 mriedem well,
20:34:34 mriedem it just skips
20:34:42 mriedem so not sure why it's litsening on nova changes
20:34:45 mriedem *listening
20:34:45 dansmith mriedem: well, we left enough +2s on there, but ... as you wish
20:35:25 sean-k-mooney im currently setting up a replacement for it at home
20:36:56 sean-k-mooney im going to dedicate one of my dev servers too it. assumeing the noise stays low ill leave it that way for at least the time being. im going to see if i can use other resource i have acess too also.
20:38:20 sean-k-mooney my plan is to add the nova core team to the list of people that can leave a comment to have it run and try to have it run on all chagne to a subset of file.
20:38:42 sean-k-mooney e.g. the libvirt dirver and neutron code
20:40:38 dtrainor mriedem, setting host_manager=host_manager didn't seen to make a difference. The deployment still fails with the same errors.
20:41:45 dtrainor blows my mind. this exact sample deployment with the exact same config on other hardware works just fine.
20:41:55 dtrainor using the exact same defaults, no less
20:42:12 sean-k-mooney dtrainor: whats the error you get?
20:45:16 dtrainor initially "No valid host was found. There are not enough hosts available., Code: 500", digging deeper in, scheduler is filtering all the hosts out http://paste.openstack.org/show/736932/
20:46:03 sean-k-mooney looks like the ramfilter is filtering out the final hosts
20:46:14 dtrainor yep
20:46:17 sean-k-mooney is this an ironic deployment or something else
20:47:03 dtrainor initially it was, but per mriedem's suggestion, i tried a host_manager deployment with the same results. i don't much understand the difference between an ironic deployment and a host deployment though i suppose
20:47:13 dtrainor also i want to point out that this is a downstream queens
20:47:22 dtrainor i know that factors in to this to a degree
20:48:04 sean-k-mooney ya i was looking at mast an saw the entrypoint was gone
20:48:11 sean-k-mooney in queens you still have https://github.com/openstack/nova/blob/stable/queens/setup.cfg#L84-L87
20:48:42 sean-k-mooney dtrainor: what do you see if you look at the hyperviors api?
20:49:34 sean-k-mooney e.g. do you have ram listed for the nodes
20:50:21 dtrainor that's what i'm looking at right now, so far no. i see 0 for all disk vcpus and memory, they are all of type 'ironic'
20:50:42 sean-k-mooney dtrainor: that would be expected in rocky
20:50:53 sean-k-mooney in rocky we disable the ram filter
20:50:58 dtrainor right, that's what i was reading
20:51:05 dtrainor in lie of resource matching
20:51:27 sean-k-mooney i wonder did a backport of the reporting change get lanned into osp 13 that could be cause the issue
20:52:40 sean-k-mooney i think there is an "openstack hypervior stats" command you can run but if the hypervior is listed as 0 ram there the scuderl would appear to be doing the right thing
20:53:58 dtrainor 'openstack hypervisor stats show' lists all zeroes
20:54:32 sean-k-mooney and the current hyperviors are ironic nodes or libvirt?
20:54:48 dtrainor they all have a hypervisor_type of 'ironic'
20:55:18 sean-k-mooney ok well from rocky on that would be expected but for queens not so much
20:56:26 dtrainor gotcha
21:05:54 mriedem the RamFilter should not be running
21:05:59 mriedem so something is messed up with your configuration
21:06:08 sean-k-mooney in queens?
21:06:35 mriedem correct
21:06:40 sean-k-mooney mriedem: the cpu disk and ram filster should be disabled in rocky definetly but i tought we used them in queens
21:06:46 mriedem no
21:06:52 sean-k-mooney oh if that change happend in queen then ya
21:06:57 mriedem haven't needed them since pike
21:07:07 sean-k-mooney ah ok
21:07:30 sean-k-mooney it should be an easy fix then to just disable them
21:07:58 sean-k-mooney not sure why triplo whould have enabled them in that case however
21:09:06 sean-k-mooney oh i think i messed that we removed the caching schduler recently https://github.com/openstack/nova/blob/master/setup.cfg#L86-L88
21:09:20 sean-k-mooney did that happen before the summit?
21:10:12 sean-k-mooney ah october 18th excelent https://github.com/openstack/nova/commit/25dadb94db37e0f1c6769bf586ec06c3b5ea3051#diff-380c6a8ebbbce17d55d50ef17d3cf906
21:10:19 mriedem dtrainor: clearly your scheduler is picking up non-default filters because TripleOCapabilitiesFilter is in the list
21:10:39 mriedem dtrainor: so you need to figure out which config file is being used for nova-scheduler and where the enabled_filters option is being set
21:10:44 mriedem because you need to remove RamFilter from that list
21:11:06 mriedem dtrainor: likely a queens version of this http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/environments/undercloud.yaml#n57
21:11:25 mriedem dtrainor: see http://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/environments/undercloud.yaml?id=04b235652b44701b8703f63aee10fac6fad13ced
21:11:30 dtrainor reading :)
21:11:33 mriedem https://bugs.launchpad.net/tripleo/+bug/1787910
21:11:33 openstack Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem)
21:11:33 dtrainor thank you both
21:11:37 mriedem i'm pretty sure that's exactly your issue
21:14:02 sean-k-mooney this is what generate the config but i dont know where the parmaters are set https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/puppet/services/nova-scheduler.yaml#L90-L91
21:14:09 dtrainor yeah, that smells about right. idk how I overlooked that bug, I had been searching about this topic for a while now
21:14:51 mriedem dansmith: in the cross-cell resize spec there are a couple of places we talked about moving the policy check from conductor to api https://review.openstack.org/#/c/616037/1/specs/stein/approved/cross-cell-resize.rst@247 - thinking about that, if we move the policy check to api and it's true (cross-cell resize is ok for a given request), do you think it's legit to change the rpc call from api->conductor into a cast? i'm thinki
21:14:51 mriedem es...
21:15:14 dtrainor this is great stuff. thank you. i need to run an errand, i'll be back at this later on.
21:15:24 dtrainor thanks for the help mriedem, sean-k-mooney. helps tremendously.
21:15:28 mriedem yw
21:16:06 sean-k-mooney dtrainor: so it look like upstrea queens enaable the ram filter https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/undercloud.yaml#L23
21:16:16 mriedem yes

Earlier   Later