Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-10
20:34:42 mriedem so not sure why it's litsening on nova changes
20:34:45 dansmith mriedem: well, we left enough +2s on there, but ... as you wish
20:34:45 mriedem *listening
20:35:25 sean-k-mooney im currently setting up a replacement for it at home
20:36:56 sean-k-mooney im going to dedicate one of my dev servers too it. assumeing the noise stays low ill leave it that way for at least the time being. im going to see if i can use other resource i have acess too also.
20:38:20 sean-k-mooney my plan is to add the nova core team to the list of people that can leave a comment to have it run and try to have it run on all chagne to a subset of file.
20:38:42 sean-k-mooney e.g. the libvirt dirver and neutron code
20:40:38 dtrainor mriedem, setting host_manager=host_manager didn't seen to make a difference. The deployment still fails with the same errors.
20:41:45 dtrainor blows my mind. this exact sample deployment with the exact same config on other hardware works just fine.
20:41:55 dtrainor using the exact same defaults, no less
20:42:12 sean-k-mooney dtrainor: whats the error you get?
20:45:16 dtrainor initially "No valid host was found. There are not enough hosts available., Code: 500", digging deeper in, scheduler is filtering all the hosts out http://paste.openstack.org/show/736932/
20:46:03 sean-k-mooney looks like the ramfilter is filtering out the final hosts
20:46:14 dtrainor yep
20:46:17 sean-k-mooney is this an ironic deployment or something else
20:47:03 dtrainor initially it was, but per mriedem's suggestion, i tried a host_manager deployment with the same results. i don't much understand the difference between an ironic deployment and a host deployment though i suppose
20:47:13 dtrainor also i want to point out that this is a downstream queens
20:47:22 dtrainor i know that factors in to this to a degree
20:48:04 sean-k-mooney ya i was looking at mast an saw the entrypoint was gone
20:48:11 sean-k-mooney in queens you still have https://github.com/openstack/nova/blob/stable/queens/setup.cfg#L84-L87
20:48:42 sean-k-mooney dtrainor: what do you see if you look at the hyperviors api?
20:49:34 sean-k-mooney e.g. do you have ram listed for the nodes
20:50:21 dtrainor that's what i'm looking at right now, so far no. i see 0 for all disk vcpus and memory, they are all of type 'ironic'
20:50:42 sean-k-mooney dtrainor: that would be expected in rocky
20:50:53 sean-k-mooney in rocky we disable the ram filter
20:50:58 dtrainor right, that's what i was reading
20:51:05 dtrainor in lie of resource matching
20:51:27 sean-k-mooney i wonder did a backport of the reporting change get lanned into osp 13 that could be cause the issue
20:52:40 sean-k-mooney i think there is an "openstack hypervior stats" command you can run but if the hypervior is listed as 0 ram there the scuderl would appear to be doing the right thing
20:53:58 dtrainor 'openstack hypervisor stats show' lists all zeroes
20:54:32 sean-k-mooney and the current hyperviors are ironic nodes or libvirt?
20:54:48 dtrainor they all have a hypervisor_type of 'ironic'
20:55:18 sean-k-mooney ok well from rocky on that would be expected but for queens not so much
20:56:26 dtrainor gotcha
21:05:54 mriedem the RamFilter should not be running
21:05:59 mriedem so something is messed up with your configuration
21:06:08 sean-k-mooney in queens?
21:06:35 mriedem correct
21:06:40 sean-k-mooney mriedem: the cpu disk and ram filster should be disabled in rocky definetly but i tought we used them in queens
21:06:46 mriedem no
21:06:52 sean-k-mooney oh if that change happend in queen then ya
21:06:57 mriedem haven't needed them since pike
21:07:07 sean-k-mooney ah ok
21:07:30 sean-k-mooney it should be an easy fix then to just disable them
21:07:58 sean-k-mooney not sure why triplo whould have enabled them in that case however
21:09:06 sean-k-mooney oh i think i messed that we removed the caching schduler recently https://github.com/openstack/nova/blob/master/setup.cfg#L86-L88
21:09:20 sean-k-mooney did that happen before the summit?
21:10:12 sean-k-mooney ah october 18th excelent https://github.com/openstack/nova/commit/25dadb94db37e0f1c6769bf586ec06c3b5ea3051#diff-380c6a8ebbbce17d55d50ef17d3cf906
21:10:19 mriedem dtrainor: clearly your scheduler is picking up non-default filters because TripleOCapabilitiesFilter is in the list
21:10:39 mriedem dtrainor: so you need to figure out which config file is being used for nova-scheduler and where the enabled_filters option is being set
21:10:44 mriedem because you need to remove RamFilter from that list
21:11:06 mriedem dtrainor: likely a queens version of this http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/environments/undercloud.yaml#n57
21:11:25 mriedem dtrainor: see http://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/environments/undercloud.yaml?id=04b235652b44701b8703f63aee10fac6fad13ced
21:11:30 dtrainor reading :)
21:11:33 dtrainor thank you both
21:11:33 openstack Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem)
21:11:33 mriedem https://bugs.launchpad.net/tripleo/+bug/1787910
21:11:37 mriedem i'm pretty sure that's exactly your issue
21:14:02 sean-k-mooney this is what generate the config but i dont know where the parmaters are set https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/puppet/services/nova-scheduler.yaml#L90-L91
21:14:09 dtrainor yeah, that smells about right. idk how I overlooked that bug, I had been searching about this topic for a while now
21:14:51 mriedem es...
21:14:51 mriedem dansmith: in the cross-cell resize spec there are a couple of places we talked about moving the policy check from conductor to api https://review.openstack.org/#/c/616037/1/specs/stein/approved/cross-cell-resize.rst@247 - thinking about that, if we move the policy check to api and it's true (cross-cell resize is ok for a given request), do you think it's legit to change the rpc call from api->conductor into a cast? i'm thinki
21:15:14 dtrainor this is great stuff. thank you. i need to run an errand, i'll be back at this later on.
21:15:24 dtrainor thanks for the help mriedem, sean-k-mooney. helps tremendously.
21:15:28 mriedem yw
21:16:06 sean-k-mooney dtrainor: so it look like upstrea queens enaable the ram filter https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/undercloud.yaml#L23
21:16:16 mriedem yes
21:16:31 dtrainor gotcha.
21:17:05 sean-k-mooney mriedem: you mentioned it should be drop after pike right so the change form rocky needs to be backported to queens
21:17:24 mriedem for tripleo sure
21:17:35 sean-k-mooney dtrainor: so you need to backport https://github.com/openstack/tripleo-heat-templates/commit/49916c09216479a8dd54e55b4c6e86dae8246fa3
21:18:25 sean-k-mooney and ya the reference the bug mriedem found
21:19:06 sean-k-mooney so dumb question. why is that not causeing the triplo gate to explode ?
21:19:16 sean-k-mooney on stable queens at least
21:35:08 mriedem i believe tripleo found the bug after we removed the ironic driver code to report vcpu/ram/disk inventory for ironic nodes in stein
21:35:19 mriedem so things "worked" before that with the RamFilter
21:35:43 mriedem and we deprecated the core/ram/disk filters in rocky
21:36:33 mriedem oh i guess that was stein as well https://review.openstack.org/#/c/596502/
21:36:41 mriedem should have been earlier
21:36:43 sean-k-mooney in that case it would seam to indicate taht OSP13 may have backported the stein change but unless that closed a bug im not sure why they would have
22:00:21 mriedem cfriesen: i've got something you might like to lose sleep over
22:00:36 cfriesen oh, goody
22:00:45 mriedem so RequestSpec has a limits field
22:00:55 mriedem which if we ever update that thing, it's on accident
22:01:09 mriedem e.g. if we resize from flavor1 to flavor2 and it's persisted, it's on accident,
22:01:19 mriedem and if we revert the resize back to flavor1, we definitely don't wipe out the limits
22:01:59 mriedem looking at something like unshelve, the limits passed down to compute to do the claim (which would only be used for pci/numa claims now) possibly does not reflect the actual flavor being used
22:02:01 openstackgerrit Jack Ding proposed openstack/nova master: [WIP] Flavor extra spec and image properties validation https://review.openstack.org/620706
22:03:15 mriedem actually unshelve might be safe here,
22:03:39 mriedem i was thinking unshelve would use stale limits https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/conductor/manager.py#L813
22:04:02 mriedem but those get overwritten in the filter_properties dict here based on the selected host from the scheduler https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/conductor/manager.py#L841
22:04:12 mriedem god it's a mess
22:04:18 mriedem tl;dr probably shouldn't ever trust RequestSpec.limits
22:04:26 mriedem and we probably shouldn't even persist it
22:04:45 cfriesen was just going to ask why it's persisted. :)
22:04:49 mriedem because limits are totally based on configured filters at any given time you scheduled
22:04:52 mriedem dansmith: ^
22:05:18 mriedem cfriesen: everything was persisted with the request spec and we've just be winding that back case by case ever since
22:05:55 mriedem because it's a turducken of original request / last flavor used but also as a glorified parameter bag to send requests to the scheduler
22:07:41 mriedem limits are just everywhere. the rpc cast to build_and_run_instance contains a (1) limits parameter, a (2) nested limits entry within filter_properties and (3) the RequestSpec.limits

Earlier   Later