| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-10 | |||
| 20:41:55 | dtrainor | using the exact same defaults, no less | |
| 20:42:12 | sean-k-mooney | dtrainor: whats the error you get? | |
| 20:45:16 | dtrainor | initially "No valid host was found. There are not enough hosts available., Code: 500", digging deeper in, scheduler is filtering all the hosts out http://paste.openstack.org/show/736932/ | |
| 20:46:03 | sean-k-mooney | looks like the ramfilter is filtering out the final hosts | |
| 20:46:14 | dtrainor | yep | |
| 20:46:17 | sean-k-mooney | is this an ironic deployment or something else | |
| 20:47:03 | dtrainor | initially it was, but per mriedem's suggestion, i tried a host_manager deployment with the same results. i don't much understand the difference between an ironic deployment and a host deployment though i suppose | |
| 20:47:13 | dtrainor | also i want to point out that this is a downstream queens | |
| 20:47:22 | dtrainor | i know that factors in to this to a degree | |
| 20:48:04 | sean-k-mooney | ya i was looking at mast an saw the entrypoint was gone | |
| 20:48:11 | sean-k-mooney | in queens you still have https://github.com/openstack/nova/blob/stable/queens/setup.cfg#L84-L87 | |
| 20:48:42 | sean-k-mooney | dtrainor: what do you see if you look at the hyperviors api? | |
| 20:49:34 | sean-k-mooney | e.g. do you have ram listed for the nodes | |
| 20:50:21 | dtrainor | that's what i'm looking at right now, so far no. i see 0 for all disk vcpus and memory, they are all of type 'ironic' | |
| 20:50:42 | sean-k-mooney | dtrainor: that would be expected in rocky | |
| 20:50:53 | sean-k-mooney | in rocky we disable the ram filter | |
| 20:50:58 | dtrainor | right, that's what i was reading | |
| 20:51:05 | dtrainor | in lie of resource matching | |
| 20:51:27 | sean-k-mooney | i wonder did a backport of the reporting change get lanned into osp 13 that could be cause the issue | |
| 20:52:40 | sean-k-mooney | i think there is an "openstack hypervior stats" command you can run but if the hypervior is listed as 0 ram there the scuderl would appear to be doing the right thing | |
| 20:53:58 | dtrainor | 'openstack hypervisor stats show' lists all zeroes | |
| 20:54:32 | sean-k-mooney | and the current hyperviors are ironic nodes or libvirt? | |
| 20:54:48 | dtrainor | they all have a hypervisor_type of 'ironic' | |
| 20:55:18 | sean-k-mooney | ok well from rocky on that would be expected but for queens not so much | |
| 20:56:26 | dtrainor | gotcha | |
| 21:05:54 | mriedem | the RamFilter should not be running | |
| 21:05:59 | mriedem | so something is messed up with your configuration | |
| 21:06:08 | sean-k-mooney | in queens? | |
| 21:06:35 | mriedem | correct | |
| 21:06:40 | sean-k-mooney | mriedem: the cpu disk and ram filster should be disabled in rocky definetly but i tought we used them in queens | |
| 21:06:46 | mriedem | no | |
| 21:06:52 | sean-k-mooney | oh if that change happend in queen then ya | |
| 21:06:57 | mriedem | haven't needed them since pike | |
| 21:07:07 | sean-k-mooney | ah ok | |
| 21:07:30 | sean-k-mooney | it should be an easy fix then to just disable them | |
| 21:07:58 | sean-k-mooney | not sure why triplo whould have enabled them in that case however | |
| 21:09:06 | sean-k-mooney | oh i think i messed that we removed the caching schduler recently https://github.com/openstack/nova/blob/master/setup.cfg#L86-L88 | |
| 21:09:20 | sean-k-mooney | did that happen before the summit? | |
| 21:10:12 | sean-k-mooney | ah october 18th excelent https://github.com/openstack/nova/commit/25dadb94db37e0f1c6769bf586ec06c3b5ea3051#diff-380c6a8ebbbce17d55d50ef17d3cf906 | |
| 21:10:19 | mriedem | dtrainor: clearly your scheduler is picking up non-default filters because TripleOCapabilitiesFilter is in the list | |
| 21:10:39 | mriedem | dtrainor: so you need to figure out which config file is being used for nova-scheduler and where the enabled_filters option is being set | |
| 21:10:44 | mriedem | because you need to remove RamFilter from that list | |
| 21:11:06 | mriedem | dtrainor: likely a queens version of this http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/environments/undercloud.yaml#n57 | |
| 21:11:25 | mriedem | dtrainor: see http://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/environments/undercloud.yaml?id=04b235652b44701b8703f63aee10fac6fad13ced | |
| 21:11:30 | dtrainor | reading :) | |
| 21:11:33 | mriedem | https://bugs.launchpad.net/tripleo/+bug/1787910 | |
| 21:11:33 | openstack | Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | |
| 21:11:33 | dtrainor | thank you both | |
| 21:11:37 | mriedem | i'm pretty sure that's exactly your issue | |
| 21:14:02 | sean-k-mooney | this is what generate the config but i dont know where the parmaters are set https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/puppet/services/nova-scheduler.yaml#L90-L91 | |
| 21:14:09 | dtrainor | yeah, that smells about right. idk how I overlooked that bug, I had been searching about this topic for a while now | |
| 21:14:51 | mriedem | dansmith: in the cross-cell resize spec there are a couple of places we talked about moving the policy check from conductor to api https://review.openstack.org/#/c/616037/1/specs/stein/approved/cross-cell-resize.rst@247 - thinking about that, if we move the policy check to api and it's true (cross-cell resize is ok for a given request), do you think it's legit to change the rpc call from api->conductor into a cast? i'm thinki | |
| 21:14:51 | mriedem | es... | |
| 21:15:14 | dtrainor | this is great stuff. thank you. i need to run an errand, i'll be back at this later on. | |
| 21:15:24 | dtrainor | thanks for the help mriedem, sean-k-mooney. helps tremendously. | |
| 21:15:28 | mriedem | yw | |
| 21:16:06 | sean-k-mooney | dtrainor: so it look like upstrea queens enaable the ram filter https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/undercloud.yaml#L23 | |
| 21:16:16 | mriedem | yes | |
| 21:16:31 | dtrainor | gotcha. | |
| 21:17:05 | sean-k-mooney | mriedem: you mentioned it should be drop after pike right so the change form rocky needs to be backported to queens | |
| 21:17:24 | mriedem | for tripleo sure | |
| 21:17:35 | sean-k-mooney | dtrainor: so you need to backport https://github.com/openstack/tripleo-heat-templates/commit/49916c09216479a8dd54e55b4c6e86dae8246fa3 | |
| 21:18:25 | sean-k-mooney | and ya the reference the bug mriedem found | |
| 21:19:06 | sean-k-mooney | so dumb question. why is that not causeing the triplo gate to explode ? | |
| 21:19:16 | sean-k-mooney | on stable queens at least | |
| 21:35:08 | mriedem | i believe tripleo found the bug after we removed the ironic driver code to report vcpu/ram/disk inventory for ironic nodes in stein | |
| 21:35:19 | mriedem | so things "worked" before that with the RamFilter | |
| 21:35:43 | mriedem | and we deprecated the core/ram/disk filters in rocky | |
| 21:36:33 | mriedem | oh i guess that was stein as well https://review.openstack.org/#/c/596502/ | |
| 21:36:41 | mriedem | should have been earlier | |
| 21:36:43 | sean-k-mooney | in that case it would seam to indicate taht OSP13 may have backported the stein change but unless that closed a bug im not sure why they would have | |
| 22:00:21 | mriedem | cfriesen: i've got something you might like to lose sleep over | |
| 22:00:36 | cfriesen | oh, goody | |
| 22:00:45 | mriedem | so RequestSpec has a limits field | |
| 22:00:55 | mriedem | which if we ever update that thing, it's on accident | |
| 22:01:09 | mriedem | e.g. if we resize from flavor1 to flavor2 and it's persisted, it's on accident, | |
| 22:01:19 | mriedem | and if we revert the resize back to flavor1, we definitely don't wipe out the limits | |
| 22:01:59 | mriedem | looking at something like unshelve, the limits passed down to compute to do the claim (which would only be used for pci/numa claims now) possibly does not reflect the actual flavor being used | |
| 22:02:01 | openstackgerrit | Jack Ding proposed openstack/nova master: [WIP] Flavor extra spec and image properties validation https://review.openstack.org/620706 | |
| 22:03:15 | mriedem | actually unshelve might be safe here, | |
| 22:03:39 | mriedem | i was thinking unshelve would use stale limits https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/conductor/manager.py#L813 | |
| 22:04:02 | mriedem | but those get overwritten in the filter_properties dict here based on the selected host from the scheduler https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/conductor/manager.py#L841 | |
| 22:04:12 | mriedem | god it's a mess | |
| 22:04:18 | mriedem | tl;dr probably shouldn't ever trust RequestSpec.limits | |
| 22:04:26 | mriedem | and we probably shouldn't even persist it | |
| 22:04:45 | cfriesen | was just going to ask why it's persisted. :) | |
| 22:04:49 | mriedem | because limits are totally based on configured filters at any given time you scheduled | |
| 22:04:52 | mriedem | dansmith: ^ | |
| 22:05:18 | mriedem | cfriesen: everything was persisted with the request spec and we've just be winding that back case by case ever since | |
| 22:05:55 | mriedem | because it's a turducken of original request / last flavor used but also as a glorified parameter bag to send requests to the scheduler | |
| 22:07:41 | mriedem | limits are just everywhere. the rpc cast to build_and_run_instance contains a (1) limits parameter, a (2) nested limits entry within filter_properties and (3) the RequestSpec.limits | |
| 22:07:44 | mriedem | it's *awesome* | |
| 22:09:10 | cfriesen | what the heck...NUMATopologyLimits contains cpu_allocation_ratio, ram_allocation_ratio, and "network_metadata"? how does that make any sense? | |
| 22:10:14 | mriedem | the network_metadata is the physnet/numa node stuff stephenfin added in rocky | |
| 22:10:22 | cfriesen | okay, so not crazy | |
| 22:12:40 | cfriesen | so it sounds like either we need to persist the limits properly and update them properly when they change, or else not persist them and create them before calling the scheduler. that sound right? | |
| 22:13:19 | mriedem | *create them *after* calling the scheduler, but yes | |
| 22:13:22 | mriedem | those are the options really | |
| 22:13:33 | mriedem | either never trust them across requests, or make sure they are always updated | |
| 22:13:40 | mriedem | i'd opt to never trust them | |