| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-10 | |||
| 21:05:54 | mriedem | the RamFilter should not be running | |
| 21:05:59 | mriedem | so something is messed up with your configuration | |
| 21:06:08 | sean-k-mooney | in queens? | |
| 21:06:35 | mriedem | correct | |
| 21:06:40 | sean-k-mooney | mriedem: the cpu disk and ram filster should be disabled in rocky definetly but i tought we used them in queens | |
| 21:06:46 | mriedem | no | |
| 21:06:52 | sean-k-mooney | oh if that change happend in queen then ya | |
| 21:06:57 | mriedem | haven't needed them since pike | |
| 21:07:07 | sean-k-mooney | ah ok | |
| 21:07:30 | sean-k-mooney | it should be an easy fix then to just disable them | |
| 21:07:58 | sean-k-mooney | not sure why triplo whould have enabled them in that case however | |
| 21:09:06 | sean-k-mooney | oh i think i messed that we removed the caching schduler recently https://github.com/openstack/nova/blob/master/setup.cfg#L86-L88 | |
| 21:09:20 | sean-k-mooney | did that happen before the summit? | |
| 21:10:12 | sean-k-mooney | ah october 18th excelent https://github.com/openstack/nova/commit/25dadb94db37e0f1c6769bf586ec06c3b5ea3051#diff-380c6a8ebbbce17d55d50ef17d3cf906 | |
| 21:10:19 | mriedem | dtrainor: clearly your scheduler is picking up non-default filters because TripleOCapabilitiesFilter is in the list | |
| 21:10:39 | mriedem | dtrainor: so you need to figure out which config file is being used for nova-scheduler and where the enabled_filters option is being set | |
| 21:10:44 | mriedem | because you need to remove RamFilter from that list | |
| 21:11:06 | mriedem | dtrainor: likely a queens version of this http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/environments/undercloud.yaml#n57 | |
| 21:11:25 | mriedem | dtrainor: see http://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/environments/undercloud.yaml?id=04b235652b44701b8703f63aee10fac6fad13ced | |
| 21:11:30 | dtrainor | reading :) | |
| 21:11:33 | dtrainor | thank you both | |
| 21:11:33 | openstack | Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | |
| 21:11:33 | mriedem | https://bugs.launchpad.net/tripleo/+bug/1787910 | |
| 21:11:37 | mriedem | i'm pretty sure that's exactly your issue | |
| 21:14:02 | sean-k-mooney | this is what generate the config but i dont know where the parmaters are set https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/puppet/services/nova-scheduler.yaml#L90-L91 | |
| 21:14:09 | dtrainor | yeah, that smells about right. idk how I overlooked that bug, I had been searching about this topic for a while now | |
| 21:14:51 | mriedem | es... | |
| 21:14:51 | mriedem | dansmith: in the cross-cell resize spec there are a couple of places we talked about moving the policy check from conductor to api https://review.openstack.org/#/c/616037/1/specs/stein/approved/cross-cell-resize.rst@247 - thinking about that, if we move the policy check to api and it's true (cross-cell resize is ok for a given request), do you think it's legit to change the rpc call from api->conductor into a cast? i'm thinki | |
| 21:15:14 | dtrainor | this is great stuff. thank you. i need to run an errand, i'll be back at this later on. | |
| 21:15:24 | dtrainor | thanks for the help mriedem, sean-k-mooney. helps tremendously. | |
| 21:15:28 | mriedem | yw | |
| 21:16:06 | sean-k-mooney | dtrainor: so it look like upstrea queens enaable the ram filter https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/undercloud.yaml#L23 | |
| 21:16:16 | mriedem | yes | |
| 21:16:31 | dtrainor | gotcha. | |
| 21:17:05 | sean-k-mooney | mriedem: you mentioned it should be drop after pike right so the change form rocky needs to be backported to queens | |
| 21:17:24 | mriedem | for tripleo sure | |
| 21:17:35 | sean-k-mooney | dtrainor: so you need to backport https://github.com/openstack/tripleo-heat-templates/commit/49916c09216479a8dd54e55b4c6e86dae8246fa3 | |
| 21:18:25 | sean-k-mooney | and ya the reference the bug mriedem found | |
| 21:19:06 | sean-k-mooney | so dumb question. why is that not causeing the triplo gate to explode ? | |
| 21:19:16 | sean-k-mooney | on stable queens at least | |
| 21:35:08 | mriedem | i believe tripleo found the bug after we removed the ironic driver code to report vcpu/ram/disk inventory for ironic nodes in stein | |
| 21:35:19 | mriedem | so things "worked" before that with the RamFilter | |
| 21:35:43 | mriedem | and we deprecated the core/ram/disk filters in rocky | |
| 21:36:33 | mriedem | oh i guess that was stein as well https://review.openstack.org/#/c/596502/ | |
| 21:36:41 | mriedem | should have been earlier | |
| 21:36:43 | sean-k-mooney | in that case it would seam to indicate taht OSP13 may have backported the stein change but unless that closed a bug im not sure why they would have | |
| 22:00:21 | mriedem | cfriesen: i've got something you might like to lose sleep over | |
| 22:00:36 | cfriesen | oh, goody | |
| 22:00:45 | mriedem | so RequestSpec has a limits field | |
| 22:00:55 | mriedem | which if we ever update that thing, it's on accident | |
| 22:01:09 | mriedem | e.g. if we resize from flavor1 to flavor2 and it's persisted, it's on accident, | |
| 22:01:19 | mriedem | and if we revert the resize back to flavor1, we definitely don't wipe out the limits | |
| 22:01:59 | mriedem | looking at something like unshelve, the limits passed down to compute to do the claim (which would only be used for pci/numa claims now) possibly does not reflect the actual flavor being used | |
| 22:02:01 | openstackgerrit | Jack Ding proposed openstack/nova master: [WIP] Flavor extra spec and image properties validation https://review.openstack.org/620706 | |
| 22:03:15 | mriedem | actually unshelve might be safe here, | |
| 22:03:39 | mriedem | i was thinking unshelve would use stale limits https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/conductor/manager.py#L813 | |
| 22:04:02 | mriedem | but those get overwritten in the filter_properties dict here based on the selected host from the scheduler https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/conductor/manager.py#L841 | |
| 22:04:12 | mriedem | god it's a mess | |
| 22:04:18 | mriedem | tl;dr probably shouldn't ever trust RequestSpec.limits | |
| 22:04:26 | mriedem | and we probably shouldn't even persist it | |
| 22:04:45 | cfriesen | was just going to ask why it's persisted. :) | |
| 22:04:49 | mriedem | because limits are totally based on configured filters at any given time you scheduled | |
| 22:04:52 | mriedem | dansmith: ^ | |
| 22:05:18 | mriedem | cfriesen: everything was persisted with the request spec and we've just be winding that back case by case ever since | |
| 22:05:55 | mriedem | because it's a turducken of original request / last flavor used but also as a glorified parameter bag to send requests to the scheduler | |
| 22:07:41 | mriedem | limits are just everywhere. the rpc cast to build_and_run_instance contains a (1) limits parameter, a (2) nested limits entry within filter_properties and (3) the RequestSpec.limits | |
| 22:07:44 | mriedem | it's *awesome* | |
| 22:09:10 | cfriesen | what the heck...NUMATopologyLimits contains cpu_allocation_ratio, ram_allocation_ratio, and "network_metadata"? how does that make any sense? | |
| 22:10:14 | mriedem | the network_metadata is the physnet/numa node stuff stephenfin added in rocky | |
| 22:10:22 | cfriesen | okay, so not crazy | |
| 22:12:40 | cfriesen | so it sounds like either we need to persist the limits properly and update them properly when they change, or else not persist them and create them before calling the scheduler. that sound right? | |
| 22:13:19 | mriedem | *create them *after* calling the scheduler, but yes | |
| 22:13:22 | mriedem | those are the options really | |
| 22:13:33 | mriedem | either never trust them across requests, or make sure they are always updated | |
| 22:13:40 | mriedem | i'd opt to never trust them | |
| 22:14:10 | mriedem | i.e. we can only revert the flavor in the request spec here https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/compute/api.py#L3489 because the instance stashes the old_flavor | |
| 22:14:16 | mriedem | we don't stash the old limits anywhere | |
| 22:15:55 | mriedem | anyway, i just needed to dump my brain here while i was looking at this limits code to see if i could re-use it | |
| 22:20:02 | cfriesen | you've probably jinxed me now, and our test guys will be firing off an email with a limit-related failure | |
| 22:20:19 | mriedem | misery loves company | |
| 22:21:53 | mriedem | this is likely also semi broken depending on how you choose to manage allocation rations now, either via config (what this would use) or via placement api https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/scheduler/filters/numa_topology_filter.py#L71 | |
| 22:21:59 | mriedem | *ratios | |
| 22:22:33 | mriedem | if i'm managing VCPU/MEMORY_MB allocation ratios via the placement api, ^ will not reflect what is actually being used | |
| 22:23:17 | mriedem | that might have been true before placement if you were only managing allocation ratios in aggregate | |
| 22:27:37 | dansmith | mriedem: is there anything in reqspec that wasn't an ill-conceived steaming pile of poo? | |
| 22:28:49 | mriedem | umm | |
| 22:29:14 | mriedem | well, it is the only place we track that is_bfv thing! | |
| 22:29:30 | mriedem | otherwise it's caused probably more harm than it was worth | |
| 22:30:54 | mriedem | oh yeah and because of that numa filter limits those are per-host, | |
| 22:31:00 | mriedem | so RequestSpec.limits is really per operation | |
| 22:31:54 | openstack | bug 1806064 in OpenStack Compute (nova) rocky "Volume remains in attaching/reserved status, if the instance is deleted after TooManyInstances exception in nova-conductor" [Medium,In progress] https://launchpad.net/bugs/1806064 - Assigned to s10 (vlad-esten) | |
| 22:31:54 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional regression test for bug 1806064 https://review.openstack.org/623931 | |
| 22:31:56 | mriedem | alex also found out that we never actually update the numa/pci requests in the reqspec when we resize to another flavor that might have different numa/pci reqs https://review.openstack.org/#/c/621077/ | |
| 22:32:02 | mriedem | just lots of fun | |
| 22:43:22 | openstackgerrit | Michael Still proposed openstack/nova master: Move nova.libvirt.utils away from using nova.utils.execute(). https://review.openstack.org/619703 | |
| 22:43:22 | openstackgerrit | Michael Still proposed openstack/nova master: Remove utils.execute() from quobyte libvirt storage driver. https://review.openstack.org/619702 | |
| 22:43:23 | openstackgerrit | Michael Still proposed openstack/nova master: Remove final users of utils.execute() in libvirt. https://review.openstack.org/619705 | |
| 22:43:23 | openstackgerrit | Michael Still proposed openstack/nova master: Imagebackend should call processutils.execute directly. https://review.openstack.org/619704 | |
| 22:43:24 | openstackgerrit | Michael Still proposed openstack/nova master: Remove utils.execute() from the hyperv driver. https://review.openstack.org/620008 | |
| 22:43:24 | openstackgerrit | Michael Still proposed openstack/nova master: Remove the final user of utils.execute() from virt.images https://review.openstack.org/620007 | |