Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-10
21:11:30 dtrainor reading :)
21:11:33 mriedem https://bugs.launchpad.net/tripleo/+bug/1787910
21:11:33 openstack Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem)
21:11:33 dtrainor thank you both
21:11:37 mriedem i'm pretty sure that's exactly your issue
21:14:02 sean-k-mooney this is what generate the config but i dont know where the parmaters are set https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/puppet/services/nova-scheduler.yaml#L90-L91
21:14:09 dtrainor yeah, that smells about right. idk how I overlooked that bug, I had been searching about this topic for a while now
21:14:51 mriedem dansmith: in the cross-cell resize spec there are a couple of places we talked about moving the policy check from conductor to api https://review.openstack.org/#/c/616037/1/specs/stein/approved/cross-cell-resize.rst@247 - thinking about that, if we move the policy check to api and it's true (cross-cell resize is ok for a given request), do you think it's legit to change the rpc call from api->conductor into a cast? i'm thinki
21:14:51 mriedem es...
21:15:14 dtrainor this is great stuff. thank you. i need to run an errand, i'll be back at this later on.
21:15:24 dtrainor thanks for the help mriedem, sean-k-mooney. helps tremendously.
21:15:28 mriedem yw
21:16:06 sean-k-mooney dtrainor: so it look like upstrea queens enaable the ram filter https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/undercloud.yaml#L23
21:16:16 mriedem yes
21:16:31 dtrainor gotcha.
21:17:05 sean-k-mooney mriedem: you mentioned it should be drop after pike right so the change form rocky needs to be backported to queens
21:17:24 mriedem for tripleo sure
21:17:35 sean-k-mooney dtrainor: so you need to backport https://github.com/openstack/tripleo-heat-templates/commit/49916c09216479a8dd54e55b4c6e86dae8246fa3
21:18:25 sean-k-mooney and ya the reference the bug mriedem found
21:19:06 sean-k-mooney so dumb question. why is that not causeing the triplo gate to explode ?
21:19:16 sean-k-mooney on stable queens at least
21:35:08 mriedem i believe tripleo found the bug after we removed the ironic driver code to report vcpu/ram/disk inventory for ironic nodes in stein
21:35:19 mriedem so things "worked" before that with the RamFilter
21:35:43 mriedem and we deprecated the core/ram/disk filters in rocky
21:36:33 mriedem oh i guess that was stein as well https://review.openstack.org/#/c/596502/
21:36:41 mriedem should have been earlier
21:36:43 sean-k-mooney in that case it would seam to indicate taht OSP13 may have backported the stein change but unless that closed a bug im not sure why they would have
22:00:21 mriedem cfriesen: i've got something you might like to lose sleep over
22:00:36 cfriesen oh, goody
22:00:45 mriedem so RequestSpec has a limits field
22:00:55 mriedem which if we ever update that thing, it's on accident
22:01:09 mriedem e.g. if we resize from flavor1 to flavor2 and it's persisted, it's on accident,
22:01:19 mriedem and if we revert the resize back to flavor1, we definitely don't wipe out the limits
22:01:59 mriedem looking at something like unshelve, the limits passed down to compute to do the claim (which would only be used for pci/numa claims now) possibly does not reflect the actual flavor being used
22:02:01 openstackgerrit Jack Ding proposed openstack/nova master: [WIP] Flavor extra spec and image properties validation https://review.openstack.org/620706
22:03:15 mriedem actually unshelve might be safe here,
22:03:39 mriedem i was thinking unshelve would use stale limits https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/conductor/manager.py#L813
22:04:02 mriedem but those get overwritten in the filter_properties dict here based on the selected host from the scheduler https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/conductor/manager.py#L841
22:04:12 mriedem god it's a mess
22:04:18 mriedem tl;dr probably shouldn't ever trust RequestSpec.limits
22:04:26 mriedem and we probably shouldn't even persist it
22:04:45 cfriesen was just going to ask why it's persisted. :)
22:04:49 mriedem because limits are totally based on configured filters at any given time you scheduled
22:04:52 mriedem dansmith: ^
22:05:18 mriedem cfriesen: everything was persisted with the request spec and we've just be winding that back case by case ever since
22:05:55 mriedem because it's a turducken of original request / last flavor used but also as a glorified parameter bag to send requests to the scheduler
22:07:41 mriedem limits are just everywhere. the rpc cast to build_and_run_instance contains a (1) limits parameter, a (2) nested limits entry within filter_properties and (3) the RequestSpec.limits
22:07:44 mriedem it's *awesome*
22:09:10 cfriesen what the heck...NUMATopologyLimits contains cpu_allocation_ratio, ram_allocation_ratio, and "network_metadata"? how does that make any sense?
22:10:14 mriedem the network_metadata is the physnet/numa node stuff stephenfin added in rocky
22:10:22 cfriesen okay, so not crazy
22:12:40 cfriesen so it sounds like either we need to persist the limits properly and update them properly when they change, or else not persist them and create them before calling the scheduler. that sound right?
22:13:19 mriedem *create them *after* calling the scheduler, but yes
22:13:22 mriedem those are the options really
22:13:33 mriedem either never trust them across requests, or make sure they are always updated
22:13:40 mriedem i'd opt to never trust them
22:14:10 mriedem i.e. we can only revert the flavor in the request spec here https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/compute/api.py#L3489 because the instance stashes the old_flavor
22:14:16 mriedem we don't stash the old limits anywhere
22:15:55 mriedem anyway, i just needed to dump my brain here while i was looking at this limits code to see if i could re-use it
22:20:02 cfriesen you've probably jinxed me now, and our test guys will be firing off an email with a limit-related failure
22:20:19 mriedem misery loves company
22:21:53 mriedem this is likely also semi broken depending on how you choose to manage allocation rations now, either via config (what this would use) or via placement api https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/scheduler/filters/numa_topology_filter.py#L71
22:21:59 mriedem *ratios
22:22:33 mriedem if i'm managing VCPU/MEMORY_MB allocation ratios via the placement api, ^ will not reflect what is actually being used
22:23:17 mriedem that might have been true before placement if you were only managing allocation ratios in aggregate
22:27:37 dansmith mriedem: is there anything in reqspec that wasn't an ill-conceived steaming pile of poo?
22:28:49 mriedem umm
22:29:14 mriedem well, it is the only place we track that is_bfv thing!
22:29:30 mriedem otherwise it's caused probably more harm than it was worth
22:30:54 mriedem oh yeah and because of that numa filter limits those are per-host,
22:31:00 mriedem so RequestSpec.limits is really per operation
22:31:54 openstackgerrit Merged openstack/nova stable/rocky: Add functional regression test for bug 1806064 https://review.openstack.org/623931
22:31:54 openstack bug 1806064 in OpenStack Compute (nova) rocky "Volume remains in attaching/reserved status, if the instance is deleted after TooManyInstances exception in nova-conductor" [Medium,In progress] https://launchpad.net/bugs/1806064 - Assigned to s10 (vlad-esten)
22:31:56 mriedem alex also found out that we never actually update the numa/pci requests in the reqspec when we resize to another flavor that might have different numa/pci reqs https://review.openstack.org/#/c/621077/
22:32:02 mriedem just lots of fun
22:43:22 openstackgerrit Michael Still proposed openstack/nova master: Remove utils.execute() from quobyte libvirt storage driver. https://review.openstack.org/619702
22:43:22 openstackgerrit Michael Still proposed openstack/nova master: Move nova.libvirt.utils away from using nova.utils.execute(). https://review.openstack.org/619703
22:43:23 openstackgerrit Michael Still proposed openstack/nova master: Imagebackend should call processutils.execute directly. https://review.openstack.org/619704
22:43:23 openstackgerrit Michael Still proposed openstack/nova master: Remove final users of utils.execute() in libvirt. https://review.openstack.org/619705
22:43:24 openstackgerrit Michael Still proposed openstack/nova master: Remove the final user of utils.execute() from virt.images https://review.openstack.org/620007
22:43:24 openstackgerrit Michael Still proposed openstack/nova master: Remove utils.execute() from the hyperv driver. https://review.openstack.org/620008
22:43:25 openstackgerrit Michael Still proposed openstack/nova master: Remove utils.execute() from virt.disk.api. https://review.openstack.org/620009
22:43:25 openstackgerrit Michael Still proposed openstack/nova master: Move a generic bridge helper to a linux_net privsep file. https://review.openstack.org/620010
22:43:26 openstackgerrit Michael Still proposed openstack/nova master: Move bridge creation to privsep. https://review.openstack.org/620180
22:43:26 openstackgerrit Michael Still proposed openstack/nova master: Move some linux network helpers to use privsep. https://review.openstack.org/621398
22:43:27 openstackgerrit Michael Still proposed openstack/nova master: Move simple execute call to processutils. https://review.openstack.org/621527
22:43:27 openstackgerrit Michael Still proposed openstack/nova master: Move interface enabling to privsep. https://review.openstack.org/621528
22:43:28 openstackgerrit Michael Still proposed openstack/nova master: Move setting mac addresses for network devices to privsep. https://review.openstack.org/621529
22:43:29 openstackgerrit Michael Still proposed openstack/nova master: Move interface disabling to privsep. https://review.openstack.org/622150
22:43:29 openstackgerrit Michael Still proposed openstack/nova master: Move binding ips to privsep. https://review.openstack.org/622151
22:43:29 openstackgerrit Michael Still proposed openstack/nova master: create_veth_pair is unused, remove it. https://review.openstack.org/624226
22:43:30 openstackgerrit Michael Still proposed openstack/nova master: Create specialist set_macaddr_and_vlan helper. https://review.openstack.org/624227
22:43:30 openstackgerrit Michael Still proposed openstack/nova master: Move create_tap_dev into privsep. https://review.openstack.org/624228
22:44:26 mriedem so much for ever getting zuul to run tests on your change now
23:03:07 mriedem dansmith: i figured out how i'm getting 1 server out of the API when listing and there is a copy in both cells: https://github.com/openstack/nova/blob/08d617084e5aa69ada0898d674022621d130aef3/nova/compute/api.py#L2684
23:04:02 dansmith mriedem: oh is that the filter for the BR if the instance is already created?
23:04:09 mriedem yeah i think that's what it's for
23:04:14 mriedem laski added it
23:04:36 dansmith yeah, well, at least it makes sense
23:49:44 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930

Earlier   Later