Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-14
14:15:36 sean-k-mooney aspiers[m]: ok just read your responces. if we can just document what will work and what wont then im more or less happy.
14:16:11 sean-k-mooney aspiers[m]: im also fine with an fix constatin + vm ram + video ram
14:16:19 aspiers[m] sean-k-mooney: absolutely - next patch set should give much more clarity over what is expected to work vs. not
14:16:25 sean-k-mooney but if we go that way i would make the constant a config value
14:16:56 aspiers[m] OK that sounds like a good idea as a short-term workaround until qemu exposes that calculation
14:17:42 aspiers[m] so you mean a fixed configurable value for the overhead per VM, right? so <hard_limit> would be calculated as flavor ram plus that value
14:17:54 sean-k-mooney the issue is qemu cant really know this up front as it depends on several things that can cahgne at runtime and from version to version
14:18:18 aspiers[m] possibly with an adjustment if guest video ram is specified via extra specs or image props
14:18:43 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941
14:18:52 sean-k-mooney aspiers[m]: ya it could be a config value for the over head or a flavor extraspec
14:19:29 sean-k-mooney just some way to reduce or increase the 512 without code change if we mesuer it is too big or two small later
14:19:56 sean-k-mooney so it would be ram + video ram + a configurable constat overhead
14:20:40 sean-k-mooney that said i think we do this somewhere already
14:23:44 sean-k-mooney aspiers[m]: we support setting the hardlimit in https://github.com/openstack/nova/blob/88951ca98e1b286b58aa1ad94f9af40b8260c01f/nova/virt/libvirt/config.py#L2102-L2133 and i belive we use it for realtime instances
14:25:00 aspiers[m] sean-k-mooney: ah yeah, I guess probably for the same reason (memory pinning)
14:25:57 sean-k-mooney aspiers[m]: yes we use it so we can memlock the memory for realtime instaces. also all hugepage backed instace will also have the hugepages memlocked which we use for ovs-dpdk so it can do dma into the guest memory
14:29:32 sean-k-mooney im trying not to be angry about the fact that apparently we only use this in unit tests.... http://codesearch.openstack.org/?q=hard_limit&i=nope&files=&repos=nova
14:30:05 aspiers[m] haha :)
14:31:13 sean-k-mooney aspiers[m]: well the code is there for you to use...
14:34:37 aspiers[m] indeed. I don't think the actual coding is gonna be too hard
15:21:53 mriedem dansmith: so belmiro gave me another issue to worry about, for which he's going to open a bug. they drop the allocation candidates limit down to 10 to make scheduling faster, but they also at times disable a large number of computes in a cell for $reasons (mnaser also does that for load balancing the cloud); the problem then is b/c of the low alloc candidates limit, placement gives back 10 disabled computes and gets novalidhost
15:22:08 mriedem to workaround it he set reserved = total for inventory in placement on the disabled nodes,
15:22:12 mriedem but that sucks for hundreds of vms
15:22:20 dansmith ouch
15:22:24 mriedem i suggested a pre-request placement filter to somehow filter out disabled nodes
15:22:40 mriedem maybe forbidden trait/aggregate, idk
15:22:48 dansmith hmm
15:22:59 dansmith they disable compute or disable a cell?
15:23:04 dansmith *computes
15:23:08 sean-k-mooney mriedem: i was going to say yes tag with a trait and use forbindin traits but then we are using placmeent to sotre metadata
15:24:34 dansmith mriedem: I think the right thing to do is to make placement results better so they don't need such a tiny result set to mitigate performance issues
15:24:42 dansmith but, regardless of what that number is, the same problem could happen
15:25:02 dansmith so if it's a disabled compute situation, I think having compute decorate its RP in some way is probably the thing to do there,
15:25:15 dansmith although it generates a lot of chatty traffic from api->compute->placement
15:25:17 sean-k-mooney stephenfin: bauzas by the way is there a nova cyborg session at the ptg? they demoed something that does not work upstream in the key note. im reviewing the spec again but it would be good to check in with them.
15:25:44 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941
15:28:05 mriedem i would really want to avoid needing to make placement api calls from the pre-placement filter for sure
15:28:30 mriedem i.e. to rope disabled computes into a resource provider aggregate and then tag a forbidden aggregate filter on the request
15:28:54 dansmith no,
15:29:01 dansmith you can't really do that I don't think
15:29:03 mriedem the compute tagging it's resource provider as disabled with a trait is one way yeah - not really the same for ironic (they set reserved = total)
15:29:14 dansmith I meant have computes decorate their own RP with a trait or reserved=total
15:29:19 openstackgerrit melanie witt proposed openstack/nova-specs master: Propose counting quota usage from placement and API database https://review.openstack.org/509042
15:29:21 mriedem sean-k-mooney: traits are going to be used as metadata in placement
15:29:28 mriedem dansmith: right
15:29:44 mriedem anyway, i told belmiro to open a bug so we don't lose track of the issue
15:30:29 mriedem we also talked about the upt stuff in compute
15:30:40 dansmith ack
15:30:42 mriedem i think he's ok with what's going on in efried's series to clean that up and sighup on changes
15:30:54 mriedem and my short history lesson on how we got here
15:31:00 mriedem my reward was his other issues
15:34:34 dansmith heh
15:39:22 sean-k-mooney mriedem: ya traits are kind of metata but im not sure how peopel would fell about a mainantnce trait or whatever we would use to make the compute node RP as down in placemetn
15:40:02 dansmith I would expect jaypipes to not want that
15:40:11 dansmith since "cant do stuff" isn't a capability
15:40:22 dansmith reserved=total is a lot more accurate
15:40:29 dansmith "these resources aren't schedulable right now"
15:40:33 sean-k-mooney yes that was my original concern with a trait
15:40:38 mriedem yeah, plus that matches what the ironic driver already does today
15:40:42 dansmith yep
15:41:19 sean-k-mooney ya that does seem alot cleaner
15:41:42 mriedem i'm not sure if that happens immediately or in a periodic? when the admin disables the compute service in the api, does the api set reserved=total for all inventory on the compute node resource provider?
15:42:09 mriedem seems it should be immediately mirrored
15:42:11 dansmith disable is an rpc call right?
15:42:16 mriedem no
15:42:21 dansmith because xen used to do stuff when you disabled it I think
15:42:47 dansmith maybe I'm mixing in the maintenance mode call
15:42:49 mriedem it's just a db change
15:42:53 mriedem yeah i think yo uare
15:43:27 dansmith okay, so I was going to say let the compute service do the disable,
15:43:33 dansmith but same difference just having api do it
15:43:44 sean-k-mooney mriedem: well renebaling it would need the virt driver to resore the old reserved value right
15:43:45 dansmith it's more better to let compute do it I think, just because it's the one that owns the resources
15:43:47 dansmith and the RP
15:43:51 mriedem set_host_enabled was the thing you're thinking of
15:43:54 mriedem in the os-hosts api
15:44:07 jaypipes sean-k-mooney, dansmith, mriedem: I'm still reviewing Virgil's Aeneid^W^Wthe Cyborg-nova interaction spec.
15:44:09 dansmith and that doesn't get called when disable?
15:44:16 sean-k-mooney mriedem: what inventor was reserver=total set on by the way
15:44:19 mriedem dansmith: no
15:44:35 mriedem PUT /os-services/{compute_node_id} is just a db change on the services record
15:44:43 sean-k-mooney jaypipes: im reviewing that currently too
15:44:52 sean-k-mooney jaypipes: this one right https://review.openstack.org/#/c/603955/10/specs/stein/approved/nova-cyborg-interaction.rst,unified
15:44:59 dansmith mriedem: okay, well, it's rather unfortunate to have the api muck with the inventory of the compute node, but mechanically it would work
15:45:13 jaypipes sean-k-mooney: yes. that one.
15:45:19 dansmith mriedem: un-disabling is a good thought though, since you'd really not want to do that from the api I think
15:45:23 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add API ref guideline for body text https://review.openstack.org/605628
15:45:35 sean-k-mooney jaypipes: i jsut pushed what i had it take a while to get through that end to end
15:45:52 mriedem dansmith: you mean fixing the reserved value once the service is re-enabled?
15:45:58 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992
15:46:16 dansmith mriedem: yeah
15:46:19 mriedem yeah i don't think we can do that from api, compute knows that info
15:46:19 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993
15:46:30 mriedem some of it is stored in the compute node record, but maybe not all inventory records
15:46:33 mriedem only ram/vcpu/disk
15:46:33 dansmith mriedem: right.. also there's a potential race I guess,
15:46:38 mriedem yeah
15:46:51 mriedem let the compute b/c the source of truth for inventory (unless the API reserves it all...?)
15:46:58 dansmith yeah
15:46:58 dansmith mriedem: which is api racing with compute to update the inventory.. they could both use generation to catch the race, but probably better to just let one own it

Earlier   Later