Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-14
13:58:36 mriedem ivaat: see my comment in the bug, it's not a bug, unless you can investigate and prove otherwise
13:58:41 mriedem it looks like a misconfiguration
13:59:15 ivaat mriedem, well i attached nova.conf as well. tried user with logging in with same user. works
14:00:04 ivaat if it looks like misconfiguration then could you point out what part in config? i have gone trough config several times. followed official doc
14:00:38 mriedem see my comment on the bug
14:00:51 ivaat bug seems to be deleted
14:01:57 ivaat sry going directly with bug link i can see it now
14:02:49 ivaat mriedem, wtf. i just confirmed i have done this several times.. just take your time when helping. conf was attached to see what is wrong
14:03:36 logan- i tried hacking instances.vm_state='resized' and now it says "Instance has not been resized." so I guess there is some other state fields that need to be updated to hack it back to a revertable state
14:05:22 mriedem ivaat: i looked at the paste in the bug, i didn't see [keystone_authtoken] in there
14:07:31 mriedem logan-: so what did you do? vm was in error and you fixed the disk and tried to confirm again but that failed b/c it was in error rather than resized status?
14:07:37 ivaat at moment when i installed was no guide to add keystone_authtoken
14:07:55 mriedem ivaat: i don't know what install guide you were looking at
14:08:02 mriedem maybe something very old
14:09:12 ivaat alright, i will test it out
14:09:16 ivaat thx for now
14:10:15 aspiers[m] mriedem: does it make sense to aim for a quick hallway discussion about SEV while we're here? sean-k-mooney too if you're around
14:10:35 ivaat even nova conf is missing such keyword.. since it is section it should have been there
14:10:48 ivaat i think i got previous version conf
14:10:59 logan- mriedem: ya, resize -> error -> unable to revert resize due to error state
14:11:20 sean-k-mooney aspiers[m]: im not in berlin unfortunetly but im on irc
14:12:05 aspiers[m] sean-k-mooney: OK thanks. We can continue on the review of course, but feel free to ping me here too if you prefer
14:12:35 logan- So looking at how to revert the resize and return it back to the source node where the disk is still present in /var/lib/nova/instances/<uuid>_resize/*
14:12:36 sean-k-mooney if im needed i could dial in to a googel hangout but im sure there are enough people to sync up with to figure things out there without me
14:12:53 aspiers[m] I'm currently liasing with our SEV engineers and AMD's to get more clarity around the questions you guys raised
14:15:08 aspiers[m] guessing a hangout won't be needed but will certainly let you know if I'm wrong
14:15:36 sean-k-mooney aspiers[m]: ok just read your responces. if we can just document what will work and what wont then im more or less happy.
14:16:11 sean-k-mooney aspiers[m]: im also fine with an fix constatin + vm ram + video ram
14:16:19 aspiers[m] sean-k-mooney: absolutely - next patch set should give much more clarity over what is expected to work vs. not
14:16:25 sean-k-mooney but if we go that way i would make the constant a config value
14:16:56 aspiers[m] OK that sounds like a good idea as a short-term workaround until qemu exposes that calculation
14:17:42 aspiers[m] so you mean a fixed configurable value for the overhead per VM, right? so <hard_limit> would be calculated as flavor ram plus that value
14:17:54 sean-k-mooney the issue is qemu cant really know this up front as it depends on several things that can cahgne at runtime and from version to version
14:18:18 aspiers[m] possibly with an adjustment if guest video ram is specified via extra specs or image props
14:18:43 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941
14:18:52 sean-k-mooney aspiers[m]: ya it could be a config value for the over head or a flavor extraspec
14:19:29 sean-k-mooney just some way to reduce or increase the 512 without code change if we mesuer it is too big or two small later
14:19:56 sean-k-mooney so it would be ram + video ram + a configurable constat overhead
14:20:40 sean-k-mooney that said i think we do this somewhere already
14:23:44 sean-k-mooney aspiers[m]: we support setting the hardlimit in https://github.com/openstack/nova/blob/88951ca98e1b286b58aa1ad94f9af40b8260c01f/nova/virt/libvirt/config.py#L2102-L2133 and i belive we use it for realtime instances
14:25:00 aspiers[m] sean-k-mooney: ah yeah, I guess probably for the same reason (memory pinning)
14:25:57 sean-k-mooney aspiers[m]: yes we use it so we can memlock the memory for realtime instaces. also all hugepage backed instace will also have the hugepages memlocked which we use for ovs-dpdk so it can do dma into the guest memory
14:29:32 sean-k-mooney im trying not to be angry about the fact that apparently we only use this in unit tests.... http://codesearch.openstack.org/?q=hard_limit&i=nope&files=&repos=nova
14:30:05 aspiers[m] haha :)
14:31:13 sean-k-mooney aspiers[m]: well the code is there for you to use...
14:34:37 aspiers[m] indeed. I don't think the actual coding is gonna be too hard
15:21:53 mriedem dansmith: so belmiro gave me another issue to worry about, for which he's going to open a bug. they drop the allocation candidates limit down to 10 to make scheduling faster, but they also at times disable a large number of computes in a cell for $reasons (mnaser also does that for load balancing the cloud); the problem then is b/c of the low alloc candidates limit, placement gives back 10 disabled computes and gets novalidhost
15:22:08 mriedem to workaround it he set reserved = total for inventory in placement on the disabled nodes,
15:22:12 mriedem but that sucks for hundreds of vms
15:22:20 dansmith ouch
15:22:24 mriedem i suggested a pre-request placement filter to somehow filter out disabled nodes
15:22:40 mriedem maybe forbidden trait/aggregate, idk
15:22:48 dansmith hmm
15:22:59 dansmith they disable compute or disable a cell?
15:23:04 dansmith *computes
15:23:08 sean-k-mooney mriedem: i was going to say yes tag with a trait and use forbindin traits but then we are using placmeent to sotre metadata
15:24:34 dansmith mriedem: I think the right thing to do is to make placement results better so they don't need such a tiny result set to mitigate performance issues
15:24:42 dansmith but, regardless of what that number is, the same problem could happen
15:25:02 dansmith so if it's a disabled compute situation, I think having compute decorate its RP in some way is probably the thing to do there,
15:25:15 dansmith although it generates a lot of chatty traffic from api->compute->placement
15:25:17 sean-k-mooney stephenfin: bauzas by the way is there a nova cyborg session at the ptg? they demoed something that does not work upstream in the key note. im reviewing the spec again but it would be good to check in with them.
15:25:44 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941
15:28:05 mriedem i would really want to avoid needing to make placement api calls from the pre-placement filter for sure
15:28:30 mriedem i.e. to rope disabled computes into a resource provider aggregate and then tag a forbidden aggregate filter on the request
15:28:54 dansmith no,
15:29:01 dansmith you can't really do that I don't think
15:29:03 mriedem the compute tagging it's resource provider as disabled with a trait is one way yeah - not really the same for ironic (they set reserved = total)
15:29:14 dansmith I meant have computes decorate their own RP with a trait or reserved=total
15:29:19 openstackgerrit melanie witt proposed openstack/nova-specs master: Propose counting quota usage from placement and API database https://review.openstack.org/509042
15:29:21 mriedem sean-k-mooney: traits are going to be used as metadata in placement
15:29:28 mriedem dansmith: right
15:29:44 mriedem anyway, i told belmiro to open a bug so we don't lose track of the issue
15:30:29 mriedem we also talked about the upt stuff in compute
15:30:40 dansmith ack
15:30:42 mriedem i think he's ok with what's going on in efried's series to clean that up and sighup on changes
15:30:54 mriedem and my short history lesson on how we got here
15:31:00 mriedem my reward was his other issues
15:34:34 dansmith heh
15:39:22 sean-k-mooney mriedem: ya traits are kind of metata but im not sure how peopel would fell about a mainantnce trait or whatever we would use to make the compute node RP as down in placemetn
15:40:02 dansmith I would expect jaypipes to not want that
15:40:11 dansmith since "cant do stuff" isn't a capability
15:40:22 dansmith reserved=total is a lot more accurate
15:40:29 dansmith "these resources aren't schedulable right now"
15:40:33 sean-k-mooney yes that was my original concern with a trait
15:40:38 mriedem yeah, plus that matches what the ironic driver already does today
15:40:42 dansmith yep
15:41:19 sean-k-mooney ya that does seem alot cleaner
15:41:42 mriedem i'm not sure if that happens immediately or in a periodic? when the admin disables the compute service in the api, does the api set reserved=total for all inventory on the compute node resource provider?
15:42:09 mriedem seems it should be immediately mirrored
15:42:11 dansmith disable is an rpc call right?
15:42:16 mriedem no
15:42:21 dansmith because xen used to do stuff when you disabled it I think
15:42:47 dansmith maybe I'm mixing in the maintenance mode call
15:42:49 mriedem it's just a db change
15:42:53 mriedem yeah i think yo uare
15:43:27 dansmith okay, so I was going to say let the compute service do the disable,
15:43:33 dansmith but same difference just having api do it
15:43:44 sean-k-mooney mriedem: well renebaling it would need the virt driver to resore the old reserved value right
15:43:45 dansmith it's more better to let compute do it I think, just because it's the one that owns the resources
15:43:47 dansmith and the RP

Earlier   Later