Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-08
20:02:14 sean-k-mooney well run validation is different
20:02:23 mriedem run_validation == enable ssh
20:02:23 sean-k-mooney in that its optional
20:02:39 mriedem sure, but it's basically on by default in tempest runs in the gate
20:02:41 sean-k-mooney e.g the test shoudl still be runale without it
20:02:47 sean-k-mooney ya
20:04:28 mriedem we should probably disable ssh validation in the nova-next job
20:04:50 mriedem we already get that in tempest-full and tempest-slow
20:05:16 sean-k-mooney well its realy jsut intened to prevent race conditions
20:05:46 sean-k-mooney e.g. in senario tests it can help with cases were you need to wait for the guest os to start before you proceed with the test
20:05:55 mriedem i don't think we're talking about the same thing
20:06:17 sean-k-mooney that is what the validation was ortinglly for as far as i knew
20:06:18 mriedem are you thinking of the nova vif_plugging_* options?
20:06:26 sean-k-mooney no
20:06:54 mriedem tempest uses it for all kinds of shit https://github.com/openstack/tempest/search?q=CONF.run_validation&unscoped_q=CONF.run_validation
20:08:25 sean-k-mooney yes but if i rememebr correctly it was originally added to make test pause on server create until you could ssh in
20:09:15 sean-k-mooney this was need for running tempst against ironic clouds
20:09:20 donnyd is the lxc driver for nova still functional?
20:09:31 sean-k-mooney am kind of
20:09:40 sean-k-mooney it was broke for 2 years
20:09:46 sean-k-mooney and got fixed recnetly
20:10:00 sean-k-mooney as in like a month ago
20:10:14 sean-k-mooney but we have not ci so maybe?
20:10:44 donnyd I see someone asking about lxd on the ml and was going to ask if we could point them at lxc instead... but figured i should ask the state of it first
20:11:40 sean-k-mooney cannoical are stoping developing nova-lxd and i dont think tere was much of a comunity outside of them maintaining it
20:12:06 sean-k-mooney the nova libvirt/lxc backend has little use or testing as far as i know
20:13:02 mriedem i'd be surprised if the lxc driver didn't shit the bed under normal usage
20:13:12 donnyd LOL mriedem
20:13:34 sean-k-mooney donnyd: the reason it was broken for ~2years was we change the args to mount
20:13:36 mriedem honestly, i tried getting a ci job going for that thing years ago and something would fail during server delete and it would just snowball and fail everything
20:13:39 donnyd seriously, that made me laugh harder than I should have
20:13:43 mriedem nbd mount issues from what i remember
20:14:30 mriedem donnyd: we could easily get a simple ci job setup for it which just runs a simple create/delete server scenario
20:14:35 mriedem and see what explodes
20:14:45 donnyd I have seen a couple lxd requests come across the ml recently and was curious to the state of it.
20:15:02 mriedem since nova-lxd isn't in tree it's not really our concern
20:15:05 mriedem our == nova
20:15:14 donnyd yea, makes sense
20:15:31 mriedem but yeah, as soon as someone kills support for something everyone comes out of the weeds
20:15:33 donnyd rather point them to something in tree anyways
20:15:40 mriedem "i was just about to go production!"
20:15:49 sean-k-mooney for what its worth i do like lxc. and it would actully be nice for developement and testing
20:16:14 donnyd I use them all the time for statefully things
20:16:36 mriedem osc is still all lxc right?
20:16:42 mriedem it's kolla that uses docker?
20:16:45 donnyd I think OSA is
20:16:49 sean-k-mooney im a bad conter person and jsut use docker for statefull things because i now how it works and can bend it to my will
20:16:50 mriedem yeah sorry, osa
20:16:58 donnyd yea, kolla, 3O
20:17:00 sean-k-mooney mriedem: kolla use docker yes
20:17:14 sean-k-mooney and osa used to support lxc for contol plane
20:17:52 donnyd I am pretty sure it still uses lxc, but i am not an osa expert so don't beat me too badly if they switched to docker
20:17:58 sean-k-mooney so osa used to install the compute nodes to bare metal but allow you to install the controler services in lxc
20:19:43 donnyd It would be great if we could get our CI to use LXC... at least for my shabby little cloud. I could probably do more
20:21:08 sean-k-mooney the first intel nfv ci was interesting... it used docker(pre 1.0) with the lxc backend to spin up super privaldges contaienr with which we then ran devstack in
20:21:53 sean-k-mooney we got really good density but it was entirly custom hacking with ducktape and sting
20:22:25 donnyd well duct tape does work... just sayin
20:23:24 sean-k-mooney it allowed use to handel every patch to neuton and respond back in less then 2 hours on a singel server with 40 cores and 128GB of ram
20:24:18 sean-k-mooney our nova ci that used vms to 3-4 servers to do the same with kvm
20:24:25 donnyd yea osa still uses lxc
20:25:34 donnyd i have 520 core and 1.1tb of memory here and I can only do 120 instances for nodepool
20:26:47 sean-k-mooney your ram limited
20:27:42 donnyd well I did have 2Tb of memory, but the other 1tb was unused
20:28:20 sean-k-mooney are you pinning the vms or disableing cpu oversuciption
20:28:33 donnyd seems like the right balance for my setup is 2:1 in cpu oversubscription
20:28:38 donnyd no
20:29:00 sean-k-mooney we used to run 4:1 for the old intel ci
20:29:12 sean-k-mooney and used hugepages for the memroy
20:29:31 donnyd I would oversubscribe more, but when i do nodepool spins up every tempest job it has in the pool and hands them all to the same hypervisor
20:30:10 donnyd literally burned up a cpu a few weeks back
20:30:40 sean-k-mooney that sound like you dont have either enough cooling or node pool is not configured correctly
20:31:10 donnyd servers run at 79 degrees
20:31:23 donnyd maybe a little on the warm side, but not too bad
20:31:54 donnyd i can't control what the CI nodepool does
20:32:25 sean-k-mooney well you can limit the quota it has on your provider
20:32:41 sean-k-mooney or you can limite the instance count on each server
20:34:17 donnyd Yes and yes - but if I take the ratio to 4:1 it will load all the tempest jobs it can find onto a single compute node.. (there is a great need for a sarcasm font)
20:34:49 sean-k-mooney you can tweek the weigher to spread by default
20:34:51 donnyd I do have the quota limited to 70 right now because I am adding in local nvmes
20:34:53 sean-k-mooney instead of pack
20:38:05 sean-k-mooney donnyd: set https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.cpu_weight_multiplier to ~8 and https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.ram_weight_multiplier to ~10
20:38:17 donnyd I don't have someone elses money to buy more compute nodes, so I try to lean more toward the conservative side
20:38:47 sean-k-mooney oh you only have a single host
20:38:55 donnyd no, I have 8
20:39:11 donnyd i just don't want to replace anymore of them
20:39:14 donnyd LOL
20:39:35 sean-k-mooney right. again it should be safe but its totally up to you.
20:39:49 donnyd And we were seeing high counts of timeouts with the ratios up higher
20:40:02 sean-k-mooney the point is when you change the allocation ratio you also want to adjust the multipliers
20:40:04 donnyd that also could have been from storage
20:40:09 donnyd oh ic
20:42:30 donnyd thanks for the help on the lxc thing sean-k-mooney
20:50:23 donnyd and i learned something new about nova today :)
20:57:17 mriedem couple of queens backports that could use a +W here https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/queens+label:Code-Review=2
21:34:11 mriedem bnemec: is there anything like this in some oslo lib? https://github.com/openstack/nova/blob/master/nova/utils.py#L530
21:35:45 bnemec mriedem: Not that I'm aware of.
21:35:59 bnemec Which is not necessarily the same thing as "no". :-)
21:36:18 mriedem yeah it's ok
21:38:51 efried mriedem: fwiw, that's effectively the same as
21:38:51 efried with mock.patch.object(...):

Earlier   Later