Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-08
20:14:30 mriedem donnyd: we could easily get a simple ci job setup for it which just runs a simple create/delete server scenario
20:14:35 mriedem and see what explodes
20:14:45 donnyd I have seen a couple lxd requests come across the ml recently and was curious to the state of it.
20:15:02 mriedem since nova-lxd isn't in tree it's not really our concern
20:15:05 mriedem our == nova
20:15:14 donnyd yea, makes sense
20:15:31 mriedem but yeah, as soon as someone kills support for something everyone comes out of the weeds
20:15:33 donnyd rather point them to something in tree anyways
20:15:40 mriedem "i was just about to go production!"
20:15:49 sean-k-mooney for what its worth i do like lxc. and it would actully be nice for developement and testing
20:16:14 donnyd I use them all the time for statefully things
20:16:36 mriedem osc is still all lxc right?
20:16:42 mriedem it's kolla that uses docker?
20:16:45 donnyd I think OSA is
20:16:49 sean-k-mooney im a bad conter person and jsut use docker for statefull things because i now how it works and can bend it to my will
20:16:50 mriedem yeah sorry, osa
20:16:58 donnyd yea, kolla, 3O
20:17:00 sean-k-mooney mriedem: kolla use docker yes
20:17:14 sean-k-mooney and osa used to support lxc for contol plane
20:17:52 donnyd I am pretty sure it still uses lxc, but i am not an osa expert so don't beat me too badly if they switched to docker
20:17:58 sean-k-mooney so osa used to install the compute nodes to bare metal but allow you to install the controler services in lxc
20:19:43 donnyd It would be great if we could get our CI to use LXC... at least for my shabby little cloud. I could probably do more
20:21:08 sean-k-mooney the first intel nfv ci was interesting... it used docker(pre 1.0) with the lxc backend to spin up super privaldges contaienr with which we then ran devstack in
20:21:53 sean-k-mooney we got really good density but it was entirly custom hacking with ducktape and sting
20:22:25 donnyd well duct tape does work... just sayin
20:23:24 sean-k-mooney it allowed use to handel every patch to neuton and respond back in less then 2 hours on a singel server with 40 cores and 128GB of ram
20:24:18 sean-k-mooney our nova ci that used vms to 3-4 servers to do the same with kvm
20:24:25 donnyd yea osa still uses lxc
20:25:34 donnyd i have 520 core and 1.1tb of memory here and I can only do 120 instances for nodepool
20:26:47 sean-k-mooney your ram limited
20:27:42 donnyd well I did have 2Tb of memory, but the other 1tb was unused
20:28:20 sean-k-mooney are you pinning the vms or disableing cpu oversuciption
20:28:33 donnyd seems like the right balance for my setup is 2:1 in cpu oversubscription
20:28:38 donnyd no
20:29:00 sean-k-mooney we used to run 4:1 for the old intel ci
20:29:12 sean-k-mooney and used hugepages for the memroy
20:29:31 donnyd I would oversubscribe more, but when i do nodepool spins up every tempest job it has in the pool and hands them all to the same hypervisor
20:30:10 donnyd literally burned up a cpu a few weeks back
20:30:40 sean-k-mooney that sound like you dont have either enough cooling or node pool is not configured correctly
20:31:10 donnyd servers run at 79 degrees
20:31:23 donnyd maybe a little on the warm side, but not too bad
20:31:54 donnyd i can't control what the CI nodepool does
20:32:25 sean-k-mooney well you can limit the quota it has on your provider
20:32:41 sean-k-mooney or you can limite the instance count on each server
20:34:17 donnyd Yes and yes - but if I take the ratio to 4:1 it will load all the tempest jobs it can find onto a single compute node.. (there is a great need for a sarcasm font)
20:34:49 sean-k-mooney you can tweek the weigher to spread by default
20:34:51 donnyd I do have the quota limited to 70 right now because I am adding in local nvmes
20:34:53 sean-k-mooney instead of pack
20:38:05 sean-k-mooney donnyd: set https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.cpu_weight_multiplier to ~8 and https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.ram_weight_multiplier to ~10
20:38:17 donnyd I don't have someone elses money to buy more compute nodes, so I try to lean more toward the conservative side
20:38:47 sean-k-mooney oh you only have a single host
20:38:55 donnyd no, I have 8
20:39:11 donnyd i just don't want to replace anymore of them
20:39:14 donnyd LOL
20:39:35 sean-k-mooney right. again it should be safe but its totally up to you.
20:39:49 donnyd And we were seeing high counts of timeouts with the ratios up higher
20:40:02 sean-k-mooney the point is when you change the allocation ratio you also want to adjust the multipliers
20:40:04 donnyd that also could have been from storage
20:40:09 donnyd oh ic
20:42:30 donnyd thanks for the help on the lxc thing sean-k-mooney
20:50:23 donnyd and i learned something new about nova today :)
20:57:17 mriedem couple of queens backports that could use a +W here https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/queens+label:Code-Review=2
21:34:11 mriedem bnemec: is there anything like this in some oslo lib? https://github.com/openstack/nova/blob/master/nova/utils.py#L530
21:35:45 bnemec mriedem: Not that I'm aware of.
21:35:59 bnemec Which is not necessarily the same thing as "no". :-)
21:36:18 mriedem yeah it's ok
21:38:51 efried with mock.patch.object(...):
21:38:51 efried mriedem: fwiw, that's effectively the same as
21:39:11 efried it would be a little weird to use mock in prod code... but it's doing the same thing.
21:39:54 mriedem yeah
21:40:11 mriedem this isn't prod code, novaclient functional tests, but not a big deal
21:52:52 sean-k-mooney do release notes need to use external link to like to the docs since they are built seperatly? i assume yes
22:01:29 efried sean-k-mooney: yes
22:01:45 efried built separately and also tied to their own release
22:02:21 efried At some point I looked into what would be involved in supporting :ref:/:doc:/:oslo.config:*:
22:02:27 efried and it was a nonstarter.
22:02:55 sean-k-mooney cool
22:03:33 efried hasta friday cats o/
22:03:52 sean-k-mooney o/
22:07:39 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117
22:08:08 openstackgerrit sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496
22:08:44 sean-k-mooney mriedem: i think i have fixed the issues you noted.
22:09:05 sean-k-mooney im off tomorow so if you see anything else ill take a look on monday
22:09:26 sean-k-mooney with that i think ill call it a night o/
22:09:49 mriedem -1!!!
22:10:02 mriedem your sleep depravation has gotten the better of you
22:11:10 sean-k-mooney haha i mess that up
22:11:30 sean-k-mooney i was working on image meta data suff before
22:13:02 openstackgerrit sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496
22:13:22 sean-k-mooney no still wrong
22:13:41 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117
22:14:44 openstackgerrit Merged openstack/nova stable/pike: Implement power_off/power_on for the FakeDriver https://review.opendev.org/674876
22:14:51 openstackgerrit sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496
22:15:29 sean-k-mooney ok fixed good catch o/
22:15:47 mriedem sean-k-mooney: ok bug me about it tomorrow
22:22:56 mnaser is the compute_nodes table exposed anywhere?
22:23:04 mnaser s/table/data/
22:23:18 sean-k-mooney its in the hyperviors api i think
22:23:25 mnaser with ironic and for some reason its complaining that there is a duplicate compute_node listed
22:25:15 mnaser Remote error: DBDuplicateEntry (pymysql.err.IntegrityError) (1062, u"Duplicate entry '77788ad5-f1a4-46ac-8132-2d88dbd4e594' for key 'compute_nodes_uuid_idx'")

Earlier   Later