| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-08 | |||
| 20:15:31 | mriedem | but yeah, as soon as someone kills support for something everyone comes out of the weeds | |
| 20:15:33 | donnyd | rather point them to something in tree anyways | |
| 20:15:40 | mriedem | "i was just about to go production!" | |
| 20:15:49 | sean-k-mooney | for what its worth i do like lxc. and it would actully be nice for developement and testing | |
| 20:16:14 | donnyd | I use them all the time for statefully things | |
| 20:16:36 | mriedem | osc is still all lxc right? | |
| 20:16:42 | mriedem | it's kolla that uses docker? | |
| 20:16:45 | donnyd | I think OSA is | |
| 20:16:49 | sean-k-mooney | im a bad conter person and jsut use docker for statefull things because i now how it works and can bend it to my will | |
| 20:16:50 | mriedem | yeah sorry, osa | |
| 20:16:58 | donnyd | yea, kolla, 3O | |
| 20:17:00 | sean-k-mooney | mriedem: kolla use docker yes | |
| 20:17:14 | sean-k-mooney | and osa used to support lxc for contol plane | |
| 20:17:52 | donnyd | I am pretty sure it still uses lxc, but i am not an osa expert so don't beat me too badly if they switched to docker | |
| 20:17:58 | sean-k-mooney | so osa used to install the compute nodes to bare metal but allow you to install the controler services in lxc | |
| 20:19:43 | donnyd | It would be great if we could get our CI to use LXC... at least for my shabby little cloud. I could probably do more | |
| 20:21:08 | sean-k-mooney | the first intel nfv ci was interesting... it used docker(pre 1.0) with the lxc backend to spin up super privaldges contaienr with which we then ran devstack in | |
| 20:21:53 | sean-k-mooney | we got really good density but it was entirly custom hacking with ducktape and sting | |
| 20:22:25 | donnyd | well duct tape does work... just sayin | |
| 20:23:24 | sean-k-mooney | it allowed use to handel every patch to neuton and respond back in less then 2 hours on a singel server with 40 cores and 128GB of ram | |
| 20:24:18 | sean-k-mooney | our nova ci that used vms to 3-4 servers to do the same with kvm | |
| 20:24:25 | donnyd | yea osa still uses lxc | |
| 20:25:34 | donnyd | i have 520 core and 1.1tb of memory here and I can only do 120 instances for nodepool | |
| 20:26:47 | sean-k-mooney | your ram limited | |
| 20:27:42 | donnyd | well I did have 2Tb of memory, but the other 1tb was unused | |
| 20:28:20 | sean-k-mooney | are you pinning the vms or disableing cpu oversuciption | |
| 20:28:33 | donnyd | seems like the right balance for my setup is 2:1 in cpu oversubscription | |
| 20:28:38 | donnyd | no | |
| 20:29:00 | sean-k-mooney | we used to run 4:1 for the old intel ci | |
| 20:29:12 | sean-k-mooney | and used hugepages for the memroy | |
| 20:29:31 | donnyd | I would oversubscribe more, but when i do nodepool spins up every tempest job it has in the pool and hands them all to the same hypervisor | |
| 20:30:10 | donnyd | literally burned up a cpu a few weeks back | |
| 20:30:40 | sean-k-mooney | that sound like you dont have either enough cooling or node pool is not configured correctly | |
| 20:31:10 | donnyd | servers run at 79 degrees | |
| 20:31:23 | donnyd | maybe a little on the warm side, but not too bad | |
| 20:31:54 | donnyd | i can't control what the CI nodepool does | |
| 20:32:25 | sean-k-mooney | well you can limit the quota it has on your provider | |
| 20:32:41 | sean-k-mooney | or you can limite the instance count on each server | |
| 20:34:17 | donnyd | Yes and yes - but if I take the ratio to 4:1 it will load all the tempest jobs it can find onto a single compute node.. (there is a great need for a sarcasm font) | |
| 20:34:49 | sean-k-mooney | you can tweek the weigher to spread by default | |
| 20:34:51 | donnyd | I do have the quota limited to 70 right now because I am adding in local nvmes | |
| 20:34:53 | sean-k-mooney | instead of pack | |
| 20:38:05 | sean-k-mooney | donnyd: set https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.cpu_weight_multiplier to ~8 and https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.ram_weight_multiplier to ~10 | |
| 20:38:17 | donnyd | I don't have someone elses money to buy more compute nodes, so I try to lean more toward the conservative side | |
| 20:38:47 | sean-k-mooney | oh you only have a single host | |
| 20:38:55 | donnyd | no, I have 8 | |
| 20:39:11 | donnyd | i just don't want to replace anymore of them | |
| 20:39:14 | donnyd | LOL | |
| 20:39:35 | sean-k-mooney | right. again it should be safe but its totally up to you. | |
| 20:39:49 | donnyd | And we were seeing high counts of timeouts with the ratios up higher | |
| 20:40:02 | sean-k-mooney | the point is when you change the allocation ratio you also want to adjust the multipliers | |
| 20:40:04 | donnyd | that also could have been from storage | |
| 20:40:09 | donnyd | oh ic | |
| 20:42:30 | donnyd | thanks for the help on the lxc thing sean-k-mooney | |
| 20:50:23 | donnyd | and i learned something new about nova today :) | |
| 20:57:17 | mriedem | couple of queens backports that could use a +W here https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/queens+label:Code-Review=2 | |
| 21:34:11 | mriedem | bnemec: is there anything like this in some oslo lib? https://github.com/openstack/nova/blob/master/nova/utils.py#L530 | |
| 21:35:45 | bnemec | mriedem: Not that I'm aware of. | |
| 21:35:59 | bnemec | Which is not necessarily the same thing as "no". :-) | |
| 21:36:18 | mriedem | yeah it's ok | |
| 21:38:51 | efried | mriedem: fwiw, that's effectively the same as | |
| 21:38:51 | efried | with mock.patch.object(...): | |
| 21:39:11 | efried | it would be a little weird to use mock in prod code... but it's doing the same thing. | |
| 21:39:54 | mriedem | yeah | |
| 21:40:11 | mriedem | this isn't prod code, novaclient functional tests, but not a big deal | |
| 21:52:52 | sean-k-mooney | do release notes need to use external link to like to the docs since they are built seperatly? i assume yes | |
| 22:01:29 | efried | sean-k-mooney: yes | |
| 22:01:45 | efried | built separately and also tied to their own release | |
| 22:02:21 | efried | At some point I looked into what would be involved in supporting :ref:/:doc:/:oslo.config:*: | |
| 22:02:27 | efried | and it was a nonstarter. | |
| 22:02:55 | sean-k-mooney | cool | |
| 22:03:33 | efried | hasta friday cats o/ | |
| 22:03:52 | sean-k-mooney | o/ | |
| 22:07:39 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117 | |
| 22:08:08 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:08:44 | sean-k-mooney | mriedem: i think i have fixed the issues you noted. | |
| 22:09:05 | sean-k-mooney | im off tomorow so if you see anything else ill take a look on monday | |
| 22:09:26 | sean-k-mooney | with that i think ill call it a night o/ | |
| 22:09:49 | mriedem | -1!!! | |
| 22:10:02 | mriedem | your sleep depravation has gotten the better of you | |
| 22:11:10 | sean-k-mooney | haha i mess that up | |
| 22:11:30 | sean-k-mooney | i was working on image meta data suff before | |
| 22:13:02 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:13:22 | sean-k-mooney | no still wrong | |
| 22:13:41 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117 | |
| 22:14:44 | openstackgerrit | Merged openstack/nova stable/pike: Implement power_off/power_on for the FakeDriver https://review.opendev.org/674876 | |
| 22:14:51 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:15:29 | sean-k-mooney | ok fixed good catch o/ | |
| 22:15:47 | mriedem | sean-k-mooney: ok bug me about it tomorrow | |
| 22:22:56 | mnaser | is the compute_nodes table exposed anywhere? | |
| 22:23:04 | mnaser | s/table/data/ | |
| 22:23:18 | sean-k-mooney | its in the hyperviors api i think | |
| 22:23:25 | mnaser | with ironic and for some reason its complaining that there is a duplicate compute_node listed | |
| 22:25:15 | mnaser | Remote error: DBDuplicateEntry (pymysql.err.IntegrityError) (1062, u"Duplicate entry '77788ad5-f1a4-46ac-8132-2d88dbd4e594' for key 'compute_nodes_uuid_idx'") | |
| 22:27:58 | mnaser | 2019-08-08 22:21:25.284 82770 WARNING nova.compute.resource_tracker [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] No compute node record for ctl1-xxxx:77788ad5-f1a4-46ac-8132-2d88dbd4e594: ComputeHostNotFound_Remote: Compute host ctl1-xxxx could not be found. | |
| 22:27:59 | mnaser | hmm | |
| 22:32:00 | mriedem | is one (soft) deleted? | |
| 22:41:07 | mnaser | yes | |
| 22:41:10 | mnaser | but i think this is tricking me | |
| 22:41:20 | mnaser | 2019-08-08 22:21:23.467 82770 ERROR ironicclient.common.http [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] Error contacting Ironic server: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409). Attempt 2 of 2: Conflict: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to | |