| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-08 | |||
| 20:26:47 | sean-k-mooney | your ram limited | |
| 20:27:42 | donnyd | well I did have 2Tb of memory, but the other 1tb was unused | |
| 20:28:20 | sean-k-mooney | are you pinning the vms or disableing cpu oversuciption | |
| 20:28:33 | donnyd | seems like the right balance for my setup is 2:1 in cpu oversubscription | |
| 20:28:38 | donnyd | no | |
| 20:29:00 | sean-k-mooney | we used to run 4:1 for the old intel ci | |
| 20:29:12 | sean-k-mooney | and used hugepages for the memroy | |
| 20:29:31 | donnyd | I would oversubscribe more, but when i do nodepool spins up every tempest job it has in the pool and hands them all to the same hypervisor | |
| 20:30:10 | donnyd | literally burned up a cpu a few weeks back | |
| 20:30:40 | sean-k-mooney | that sound like you dont have either enough cooling or node pool is not configured correctly | |
| 20:31:10 | donnyd | servers run at 79 degrees | |
| 20:31:23 | donnyd | maybe a little on the warm side, but not too bad | |
| 20:31:54 | donnyd | i can't control what the CI nodepool does | |
| 20:32:25 | sean-k-mooney | well you can limit the quota it has on your provider | |
| 20:32:41 | sean-k-mooney | or you can limite the instance count on each server | |
| 20:34:17 | donnyd | Yes and yes - but if I take the ratio to 4:1 it will load all the tempest jobs it can find onto a single compute node.. (there is a great need for a sarcasm font) | |
| 20:34:49 | sean-k-mooney | you can tweek the weigher to spread by default | |
| 20:34:51 | donnyd | I do have the quota limited to 70 right now because I am adding in local nvmes | |
| 20:34:53 | sean-k-mooney | instead of pack | |
| 20:38:05 | sean-k-mooney | donnyd: set https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.cpu_weight_multiplier to ~8 and https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.ram_weight_multiplier to ~10 | |
| 20:38:17 | donnyd | I don't have someone elses money to buy more compute nodes, so I try to lean more toward the conservative side | |
| 20:38:47 | sean-k-mooney | oh you only have a single host | |
| 20:38:55 | donnyd | no, I have 8 | |
| 20:39:11 | donnyd | i just don't want to replace anymore of them | |
| 20:39:14 | donnyd | LOL | |
| 20:39:35 | sean-k-mooney | right. again it should be safe but its totally up to you. | |
| 20:39:49 | donnyd | And we were seeing high counts of timeouts with the ratios up higher | |
| 20:40:02 | sean-k-mooney | the point is when you change the allocation ratio you also want to adjust the multipliers | |
| 20:40:04 | donnyd | that also could have been from storage | |
| 20:40:09 | donnyd | oh ic | |
| 20:42:30 | donnyd | thanks for the help on the lxc thing sean-k-mooney | |
| 20:50:23 | donnyd | and i learned something new about nova today :) | |
| 20:57:17 | mriedem | couple of queens backports that could use a +W here https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/queens+label:Code-Review=2 | |
| 21:34:11 | mriedem | bnemec: is there anything like this in some oslo lib? https://github.com/openstack/nova/blob/master/nova/utils.py#L530 | |
| 21:35:45 | bnemec | mriedem: Not that I'm aware of. | |
| 21:35:59 | bnemec | Which is not necessarily the same thing as "no". :-) | |
| 21:36:18 | mriedem | yeah it's ok | |
| 21:38:51 | efried | mriedem: fwiw, that's effectively the same as | |
| 21:38:51 | efried | with mock.patch.object(...): | |
| 21:39:11 | efried | it would be a little weird to use mock in prod code... but it's doing the same thing. | |
| 21:39:54 | mriedem | yeah | |
| 21:40:11 | mriedem | this isn't prod code, novaclient functional tests, but not a big deal | |
| 21:52:52 | sean-k-mooney | do release notes need to use external link to like to the docs since they are built seperatly? i assume yes | |
| 22:01:29 | efried | sean-k-mooney: yes | |
| 22:01:45 | efried | built separately and also tied to their own release | |
| 22:02:21 | efried | At some point I looked into what would be involved in supporting :ref:/:doc:/:oslo.config:*: | |
| 22:02:27 | efried | and it was a nonstarter. | |
| 22:02:55 | sean-k-mooney | cool | |
| 22:03:33 | efried | hasta friday cats o/ | |
| 22:03:52 | sean-k-mooney | o/ | |
| 22:07:39 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117 | |
| 22:08:08 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:08:44 | sean-k-mooney | mriedem: i think i have fixed the issues you noted. | |
| 22:09:05 | sean-k-mooney | im off tomorow so if you see anything else ill take a look on monday | |
| 22:09:26 | sean-k-mooney | with that i think ill call it a night o/ | |
| 22:09:49 | mriedem | -1!!! | |
| 22:10:02 | mriedem | your sleep depravation has gotten the better of you | |
| 22:11:10 | sean-k-mooney | haha i mess that up | |
| 22:11:30 | sean-k-mooney | i was working on image meta data suff before | |
| 22:13:02 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:13:22 | sean-k-mooney | no still wrong | |
| 22:13:41 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117 | |
| 22:14:44 | openstackgerrit | Merged openstack/nova stable/pike: Implement power_off/power_on for the FakeDriver https://review.opendev.org/674876 | |
| 22:14:51 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:15:29 | sean-k-mooney | ok fixed good catch o/ | |
| 22:15:47 | mriedem | sean-k-mooney: ok bug me about it tomorrow | |
| 22:22:56 | mnaser | is the compute_nodes table exposed anywhere? | |
| 22:23:04 | mnaser | s/table/data/ | |
| 22:23:18 | sean-k-mooney | its in the hyperviors api i think | |
| 22:23:25 | mnaser | with ironic and for some reason its complaining that there is a duplicate compute_node listed | |
| 22:25:15 | mnaser | Remote error: DBDuplicateEntry (pymysql.err.IntegrityError) (1062, u"Duplicate entry '77788ad5-f1a4-46ac-8132-2d88dbd4e594' for key 'compute_nodes_uuid_idx'") | |
| 22:27:58 | mnaser | 2019-08-08 22:21:25.284 82770 WARNING nova.compute.resource_tracker [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] No compute node record for ctl1-xxxx:77788ad5-f1a4-46ac-8132-2d88dbd4e594: ComputeHostNotFound_Remote: Compute host ctl1-xxxx could not be found. | |
| 22:27:59 | mnaser | hmm | |
| 22:32:00 | mriedem | is one (soft) deleted? | |
| 22:41:07 | mnaser | yes | |
| 22:41:10 | mnaser | but i think this is tricking me | |
| 22:41:20 | mnaser | 2019-08-08 22:21:23.467 82770 ERROR ironicclient.common.http [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] Error contacting Ironic server: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409). Attempt 2 of 2: Conflict: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to | |
| 22:41:20 | mnaser | Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409) | |
| 22:41:39 | mnaser | i see that happen right before it, so i wonder if those warnings/errors make nova-compute blow up and bubble up the wrong exception | |
| 22:43:39 | mnaser | there is a soft deleted one though, i wonder if thats the issue | |
| 22:44:12 | mriedem | yeah the unique index is on uuid not including the deleted column | |
| 22:44:22 | mriedem | unlike uniq_compute_nodes0host0hypervisor_hostname0deleted | |
| 22:44:44 | mriedem | i can't remember what release it was, but semi-recently we started using the ironic node uuid as the compute node uuid for easier correlation of those records | |
| 22:44:57 | mnaser | well deleting it fixing it but that sounds like a bug then i guess | |
| 22:44:59 | mnaser | yeah that might explain it | |
| 22:45:08 | mriedem | but, since the compute node uuid is not unique now when it gets created, it'll be a problem if the ironic node / compute node is soft-deleted and not archived... | |
| 22:45:33 | mnaser | mriedem: https://github.com/openstack/nova/commit/9f28727eb75e05e07bad51b6eecce667d09dfb65 | |
| 22:45:53 | mriedem | which...does that mean a recreate is as simple as creating a server from a specific bm node, deleting the server, and then creating another server from the same node? | |
| 22:46:05 | mriedem | yes that's how i knew, i wrote it :) | |
| 22:46:50 | mriedem | went into rocky so it must not be that common of an issue | |
| 22:48:09 | mnaser | i think this might have been after moving a node to maintenance | |
| 22:48:10 | mnaser | and then out of it | |
| 22:48:11 | mnaser | i think? | |
| 22:48:33 | mnaser | does that result in the compute node being deleted? ironic is still new to me | |
| 22:48:35 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for regression bug 1825537 https://review.opendev.org/675355 | |
| 22:48:35 | openstack | bug 1825537 in OpenStack Compute (nova) queens "finish_resize failures incorrectly revert allocations" [Medium,Confirmed] https://launchpad.net/bugs/1825537 | |
| 22:48:51 | mriedem | do you see this for that node? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8306 | |
| 22:50:59 | mnaser | mriedem: 2019-08-08 17:20:13.921 6379 INFO nova.compute.manager [req-c71e5c81-eb34-4f72-a260-6aa7e802f490 - - - - -] Deleting orphan compute node 31 hypervisor host is 77788ad5-f1a4-46ac-8132-2d88dbd4e594, nodes are set([u'6d556617-2bdc-42b3-a3fe-b9218a1ebf0e', u'a634fab2-ecea-4cfa-be09-032dce6eaf51', u'2dee290d-ef73-46bc-8fc2-af248841ca12']) | |
| 22:51:07 | mriedem | heh, bingo | |
| 22:51:11 | mnaser | yep, and i think this is when we set it to maintenance mode i think | |