| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-08 | |||
| 21:34:11 | mriedem | bnemec: is there anything like this in some oslo lib? https://github.com/openstack/nova/blob/master/nova/utils.py#L530 | |
| 21:35:45 | bnemec | mriedem: Not that I'm aware of. | |
| 21:35:59 | bnemec | Which is not necessarily the same thing as "no". :-) | |
| 21:36:18 | mriedem | yeah it's ok | |
| 21:38:51 | efried | with mock.patch.object(...): | |
| 21:38:51 | efried | mriedem: fwiw, that's effectively the same as | |
| 21:39:11 | efried | it would be a little weird to use mock in prod code... but it's doing the same thing. | |
| 21:39:54 | mriedem | yeah | |
| 21:40:11 | mriedem | this isn't prod code, novaclient functional tests, but not a big deal | |
| 21:52:52 | sean-k-mooney | do release notes need to use external link to like to the docs since they are built seperatly? i assume yes | |
| 22:01:29 | efried | sean-k-mooney: yes | |
| 22:01:45 | efried | built separately and also tied to their own release | |
| 22:02:21 | efried | At some point I looked into what would be involved in supporting :ref:/:doc:/:oslo.config:*: | |
| 22:02:27 | efried | and it was a nonstarter. | |
| 22:02:55 | sean-k-mooney | cool | |
| 22:03:33 | efried | hasta friday cats o/ | |
| 22:03:52 | sean-k-mooney | o/ | |
| 22:07:39 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117 | |
| 22:08:08 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:08:44 | sean-k-mooney | mriedem: i think i have fixed the issues you noted. | |
| 22:09:05 | sean-k-mooney | im off tomorow so if you see anything else ill take a look on monday | |
| 22:09:26 | sean-k-mooney | with that i think ill call it a night o/ | |
| 22:09:49 | mriedem | -1!!! | |
| 22:10:02 | mriedem | your sleep depravation has gotten the better of you | |
| 22:11:10 | sean-k-mooney | haha i mess that up | |
| 22:11:30 | sean-k-mooney | i was working on image meta data suff before | |
| 22:13:02 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:13:22 | sean-k-mooney | no still wrong | |
| 22:13:41 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117 | |
| 22:14:44 | openstackgerrit | Merged openstack/nova stable/pike: Implement power_off/power_on for the FakeDriver https://review.opendev.org/674876 | |
| 22:14:51 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:15:29 | sean-k-mooney | ok fixed good catch o/ | |
| 22:15:47 | mriedem | sean-k-mooney: ok bug me about it tomorrow | |
| 22:22:56 | mnaser | is the compute_nodes table exposed anywhere? | |
| 22:23:04 | mnaser | s/table/data/ | |
| 22:23:18 | sean-k-mooney | its in the hyperviors api i think | |
| 22:23:25 | mnaser | with ironic and for some reason its complaining that there is a duplicate compute_node listed | |
| 22:25:15 | mnaser | Remote error: DBDuplicateEntry (pymysql.err.IntegrityError) (1062, u"Duplicate entry '77788ad5-f1a4-46ac-8132-2d88dbd4e594' for key 'compute_nodes_uuid_idx'") | |
| 22:27:58 | mnaser | 2019-08-08 22:21:25.284 82770 WARNING nova.compute.resource_tracker [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] No compute node record for ctl1-xxxx:77788ad5-f1a4-46ac-8132-2d88dbd4e594: ComputeHostNotFound_Remote: Compute host ctl1-xxxx could not be found. | |
| 22:27:59 | mnaser | hmm | |
| 22:32:00 | mriedem | is one (soft) deleted? | |
| 22:41:07 | mnaser | yes | |
| 22:41:10 | mnaser | but i think this is tricking me | |
| 22:41:20 | mnaser | Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409) | |
| 22:41:20 | mnaser | 2019-08-08 22:21:23.467 82770 ERROR ironicclient.common.http [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] Error contacting Ironic server: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409). Attempt 2 of 2: Conflict: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to | |
| 22:41:39 | mnaser | i see that happen right before it, so i wonder if those warnings/errors make nova-compute blow up and bubble up the wrong exception | |
| 22:43:39 | mnaser | there is a soft deleted one though, i wonder if thats the issue | |
| 22:44:12 | mriedem | yeah the unique index is on uuid not including the deleted column | |
| 22:44:22 | mriedem | unlike uniq_compute_nodes0host0hypervisor_hostname0deleted | |
| 22:44:44 | mriedem | i can't remember what release it was, but semi-recently we started using the ironic node uuid as the compute node uuid for easier correlation of those records | |
| 22:44:57 | mnaser | well deleting it fixing it but that sounds like a bug then i guess | |
| 22:44:59 | mnaser | yeah that might explain it | |
| 22:45:08 | mriedem | but, since the compute node uuid is not unique now when it gets created, it'll be a problem if the ironic node / compute node is soft-deleted and not archived... | |
| 22:45:33 | mnaser | mriedem: https://github.com/openstack/nova/commit/9f28727eb75e05e07bad51b6eecce667d09dfb65 | |
| 22:45:53 | mriedem | which...does that mean a recreate is as simple as creating a server from a specific bm node, deleting the server, and then creating another server from the same node? | |
| 22:46:05 | mriedem | yes that's how i knew, i wrote it :) | |
| 22:46:50 | mriedem | went into rocky so it must not be that common of an issue | |
| 22:48:09 | mnaser | i think this might have been after moving a node to maintenance | |
| 22:48:10 | mnaser | and then out of it | |
| 22:48:11 | mnaser | i think? | |
| 22:48:33 | mnaser | does that result in the compute node being deleted? ironic is still new to me | |
| 22:48:35 | openstack | bug 1825537 in OpenStack Compute (nova) queens "finish_resize failures incorrectly revert allocations" [Medium,Confirmed] https://launchpad.net/bugs/1825537 | |
| 22:48:35 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for regression bug 1825537 https://review.opendev.org/675355 | |
| 22:48:51 | mriedem | do you see this for that node? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8306 | |
| 22:50:59 | mnaser | mriedem: 2019-08-08 17:20:13.921 6379 INFO nova.compute.manager [req-c71e5c81-eb34-4f72-a260-6aa7e802f490 - - - - -] Deleting orphan compute node 31 hypervisor host is 77788ad5-f1a4-46ac-8132-2d88dbd4e594, nodes are set([u'6d556617-2bdc-42b3-a3fe-b9218a1ebf0e', u'a634fab2-ecea-4cfa-be09-032dce6eaf51', u'2dee290d-ef73-46bc-8fc2-af248841ca12']) | |
| 22:51:07 | mriedem | heh, bingo | |
| 22:51:11 | mnaser | yep, and i think this is when we set it to maintenance mode i think | |
| 22:51:21 | mriedem | which we likely don't test in ironic+nova ci | |
| 22:51:35 | mnaser | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8298 | |
| 22:52:04 | mriedem | right... | |
| 22:52:07 | mriedem | well, report a bug first, | |
| 22:52:13 | mriedem | we could revert https://review.opendev.org/#/c/571535/ which would be unfortunate, | |
| 22:52:25 | mriedem | or change https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8311 to be a hard delete for ironic, | |
| 22:52:29 | mriedem | but i'm not sure how people would feel about that | |
| 22:52:36 | mriedem | we like to pack rat | |
| 22:53:06 | mriedem | i sorta feel that ironic compute nodes are special though unlike hypervisor compute ndoes (libvirt) and are more ephemeral | |
| 22:53:17 | mriedem | meaning hard delete isn't so bad | |
| 22:53:20 | mnaser | i agree | |
| 22:53:31 | mriedem | you're just going to archive those dorks anyway | |
| 22:54:35 | mriedem | ehh...the shitty thing is ComptueNode.destroy is remotable so adding a hard_delete kwarg to that or something would be an rpc version change, which means not backportable | |
| 22:55:02 | mriedem | not even sure about changing the behavior within ComputeNode.destroy... | |
| 22:55:21 | mnaser | unless the conductor does the hard delete if it knows that the destroy is for an ironic h v | |
| 22:55:31 | mriedem | right that's what i mean by within the method | |
| 22:55:41 | mriedem | would need to run it past dansmith in the morning probably | |
| 22:55:48 | mriedem | for now you can archive the db to workaround it | |
| 22:55:59 | mriedem | report a bug and i can talk with dan in the early morn | |
| 22:57:35 | mnaser | or maybe you can pull *all* baremetal nodes and only delete those that are missing (and ignore disabled/maintenance oness) | |
| 22:58:05 | mriedem | umm | |
| 22:58:28 | mriedem | i don't know that disabled/maintenance mode is a thing we track in the compute node object | |
| 22:58:35 | mriedem | unless it's in one of those json-ified fields | |
| 22:58:50 | mnaser | i meant when the ironic driver pulls the list of nodes | |
| 22:59:06 | mriedem | oh side-step the cn.destroy | |
| 22:59:26 | mriedem | yeah maybe, not sure the implications of that though | |
| 22:59:39 | mriedem | if you're host is managing 1000 nodes, that could be an expensive pull every 60 seconds :) | |
| 22:59:46 | mriedem | *your | |
| 23:00:03 | mnaser | you were gonna pull 1000 anyways, the only thing where it would be an issue is if you have 1000 with 990 in maint mode | |
| 23:00:23 | mriedem | 999 problems but the node ain't one? | |
| 23:00:33 | mnaser | had to go with 990 to not let you get away with that one ha | |
| 23:00:38 | openstack | Launchpad bug 1839560 in OpenStack Compute (nova) "ironic: moving node to maintenance makes it unusable afterwards" [Undecided,New] | |
| 23:00:38 | mnaser | https://bugs.launchpad.net/nova/+bug/1839560 | |