| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-08 | |||
| 22:09:05 | sean-k-mooney | im off tomorow so if you see anything else ill take a look on monday | |
| 22:09:26 | sean-k-mooney | with that i think ill call it a night o/ | |
| 22:09:49 | mriedem | -1!!! | |
| 22:10:02 | mriedem | your sleep depravation has gotten the better of you | |
| 22:11:10 | sean-k-mooney | haha i mess that up | |
| 22:11:30 | sean-k-mooney | i was working on image meta data suff before | |
| 22:13:02 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:13:22 | sean-k-mooney | no still wrong | |
| 22:13:41 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117 | |
| 22:14:44 | openstackgerrit | Merged openstack/nova stable/pike: Implement power_off/power_on for the FakeDriver https://review.opendev.org/674876 | |
| 22:14:51 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 22:15:29 | sean-k-mooney | ok fixed good catch o/ | |
| 22:15:47 | mriedem | sean-k-mooney: ok bug me about it tomorrow | |
| 22:22:56 | mnaser | is the compute_nodes table exposed anywhere? | |
| 22:23:04 | mnaser | s/table/data/ | |
| 22:23:18 | sean-k-mooney | its in the hyperviors api i think | |
| 22:23:25 | mnaser | with ironic and for some reason its complaining that there is a duplicate compute_node listed | |
| 22:25:15 | mnaser | Remote error: DBDuplicateEntry (pymysql.err.IntegrityError) (1062, u"Duplicate entry '77788ad5-f1a4-46ac-8132-2d88dbd4e594' for key 'compute_nodes_uuid_idx'") | |
| 22:27:58 | mnaser | 2019-08-08 22:21:25.284 82770 WARNING nova.compute.resource_tracker [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] No compute node record for ctl1-xxxx:77788ad5-f1a4-46ac-8132-2d88dbd4e594: ComputeHostNotFound_Remote: Compute host ctl1-xxxx could not be found. | |
| 22:27:59 | mnaser | hmm | |
| 22:32:00 | mriedem | is one (soft) deleted? | |
| 22:41:07 | mnaser | yes | |
| 22:41:10 | mnaser | but i think this is tricking me | |
| 22:41:20 | mnaser | 2019-08-08 22:21:23.467 82770 ERROR ironicclient.common.http [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] Error contacting Ironic server: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409). Attempt 2 of 2: Conflict: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to | |
| 22:41:20 | mnaser | Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409) | |
| 22:41:39 | mnaser | i see that happen right before it, so i wonder if those warnings/errors make nova-compute blow up and bubble up the wrong exception | |
| 22:43:39 | mnaser | there is a soft deleted one though, i wonder if thats the issue | |
| 22:44:12 | mriedem | yeah the unique index is on uuid not including the deleted column | |
| 22:44:22 | mriedem | unlike uniq_compute_nodes0host0hypervisor_hostname0deleted | |
| 22:44:44 | mriedem | i can't remember what release it was, but semi-recently we started using the ironic node uuid as the compute node uuid for easier correlation of those records | |
| 22:44:57 | mnaser | well deleting it fixing it but that sounds like a bug then i guess | |
| 22:44:59 | mnaser | yeah that might explain it | |
| 22:45:08 | mriedem | but, since the compute node uuid is not unique now when it gets created, it'll be a problem if the ironic node / compute node is soft-deleted and not archived... | |
| 22:45:33 | mnaser | mriedem: https://github.com/openstack/nova/commit/9f28727eb75e05e07bad51b6eecce667d09dfb65 | |
| 22:45:53 | mriedem | which...does that mean a recreate is as simple as creating a server from a specific bm node, deleting the server, and then creating another server from the same node? | |
| 22:46:05 | mriedem | yes that's how i knew, i wrote it :) | |
| 22:46:50 | mriedem | went into rocky so it must not be that common of an issue | |
| 22:48:09 | mnaser | i think this might have been after moving a node to maintenance | |
| 22:48:10 | mnaser | and then out of it | |
| 22:48:11 | mnaser | i think? | |
| 22:48:33 | mnaser | does that result in the compute node being deleted? ironic is still new to me | |
| 22:48:35 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for regression bug 1825537 https://review.opendev.org/675355 | |
| 22:48:35 | openstack | bug 1825537 in OpenStack Compute (nova) queens "finish_resize failures incorrectly revert allocations" [Medium,Confirmed] https://launchpad.net/bugs/1825537 | |
| 22:48:51 | mriedem | do you see this for that node? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8306 | |
| 22:50:59 | mnaser | mriedem: 2019-08-08 17:20:13.921 6379 INFO nova.compute.manager [req-c71e5c81-eb34-4f72-a260-6aa7e802f490 - - - - -] Deleting orphan compute node 31 hypervisor host is 77788ad5-f1a4-46ac-8132-2d88dbd4e594, nodes are set([u'6d556617-2bdc-42b3-a3fe-b9218a1ebf0e', u'a634fab2-ecea-4cfa-be09-032dce6eaf51', u'2dee290d-ef73-46bc-8fc2-af248841ca12']) | |
| 22:51:07 | mriedem | heh, bingo | |
| 22:51:11 | mnaser | yep, and i think this is when we set it to maintenance mode i think | |
| 22:51:21 | mriedem | which we likely don't test in ironic+nova ci | |
| 22:51:35 | mnaser | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8298 | |
| 22:52:04 | mriedem | right... | |
| 22:52:07 | mriedem | well, report a bug first, | |
| 22:52:13 | mriedem | we could revert https://review.opendev.org/#/c/571535/ which would be unfortunate, | |
| 22:52:25 | mriedem | or change https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8311 to be a hard delete for ironic, | |
| 22:52:29 | mriedem | but i'm not sure how people would feel about that | |
| 22:52:36 | mriedem | we like to pack rat | |
| 22:53:06 | mriedem | i sorta feel that ironic compute nodes are special though unlike hypervisor compute ndoes (libvirt) and are more ephemeral | |
| 22:53:17 | mriedem | meaning hard delete isn't so bad | |
| 22:53:20 | mnaser | i agree | |
| 22:53:31 | mriedem | you're just going to archive those dorks anyway | |
| 22:54:35 | mriedem | ehh...the shitty thing is ComptueNode.destroy is remotable so adding a hard_delete kwarg to that or something would be an rpc version change, which means not backportable | |
| 22:55:02 | mriedem | not even sure about changing the behavior within ComputeNode.destroy... | |
| 22:55:21 | mnaser | unless the conductor does the hard delete if it knows that the destroy is for an ironic h v | |
| 22:55:31 | mriedem | right that's what i mean by within the method | |
| 22:55:41 | mriedem | would need to run it past dansmith in the morning probably | |
| 22:55:48 | mriedem | for now you can archive the db to workaround it | |
| 22:55:59 | mriedem | report a bug and i can talk with dan in the early morn | |
| 22:57:35 | mnaser | or maybe you can pull *all* baremetal nodes and only delete those that are missing (and ignore disabled/maintenance oness) | |
| 22:58:05 | mriedem | umm | |
| 22:58:28 | mriedem | i don't know that disabled/maintenance mode is a thing we track in the compute node object | |
| 22:58:35 | mriedem | unless it's in one of those json-ified fields | |
| 22:58:50 | mnaser | i meant when the ironic driver pulls the list of nodes | |
| 22:59:06 | mriedem | oh side-step the cn.destroy | |
| 22:59:26 | mriedem | yeah maybe, not sure the implications of that though | |
| 22:59:39 | mriedem | if you're host is managing 1000 nodes, that could be an expensive pull every 60 seconds :) | |
| 22:59:46 | mriedem | *your | |
| 23:00:03 | mnaser | you were gonna pull 1000 anyways, the only thing where it would be an issue is if you have 1000 with 990 in maint mode | |
| 23:00:23 | mriedem | 999 problems but the node ain't one? | |
| 23:00:33 | mnaser | had to go with 990 to not let you get away with that one ha | |
| 23:00:38 | mnaser | https://bugs.launchpad.net/nova/+bug/1839560 | |
| 23:00:38 | openstack | Launchpad bug 1839560 in OpenStack Compute (nova) "ironic: moving node to maintenance makes it unusable afterwards" [Undecided,New] | |
| 23:00:52 | mnaser | oh wait | |
| 23:01:06 | mnaser | i just realized the update_available_resource thing runs inside manager.py so its not ironic specific | |
| 23:01:13 | mriedem | you'd still have an issue if you deleted the compute service record which will (soft) delete all of the compute nodes associated with it | |
| 23:01:18 | mriedem | right | |
| 23:01:24 | mriedem | thought that node destroy stuff is really for ironic | |
| 23:01:30 | mriedem | in update_available_resource | |
| 23:01:34 | mriedem | *though | |
| 23:01:58 | mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L401 | |
| 23:02:00 | mnaser | ah yeah but i figured thats you burning yourself if you delete a compute service record (but i can imagine you would have the assumption that it automatically appears if you deleted it again so yeah) | |
| 23:02:08 | mriedem | ^ is a problem when deleting the managing nova-compute service record | |
| 23:02:23 | mriedem | it will when you restart the nova-compute service | |
| 23:02:36 | mriedem | or if you leave nova-compute running when you destroy the service | |
| 23:02:49 | mriedem | see note on https://docs.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service | |
| 23:03:28 | mriedem | so, i guess the revert is probably easiest backportable thing | |
| 23:04:29 | mnaser | yeah i cant think of anything better, anything else makes the api layer code too ironic-y | |
| 23:10:17 | mriedem | i'll post a revert for discussion anyway | |
| 23:12:14 | mriedem | warren zevon is going to help me get through this | |
| 23:23:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Match ComputeNode.uuid to ironic node uuid in RT" https://review.opendev.org/675496 | |
| 23:28:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/675417 | |
| 23:28:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Dump versioned notifications when len assertions fail https://review.opendev.org/675404 | |