| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-08 | |||
| 22:41:07 | mnaser | yes | |
| 22:41:10 | mnaser | but i think this is tricking me | |
| 22:41:20 | mnaser | 2019-08-08 22:21:23.467 82770 ERROR ironicclient.common.http [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] Error contacting Ironic server: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409). Attempt 2 of 2: Conflict: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to | |
| 22:41:20 | mnaser | Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409) | |
| 22:41:39 | mnaser | i see that happen right before it, so i wonder if those warnings/errors make nova-compute blow up and bubble up the wrong exception | |
| 22:43:39 | mnaser | there is a soft deleted one though, i wonder if thats the issue | |
| 22:44:12 | mriedem | yeah the unique index is on uuid not including the deleted column | |
| 22:44:22 | mriedem | unlike uniq_compute_nodes0host0hypervisor_hostname0deleted | |
| 22:44:44 | mriedem | i can't remember what release it was, but semi-recently we started using the ironic node uuid as the compute node uuid for easier correlation of those records | |
| 22:44:57 | mnaser | well deleting it fixing it but that sounds like a bug then i guess | |
| 22:44:59 | mnaser | yeah that might explain it | |
| 22:45:08 | mriedem | but, since the compute node uuid is not unique now when it gets created, it'll be a problem if the ironic node / compute node is soft-deleted and not archived... | |
| 22:45:33 | mnaser | mriedem: https://github.com/openstack/nova/commit/9f28727eb75e05e07bad51b6eecce667d09dfb65 | |
| 22:45:53 | mriedem | which...does that mean a recreate is as simple as creating a server from a specific bm node, deleting the server, and then creating another server from the same node? | |
| 22:46:05 | mriedem | yes that's how i knew, i wrote it :) | |
| 22:46:50 | mriedem | went into rocky so it must not be that common of an issue | |
| 22:48:09 | mnaser | i think this might have been after moving a node to maintenance | |
| 22:48:10 | mnaser | and then out of it | |
| 22:48:11 | mnaser | i think? | |
| 22:48:33 | mnaser | does that result in the compute node being deleted? ironic is still new to me | |
| 22:48:35 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for regression bug 1825537 https://review.opendev.org/675355 | |
| 22:48:35 | openstack | bug 1825537 in OpenStack Compute (nova) queens "finish_resize failures incorrectly revert allocations" [Medium,Confirmed] https://launchpad.net/bugs/1825537 | |
| 22:48:51 | mriedem | do you see this for that node? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8306 | |
| 22:50:59 | mnaser | mriedem: 2019-08-08 17:20:13.921 6379 INFO nova.compute.manager [req-c71e5c81-eb34-4f72-a260-6aa7e802f490 - - - - -] Deleting orphan compute node 31 hypervisor host is 77788ad5-f1a4-46ac-8132-2d88dbd4e594, nodes are set([u'6d556617-2bdc-42b3-a3fe-b9218a1ebf0e', u'a634fab2-ecea-4cfa-be09-032dce6eaf51', u'2dee290d-ef73-46bc-8fc2-af248841ca12']) | |
| 22:51:07 | mriedem | heh, bingo | |
| 22:51:11 | mnaser | yep, and i think this is when we set it to maintenance mode i think | |
| 22:51:21 | mriedem | which we likely don't test in ironic+nova ci | |
| 22:51:35 | mnaser | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8298 | |
| 22:52:04 | mriedem | right... | |
| 22:52:07 | mriedem | well, report a bug first, | |
| 22:52:13 | mriedem | we could revert https://review.opendev.org/#/c/571535/ which would be unfortunate, | |
| 22:52:25 | mriedem | or change https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8311 to be a hard delete for ironic, | |
| 22:52:29 | mriedem | but i'm not sure how people would feel about that | |
| 22:52:36 | mriedem | we like to pack rat | |
| 22:53:06 | mriedem | i sorta feel that ironic compute nodes are special though unlike hypervisor compute ndoes (libvirt) and are more ephemeral | |
| 22:53:17 | mriedem | meaning hard delete isn't so bad | |
| 22:53:20 | mnaser | i agree | |
| 22:53:31 | mriedem | you're just going to archive those dorks anyway | |
| 22:54:35 | mriedem | ehh...the shitty thing is ComptueNode.destroy is remotable so adding a hard_delete kwarg to that or something would be an rpc version change, which means not backportable | |
| 22:55:02 | mriedem | not even sure about changing the behavior within ComputeNode.destroy... | |
| 22:55:21 | mnaser | unless the conductor does the hard delete if it knows that the destroy is for an ironic h v | |
| 22:55:31 | mriedem | right that's what i mean by within the method | |
| 22:55:41 | mriedem | would need to run it past dansmith in the morning probably | |
| 22:55:48 | mriedem | for now you can archive the db to workaround it | |
| 22:55:59 | mriedem | report a bug and i can talk with dan in the early morn | |
| 22:57:35 | mnaser | or maybe you can pull *all* baremetal nodes and only delete those that are missing (and ignore disabled/maintenance oness) | |
| 22:58:05 | mriedem | umm | |
| 22:58:28 | mriedem | i don't know that disabled/maintenance mode is a thing we track in the compute node object | |
| 22:58:35 | mriedem | unless it's in one of those json-ified fields | |
| 22:58:50 | mnaser | i meant when the ironic driver pulls the list of nodes | |
| 22:59:06 | mriedem | oh side-step the cn.destroy | |
| 22:59:26 | mriedem | yeah maybe, not sure the implications of that though | |
| 22:59:39 | mriedem | if you're host is managing 1000 nodes, that could be an expensive pull every 60 seconds :) | |
| 22:59:46 | mriedem | *your | |
| 23:00:03 | mnaser | you were gonna pull 1000 anyways, the only thing where it would be an issue is if you have 1000 with 990 in maint mode | |
| 23:00:23 | mriedem | 999 problems but the node ain't one? | |
| 23:00:33 | mnaser | had to go with 990 to not let you get away with that one ha | |
| 23:00:38 | mnaser | https://bugs.launchpad.net/nova/+bug/1839560 | |
| 23:00:38 | openstack | Launchpad bug 1839560 in OpenStack Compute (nova) "ironic: moving node to maintenance makes it unusable afterwards" [Undecided,New] | |
| 23:00:52 | mnaser | oh wait | |
| 23:01:06 | mnaser | i just realized the update_available_resource thing runs inside manager.py so its not ironic specific | |
| 23:01:13 | mriedem | you'd still have an issue if you deleted the compute service record which will (soft) delete all of the compute nodes associated with it | |
| 23:01:18 | mriedem | right | |
| 23:01:24 | mriedem | thought that node destroy stuff is really for ironic | |
| 23:01:30 | mriedem | in update_available_resource | |
| 23:01:34 | mriedem | *though | |
| 23:01:58 | mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L401 | |
| 23:02:00 | mnaser | ah yeah but i figured thats you burning yourself if you delete a compute service record (but i can imagine you would have the assumption that it automatically appears if you deleted it again so yeah) | |
| 23:02:08 | mriedem | ^ is a problem when deleting the managing nova-compute service record | |
| 23:02:23 | mriedem | it will when you restart the nova-compute service | |
| 23:02:36 | mriedem | or if you leave nova-compute running when you destroy the service | |
| 23:02:49 | mriedem | see note on https://docs.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service | |
| 23:03:28 | mriedem | so, i guess the revert is probably easiest backportable thing | |
| 23:04:29 | mnaser | yeah i cant think of anything better, anything else makes the api layer code too ironic-y | |
| 23:10:17 | mriedem | i'll post a revert for discussion anyway | |
| 23:12:14 | mriedem | warren zevon is going to help me get through this | |
| 23:23:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Match ComputeNode.uuid to ironic node uuid in RT" https://review.opendev.org/675496 | |
| 23:28:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/675417 | |
| 23:28:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Dump versioned notifications when len assertions fail https://review.opendev.org/675404 | |
| 23:28:10 | mriedem | god i must break more things than i fix | |
| #openstack-nova - 2019-08-09 | |||
| 00:38:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add database schema upgrade check https://review.opendev.org/667047 | |
| 02:31:23 | openstackgerrit | Merged openstack/nova stable/rocky: Don't generate service UUID for deleted services https://review.opendev.org/673821 | |
| 02:31:29 | openstackgerrit | Merged openstack/nova master: Skip test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/675417 | |
| 02:31:37 | openstackgerrit | Merged openstack/nova master: Dump versioned notifications when len assertions fail https://review.opendev.org/675404 | |
| 02:49:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix rebuild of baremetal instance when vm_state is ERROR https://review.opendev.org/523559 | |
| 02:49:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: run test_rebuild_server_in_error_state in ironic CI job https://review.opendev.org/675391 | |
| 05:06:09 | openstackgerrit | Merged openstack/nova stable/queens: Replace non-nova server fault message https://review.opendev.org/674859 | |
| 05:48:07 | openstackgerrit | Yongli He proposed openstack/nova master: Add server sub-resource topology API https://review.opendev.org/621476 | |
| 06:08:30 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698 | |
| 06:24:01 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: trivial: Use NoDBTestCase instead of TestCase https://review.opendev.org/675529 | |
| 06:35:27 | openstackgerrit | ya.wang proposed openstack/nova master: Add releasenote and update docs. https://review.opendev.org/674526 | |
| 07:25:49 | openstackgerrit | zhufl proposed openstack/nova master: [Trivial]Remove unused _last_bw_usage_cell_update https://review.opendev.org/675543 | |
| 07:42:49 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | |
| 08:06:41 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent init_host test to interfere with other tests https://review.opendev.org/675553 | |
| 08:26:16 | openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | |
| 09:29:58 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Correct nit with cpu-resources https://review.opendev.org/675276 | |
| 09:34:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test heal port allocations in nova-next https://review.opendev.org/669879 | |
| 09:45:00 | openstackgerrit | Adam Spiers proposed openstack/nova master: Drop usage of lxml's deprecated getchildren() method https://review.opendev.org/675566 | |
| 09:54:32 | alex_xu | sean-k-mooney: i feel we have problem for confirm and revert resize https://review.opendev.org/#/c/621646/23/nova/virt/libvirt/driver.py@1099 | |
| 09:56:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls https://review.opendev.org/655721 | |