Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-08
22:45:33 mnaser mriedem: https://github.com/openstack/nova/commit/9f28727eb75e05e07bad51b6eecce667d09dfb65
22:45:53 mriedem which...does that mean a recreate is as simple as creating a server from a specific bm node, deleting the server, and then creating another server from the same node?
22:46:05 mriedem yes that's how i knew, i wrote it :)
22:46:50 mriedem went into rocky so it must not be that common of an issue
22:48:09 mnaser i think this might have been after moving a node to maintenance
22:48:10 mnaser and then out of it
22:48:11 mnaser i think?
22:48:33 mnaser does that result in the compute node being deleted? ironic is still new to me
22:48:35 openstack bug 1825537 in OpenStack Compute (nova) queens "finish_resize failures incorrectly revert allocations" [Medium,Confirmed] https://launchpad.net/bugs/1825537
22:48:35 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for regression bug 1825537 https://review.opendev.org/675355
22:48:51 mriedem do you see this for that node? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8306
22:50:59 mnaser mriedem: 2019-08-08 17:20:13.921 6379 INFO nova.compute.manager [req-c71e5c81-eb34-4f72-a260-6aa7e802f490 - - - - -] Deleting orphan compute node 31 hypervisor host is 77788ad5-f1a4-46ac-8132-2d88dbd4e594, nodes are set([u'6d556617-2bdc-42b3-a3fe-b9218a1ebf0e', u'a634fab2-ecea-4cfa-be09-032dce6eaf51', u'2dee290d-ef73-46bc-8fc2-af248841ca12'])
22:51:07 mriedem heh, bingo
22:51:11 mnaser yep, and i think this is when we set it to maintenance mode i think
22:51:21 mriedem which we likely don't test in ironic+nova ci
22:51:35 mnaser https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8298
22:52:04 mriedem right...
22:52:07 mriedem well, report a bug first,
22:52:13 mriedem we could revert https://review.opendev.org/#/c/571535/ which would be unfortunate,
22:52:25 mriedem or change https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8311 to be a hard delete for ironic,
22:52:29 mriedem but i'm not sure how people would feel about that
22:52:36 mriedem we like to pack rat
22:53:06 mriedem i sorta feel that ironic compute nodes are special though unlike hypervisor compute ndoes (libvirt) and are more ephemeral
22:53:17 mriedem meaning hard delete isn't so bad
22:53:20 mnaser i agree
22:53:31 mriedem you're just going to archive those dorks anyway
22:54:35 mriedem ehh...the shitty thing is ComptueNode.destroy is remotable so adding a hard_delete kwarg to that or something would be an rpc version change, which means not backportable
22:55:02 mriedem not even sure about changing the behavior within ComputeNode.destroy...
22:55:21 mnaser unless the conductor does the hard delete if it knows that the destroy is for an ironic h v
22:55:31 mriedem right that's what i mean by within the method
22:55:41 mriedem would need to run it past dansmith in the morning probably
22:55:48 mriedem for now you can archive the db to workaround it
22:55:59 mriedem report a bug and i can talk with dan in the early morn
22:57:35 mnaser or maybe you can pull *all* baremetal nodes and only delete those that are missing (and ignore disabled/maintenance oness)
22:58:05 mriedem umm
22:58:28 mriedem i don't know that disabled/maintenance mode is a thing we track in the compute node object
22:58:35 mriedem unless it's in one of those json-ified fields
22:58:50 mnaser i meant when the ironic driver pulls the list of nodes
22:59:06 mriedem oh side-step the cn.destroy
22:59:26 mriedem yeah maybe, not sure the implications of that though
22:59:39 mriedem if you're host is managing 1000 nodes, that could be an expensive pull every 60 seconds :)
22:59:46 mriedem *your
23:00:03 mnaser you were gonna pull 1000 anyways, the only thing where it would be an issue is if you have 1000 with 990 in maint mode
23:00:23 mriedem 999 problems but the node ain't one?
23:00:33 mnaser had to go with 990 to not let you get away with that one ha
23:00:38 openstack Launchpad bug 1839560 in OpenStack Compute (nova) "ironic: moving node to maintenance makes it unusable afterwards" [Undecided,New]
23:00:38 mnaser https://bugs.launchpad.net/nova/+bug/1839560
23:00:52 mnaser oh wait
23:01:06 mnaser i just realized the update_available_resource thing runs inside manager.py so its not ironic specific
23:01:13 mriedem you'd still have an issue if you deleted the compute service record which will (soft) delete all of the compute nodes associated with it
23:01:18 mriedem right
23:01:24 mriedem thought that node destroy stuff is really for ironic
23:01:30 mriedem in update_available_resource
23:01:34 mriedem *though
23:01:58 mriedem https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L401
23:02:00 mnaser ah yeah but i figured thats you burning yourself if you delete a compute service record (but i can imagine you would have the assumption that it automatically appears if you deleted it again so yeah)
23:02:08 mriedem ^ is a problem when deleting the managing nova-compute service record
23:02:23 mriedem it will when you restart the nova-compute service
23:02:36 mriedem or if you leave nova-compute running when you destroy the service
23:02:49 mriedem see note on https://docs.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service
23:03:28 mriedem so, i guess the revert is probably easiest backportable thing
23:04:29 mnaser yeah i cant think of anything better, anything else makes the api layer code too ironic-y
23:10:17 mriedem i'll post a revert for discussion anyway
23:12:14 mriedem warren zevon is going to help me get through this
23:23:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert "Match ComputeNode.uuid to ironic node uuid in RT" https://review.opendev.org/675496
23:28:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/675417
23:28:03 openstackgerrit Matt Riedemann proposed openstack/nova master: Dump versioned notifications when len assertions fail https://review.opendev.org/675404
23:28:10 mriedem god i must break more things than i fix
#openstack-nova - 2019-08-09
00:38:45 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add database schema upgrade check https://review.opendev.org/667047
02:31:23 openstackgerrit Merged openstack/nova stable/rocky: Don't generate service UUID for deleted services https://review.opendev.org/673821
02:31:29 openstackgerrit Merged openstack/nova master: Skip test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/675417
02:31:37 openstackgerrit Merged openstack/nova master: Dump versioned notifications when len assertions fail https://review.opendev.org/675404
02:49:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix rebuild of baremetal instance when vm_state is ERROR https://review.opendev.org/523559
02:49:26 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: run test_rebuild_server_in_error_state in ironic CI job https://review.opendev.org/675391
05:06:09 openstackgerrit Merged openstack/nova stable/queens: Replace non-nova server fault message https://review.opendev.org/674859
05:48:07 openstackgerrit Yongli He proposed openstack/nova master: Add server sub-resource topology API https://review.opendev.org/621476
06:08:30 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698
06:24:01 openstackgerrit Takashi NATSUME proposed openstack/nova master: trivial: Use NoDBTestCase instead of TestCase https://review.opendev.org/675529
06:35:27 openstackgerrit ya.wang proposed openstack/nova master: Add releasenote and update docs. https://review.opendev.org/674526
07:25:49 openstackgerrit zhufl proposed openstack/nova master: [Trivial]Remove unused _last_bw_usage_cell_update https://review.opendev.org/675543
07:42:49 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
08:06:41 openstackgerrit Balazs Gibizer proposed openstack/nova master: Prevent init_host test to interfere with other tests https://review.opendev.org/675553
08:26:16 openstackgerrit Bhagyashri Shewale proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895
09:29:58 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Correct nit with cpu-resources https://review.opendev.org/675276
09:34:58 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test heal port allocations in nova-next https://review.opendev.org/669879
09:45:00 openstackgerrit Adam Spiers proposed openstack/nova master: Drop usage of lxml's deprecated getchildren() method https://review.opendev.org/675566
09:54:32 alex_xu sean-k-mooney: i feel we have problem for confirm and revert resize https://review.opendev.org/#/c/621646/23/nova/virt/libvirt/driver.py@1099
09:56:08 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls https://review.opendev.org/655721
09:57:49 openstackgerrit Merged openstack/nova-specs master: Correct nit with cpu-resources https://review.opendev.org/675276
10:06:14 openstackgerrit Stephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894
10:06:14 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338
10:06:15 openstackgerrit Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895
10:06:15 openstackgerrit Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801
10:06:16 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571
10:06:16 openstackgerrit Stephen Finucane proposed openstack/nova master: fakelibvirt: Stop distinguishing between NUMA, non-NUMA https://review.opendev.org/675570
10:07:19 stephenfin alex_xu, efried, sean-k-mooney: Okay, with the exception of the reshaper, which still has some todo's in it, I _think_ the PCPU stuff is ready for review. It might look big but the bulk of it is handling tech debt. Let me know if you need advice on any part of it
10:07:47 alex_xu stephenfin: cool~
10:08:41 stephenfin alex_xu, efried, sean-k-mooney: I'm going to continue working on the reshaper and then start manual testing once that's done. I particularly want to check Train controller <-> Stein compute node compatibility and upgrades
10:10:50 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add database schema upgrade check https://review.opendev.org/667047
10:37:41 openstackgerrit Adam Spiers proposed openstack/nova master: Rename 'map' variable to avoid shadowing keywords https://review.opendev.org/675578

Earlier   Later