Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-17
21:21:34 openstackgerrit sean mooney proposed openstack/nova stable/rocky: [WIP] only update port binding if there is not an inactive binding https://review.openstack.org/653609
21:24:02 sean-k-mooney ^ is for testing
22:01:06 mriedem here it comes
22:02:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382
22:02:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179
22:02:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521
22:02:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806
22:02:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792
22:02:07 openstack bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem)
22:02:08 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Add instance hard delete https://review.openstack.org/650984
22:02:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036
22:02:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012
22:02:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123
22:02:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581
22:02:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892
22:02:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313
22:02:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853
22:02:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890
22:02:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293
22:02:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605
22:02:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832
22:02:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891
22:02:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047
22:02:25 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.openstack.org/638046
22:02:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268
22:02:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048
22:02:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353
22:02:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650
22:02:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269
22:03:57 mriedem time to scram http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&fullscreen&panelId=3
22:04:47 mriedem btw, ^ is in a runway slot and the bottom 3 changes have a +2 on them, should be relatively easy if another core can help out
22:29:25 mnaser hmm
22:29:44 mnaser is there a possible race condition around cell discover hosts?
22:29:47 mnaser this is really weird
22:29:48 mnaser http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/ara-report/result/2a432934-7b70-491a-9571-a40e85cd15c6/
22:30:05 mnaser "Checking host mapping for compute host" repeated twice, but with the same uuid?
22:31:21 mnaser the heck
22:31:22 mnaser http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-compute.log.txt.gz#_2019-04-17_21_52_21_727
22:31:28 mnaser http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-compute.log.txt.gz#_2019-04-17_21_52_21_819
22:32:17 mnaser in the same pid..
22:33:42 melwitt are you running discover_hosts in parallel during a deployment?
22:34:59 melwitt https://bugs.launchpad.net/nova/+bug/1824445
22:35:00 openstack Launchpad bug 1824445 in OpenStack Compute (nova) "nova-manage cellv2 discover_hosts traces when run in parallel" [Low,In progress] - Assigned to melanie witt (melwitt)
22:36:34 mnaser melwitt: nope, should be once only
22:36:55 mnaser but also is it normal that there is two log lines that say compute node created
22:37:58 mnaser melwitt: we also delegate_to the nova-conductor host so we run it once
22:39:05 melwitt I've only seen this before when the command is run in parallel presumably on different hosts
22:39:40 melwitt that's not normal, it shouldn't be trying to create the host mapping for the same host twice
22:39:52 melwitt I don't even get how that could happen non-parallel
22:39:58 mnaser yeah..
22:40:05 mnaser and the pid is the same too
22:41:17 melwitt I dunno, seems like dark magic
22:42:09 melwitt hm, ok the last log is ringing a bell
22:42:36 melwitt did you delete a service via the API and then not restart the nova-compute that goes with it?
22:43:16 melwitt because nova-compute will only create a compute node record when it starts up, it will not recover if the compute node record gets deleted out from under it while it's running
22:46:19 mnaser we don't delete anything afaik :\
22:46:27 mnaser during the CI run at least
22:46:51 melwitt ok. I don't see how that would be related to the discover_hosts thing anyway if it were happening
22:48:29 mnaser https://github.com/openstack/openstack-ansible-os_nova/blob/d1aa5839b482c7412404ebb7bf050ee806fbf281/tasks/main.yml#L233-L243
22:48:42 melwitt mnaser: oh, sorry, I see now it's saying Checking first and then Creating, so that's probably normal
22:48:51 mnaser oh oops
22:48:54 melwitt I thought it was the same message twice but it's not
22:49:10 mnaser you're right yes
22:49:12 mnaser "Compute node record created" is just once
22:49:20 melwitt but the duplicate, I've only seen that when one nova-manage is racing against another. do you have the discover_hosts scheduler periodic enabled?
22:49:44 mnaser I think we enable that based on something
22:49:45 mnaser one sec
22:49:48 openstackgerrit Eric Fried proposed openstack/nova master: Introduces the openstacksdk to nova https://review.openstack.org/643664
22:50:08 melwitt that's another possibility, if you were unlucky enough to have that fire at the same time as another scheduler, or if you were to run 'nova-manage cellv2 discover_hosts' at the same time a scheduler periodic discover_hosts fires
22:50:39 mnaser scheduler.discover_hosts_in_cells_interval = 60
22:50:55 openstackgerrit Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899
22:51:07 mnaser does discover_hosts run in the conductor I assume?
22:51:07 openstackgerrit Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899
22:51:20 melwitt ok, and is that set on all schedulers or just one? (recommendation is to enable it on only one) see https://review.openstack.org/651947
22:51:25 melwitt scheduler
22:52:25 mnaser http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-scheduler.log.txt.gz#_2019-04-17_21_52_03_776
22:52:30 mnaser yeah its around the same time
22:52:35 mnaser we have it enabled on all hosts :X
22:53:13 melwitt ok. yeah, I don't think we knew this was going to be a potential problem, or at least I didn't know
22:53:22 mnaser I mean its the first time I see this
22:53:24 mnaser but at least we kno wwhy
22:53:26 melwitt the trace isn't harmful, it just means discover_hosts collided with another one
22:53:43 melwitt so the patch I linked earlier aims to warn and give guidance instead of tracing
22:54:15 mnaser yeah that would be good because the ansible obviously fails when it sees that
22:54:18 mnaser so that would be good to know
22:54:24 melwitt commit message explains why to do that too vs ignore DBDuplicateEntry
22:54:30 mnaser neat
22:54:56 melwitt oh it does? how do yall do discover_hosts? keep running until you receive status code 0? that's what tripleo does
22:55:39 melwitt the status code is a little weird on that command where non-zero means "there's more work to do" and zero means "I'm done"
22:55:54 melwitt and I think that's the review feedback I got on that patch currently is to enhance the doc to explain that
22:57:13 openstackgerrit Merged openstack/nova master: Stop handling cells v1 in '/os-servers' API https://review.openstack.org/651293
22:57:21 openstackgerrit Merged openstack/nova stable/rocky: Add functional recreate test for bug 1819963 https://review.openstack.org/648409
22:57:22 openstack bug 1819963 in OpenStack Compute (nova) rocky "Reverting a resize does not update the instance.availability_zone value to the source az" [Medium,In progress] https://launchpad.net/bugs/1819963 - Assigned to Matt Riedemann (mriedem)
22:57:36 openstackgerrit Merged openstack/nova stable/rocky: Update instance.availability_zone on revertResize https://review.openstack.org/648410
22:57:48 mnaser melwitt: we run discover_hosts every time once only on one of the conductors
22:58:03 mnaser I didn't know discover_hosts does work in batches
22:58:06 melwitt oh, so always expect to receive 0
22:58:29 melwitt I don't think it does. tripleo retries everything
22:58:43 mnaser https://github.com/openstack/openstack-ansible-os_nova/blob/d1aa5839b482c7412404ebb7bf050ee806fbf281/tasks/nova_db_post_setup.yml#L21-L26
22:59:06 mnaser ya I try to avoid, uh, brute forcing stuff :p

Earlier   Later