Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-17
18:56:27 mriedem gmann: done https://review.openstack.org/#/c/603969/
18:57:21 openstackgerrit melanie witt proposed openstack/nova stable/stein: libvirt: set device address tag only if setting disk unit https://review.openstack.org/653510
18:59:45 openstackgerrit melanie witt proposed openstack/nova stable/rocky: libvirt: set device address tag only if setting disk unit https://review.openstack.org/653511
19:00:35 openstackgerrit melanie witt proposed openstack/nova stable/queens: libvirt: set device address tag only if setting disk unit https://review.openstack.org/653512
19:05:37 openstackgerrit melanie witt proposed openstack/nova stable/pike: libvirt: set device address tag only if setting disk unit https://review.openstack.org/653514
19:08:09 openstackgerrit Eric Fried proposed openstack/nova master: DEMO: Functional recreate test for bug 1819963 https://review.openstack.org/653504
19:08:10 openstack bug 1819963 in OpenStack Compute (nova) rocky "Reverting a resize does not update the instance.availability_zone value to the source az" [Medium,In progress] https://launchpad.net/bugs/1819963 - Assigned to Matt Riedemann (mriedem)
19:08:10 openstackgerrit Eric Fried proposed openstack/nova master: DEMO: Update instance.availability_zone on revertResize https://review.openstack.org/653505
19:08:35 efried mriedem: Thanks. I've done a couple of things to improve that test case ^ - lmk if you think it's worth pulling those out into a real patch.
19:09:29 efried actually, the first thing - checking the db explicitly - would have only applied to the interim recreate patch. So it's just the part where we do that GET /servers?availability_zone=* query that would be added now.
19:20:49 openstackgerrit Ghanshyam Mann proposed openstack/nova-specs master: Spec for API inconsistency cleanup https://review.openstack.org/603969
19:21:29 gmann mriedem: thanks. ^^ updated.
20:16:13 openstackgerrit Merged openstack/nova master: Remove '/os-cells' REST APIs https://review.openstack.org/651291
20:16:20 openstackgerrit Merged openstack/nova master: Stop handling cells v1 in '/os-hypervisors' API https://review.openstack.org/651292
20:29:40 openstackgerrit Merged openstack/nova stable/rocky: Fix incomplete instance data returned after build failure https://review.openstack.org/647911
20:29:48 openstackgerrit Merged openstack/nova stable/queens: Update instance.availability_zone during live migration https://review.openstack.org/647623
20:55:52 efried mriedem: Prior to "handle down cells", would the APIs just return failures?
20:57:03 mriedem efried: by default no, but it's configurable
20:57:32 mriedem https://docs.openstack.org/nova/latest/admin/cells.html#handling-cell-failures
20:57:35 openstackgerrit sean mooney proposed openstack/nova master: [WIP] only update port binding if there is not an inactive binding https://review.openstack.org/653506
20:57:40 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#api.list_records_by_skipping_down_cells
20:59:32 efried okay, so previously we would either (a) show information only for up cells, completely skipping down cells as if they weren't there; or (b) error the API call
20:59:45 efried and now we (c) show partial information from the down cells
21:00:46 mriedem (c) if you're using microversion >= 2.69
21:02:18 efried ack, thx
21:21:34 openstackgerrit sean mooney proposed openstack/nova stable/rocky: [WIP] only update port binding if there is not an inactive binding https://review.openstack.org/653609
21:24:02 sean-k-mooney ^ is for testing
22:01:06 mriedem here it comes
22:02:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179
22:02:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382
22:02:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521
22:02:07 openstack bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem)
22:02:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792
22:02:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806
22:02:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012
22:02:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036
22:02:08 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Add instance hard delete https://review.openstack.org/650984
22:02:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123
22:02:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892
22:02:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581
22:02:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853
22:02:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313
22:02:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293
22:02:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890
22:02:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832
22:02:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605
22:02:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891
22:02:25 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.openstack.org/638046
22:02:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047
22:02:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048
22:02:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268
22:02:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650
22:02:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353
22:02:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269
22:03:57 mriedem time to scram http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&fullscreen&panelId=3
22:04:47 mriedem btw, ^ is in a runway slot and the bottom 3 changes have a +2 on them, should be relatively easy if another core can help out
22:29:25 mnaser hmm
22:29:44 mnaser is there a possible race condition around cell discover hosts?
22:29:47 mnaser this is really weird
22:29:48 mnaser http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/ara-report/result/2a432934-7b70-491a-9571-a40e85cd15c6/
22:30:05 mnaser "Checking host mapping for compute host" repeated twice, but with the same uuid?
22:31:21 mnaser the heck
22:31:22 mnaser http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-compute.log.txt.gz#_2019-04-17_21_52_21_727
22:31:28 mnaser http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-compute.log.txt.gz#_2019-04-17_21_52_21_819
22:32:17 mnaser in the same pid..
22:33:42 melwitt are you running discover_hosts in parallel during a deployment?
22:34:59 melwitt https://bugs.launchpad.net/nova/+bug/1824445
22:35:00 openstack Launchpad bug 1824445 in OpenStack Compute (nova) "nova-manage cellv2 discover_hosts traces when run in parallel" [Low,In progress] - Assigned to melanie witt (melwitt)
22:36:34 mnaser melwitt: nope, should be once only
22:36:55 mnaser but also is it normal that there is two log lines that say compute node created
22:37:58 mnaser melwitt: we also delegate_to the nova-conductor host so we run it once
22:39:05 melwitt I've only seen this before when the command is run in parallel presumably on different hosts
22:39:40 melwitt that's not normal, it shouldn't be trying to create the host mapping for the same host twice
22:39:52 melwitt I don't even get how that could happen non-parallel
22:39:58 mnaser yeah..
22:40:05 mnaser and the pid is the same too
22:41:17 melwitt I dunno, seems like dark magic
22:42:09 melwitt hm, ok the last log is ringing a bell
22:42:36 melwitt did you delete a service via the API and then not restart the nova-compute that goes with it?
22:43:16 melwitt because nova-compute will only create a compute node record when it starts up, it will not recover if the compute node record gets deleted out from under it while it's running
22:46:19 mnaser we don't delete anything afaik :\
22:46:27 mnaser during the CI run at least
22:46:51 melwitt ok. I don't see how that would be related to the discover_hosts thing anyway if it were happening
22:48:29 mnaser https://github.com/openstack/openstack-ansible-os_nova/blob/d1aa5839b482c7412404ebb7bf050ee806fbf281/tasks/main.yml#L233-L243
22:48:42 melwitt mnaser: oh, sorry, I see now it's saying Checking first and then Creating, so that's probably normal
22:48:51 mnaser oh oops
22:48:54 melwitt I thought it was the same message twice but it's not
22:49:10 mnaser you're right yes
22:49:12 mnaser "Compute node record created" is just once
22:49:20 melwitt but the duplicate, I've only seen that when one nova-manage is racing against another. do you have the discover_hosts scheduler periodic enabled?
22:49:44 mnaser I think we enable that based on something
22:49:45 mnaser one sec
22:49:48 openstackgerrit Eric Fried proposed openstack/nova master: Introduces the openstacksdk to nova https://review.openstack.org/643664
22:50:08 melwitt that's another possibility, if you were unlucky enough to have that fire at the same time as another scheduler, or if you were to run 'nova-manage cellv2 discover_hosts' at the same time a scheduler periodic discover_hosts fires
22:50:39 mnaser scheduler.discover_hosts_in_cells_interval = 60
22:50:55 openstackgerrit Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899
22:51:07 openstackgerrit Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899
22:51:07 mnaser does discover_hosts run in the conductor I assume?
22:51:20 melwitt ok, and is that set on all schedulers or just one? (recommendation is to enable it on only one) see https://review.openstack.org/651947
22:51:25 melwitt scheduler

Earlier   Later