Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-19
17:52:05 sean-k-mooney if the issue still exists with pci passthough and vgpus disabled in the conf(assuming you were using either) that narrows down the code that could be at falut
17:52:07 rouk nope, same config as all other nodes, just rbd storage, oslo memcached, lock_path = /var/lib/nova/tmp,
17:52:27 sean-k-mooney ok so we can ignore those codepaths then
17:52:28 rouk no major customization other than standard kolla config for a compute node.
17:52:42 sean-k-mooney what relese did you say you were using
17:52:44 rouk can dump you configs if it would help
17:53:33 rouk rocky, nova==18.1.0, it was working on this node up until monday, no changes in version, got some stein deployments that are happy too.
17:54:11 rouk could move up to 18.2.1 i guess, if a fresh one would help maybe?
17:58:55 openstackgerrit Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420
17:59:09 sean-k-mooney am can you file a bug for this and inclode as much info as you can
17:59:52 rouk well if nobody has heard of it, and nobody has had similar, i would need to find the exact cause before a bugreport would be any use...
17:59:57 sean-k-mooney it sound like there is a deadlock or somethin that is preventing thinks form correctly yeilding
18:00:06 openstackgerrit Adam Spiers proposed openstack/nova master: Use fake flavor instead of empty dict in test https://review.opendev.org/662555
18:00:06 openstackgerrit Adam Spiers proposed openstack/nova master: Pass extra_specs to flavor in vif tests https://review.opendev.org/662556
18:00:07 openstackgerrit Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334
18:00:07 openstackgerrit Adam Spiers proposed openstack/nova master: Add element to guest config for AMD SEV https://review.opendev.org/636318
18:00:08 openstackgerrit Adam Spiers proposed openstack/nova master: Allow guest devices to include https://review.opendev.org/644564
18:00:08 openstackgerrit Adam Spiers proposed openstack/nova master: Detect that SEV is required and enable iommu for devices https://review.opendev.org/644565
18:00:09 openstackgerrit Adam Spiers proposed openstack/nova master: Use element when SEV is required https://review.opendev.org/662557
18:00:09 openstackgerrit Adam Spiers proposed openstack/nova master: Enable memory locking if SEV is requested https://review.opendev.org/662558
18:00:33 sean-k-mooney rouk: well you dont need to fuly root cause it to open the bug you could update it as you go with your findings
18:00:56 aspiers sean-k-mooney, mriedem: this is the best I could come up with for the fake dict issue: https://review.opendev.org/#/c/664420/10/nova/virt/hardware.py@1142
18:13:58 mriedem sean-k-mooney: the 2.7.15 python thing recently was that hacking check unit test failure
18:14:23 mriedem https://bugs.launchpad.net/nova/+bug/1832392
18:14:24 openstack Launchpad bug 1804062 in nova (Ubuntu Eoan) "duplicate for #1832392 test_hacking fails for python 3.6.7 and newer" [High,Triaged]
18:16:04 mriedem aspiers: comment inline
18:25:24 sean-k-mooney mriedem: kolla-ansible also had some other issue but i think rouk issue is related to ceph network connectinvity
18:26:08 aspiers mriedem: thanks!
18:26:37 sean-k-mooney not 100% sure but when they tried to conenct to ceph form the node the ceph status commnd was filling speradically so that might be cause the agent to hang when it tries to get the capasity fo the rbd pool
18:31:21 rouk yeah, checking for ip conflicts that somehow got into the network managed only by kolla...
18:31:32 rouk see what else we can find, its flapping like an ip conflict.
18:32:10 sean-k-mooney you mentioned you changed the host name between redeploying right
18:32:25 sean-k-mooney did you use the kolla ansible bootstrap playbook
18:32:49 sean-k-mooney that templates out a /etc/hosts file with static assignments for all nodes
18:33:10 sean-k-mooney but if you are also using dhcp that could maybe cause issues
18:33:36 sean-k-mooney its been a while since i woked on kolla so they may have changed that
18:41:40 mriedem sean-k-mooney: mlavalle: comments on https://review.opendev.org/#/c/644881/ and question about the assertion that ovs hybrid plug vif types are "neutron wide" configuration - that seems surprising to me
18:41:48 mriedem i thought that conifguration was per neutron agent
18:48:15 artom mriedem, thanks for the review - we're in the final throws of packing, so sean-k-mooney will have to follow up. I've left a reply for 1 thing though.
19:40:47 mriedem https://review.opendev.org/#/c/571265/ adds a functional test for a scheduler filter that was otherwise not tested (i don't think when i wrote it anyway), open for over a year now, has a +2 if someone can look
19:50:27 mriedem is melwitt out this week?
19:51:32 mriedem dansmith: looking back on https://bugs.launchpad.net/nova/+bug/1773945 and https://bugs.launchpad.net/nova/+bug/1784074 - if we get here in conductor where the instance is deleted after the build request is created and we've done scheduling but before the instance is created in a cell, we delete the build request here and continue to the next instance we're building: https://github.com/openstack/nova/blob/74aebe0d4e5a978a4001
19:51:35 openstack Launchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
19:51:35 mriedem 0aee9e70e98246c4/nova/conductor/manager.py#L1350
19:51:36 openstack Launchpad bug 1784074 in OpenStack Compute (nova) "Instances end up with no cell assigned in instance_mappings" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
19:51:46 mriedem we don't bury that instance in cell0 presumably because it's already been deleted via build request
19:52:05 mriedem however, since we don't bury it, and the build request is gone, and we didn't create the instance, we could have an orphaned instance mapping, yeah?
19:52:33 jgwentworth no, just didn't want to lose my other nick from not using it
19:54:31 melwitt concatenating not working
19:54:43 mriedem ok, was going to ping you on ^ as well
19:55:02 melwitt what you're saying makes sense though
19:55:44 melwitt interestingly recently I was trying to figure out how an instance could possibly be in both cell0 and cell1 in a single cell env, since that's happened to rdo cloud several times. but I found no way
19:56:05 melwitt so if you saw how that could be, lmk
19:59:02 dansmith mriedem: maybe?
19:59:19 melwitt so, we create instance mapping with cell_id=NULL in compute/api. so if we get to conductor and no build request found, we add a placeholder to the instance list
20:00:15 melwitt we don't create an instance record
20:01:02 mriedem right we'd skip those here https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/conductor/manager.py#L1391
20:01:29 dansmith yeah was just reading that loop
20:01:40 dansmith we assume that if instance is none that it's because we buried it already I think
20:01:42 mriedem while i'm looking at this code, we should probably make this fatal now :) https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/conductor/manager.py#L1238
20:01:58 melwitt we delete all the stuff if quota recheck fails. but if someone has requested a delete of the thing, wouldn't the compute/api delete the instance mapping? I can't remember where we do that for delete
20:02:03 mriedem dansmith: or it's already been deleted via the build request before we got that far
20:02:23 mriedem melwitt: no we don't delete the instance mapping when deleting the instance (or build request) in the api
20:02:41 mriedem that's why surya had to add it to the archive_deleted_rows cmd
20:02:51 dansmith mriedem: yeah, we get instance=None if the build request failed, do we get it for other reasons?
20:03:04 melwitt oh, ever? if we never do, then is it an issue?
20:03:06 dansmith ohh,
20:03:08 dansmith I see,
20:03:09 openstackgerrit Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420
20:03:18 dansmith we bury it right away if we get hostmappingnotfound
20:03:33 dansmith but if we get instance mapping not found, we don't bury, but instances.append(None)
20:03:40 dansmith the loop at the bottom assumes the former behavior
20:04:07 dansmith the latter looks like it was added for quota counting reasons
20:04:22 melwitt yeah, it was
20:04:26 mriedem dansmith: you mean build request not found
20:04:34 melwitt the append None thing
20:04:36 mriedem https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/conductor/manager.py#L1350
20:04:42 openstackgerrit Adam Spiers proposed openstack/nova master: Use fake flavor instead of empty dict in test https://review.opendev.org/662555
20:04:43 openstackgerrit Adam Spiers proposed openstack/nova master: Pass extra_specs to flavor in vif tests https://review.opendev.org/662556
20:04:43 openstackgerrit Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334
20:04:44 openstackgerrit Adam Spiers proposed openstack/nova master: Add element to guest config for AMD SEV https://review.opendev.org/636318
20:04:44 openstackgerrit Adam Spiers proposed openstack/nova master: Allow guest devices to include https://review.opendev.org/644564
20:04:45 openstackgerrit Adam Spiers proposed openstack/nova master: Detect that SEV is required and enable iommu for devices https://review.opendev.org/644565
20:04:45 openstackgerrit Adam Spiers proposed openstack/nova master: Use element when SEV is required https://review.opendev.org/662557
20:04:46 openstackgerrit Adam Spiers proposed openstack/nova master: Enable memory locking if SEV is requested https://review.opendev.org/662558
20:04:48 dansmith mriedem: yeah sorry
20:04:56 dansmith mriedem: yeah that was melwitt's main counting quotas patch
20:05:53 dansmith which added the instances.append(None) for the host not found bury case as well
20:06:04 dansmith https://github.com/openstack/nova/commit/5c90b25e49d47deb7dc6695333d9d5e46efe8665#diff-378a96ec6159d0a2f8ec7ab71bc3843bR946
20:06:40 dansmith although it looks like we were skipping those both anyway
20:07:11 mriedem yeah, and at some point in talking about https://bugs.launchpad.net/nova/+bug/1784074 we had talked about squashing those two for loops into one again, but i've lost track of the discussion about that (though i think i know how to find it)
20:07:12 openstack Launchpad bug 1784074 in OpenStack Compute (nova) "Instances end up with no cell assigned in instance_mappings" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
20:07:15 openstackgerrit Eric Fried proposed openstack/nova master: Clarify --before help text in nova manage https://review.opendev.org/661289
20:07:18 mriedem i'm not really signing up to do that refactor now though,
20:07:28 dansmith yeah
20:07:32 mriedem just thinking it makes sense for us to delete the instance mapping in this block https://github.com/openstack/nova/blob/74aebe0d4e5a978a40011e890aee9e70e98246c4/nova/conductor/manager.py#L1350
20:07:45 dansmith I think we should just bury in that case, no?
20:07:47 mriedem b/c if the BR is gone, and the instance isn't created in a cell yet, the instance mapping shouldn't exist either - it's orphaned
20:07:54 melwitt I've been wanting ("wanting") to do that refactor but, you know how it goes
20:07:55 mriedem burying it would bring it back

Earlier   Later