Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-30
19:58:39 tzumainn mriedem, haha, yep - I started out talking with some ironic folks, and confusion all around has led me here : )
19:58:55 tzumainn thanks for the information, I really appreciate it!
19:59:30 mriedem maybe related to https://review.openstack.org/#/c/587922/ ?
20:00:19 jungleboyj mriedem: I don't have a separate one. Asked people to mark the topics and then I was going to collect them up.
20:00:22 mriedem remove_node should likely be in the same semaphore as _update_available_resource_for_node
20:00:52 mriedem i guess i already said that https://review.openstack.org/#/c/587922/2/nova/compute/resource_tracker.py
20:01:21 mriedem so self.old_resources will default a ComputeNode object if an entry isn't in the dict...
20:02:25 zigo mriedem: Should I try your patch at https://review.openstack.org/#/c/598176/ and report the result?
20:02:45 melwitt does self.compute_nodes refer to self.old_resources at all?
20:03:01 mriedem they are compared in _resource_change
20:03:11 mriedem to determine if we should call ComputeNode.save()
20:03:28 mriedem zigo: we aren't going to ship that revert i don't think so probably would be a waste of your time
20:04:13 zigo mriedem: If it's only a temporary fix that I can use to validate all of Rocky, that's nice already, then I can still remove the patch...
20:04:34 zigo Hum...
20:04:39 zigo It doesn't apply at all anyway.
20:05:12 zigo mriedem: This wasn't in rocky.
20:05:16 mriedem cdent: melwitt: was also wondering if this somehow is contributing to the allocation ratio bug https://review.openstack.org/#/c/518294/
20:05:19 mriedem but that was in queens
20:05:27 mriedem zigo: right
20:05:56 melwitt ack
20:06:44 zigo mriedem: Could you see something with the added logs in https://review.openstack.org/#/c/597175/ ?
20:06:54 zigo Both your patches were added there ...
20:06:58 zigo (the ones for logging...)
20:08:07 mriedem i added more debug logs this morning after a recreate in the xen ci, was just about to check those results
20:08:42 mriedem tzumainn: you might want to report a nova bug regardless so we don't lose track of what you hit
20:08:52 mriedem i'm not sure *how* you're hitting it, but that seems to be the theme this week with all bugs
20:09:00 mriedem "how in the hell is this even possible?"
20:09:12 tzumainn mriedem, hahaha - okay, I'll do that, thanks!
20:09:58 openstackgerrit Merged openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034
20:10:00 mriedem dansmith: is it weird that ComputeNode.save() doesn't check to see if 'id' is changed?
20:10:01 mriedem or not set?
20:10:31 dansmith create or save?
20:10:32 mriedem i guess if it weren't set we'd blow up
20:10:33 mriedem save
20:10:36 melwitt I was thinking save() would just fail if there's no id, because then how could it find the thing to update
20:10:46 dansmith not sure we usually check that it's set on save, but we could
20:11:01 dansmith on create we usually check to avoid re-create
20:11:03 mriedem melwitt: right we'd blow up if id wasn't set on save
20:11:07 melwitt yeah
20:11:08 mriedem yeah we check on create
20:11:18 mriedem i was wondering if id changed though and we saved
20:11:51 mriedem i guess we don't really check for that anywhere
20:11:54 dansmith we pop id out of the changes,
20:12:03 melwitt even if we did, that doesn't explain the complete lack of an 'id' on a ComputeNode object in self.compute_nodes
20:12:04 dansmith but I guess we try to save anyway even if that was the only thing
20:12:50 melwitt I still don't see how we could lose that, even with the self.old_resources compare
20:13:23 dansmith if id is actually not set then it's either an object created with no id (not from the db) or someone del'd it off an objet
20:13:24 mriedem i don't either, i was wondering if we were getting a blank ComputeNode from old_resources which uses defaultdict and somehow shoved that blank one into self.compute_nodes
20:13:25 mriedem but we don't
20:13:51 melwitt yeah
20:16:17 mriedem there is a very small window where self.compute_nodes could have a ComputeNode in it without an id
20:16:20 mriedem in _init_compute_node
20:16:21 mriedem self.compute_nodes[nodename] = cn
20:16:21 mriedem cn.create()
20:16:25 mriedem b/c cn.create() is what sets the id
20:16:46 mriedem but, that code is all in a lock on the same host when we call update_available_resource
20:16:52 mriedem so not sure how anything could race and hit that
20:16:54 melwitt yeah, I wondered about that
20:17:46 mriedem tzumainn: did you by chance have multiple nova-compute services running on the same host?
20:17:59 mriedem no that still wouldn't do this
20:18:05 mriedem b/c the compute nodes map is in memory
20:18:19 mriedem i give up
20:18:20 tzumainn mriedem, I only have one, in any case
20:18:27 tzumainn mriedem, no worries, thanks for taking a look : )
20:20:16 mriedem so uh, the xen ci passed this time with the debug patch https://review.openstack.org/#/c/597613/
20:20:21 mriedem that's nice and consistent
20:23:46 melwitt great
20:26:28 melwitt keeping in the theme of Impossible Bug Week
20:26:29 Sundar melwitt: I am proposing a session for Cyborg/Nova to sort out os-acc. The main session could be in Cyborg time (Mon/Tue) and hopefully placement folks would attend. It may be helpful to get some time on Nova schedule as well to get everybody on board. What do you think?
20:26:52 cdent sorry mriedem, I'm sort of afk: if it was was that patch from back in queens, then the twisted narrow passageways are way twisted and have a lot of explaining to do to say why it's only showing up now
20:29:54 melwitt Sundar: do you think we need two sessions? we could just join the monday or tuesday session since nova doesn't start until wednesday
20:29:55 cdent mriedem: can you think of a way we could model the issue in a functional test? the feedback latency is hurting my head
20:30:50 mriedem cdent: given the xen ci is now passing, i don't really know
20:31:03 mriedem and why this isn't failing with the libvirt driver in master CI, again don't know
20:31:10 cdent fun fun
20:31:12 mriedem efried: we now have a new ERROR in the n-cpu logs on startup http://logs.openstack.org/98/584598/21/check/tempest-full/85acbda/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Aug_29_21_43_10_675029
20:31:22 mriedem efried: b/c we hit ^ before the resource provider is created
20:31:50 mriedem new from https://review.openstack.org/#/c/584598/
20:32:17 dansmith hmm, I got that disk not found one the other day locally
20:32:26 dansmith I thought it was residue from my old machine
20:33:04 dansmith is that because we're re-raising too?
20:34:18 mriedem https://bugs.launchpad.net/nova/+bug/1789998
20:34:18 openstack Launchpad bug 1789998 in OpenStack Compute (nova) "ResourceProviderAllocationRetrievalFailed ERROR log message on fresh n-cpu startup" [Low,Triaged]
20:34:50 mriedem the DiskNotFound during periodic is usually b/c we're deleting an instance on that host at the same time
20:35:11 mriedem http://logs.openstack.org/98/584598/21/check/tempest-full/85acbda/controller/logs/screen-n-cpu.txt.gz#_Aug_29_21_51_13_055225
20:35:14 dansmith okay this was on startup for me, which prevented it from every reporting initial inventory to placement
20:35:16 mriedem right before the DiskNotFound
20:35:25 mriedem Aug 29 21:51:13.055225 ubuntu-xenial-rax-iad-0001643010 nova-compute[16853]: INFO nova.virt.libvirt.driver [None req-af17b138-d942-4bbf-bf67-4d03b492ed3c tempest-ImagesTestJSON-1980657675 tempest-ImagesTestJSON-1980657675] [instance: 21227895-e216-463f-8e76-998fe637bab2] Deletion of /opt/stack/data/nova/instances/21227895-e216-463f-8e76-998fe637bab2_del complete
20:35:28 dansmith but I had what I think was a stale disk image in my instances directory
20:35:34 mriedem Aug 29 21:51:13.167422 ubuntu-xenial-rax-iad-0001643010 nova-compute[16853]: ERROR nova.compute.manager [None req-6f4285cc-ce12-4c29-b879-99fdaaae59be None None] Error updating resources for node ubuntu-xenial-rax-iad-0001643010.: DiskNotFound: No disk at /opt/stack/data/nova/instances/21227895-e216-463f-8e76-998fe637bab2/disk
20:35:42 dansmith nuking that fixed it, but I just had to do one more thing and then killed it all
20:35:46 mriedem yeah on startup would be a different weirdness
20:37:39 Sundar melwitt: If we can settle everything in one session, that would be great. Given the quantum of reviews, we want to make sure that there is enough convergence to close the spec https://review.openstack.org/#/c/577438 after the PTG.
20:38:02 Sundar Could we keep a 30-minute placeholder?
20:40:31 mriedem so on the 2nd go around in the RT.update_available_resource in this xen CI run, we see the RT say the tracked compute node has changed:
20:40:33 mriedem http://paste.openstack.org/show/729180/
20:40:44 mriedem b/c of _copy_resources updating the *_allocation_ratio values to 0.0
20:42:13 mriedem Aug 30 08:03:09.458344 dsvm-devstack-citrix-lon-nodepool-1379396 nova-compute[24292]: INFO nova.compute.resource_tracker [None req-9b1b9924-b89e-4a03-9a69-c9fff17594e3 None None] ComputeNode.cpu_allocation_ratio changed from 16.0 to 0.0 in _copy_resources.
20:44:50 melwitt Sundar: 30-minute placeholder for monday or tuesday?
20:46:05 melwitt Sundar: also, I think we should have the nova bits of the interaction proposed to nova-specs for review, that way we can more easily organize our review on that part

Earlier   Later