Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-15
19:35:35 efried okay. be warned I'm likely to be -1 on a solution involving retrying.
19:36:11 mriedem i'm not retrying
19:36:19 efried cool
19:44:24 artom efried, mriedem, I... don't remember?
19:47:09 artom I went IRC log diving
19:47:10 artom Jul 16 10:07:27 efried, hah, see where else that error popped up: http://logs.openstack.org/09/666409/8/check/tempest-full-py3/38bf84e/job-output.txt#_2019-07-14_17_19_10_677555
19:47:19 artom Jul 16 10:08:33 <artom> efried, there aren't that many hits, but yeah, our theory from last night is pretty much confirmed
19:47:57 efried okay, well anyway, mriedem is working on a fix.
19:48:18 artom Jul 15 16:05:11 artom: This is interesting, the failure on that skip patch http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/testr_results.html.gz
19:48:37 artom So, I think we first hit this in CI back when we were testing that hybrid plug revert resize thing
19:48:40 artom So yeah, CI only
19:48:46 efried ack
19:48:49 artom That we know of, anyways
19:49:02 mriedem it's an extremely tight window between GETing the allocations and PUTing them back with allocations={}
19:49:04 efried I would think it would be fairly tough to hit otherwise
19:49:19 mriedem tempest creates a server and then immediately deletes it
19:49:21 mriedem when we hit this
19:49:21 efried yeah, there's basically nothing in that window.
19:51:06 artom Right, it's coming back
19:51:10 artom It's a specific tempest tes
19:56:21 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add force kwarg to delete_allocation_for_instance https://review.opendev.org/688802
19:56:30 mriedem i'm a bit troubled by the various places that should use force=True in ^
19:56:50 mriedem meaning, wtf did we change the behavior of delete_allocation_for_instance in the first place? it feels like we did because we could.
20:00:25 melwitt efried, gibi, mriedem: can confirm I ran into the bug but didn't file one (and I saw in the backscroll that artom filed one)
20:00:59 efried mriedem: agreed, given the change we actually ended up with, we would have been better off not doing it.
20:03:40 mriedem i've annoted that places i'm using force=True now to try and justify the reasoning
20:03:45 mriedem *the places
20:03:58 mriedem maybe gibi can say "no we shouldn't force b/c resource requests" or something, idk
20:04:04 efried *for spawn, we should always be using NULL, and if we get a conflict, it means we're racing with some other operation (delete, resize, etc) and should abort the spawn
20:04:04 efried mriedem: if we're going to use generation-based allocation management at all, we should really be doing the GET early in the flow (except for spawn*) so that the race window actually means what it should.
20:04:05 mriedem *port resource requests and nested allocations
20:04:26 efried but the existing use of generations is worse than useless.
20:04:50 efried (I should have objected harder instead of +2ing that patch on the promise of "we'll improve it later")
20:08:35 mriedem yeah idk, i don't remember being very involved in this, in irc, or the meetings. i didn't comment in the ML thread and i didn't get into the "why"s in the patch, just reviewed it, likely to keep the series moving and trust gibi and everyone else's decisions on this (since you, chris, gibi and jay were all involved)
20:09:04 mriedem going back to the feeling of "we did it because we could"
20:11:35 efried we did it because we *should*, but we should do it right, and were planning on doing so eventually.
20:12:50 mriedem that makes more sense in the other patches in that series which dealt with PUTing allocations with updates rather than removing them when deleting a server
20:24:06 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341
20:24:07 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522
20:24:07 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029
20:24:13 mriedem dansmith: a few things on https://review.opendev.org/#/c/687140/
20:24:17 mriedem did you intend to drop the release note?
20:25:17 dansmith mriedem: ah thanks. No, I tried to get all smart with the renumber on the microversion which caused me to lose several things.. it was a real timesaver
20:26:36 KeithMnemonic melwitt, mriedem, that patch from hemna finally verified. Thanks for all of you help. Reviews when someone gets time are appreciated in advance https://review.opendev.org/#/c/683008/
20:27:33 melwitt thanks for the heads up
20:30:04 efried what's the ironic ring thing called?
20:30:33 efried rebalance puts your node in another.... "X"?
20:31:11 efried yeah, it's a "ring", I'm not too crazy.
20:31:20 openstackgerrit Dan Smith proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140
20:31:22 dansmith mriedem: ^
20:31:32 dansmith I gotta step away for a bit..got a raging headache
20:31:54 dansmith and no matter what my wife says, it is NOT because I emptied a whole can of Brakleen on my valve cover last night in the closed-up garage
20:32:56 mriedem some are saying fumes are good for the brain
20:33:01 mriedem efried: yeah hashring
20:33:13 efried thx
20:33:31 KeithMnemonic files accessibility /var/lib/nova/instances/... nova/virt/libvirt/driver.py:8893 " my guess is maybe a permission or something but was wondering if anyone ever ran into something like this
20:33:31 KeithMnemonic looking for some tips/suggestions on another odd issue I am investigating. This is Rocky with ceph backed instances. i.e instance boots from ceph directly. per https://docs.ceph.com/docs/master/rbd/rbd-openstack/ . When doing an evacuate from a compute that is powered off it fails with "Invalid state of instance files on shared storage" it looks like somewhere here it is failing on this "Checking instance
20:34:17 KeithMnemonic the instance ran fine of the source compute. my next step is to try and see if the same happens with a migrate
20:34:32 KeithMnemonic and other instances in the same ceph pool are running on the target
20:34:46 KeithMnemonic so both computes can talk to ceph
20:34:53 efried mriedem: do you have to disable a compute service before you delete it?
20:35:14 mriedem nope
20:35:29 mriedem you should stop the actual process though
20:35:44 mriedem see https://docs.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service
20:36:13 mriedem which is somewhat related to our old friend https://review.opendev.org/#/c/678100/
20:36:29 efried right, so technically you could race service deletion with an instance operation.
20:36:34 mriedem yup
20:36:36 efried even though it means you were bad.
20:36:40 mriedem and we fail to delete the providers
20:36:45 efried rite
20:36:50 efried swhat I'm looking at now.
20:37:58 mriedem in the related ML thread for that patch we talked about making the API only proceed if the service was down but nacked that idea for some reason
20:44:10 mriedem fun it looks like my rechecks are being ignored
20:44:42 mriedem dan rechecked https://review.opendev.org/#/c/634832/ hours ago with no results and it's not queued and i just rechecked it and it's still not queued
20:44:47 mriedem fungi: ^
20:45:34 mriedem my guess is because the comment doesn't start with "recheck", it starts with "(3 comments)"
20:45:50 mriedem yup, now it's queued
20:45:54 mriedem is that new behavior?
20:46:48 fungi nope, it's just the way zuul is configured via a regular expression on the text of the comment event gerrit emits
20:47:15 mriedem could have sworn i've issued rechecks while leaving comments before, but can't say for certain
20:47:34 fungi if you leave a vote at the same time you add a recheck comment, it won't match the regex
20:48:16 mriedem ok
20:48:19 fungi https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L24
20:48:26 fungi that's the current regex
20:48:45 fungi i've noticed before that it ignores the recheck if i leave a vote with the same comment
20:49:34 fungi i've never dug in with a sample comment event to see if that regex could be extended to accommodate it
20:50:00 openstack bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] https://launchpad.net/bugs/1841481 - Assigned to Matt Riedemann (mriedem)
20:50:00 mriedem efried: are you working through bug 1841481 ?
20:50:20 efried mriedem: stale, but in my backlog
20:50:28 efried is it time to get back to it?
20:50:35 mriedem well, was just going to point out https://review.opendev.org/#/c/684840/ and above
20:50:37 efried I feel guilty any time I do real code.
20:50:54 efried there's always some f'in ptl thing that needs doing.
20:51:00 mriedem your series deals with the corrupt provider tree cache, mine deals with the corrupt RT.compute_nodes cache
20:51:23 efried so they need to be combined?
20:51:40 efried or at least reconciled
20:53:01 mriedem last i looked at yours i said,
20:53:02 mriedem "Simply dealing with the ResourceTracker.compute_nodes invalid cache (issue #1 in the bug report) resolves the issue assuming the ProviderTree associations are considered stale. If the associations are stale in the ProviderTree cache, we likely still have a problem which is what Eric's series here is dealing with (but Eric's series doesn't deal with the ResourceTracker.compute_nodes aspect of the bug)."
20:54:25 mriedem i'm not totally sure my functional recreate test hits all of the nuance with the provider tree cache but it does check _associations_stale
20:55:15 mriedem the actual RT.compute_nodes cache fix is simple https://review.opendev.org/#/c/684849/2/nova/compute/resource_tracker.py

Earlier   Later