Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-15
20:04:04 efried mriedem: if we're going to use generation-based allocation management at all, we should really be doing the GET early in the flow (except for spawn*) so that the race window actually means what it should.
20:04:04 efried *for spawn, we should always be using NULL, and if we get a conflict, it means we're racing with some other operation (delete, resize, etc) and should abort the spawn
20:04:05 mriedem *port resource requests and nested allocations
20:04:26 efried but the existing use of generations is worse than useless.
20:04:50 efried (I should have objected harder instead of +2ing that patch on the promise of "we'll improve it later")
20:08:35 mriedem yeah idk, i don't remember being very involved in this, in irc, or the meetings. i didn't comment in the ML thread and i didn't get into the "why"s in the patch, just reviewed it, likely to keep the series moving and trust gibi and everyone else's decisions on this (since you, chris, gibi and jay were all involved)
20:09:04 mriedem going back to the feeling of "we did it because we could"
20:11:35 efried we did it because we *should*, but we should do it right, and were planning on doing so eventually.
20:12:50 mriedem that makes more sense in the other patches in that series which dealt with PUTing allocations with updates rather than removing them when deleting a server
20:24:06 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341
20:24:07 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029
20:24:07 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522
20:24:13 mriedem dansmith: a few things on https://review.opendev.org/#/c/687140/
20:24:17 mriedem did you intend to drop the release note?
20:25:17 dansmith mriedem: ah thanks. No, I tried to get all smart with the renumber on the microversion which caused me to lose several things.. it was a real timesaver
20:26:36 KeithMnemonic melwitt, mriedem, that patch from hemna finally verified. Thanks for all of you help. Reviews when someone gets time are appreciated in advance https://review.opendev.org/#/c/683008/
20:27:33 melwitt thanks for the heads up
20:30:04 efried what's the ironic ring thing called?
20:30:33 efried rebalance puts your node in another.... "X"?
20:31:11 efried yeah, it's a "ring", I'm not too crazy.
20:31:20 openstackgerrit Dan Smith proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140
20:31:22 dansmith mriedem: ^
20:31:32 dansmith I gotta step away for a bit..got a raging headache
20:31:54 dansmith and no matter what my wife says, it is NOT because I emptied a whole can of Brakleen on my valve cover last night in the closed-up garage
20:32:56 mriedem some are saying fumes are good for the brain
20:33:01 mriedem efried: yeah hashring
20:33:13 efried thx
20:33:31 KeithMnemonic looking for some tips/suggestions on another odd issue I am investigating. This is Rocky with ceph backed instances. i.e instance boots from ceph directly. per https://docs.ceph.com/docs/master/rbd/rbd-openstack/ . When doing an evacuate from a compute that is powered off it fails with "Invalid state of instance files on shared storage" it looks like somewhere here it is failing on this "Checking instance
20:33:31 KeithMnemonic files accessibility /var/lib/nova/instances/... nova/virt/libvirt/driver.py:8893 " my guess is maybe a permission or something but was wondering if anyone ever ran into something like this
20:34:17 KeithMnemonic the instance ran fine of the source compute. my next step is to try and see if the same happens with a migrate
20:34:32 KeithMnemonic and other instances in the same ceph pool are running on the target
20:34:46 KeithMnemonic so both computes can talk to ceph
20:34:53 efried mriedem: do you have to disable a compute service before you delete it?
20:35:14 mriedem nope
20:35:29 mriedem you should stop the actual process though
20:35:44 mriedem see https://docs.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service
20:36:13 mriedem which is somewhat related to our old friend https://review.opendev.org/#/c/678100/
20:36:29 efried right, so technically you could race service deletion with an instance operation.
20:36:34 mriedem yup
20:36:36 efried even though it means you were bad.
20:36:40 mriedem and we fail to delete the providers
20:36:45 efried rite
20:36:50 efried swhat I'm looking at now.
20:37:58 mriedem in the related ML thread for that patch we talked about making the API only proceed if the service was down but nacked that idea for some reason
20:44:10 mriedem fun it looks like my rechecks are being ignored
20:44:42 mriedem dan rechecked https://review.opendev.org/#/c/634832/ hours ago with no results and it's not queued and i just rechecked it and it's still not queued
20:44:47 mriedem fungi: ^
20:45:34 mriedem my guess is because the comment doesn't start with "recheck", it starts with "(3 comments)"
20:45:50 mriedem yup, now it's queued
20:45:54 mriedem is that new behavior?
20:46:48 fungi nope, it's just the way zuul is configured via a regular expression on the text of the comment event gerrit emits
20:47:15 mriedem could have sworn i've issued rechecks while leaving comments before, but can't say for certain
20:47:34 fungi if you leave a vote at the same time you add a recheck comment, it won't match the regex
20:48:16 mriedem ok
20:48:19 fungi https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L24
20:48:26 fungi that's the current regex
20:48:45 fungi i've noticed before that it ignores the recheck if i leave a vote with the same comment
20:49:34 fungi i've never dug in with a sample comment event to see if that regex could be extended to accommodate it
20:50:00 mriedem efried: are you working through bug 1841481 ?
20:50:00 openstack bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] https://launchpad.net/bugs/1841481 - Assigned to Matt Riedemann (mriedem)
20:50:20 efried mriedem: stale, but in my backlog
20:50:28 efried is it time to get back to it?
20:50:35 mriedem well, was just going to point out https://review.opendev.org/#/c/684840/ and above
20:50:37 efried I feel guilty any time I do real code.
20:50:54 efried there's always some f'in ptl thing that needs doing.
20:51:00 mriedem your series deals with the corrupt provider tree cache, mine deals with the corrupt RT.compute_nodes cache
20:51:23 efried so they need to be combined?
20:51:40 efried or at least reconciled
20:53:01 mriedem last i looked at yours i said,
20:53:02 mriedem "Simply dealing with the ResourceTracker.compute_nodes invalid cache (issue #1 in the bug report) resolves the issue assuming the ProviderTree associations are considered stale. If the associations are stale in the ProviderTree cache, we likely still have a problem which is what Eric's series here is dealing with (but Eric's series doesn't deal with the ResourceTracker.compute_nodes aspect of the bug)."
20:54:25 mriedem i'm not totally sure my functional recreate test hits all of the nuance with the provider tree cache but it does check _associations_stale
20:55:15 mriedem the actual RT.compute_nodes cache fix is simple https://review.opendev.org/#/c/684849/2/nova/compute/resource_tracker.py
20:55:26 mriedem don't cache the node unless RT._update is OK
21:04:05 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Skipping%20removal%20of%20allocations%20for%20deleted%20instances%3A%20Failed%20to%20retrieve%20allocations%20for%20resource%20provider%5C%22%20AND%20message%3A%5C%22No%20resource%20provider%20with%20uuid%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20project%3A%5C%22openstack%2Fironic%5C%22&from=7d
21:04:12 mriedem we do see it in ironic multinode jobs
21:08:10 mriedem maybe the functional test is too much in there, idk - could just do a simple unit test to make sure we don't save the node in RT.compute_nodes if _update fails like we did here https://review.opendev.org/#/c/675704/
21:14:12 openstackgerrit Dan Smith proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140
21:32:29 efried clearly I'll need to set aside some time to swap that all in
21:32:36 efried which isn't today unfortunately.
22:56:35 openstackgerrit Merged openstack/nova master: Add cache_images() to conductor https://review.opendev.org/687139
22:56:51 openstackgerrit Merged openstack/nova master: Fix legacy issues in filter migrations by user_id/project_id https://review.opendev.org/682198
23:10:45 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add negative test to delete server during cross-cell resize claim https://review.opendev.org/688832
23:16:56 mriedem drats, have to rebase the cross-cell resize series again
23:17:47 mriedem dansmith: we can talk about it tomorrow or whenever but i wrote that negative test you asked for https://review.opendev.org/688832 and it exposes a latent bug in how MigrationTask.rollback works, which i think affects same-cell resize as well wrt leaked allocations on the source host
23:17:56 mriedem but i'm basically done for the day as well
#openstack-nova - 2019-10-16
00:31:14 brinzhang good morning, nova.
00:31:50 brinzhang efried: good morning ^^
01:01:38 openstackgerrit Brin Zhang proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140
01:05:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.opendev.org/627890
01:05:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.opendev.org/634832
01:05:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.opendev.org/627891
01:05:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080
01:05:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646
01:05:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668
01:05:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684
01:05:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224
01:06:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253
01:06:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591
01:06:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592
01:06:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450

Earlier   Later