| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-15 | |||
| 20:24:07 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | |
| 20:24:13 | mriedem | dansmith: a few things on https://review.opendev.org/#/c/687140/ | |
| 20:24:17 | mriedem | did you intend to drop the release note? | |
| 20:25:17 | dansmith | mriedem: ah thanks. No, I tried to get all smart with the renumber on the microversion which caused me to lose several things.. it was a real timesaver | |
| 20:26:36 | KeithMnemonic | melwitt, mriedem, that patch from hemna finally verified. Thanks for all of you help. Reviews when someone gets time are appreciated in advance https://review.opendev.org/#/c/683008/ | |
| 20:27:33 | melwitt | thanks for the heads up | |
| 20:30:04 | efried | what's the ironic ring thing called? | |
| 20:30:33 | efried | rebalance puts your node in another.... "X"? | |
| 20:31:11 | efried | yeah, it's a "ring", I'm not too crazy. | |
| 20:31:20 | openstackgerrit | Dan Smith proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140 | |
| 20:31:22 | dansmith | mriedem: ^ | |
| 20:31:32 | dansmith | I gotta step away for a bit..got a raging headache | |
| 20:31:54 | dansmith | and no matter what my wife says, it is NOT because I emptied a whole can of Brakleen on my valve cover last night in the closed-up garage | |
| 20:32:56 | mriedem | some are saying fumes are good for the brain | |
| 20:33:01 | mriedem | efried: yeah hashring | |
| 20:33:13 | efried | thx | |
| 20:33:31 | KeithMnemonic | files accessibility /var/lib/nova/instances/... nova/virt/libvirt/driver.py:8893 " my guess is maybe a permission or something but was wondering if anyone ever ran into something like this | |
| 20:33:31 | KeithMnemonic | looking for some tips/suggestions on another odd issue I am investigating. This is Rocky with ceph backed instances. i.e instance boots from ceph directly. per https://docs.ceph.com/docs/master/rbd/rbd-openstack/ . When doing an evacuate from a compute that is powered off it fails with "Invalid state of instance files on shared storage" it looks like somewhere here it is failing on this "Checking instance | |
| 20:34:17 | KeithMnemonic | the instance ran fine of the source compute. my next step is to try and see if the same happens with a migrate | |
| 20:34:32 | KeithMnemonic | and other instances in the same ceph pool are running on the target | |
| 20:34:46 | KeithMnemonic | so both computes can talk to ceph | |
| 20:34:53 | efried | mriedem: do you have to disable a compute service before you delete it? | |
| 20:35:14 | mriedem | nope | |
| 20:35:29 | mriedem | you should stop the actual process though | |
| 20:35:44 | mriedem | see https://docs.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service | |
| 20:36:13 | mriedem | which is somewhat related to our old friend https://review.opendev.org/#/c/678100/ | |
| 20:36:29 | efried | right, so technically you could race service deletion with an instance operation. | |
| 20:36:34 | mriedem | yup | |
| 20:36:36 | efried | even though it means you were bad. | |
| 20:36:40 | mriedem | and we fail to delete the providers | |
| 20:36:45 | efried | rite | |
| 20:36:50 | efried | swhat I'm looking at now. | |
| 20:37:58 | mriedem | in the related ML thread for that patch we talked about making the API only proceed if the service was down but nacked that idea for some reason | |
| 20:44:10 | mriedem | fun it looks like my rechecks are being ignored | |
| 20:44:42 | mriedem | dan rechecked https://review.opendev.org/#/c/634832/ hours ago with no results and it's not queued and i just rechecked it and it's still not queued | |
| 20:44:47 | mriedem | fungi: ^ | |
| 20:45:34 | mriedem | my guess is because the comment doesn't start with "recheck", it starts with "(3 comments)" | |
| 20:45:50 | mriedem | yup, now it's queued | |
| 20:45:54 | mriedem | is that new behavior? | |
| 20:46:48 | fungi | nope, it's just the way zuul is configured via a regular expression on the text of the comment event gerrit emits | |
| 20:47:15 | mriedem | could have sworn i've issued rechecks while leaving comments before, but can't say for certain | |
| 20:47:34 | fungi | if you leave a vote at the same time you add a recheck comment, it won't match the regex | |
| 20:48:16 | mriedem | ok | |
| 20:48:19 | fungi | https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L24 | |
| 20:48:26 | fungi | that's the current regex | |
| 20:48:45 | fungi | i've noticed before that it ignores the recheck if i leave a vote with the same comment | |
| 20:49:34 | fungi | i've never dug in with a sample comment event to see if that regex could be extended to accommodate it | |
| 20:50:00 | openstack | bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] https://launchpad.net/bugs/1841481 - Assigned to Matt Riedemann (mriedem) | |
| 20:50:00 | mriedem | efried: are you working through bug 1841481 ? | |
| 20:50:20 | efried | mriedem: stale, but in my backlog | |
| 20:50:28 | efried | is it time to get back to it? | |
| 20:50:35 | mriedem | well, was just going to point out https://review.opendev.org/#/c/684840/ and above | |
| 20:50:37 | efried | I feel guilty any time I do real code. | |
| 20:50:54 | efried | there's always some f'in ptl thing that needs doing. | |
| 20:51:00 | mriedem | your series deals with the corrupt provider tree cache, mine deals with the corrupt RT.compute_nodes cache | |
| 20:51:23 | efried | so they need to be combined? | |
| 20:51:40 | efried | or at least reconciled | |
| 20:53:01 | mriedem | last i looked at yours i said, | |
| 20:53:02 | mriedem | "Simply dealing with the ResourceTracker.compute_nodes invalid cache (issue #1 in the bug report) resolves the issue assuming the ProviderTree associations are considered stale. If the associations are stale in the ProviderTree cache, we likely still have a problem which is what Eric's series here is dealing with (but Eric's series doesn't deal with the ResourceTracker.compute_nodes aspect of the bug)." | |
| 20:54:25 | mriedem | i'm not totally sure my functional recreate test hits all of the nuance with the provider tree cache but it does check _associations_stale | |
| 20:55:15 | mriedem | the actual RT.compute_nodes cache fix is simple https://review.opendev.org/#/c/684849/2/nova/compute/resource_tracker.py | |
| 20:55:26 | mriedem | don't cache the node unless RT._update is OK | |
| 21:04:05 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Skipping%20removal%20of%20allocations%20for%20deleted%20instances%3A%20Failed%20to%20retrieve%20allocations%20for%20resource%20provider%5C%22%20AND%20message%3A%5C%22No%20resource%20provider%20with%20uuid%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20project%3A%5C%22openstack%2Fironic%5C%22&from=7d | |
| 21:04:12 | mriedem | we do see it in ironic multinode jobs | |
| 21:08:10 | mriedem | maybe the functional test is too much in there, idk - could just do a simple unit test to make sure we don't save the node in RT.compute_nodes if _update fails like we did here https://review.opendev.org/#/c/675704/ | |
| 21:14:12 | openstackgerrit | Dan Smith proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140 | |
| 21:32:29 | efried | clearly I'll need to set aside some time to swap that all in | |
| 21:32:36 | efried | which isn't today unfortunately. | |
| 22:56:35 | openstackgerrit | Merged openstack/nova master: Add cache_images() to conductor https://review.opendev.org/687139 | |
| 22:56:51 | openstackgerrit | Merged openstack/nova master: Fix legacy issues in filter migrations by user_id/project_id https://review.opendev.org/682198 | |
| 23:10:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add negative test to delete server during cross-cell resize claim https://review.opendev.org/688832 | |
| 23:16:56 | mriedem | drats, have to rebase the cross-cell resize series again | |
| 23:17:47 | mriedem | dansmith: we can talk about it tomorrow or whenever but i wrote that negative test you asked for https://review.opendev.org/688832 and it exposes a latent bug in how MigrationTask.rollback works, which i think affects same-cell resize as well wrt leaked allocations on the source host | |
| 23:17:56 | mriedem | but i'm basically done for the day as well | |
| #openstack-nova - 2019-10-16 | |||
| 00:31:14 | brinzhang | good morning, nova. | |
| 00:31:50 | brinzhang | efried: good morning ^^ | |
| 01:01:38 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140 | |
| 01:05:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.opendev.org/634832 | |
| 01:05:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.opendev.org/627890 | |
| 01:05:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080 | |
| 01:05:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.opendev.org/627891 | |
| 01:05:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668 | |
| 01:05:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646 | |
| 01:05:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | |
| 01:05:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684 | |
| 01:06:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253 | |
| 01:06:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592 | |
| 01:06:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591 | |
| 01:06:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451 | |
| 01:06:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450 | |
| 01:06:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013 | |
| 01:06:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012 | |
| 01:06:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.opendev.org/637070 | |
| 01:06:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058 | |
| 01:06:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075 | |
| 01:22:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140 | |
| 02:11:39 | openstackgerrit | zhangyujun proposed openstack/nova master: Should not raise when restore power on failed https://review.opendev.org/624854 | |
| 02:21:21 | openstackgerrit | Brin Zhang proposed openstack/nova master: Follow-up: Add delete_on_termination to volume-attach API https://review.opendev.org/680655 | |
| 06:30:51 | openstackgerrit | Merged openstack/nova master: Update compute rpc version alias for train https://review.opendev.org/688418 | |
| 07:22:39 | openstackgerrit | Brin Zhang proposed openstack/nova master: Follow-up: Add delete_on_termination to volume-attach API https://review.opendev.org/680655 | |