Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-15
20:36:45 efried rite
20:36:50 efried swhat I'm looking at now.
20:37:58 mriedem in the related ML thread for that patch we talked about making the API only proceed if the service was down but nacked that idea for some reason
20:44:10 mriedem fun it looks like my rechecks are being ignored
20:44:42 mriedem dan rechecked https://review.opendev.org/#/c/634832/ hours ago with no results and it's not queued and i just rechecked it and it's still not queued
20:44:47 mriedem fungi: ^
20:45:34 mriedem my guess is because the comment doesn't start with "recheck", it starts with "(3 comments)"
20:45:50 mriedem yup, now it's queued
20:45:54 mriedem is that new behavior?
20:46:48 fungi nope, it's just the way zuul is configured via a regular expression on the text of the comment event gerrit emits
20:47:15 mriedem could have sworn i've issued rechecks while leaving comments before, but can't say for certain
20:47:34 fungi if you leave a vote at the same time you add a recheck comment, it won't match the regex
20:48:16 mriedem ok
20:48:19 fungi https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L24
20:48:26 fungi that's the current regex
20:48:45 fungi i've noticed before that it ignores the recheck if i leave a vote with the same comment
20:49:34 fungi i've never dug in with a sample comment event to see if that regex could be extended to accommodate it
20:50:00 mriedem efried: are you working through bug 1841481 ?
20:50:00 openstack bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] https://launchpad.net/bugs/1841481 - Assigned to Matt Riedemann (mriedem)
20:50:20 efried mriedem: stale, but in my backlog
20:50:28 efried is it time to get back to it?
20:50:35 mriedem well, was just going to point out https://review.opendev.org/#/c/684840/ and above
20:50:37 efried I feel guilty any time I do real code.
20:50:54 efried there's always some f'in ptl thing that needs doing.
20:51:00 mriedem your series deals with the corrupt provider tree cache, mine deals with the corrupt RT.compute_nodes cache
20:51:23 efried so they need to be combined?
20:51:40 efried or at least reconciled
20:53:01 mriedem last i looked at yours i said,
20:53:02 mriedem "Simply dealing with the ResourceTracker.compute_nodes invalid cache (issue #1 in the bug report) resolves the issue assuming the ProviderTree associations are considered stale. If the associations are stale in the ProviderTree cache, we likely still have a problem which is what Eric's series here is dealing with (but Eric's series doesn't deal with the ResourceTracker.compute_nodes aspect of the bug)."
20:54:25 mriedem i'm not totally sure my functional recreate test hits all of the nuance with the provider tree cache but it does check _associations_stale
20:55:15 mriedem the actual RT.compute_nodes cache fix is simple https://review.opendev.org/#/c/684849/2/nova/compute/resource_tracker.py
20:55:26 mriedem don't cache the node unless RT._update is OK
21:04:05 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Skipping%20removal%20of%20allocations%20for%20deleted%20instances%3A%20Failed%20to%20retrieve%20allocations%20for%20resource%20provider%5C%22%20AND%20message%3A%5C%22No%20resource%20provider%20with%20uuid%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20project%3A%5C%22openstack%2Fironic%5C%22&from=7d
21:04:12 mriedem we do see it in ironic multinode jobs
21:08:10 mriedem maybe the functional test is too much in there, idk - could just do a simple unit test to make sure we don't save the node in RT.compute_nodes if _update fails like we did here https://review.opendev.org/#/c/675704/
21:14:12 openstackgerrit Dan Smith proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140
21:32:29 efried clearly I'll need to set aside some time to swap that all in
21:32:36 efried which isn't today unfortunately.
22:56:35 openstackgerrit Merged openstack/nova master: Add cache_images() to conductor https://review.opendev.org/687139
22:56:51 openstackgerrit Merged openstack/nova master: Fix legacy issues in filter migrations by user_id/project_id https://review.opendev.org/682198
23:10:45 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add negative test to delete server during cross-cell resize claim https://review.opendev.org/688832
23:16:56 mriedem drats, have to rebase the cross-cell resize series again
23:17:47 mriedem dansmith: we can talk about it tomorrow or whenever but i wrote that negative test you asked for https://review.opendev.org/688832 and it exposes a latent bug in how MigrationTask.rollback works, which i think affects same-cell resize as well wrt leaked allocations on the source host
23:17:56 mriedem but i'm basically done for the day as well
#openstack-nova - 2019-10-16
00:31:14 brinzhang good morning, nova.
00:31:50 brinzhang efried: good morning ^^
01:01:38 openstackgerrit Brin Zhang proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140
01:05:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.opendev.org/627890
01:05:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.opendev.org/634832
01:05:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.opendev.org/627891
01:05:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080
01:05:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646
01:05:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668
01:05:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684
01:05:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224
01:06:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253
01:06:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591
01:06:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592
01:06:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450
01:06:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451
01:06:03 openstackgerrit Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012
01:06:03 openstackgerrit Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013
01:06:04 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058
01:06:04 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.opendev.org/637070
01:06:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075
01:22:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140
02:11:39 openstackgerrit zhangyujun proposed openstack/nova master: Should not raise when restore power on failed https://review.opendev.org/624854
02:21:21 openstackgerrit Brin Zhang proposed openstack/nova master: Follow-up: Add delete_on_termination to volume-attach API https://review.opendev.org/680655
06:30:51 openstackgerrit Merged openstack/nova master: Update compute rpc version alias for train https://review.opendev.org/688418
07:22:39 openstackgerrit Brin Zhang proposed openstack/nova master: Follow-up: Add delete_on_termination to volume-attach API https://review.opendev.org/680655
07:27:42 gibi good morning
07:47:11 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add minor version [21] to the test_versions https://review.opendev.org/688599
08:06:50 LiangFang hi, where can I find the plan of Shanghai PTG? Thanks
08:07:02 LiangFang e.g. the topics
08:07:18 LiangFang how to add topic please?
08:15:41 melwitt LiangFang: http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010126.html
08:18:39 kashyap melwitt: Wauw, you're still awake?
08:20:08 LiangFang melwitt: thanks
08:23:37 melwitt kashyap: yeah :( going to bed soon
08:24:13 melwitt thanks o/
08:43:58 frickler KeithMnemonic: I haven't tested that scenario yet, but we're running the same setup, so I'd be interested in hearing more about your findings. If you discover a reproducible issue, it would probably be best to create a bug report on lp
09:13:12 openstackgerrit sunjiyun-inspur proposed openstack/nova master: first modify https://review.opendev.org/688884
09:31:30 openstackgerrit Huachang Wang proposed openstack/nova master: Set instance CPU policy to 'share' through image property https://review.opendev.org/688603
11:07:22 openstackgerrit wangfaxin proposed openstack/nova master: Fix misspell word https://review.opendev.org/688901
12:53:04 openstackgerrit Balazs Gibizer proposed openstack/nova master: Mask the token used to allow access to consoles https://review.opendev.org/220622
13:29:06 mriedem bauzas: another scenario where we could leak allocations https://bugs.launchpad.net/nova/+bug/1848343
13:29:07 openstack Launchpad bug 1848343 in OpenStack Compute (nova) "MigrationTask rollback can leak allocations for a deleted server" [Undecided,Triaged]
13:29:59 mriedem does your audit command detect the scenario that a resource provider has VCPU allocations but the instance consumer does not exist?
13:30:43 dansmith mriedem: guh, sorry about those last few issues, but thanks for fixing them
13:32:00 mriedem np
13:32:31 mriedem i reckon i'll spend some time today to write a recreate test for same-cell resize for that latent bug above
13:48:55 openstackgerrit Huachang Wang proposed openstack/nova master: [WIP] To create single NUMA node instance in function '_get_numa_topology_auto' https://review.opendev.org/688932
13:48:55 openstackgerrit Huachang Wang proposed openstack/nova master: [WIP] Assign and track instance pinning cpu through 'cpu_pinning' field https://review.opendev.org/688933
13:48:56 openstackgerrit Huachang Wang proposed openstack/nova master: [WIP] Add a new instance CPU allocation policy: mixed https://review.opendev.org/688934
13:48:57 openstackgerrit Huachang Wang proposed openstack/nova master: [WIP] virt/libvirt: Get host pin cpuset according instance cpu_pinning https://review.opendev.org/688935
13:48:57 openstackgerrit Huachang Wang proposed openstack/nova master: [WIP] metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
13:57:56 KeithMnemonic frickler i will let you know what i find out. so far it seems some sort of access issue with the backend rbd disk during an evacuation
14:06:30 openstackgerrit Dan Smith proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines https://review.opendev.org/687141
14:07:55 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Microversion 2.80: Add user_id/project_id to migration-list API https://review.opendev.org/675023
14:09:30 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Add aggregate-cache-images command and client routines https://review.opendev.org/687141

Earlier   Later