| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-07 | |||
| 12:18:30 | gibi | making the bdm.destroy even more delayed | |
| 12:18:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Merge 'test_hypervisor_status' into 'test_hypervisors' https://review.opendev.org/c/openstack/nova/+/764039 | |
| 12:18:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Drop statistics-style fields from os-hypervisors https://review.opendev.org/c/openstack/nova/+/764040 | |
| 12:18:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Change default policy for '/os-hypervisors' https://review.opendev.org/c/openstack/nova/+/765798 | |
| 12:18:57 | sean-k-mooney | well its load related, so 10 seconds was just a guess based on what we were seeing | |
| 12:19:10 | sean-k-mooney | but a slighly slower node could still nit it | |
| 12:19:37 | sean-k-mooney | it may be something else but it sounds related | |
| 12:21:35 | lyarwood | gmann: ^ btw, smells like an issue with https://review.opendev.org/c/openstack/tempest/+/764407 | |
| 12:22:02 | lyarwood | I don't think this is an issue with https://review.opendev.org/c/openstack/nova/+/755799 fwiw sean-k-mooney, at least at first glance | |
| 12:22:36 | lyarwood | gmann: ignore that ping, the assertion is in the test itself, not cleanup. | |
| 12:23:24 | sean-k-mooney | that test chagne merged 9 days ago, gibi are you still seeing it after that? | |
| 12:23:51 | gibi | sean-k-mooney: yepp it is actively happening | |
| 12:24:00 | gibi | logstash link is in the bugreport | |
| 12:24:29 | gibi | but I now 70% confident that the cinder volume becomes available _befor_ nova deletes the BDM | |
| 12:24:35 | gibi | so the tempest test is simply racing on this | |
| 12:24:43 | lyarwood | that shouldn't be possible | |
| 12:25:25 | gibi | lyarwood: I think this is the relevant code path https://github.com/openstack/nova/blob/97729aabaa7768e1af2fa4a0bdda0df7ad598426/nova/compute/manager.py#L7071-L7086 | |
| 12:25:49 | gibi | the bdm.destory() is the last thing nova does during volume detach | |
| 12:26:50 | lyarwood | gibi: right and that point we've already detached from the instance in libvirt and detached the volume from the instance via c-api | |
| 12:26:59 | gibi | yepp | |
| 12:27:21 | gibi | so there is a timewindow where the volume is available in cinder, but still has a BDM in nova | |
| 12:27:36 | gibi | hence the nova volume attachment list will show the attachment | |
| 12:27:44 | lyarwood | ah right | |
| 12:27:48 | lyarwood | it's using servers_client | |
| 12:27:53 | lyarwood | instead of volumes_client | |
| 12:28:07 | lyarwood | so we could race the bdm.destroy in the db | |
| 12:28:26 | sean-k-mooney | for what its worth that might also be an issue with shift on stack | |
| 12:28:52 | sean-k-mooney | they were reporting similar behavior where the volumes were atached in nova but not in cinder | |
| 12:29:32 | sean-k-mooney | although in there case i think that was a persitent state | |
| 12:29:40 | sean-k-mooney | not just an interval where it was racy | |
| 12:30:30 | gibi | sean-k-mooney: we will see if this is a persistent bug after I propose a re-query in the tempest test | |
| 12:30:44 | gibi | if the re-query fails then we know we have a problem where the bdm is not deleted | |
| 12:31:00 | lyarwood | gibi: the way to fix this is to add a waiter for volume attachments to the servers_clienty | |
| 12:31:02 | lyarwood | client* | |
| 12:31:18 | gibi | lyarwood: yes | |
| 12:31:20 | sean-k-mooney | i prefer clienty | |
| 12:31:21 | gibi | that is my plan | |
| 12:31:44 | lyarwood | cool, let me know when you've posted a review and I'll happily review | |
| 12:31:45 | sean-k-mooney | espcially the clinety thingy | |
| 12:31:51 | gibi | sean-k-mooney: :) | |
| 12:31:55 | gibi | lyarwood: ack | |
| 12:33:52 | lyarwood | urgh | |
| 12:33:54 | lyarwood | I see | |
| 12:54:06 | openstackgerrit | Lee Yarwood proposed openstack/nova master: tools: Allow check-cherry-picks.sh to be disabled by an env var https://review.opendev.org/c/openstack/nova/+/765802 | |
| 12:54:36 | lyarwood | ^ workaround for our downstream CI so we can use jobs and tox envs directly from upstream for a change | |
| 13:02:56 | sean-k-mooney | instead of just using sed to remove it | |
| 13:03:11 | sean-k-mooney | that is cleaner then our current hack i guess | |
| 13:14:40 | teoobo_ | gibi: I am working on this bug (https://bugs.launchpad.net/nova/+bug/1751784) but I cannot reproduce it in my devstack | |
| 13:14:40 | openstack | Launchpad bug 1751784 in OpenStack Compute (nova) "Add more LOG.debug in nova.virt.hardware" [Wishlist,Confirmed] - Assigned to Matteo Sposato (teoobo) | |
| 13:14:50 | teoobo_ | gibi: I have already filled out the form to request a mentor, but in the meantime I would like to continue working on it | |
| 13:21:12 | openstackgerrit | Lee Yarwood proposed openstack/nova master: [WIP] Migrate nova-grenade-multinode job to zuulv3 native https://review.opendev.org/c/openstack/nova/+/742056 | |
| 13:28:56 | gibi | teoobo_: on a call, I will get back to you in 30 minutes | |
| 13:42:39 | lyarwood | `ERROR: No matching distribution found for packaging>=20.4` fun | |
| 13:50:28 | gibi | lyarwood: hm, pypi has packaging-20.4-py2.py3-none-any.whl | |
| 13:51:13 | sean-k-mooney | ya i was checkin ghtat before i grabed coffee | |
| 13:51:34 | sean-k-mooney | i would guess this is just an issue with the mirros | |
| 13:53:01 | sean-k-mooney | stephenfin: by the way are we going to remove vcpu_pin_set and the fallback query this cycle? | |
| 13:53:14 | sean-k-mooney | we deprecated them in train | |
| 13:53:31 | sean-k-mooney | just trying to figure out if they will be present in 17 or not | |
| 13:54:18 | lyarwood | gibi / sean-k-mooney ; yeah I've asked in #openstack-infra, I assume 20.6 being yanked has caused this | |
| 13:54:22 | sean-k-mooney | we already dont support the deprecated option for new deployments downstream just for upgrades | |
| 13:54:27 | lyarwood | as in we haven't sync'd the newer version yet | |
| 13:54:51 | sean-k-mooney | lyarwood: well 20.5 and 20.7 should be valid | |
| 13:55:12 | sean-k-mooney | lyarwood: they are not public mirrors as far as i know | |
| 13:55:34 | sean-k-mooney | they are limited to the the ci vms | |
| 13:55:46 | lyarwood | sean-k-mooney: yeah I assume we had 20.6 cached in the mirror | |
| 13:55:50 | lyarwood | sean-k-mooney: that was then yanked | |
| 13:56:02 | lyarwood | sean-k-mooney: and for some reason we don't have 20.5 or 20.7 now | |
| 13:58:11 | sean-k-mooney | yep so normally we would jsut hop on the infra channel and ask the ci provier to check | |
| 13:58:47 | sean-k-mooney | infra can also hold a vm and ssh in if needed i guess | |
| 14:00:21 | lyarwood | cool cool | |
| 14:00:25 | lyarwood | lets see what they say | |
| 14:05:44 | gibi | teoobo_: when you say that you cannot reproduce what do you really mean? For me the bug is pretty generic: "This filter is difficult to troubleshoot because there are not much LOG.debug functions in nova.virt.hardware" | |
| 14:06:33 | gibi | teoobo_: do you mean that since then we added plenty of debug logs and now it is clear why the NUMATopologyFilter fails when it fails? | |
| 14:09:51 | teoobo_ | gibi: Yes, I mean that, but since I'm not a nova expert I'm not sure if my judgment is valid | |
| 14:11:40 | gibi | teoobo_: it is nice to here that the filter is now better about logging. I think it would be nice to summarize your experience in the bug. Maybe an example about the logs now emitted helping the troubleshooting | |
| 14:11:44 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/752284 | |
| 14:14:18 | teoobo_ | gibi: thanks for the advice | |
| 14:15:45 | gibi | teoobo_: we could even close the bug with your comment and realy on other users still feeling bad about the logging to open a new one with a specific case when the logging is not enough | |
| 14:17:01 | stephenfin | sean-k-mooney: I have draft patches written but I haven't decided what to do with the reshaper yet | |
| 14:21:06 | mgariepy | lyarwood, do you have a few minutes to review the ussuri cpu feature backport please ? https://review.opendev.org/c/openstack/nova/+/758761 | |
| 14:21:21 | lyarwood | mgariepy: yeah I can take a look now | |
| 14:21:29 | mgariepy | thanks | |
| 14:24:46 | lyarwood | mgariepy: done, LGTM. | |
| 14:25:00 | mgariepy | Thanks a lot :D | |
| 14:27:20 | gibi | teoobo_: do you need some low hanging improvement that could be worked on? | |
| 14:27:50 | gibi | teoobo_: I think there is a list of functional test improvement like this https://review.opendev.org/c/openstack/nova/+/758210 to be done | |
| 14:28:32 | gibi | teoobo_: let me know if you are interested | |
| 14:40:15 | gibi | lyarwood: tempest fix for the racy detach https://review.opendev.org/c/openstack/tempest/+/765814 | |
| 14:50:08 | sean-k-mooney | stephenfin: ah ok ya makes sense | |
| 15:02:09 | lyarwood | gibi: ack LGTM, quick naming nit and we will need some tests but this should be good | |
| 15:02:23 | gibi | we need test for tempest tests? whaaat :D | |
| 15:02:29 | gibi | looking ... | |
| 15:02:46 | lyarwood | gibi: but who is going to test the test that tests the test?!! | |
| 15:02:48 | lyarwood | gibi: /s | |
| 15:02:58 | lyarwood | gibi: yeah just basic unit tests for the waiter | |
| 15:03:10 | gibi | yeah, I need self testing tests in a recursive way... :D | |
| 15:03:10 | lyarwood | gibi: I was asked to do it before so I'm assuming someone will ask for them here | |
| 15:03:24 | gibi | lyarwood: sure, thanks. I will fix the patch up | |
| 15:10:37 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/752284 | |