| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-07 | |||
| 12:24:43 | lyarwood | that shouldn't be possible | |
| 12:25:25 | gibi | lyarwood: I think this is the relevant code path https://github.com/openstack/nova/blob/97729aabaa7768e1af2fa4a0bdda0df7ad598426/nova/compute/manager.py#L7071-L7086 | |
| 12:25:49 | gibi | the bdm.destory() is the last thing nova does during volume detach | |
| 12:26:50 | lyarwood | gibi: right and that point we've already detached from the instance in libvirt and detached the volume from the instance via c-api | |
| 12:26:59 | gibi | yepp | |
| 12:27:21 | gibi | so there is a timewindow where the volume is available in cinder, but still has a BDM in nova | |
| 12:27:36 | gibi | hence the nova volume attachment list will show the attachment | |
| 12:27:44 | lyarwood | ah right | |
| 12:27:48 | lyarwood | it's using servers_client | |
| 12:27:53 | lyarwood | instead of volumes_client | |
| 12:28:07 | lyarwood | so we could race the bdm.destroy in the db | |
| 12:28:26 | sean-k-mooney | for what its worth that might also be an issue with shift on stack | |
| 12:28:52 | sean-k-mooney | they were reporting similar behavior where the volumes were atached in nova but not in cinder | |
| 12:29:32 | sean-k-mooney | although in there case i think that was a persitent state | |
| 12:29:40 | sean-k-mooney | not just an interval where it was racy | |
| 12:30:30 | gibi | sean-k-mooney: we will see if this is a persistent bug after I propose a re-query in the tempest test | |
| 12:30:44 | gibi | if the re-query fails then we know we have a problem where the bdm is not deleted | |
| 12:31:00 | lyarwood | gibi: the way to fix this is to add a waiter for volume attachments to the servers_clienty | |
| 12:31:02 | lyarwood | client* | |
| 12:31:18 | gibi | lyarwood: yes | |
| 12:31:20 | sean-k-mooney | i prefer clienty | |
| 12:31:21 | gibi | that is my plan | |
| 12:31:44 | lyarwood | cool, let me know when you've posted a review and I'll happily review | |
| 12:31:45 | sean-k-mooney | espcially the clinety thingy | |
| 12:31:51 | gibi | sean-k-mooney: :) | |
| 12:31:55 | gibi | lyarwood: ack | |
| 12:33:52 | lyarwood | urgh | |
| 12:33:54 | lyarwood | I see | |
| 12:54:06 | openstackgerrit | Lee Yarwood proposed openstack/nova master: tools: Allow check-cherry-picks.sh to be disabled by an env var https://review.opendev.org/c/openstack/nova/+/765802 | |
| 12:54:36 | lyarwood | ^ workaround for our downstream CI so we can use jobs and tox envs directly from upstream for a change | |
| 13:02:56 | sean-k-mooney | instead of just using sed to remove it | |
| 13:03:11 | sean-k-mooney | that is cleaner then our current hack i guess | |
| 13:14:40 | openstack | Launchpad bug 1751784 in OpenStack Compute (nova) "Add more LOG.debug in nova.virt.hardware" [Wishlist,Confirmed] - Assigned to Matteo Sposato (teoobo) | |
| 13:14:40 | teoobo_ | gibi: I am working on this bug (https://bugs.launchpad.net/nova/+bug/1751784) but I cannot reproduce it in my devstack | |
| 13:14:50 | teoobo_ | gibi: I have already filled out the form to request a mentor, but in the meantime I would like to continue working on it | |
| 13:21:12 | openstackgerrit | Lee Yarwood proposed openstack/nova master: [WIP] Migrate nova-grenade-multinode job to zuulv3 native https://review.opendev.org/c/openstack/nova/+/742056 | |
| 13:28:56 | gibi | teoobo_: on a call, I will get back to you in 30 minutes | |
| 13:42:39 | lyarwood | `ERROR: No matching distribution found for packaging>=20.4` fun | |
| 13:50:28 | gibi | lyarwood: hm, pypi has packaging-20.4-py2.py3-none-any.whl | |
| 13:51:13 | sean-k-mooney | ya i was checkin ghtat before i grabed coffee | |
| 13:51:34 | sean-k-mooney | i would guess this is just an issue with the mirros | |
| 13:53:01 | sean-k-mooney | stephenfin: by the way are we going to remove vcpu_pin_set and the fallback query this cycle? | |
| 13:53:14 | sean-k-mooney | we deprecated them in train | |
| 13:53:31 | sean-k-mooney | just trying to figure out if they will be present in 17 or not | |
| 13:54:18 | lyarwood | gibi / sean-k-mooney ; yeah I've asked in #openstack-infra, I assume 20.6 being yanked has caused this | |
| 13:54:22 | sean-k-mooney | we already dont support the deprecated option for new deployments downstream just for upgrades | |
| 13:54:27 | lyarwood | as in we haven't sync'd the newer version yet | |
| 13:54:51 | sean-k-mooney | lyarwood: well 20.5 and 20.7 should be valid | |
| 13:55:12 | sean-k-mooney | lyarwood: they are not public mirrors as far as i know | |
| 13:55:34 | sean-k-mooney | they are limited to the the ci vms | |
| 13:55:46 | lyarwood | sean-k-mooney: yeah I assume we had 20.6 cached in the mirror | |
| 13:55:50 | lyarwood | sean-k-mooney: that was then yanked | |
| 13:56:02 | lyarwood | sean-k-mooney: and for some reason we don't have 20.5 or 20.7 now | |
| 13:58:11 | sean-k-mooney | yep so normally we would jsut hop on the infra channel and ask the ci provier to check | |
| 13:58:47 | sean-k-mooney | infra can also hold a vm and ssh in if needed i guess | |
| 14:00:21 | lyarwood | cool cool | |
| 14:00:25 | lyarwood | lets see what they say | |
| 14:05:44 | gibi | teoobo_: when you say that you cannot reproduce what do you really mean? For me the bug is pretty generic: "This filter is difficult to troubleshoot because there are not much LOG.debug functions in nova.virt.hardware" | |
| 14:06:33 | gibi | teoobo_: do you mean that since then we added plenty of debug logs and now it is clear why the NUMATopologyFilter fails when it fails? | |
| 14:09:51 | teoobo_ | gibi: Yes, I mean that, but since I'm not a nova expert I'm not sure if my judgment is valid | |
| 14:11:40 | gibi | teoobo_: it is nice to here that the filter is now better about logging. I think it would be nice to summarize your experience in the bug. Maybe an example about the logs now emitted helping the troubleshooting | |
| 14:11:44 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/752284 | |
| 14:14:18 | teoobo_ | gibi: thanks for the advice | |
| 14:15:45 | gibi | teoobo_: we could even close the bug with your comment and realy on other users still feeling bad about the logging to open a new one with a specific case when the logging is not enough | |
| 14:17:01 | stephenfin | sean-k-mooney: I have draft patches written but I haven't decided what to do with the reshaper yet | |
| 14:21:06 | mgariepy | lyarwood, do you have a few minutes to review the ussuri cpu feature backport please ? https://review.opendev.org/c/openstack/nova/+/758761 | |
| 14:21:21 | lyarwood | mgariepy: yeah I can take a look now | |
| 14:21:29 | mgariepy | thanks | |
| 14:24:46 | lyarwood | mgariepy: done, LGTM. | |
| 14:25:00 | mgariepy | Thanks a lot :D | |
| 14:27:20 | gibi | teoobo_: do you need some low hanging improvement that could be worked on? | |
| 14:27:50 | gibi | teoobo_: I think there is a list of functional test improvement like this https://review.opendev.org/c/openstack/nova/+/758210 to be done | |
| 14:28:32 | gibi | teoobo_: let me know if you are interested | |
| 14:40:15 | gibi | lyarwood: tempest fix for the racy detach https://review.opendev.org/c/openstack/tempest/+/765814 | |
| 14:50:08 | sean-k-mooney | stephenfin: ah ok ya makes sense | |
| 15:02:09 | lyarwood | gibi: ack LGTM, quick naming nit and we will need some tests but this should be good | |
| 15:02:23 | gibi | we need test for tempest tests? whaaat :D | |
| 15:02:29 | gibi | looking ... | |
| 15:02:46 | lyarwood | gibi: but who is going to test the test that tests the test?!! | |
| 15:02:48 | lyarwood | gibi: /s | |
| 15:02:58 | lyarwood | gibi: yeah just basic unit tests for the waiter | |
| 15:03:10 | lyarwood | gibi: I was asked to do it before so I'm assuming someone will ask for them here | |
| 15:03:10 | gibi | yeah, I need self testing tests in a recursive way... :D | |
| 15:03:24 | gibi | lyarwood: sure, thanks. I will fix the patch up | |
| 15:10:37 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/752284 | |
| 15:36:53 | teoobo_ | gibi: I am interested in helping out! | |
| 15:38:55 | kashyap | lyarwood: A stupid question - upstream CI runs instance deletion (functional) tests, right? Got a pointer to a latest run for it? | |
| 15:39:06 | kashyap | I think in any CI run I should be able to find it | |
| 15:40:38 | gibi | teoobo_: so there are those nice helpers for the functional test but they are not always used in the tests, so the work would be to find where direct api calls are used for those server actions and replace it with the call to the helper | |
| 15:40:44 | lyarwood | kashyap: remember that we change the real meaning of functional tests in Nova | |
| 15:41:03 | lyarwood | kashyap: do you mean actual instance deletion including the removal of a libvirt domain etc? | |
| 15:41:19 | lyarwood | kashyap: if so that's actually under integration testing with tempest and yeah there should be loads of examples | |
| 15:41:35 | lyarwood | kashyap: each time we create and instance we should eventually clean it up | |
| 15:41:40 | lyarwood | create an* | |
| 15:41:48 | kashyap | lyarwood: Yeah, indeed - actual instance deletion including the removal | |
| 15:41:52 | kashyap | ... of libvirt domain | |
| 15:41:57 | kashyap | lyarwood: The reason why I ask is, | |
| 15:42:04 | lyarwood | kashyap: kk, should be loads of examples in any tempest run | |
| 15:42:21 | kashyap | I was debugging an NVDIMM issue with OSP QE; and Nova (upstream) code currently assumes you have 'daxio' tool installed | |
| 15:42:38 | kashyap | But ... some enterprise distros don't have it installed in their containers :-) | |