| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-07 | |||
| 09:38:47 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: libvirt: Skip encryption metadata lookups if secret already exists on host https://review.opendev.org/c/openstack/nova/+/765770 | |
| 09:39:15 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: libvirt: Skip encryption metadata lookups if secret already exists on host https://review.opendev.org/c/openstack/nova/+/765771 | |
| 09:40:07 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: libvirt: Skip encryption metadata lookups if secret already exists on host https://review.opendev.org/c/openstack/nova/+/765772 | |
| 09:40:32 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: libvirt: Skip encryption metadata lookups if secret already exists on host https://review.opendev.org/c/openstack/nova/+/765773 | |
| 09:41:07 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: Skip encryption metadata lookups if secret already exists on host https://review.opendev.org/c/openstack/nova/+/765774 | |
| 09:43:22 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/victoria: Add regression test for bug #1899649 https://review.opendev.org/c/openstack/nova/+/764611 | |
| 09:43:22 | openstack | bug 1899649 in OpenStack Compute (nova) "Volume marked as available after a failure to build" [Undecided,In progress] https://launchpad.net/bugs/1899649 - Assigned to Lee Yarwood (lyarwood) | |
| 09:43:24 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/victoria: compute: Don't detach volumes when RescheduledException raised without retry https://review.opendev.org/c/openstack/nova/+/764612 | |
| 09:50:19 | openstackgerrit | chengsheng proposed openstack/nova master: CPU selection with hypervisor consideration https://review.opendev.org/c/openstack/nova/+/762330 | |
| 09:54:27 | gibi | aarents: I have a question in https://review.opendev.org/c/openstack/nova/+/765561 | |
| 09:55:06 | gibi | lyarwood: welcome back o/ | |
| 10:00:06 | openstackgerrit | Aleksey Ulyanov proposed openstack/nova master: Fix graceful shutdown timeout https://review.opendev.org/c/openstack/nova/+/764927 | |
| 10:22:08 | aarents | Hi gibi lyarwood | |
| 10:22:13 | aarents | gibi: I just replied | |
| 10:22:17 | gibi | looking | |
| 10:26:13 | gibi | aarents: so resize depends on image_base_image_ref being present in metadata and some instances doesn't have it as the previous fix only fixed it for new instances | |
| 10:26:17 | gibi | Am I correct? | |
| 10:26:42 | aarents | gibi: correct | |
| 10:27:44 | gibi | then I think only the comment need a bit of rewording to state this | |
| 10:27:52 | gibi | as from the bug link it was not totally clear | |
| 10:28:02 | gibi | but your review answer cleared it | |
| 10:29:54 | gibi | aarents: replied in the review, thanks for the clarification | |
| 10:33:23 | aarents | gibi: oky | |
| 10:33:39 | aarents | lyarwood: thnsk I will address the nit | |
| 12:03:43 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Update modernize-os-hypervisors-api spec, pt. 2 https://review.opendev.org/c/openstack/nova-specs/+/765796 | |
| 12:03:44 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Update modernize-os-hypervisors-api spec, pt. 3 https://review.opendev.org/c/openstack/nova-specs/+/765797 | |
| 12:11:02 | gibi | fyi I see this happening quite a lot recently on the gate https://bugs.launchpad.net/tempest/+bug/1907084 | |
| 12:11:02 | openstack | Launchpad bug 1907084 in tempest "test_attach_scsi_disk_with_config_drive intermittently fails at detaching volume" [Undecided,New] | |
| 12:11:40 | gibi | I think it is just a race condition in the test. So I will try to propose a re-query of the volume attachemnts in tempest | |
| 12:14:27 | sean-k-mooney | gibi: well that or its realted to the race lyarwood was working around with the retry interval | |
| 12:15:27 | gibi | sean-k-mooney: do you have a link for that? | |
| 12:16:56 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/755799 was the workaround | |
| 12:17:10 | sean-k-mooney | the fix is to use libvirt events | |
| 12:18:17 | gibi | sean-k-mooney: ahh, could be related to the fact that now the race happens more than before as I guess we delayed the detach on the libvirt side | |
| 12:18:30 | gibi | making the bdm.destroy even more delayed | |
| 12:18:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Merge 'test_hypervisor_status' into 'test_hypervisors' https://review.opendev.org/c/openstack/nova/+/764039 | |
| 12:18:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Drop statistics-style fields from os-hypervisors https://review.opendev.org/c/openstack/nova/+/764040 | |
| 12:18:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Change default policy for '/os-hypervisors' https://review.opendev.org/c/openstack/nova/+/765798 | |
| 12:18:57 | sean-k-mooney | well its load related, so 10 seconds was just a guess based on what we were seeing | |
| 12:19:10 | sean-k-mooney | but a slighly slower node could still nit it | |
| 12:19:37 | sean-k-mooney | it may be something else but it sounds related | |
| 12:21:35 | lyarwood | gmann: ^ btw, smells like an issue with https://review.opendev.org/c/openstack/tempest/+/764407 | |
| 12:22:02 | lyarwood | I don't think this is an issue with https://review.opendev.org/c/openstack/nova/+/755799 fwiw sean-k-mooney, at least at first glance | |
| 12:22:36 | lyarwood | gmann: ignore that ping, the assertion is in the test itself, not cleanup. | |
| 12:23:24 | sean-k-mooney | that test chagne merged 9 days ago, gibi are you still seeing it after that? | |
| 12:23:51 | gibi | sean-k-mooney: yepp it is actively happening | |
| 12:24:00 | gibi | logstash link is in the bugreport | |
| 12:24:29 | gibi | but I now 70% confident that the cinder volume becomes available _befor_ nova deletes the BDM | |
| 12:24:35 | gibi | so the tempest test is simply racing on this | |
| 12:24:43 | lyarwood | that shouldn't be possible | |
| 12:25:25 | gibi | lyarwood: I think this is the relevant code path https://github.com/openstack/nova/blob/97729aabaa7768e1af2fa4a0bdda0df7ad598426/nova/compute/manager.py#L7071-L7086 | |
| 12:25:49 | gibi | the bdm.destory() is the last thing nova does during volume detach | |
| 12:26:50 | lyarwood | gibi: right and that point we've already detached from the instance in libvirt and detached the volume from the instance via c-api | |
| 12:26:59 | gibi | yepp | |
| 12:27:21 | gibi | so there is a timewindow where the volume is available in cinder, but still has a BDM in nova | |
| 12:27:36 | gibi | hence the nova volume attachment list will show the attachment | |
| 12:27:44 | lyarwood | ah right | |
| 12:27:48 | lyarwood | it's using servers_client | |
| 12:27:53 | lyarwood | instead of volumes_client | |
| 12:28:07 | lyarwood | so we could race the bdm.destroy in the db | |
| 12:28:26 | sean-k-mooney | for what its worth that might also be an issue with shift on stack | |
| 12:28:52 | sean-k-mooney | they were reporting similar behavior where the volumes were atached in nova but not in cinder | |
| 12:29:32 | sean-k-mooney | although in there case i think that was a persitent state | |
| 12:29:40 | sean-k-mooney | not just an interval where it was racy | |
| 12:30:30 | gibi | sean-k-mooney: we will see if this is a persistent bug after I propose a re-query in the tempest test | |
| 12:30:44 | gibi | if the re-query fails then we know we have a problem where the bdm is not deleted | |
| 12:31:00 | lyarwood | gibi: the way to fix this is to add a waiter for volume attachments to the servers_clienty | |
| 12:31:02 | lyarwood | client* | |
| 12:31:18 | gibi | lyarwood: yes | |
| 12:31:20 | sean-k-mooney | i prefer clienty | |
| 12:31:21 | gibi | that is my plan | |
| 12:31:44 | lyarwood | cool, let me know when you've posted a review and I'll happily review | |
| 12:31:45 | sean-k-mooney | espcially the clinety thingy | |
| 12:31:51 | gibi | sean-k-mooney: :) | |
| 12:31:55 | gibi | lyarwood: ack | |
| 12:33:52 | lyarwood | urgh | |
| 12:33:54 | lyarwood | I see | |
| 12:54:06 | openstackgerrit | Lee Yarwood proposed openstack/nova master: tools: Allow check-cherry-picks.sh to be disabled by an env var https://review.opendev.org/c/openstack/nova/+/765802 | |
| 12:54:36 | lyarwood | ^ workaround for our downstream CI so we can use jobs and tox envs directly from upstream for a change | |
| 13:02:56 | sean-k-mooney | instead of just using sed to remove it | |
| 13:03:11 | sean-k-mooney | that is cleaner then our current hack i guess | |
| 13:14:40 | teoobo_ | gibi: I am working on this bug (https://bugs.launchpad.net/nova/+bug/1751784) but I cannot reproduce it in my devstack | |
| 13:14:40 | openstack | Launchpad bug 1751784 in OpenStack Compute (nova) "Add more LOG.debug in nova.virt.hardware" [Wishlist,Confirmed] - Assigned to Matteo Sposato (teoobo) | |
| 13:14:50 | teoobo_ | gibi: I have already filled out the form to request a mentor, but in the meantime I would like to continue working on it | |
| 13:21:12 | openstackgerrit | Lee Yarwood proposed openstack/nova master: [WIP] Migrate nova-grenade-multinode job to zuulv3 native https://review.opendev.org/c/openstack/nova/+/742056 | |
| 13:28:56 | gibi | teoobo_: on a call, I will get back to you in 30 minutes | |
| 13:42:39 | lyarwood | `ERROR: No matching distribution found for packaging>=20.4` fun | |
| 13:50:28 | gibi | lyarwood: hm, pypi has packaging-20.4-py2.py3-none-any.whl | |
| 13:51:13 | sean-k-mooney | ya i was checkin ghtat before i grabed coffee | |
| 13:51:34 | sean-k-mooney | i would guess this is just an issue with the mirros | |
| 13:53:01 | sean-k-mooney | stephenfin: by the way are we going to remove vcpu_pin_set and the fallback query this cycle? | |
| 13:53:14 | sean-k-mooney | we deprecated them in train | |
| 13:53:31 | sean-k-mooney | just trying to figure out if they will be present in 17 or not | |
| 13:54:18 | lyarwood | gibi / sean-k-mooney ; yeah I've asked in #openstack-infra, I assume 20.6 being yanked has caused this | |
| 13:54:22 | sean-k-mooney | we already dont support the deprecated option for new deployments downstream just for upgrades | |
| 13:54:27 | lyarwood | as in we haven't sync'd the newer version yet | |
| 13:54:51 | sean-k-mooney | lyarwood: well 20.5 and 20.7 should be valid | |
| 13:55:12 | sean-k-mooney | lyarwood: they are not public mirrors as far as i know | |
| 13:55:34 | sean-k-mooney | they are limited to the the ci vms | |
| 13:55:46 | lyarwood | sean-k-mooney: yeah I assume we had 20.6 cached in the mirror | |