| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-04 | |||
| 05:55:22 | Jorhson | lyarwood: and change the code with minimal cost. | |
| 05:58:32 | Jorhson | lyarwood:https://review.opendev.org/#/c/758920/ @75 | |
| 06:01:04 | Jorhson | lyarwood:https://review.opendev.org/#/c/758920/11/specs/wallaby/approved/ensure-volume-attachment-successful.rst@75 | |
| 06:05:42 | brinzhang_ | lyarwood: jorhson's mean your point in PS9 in this specification | |
| 07:46:08 | openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | |
| 09:14:44 | openstackgerrit | Elod Illes proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055 | |
| 09:24:29 | lyarwood | johanssone / brinzhang_ ; morning | |
| 09:34:06 | brinzhang_ | lyarwood: good morning | |
| 09:34:22 | brinzhang_ | jorhson^ | |
| 09:36:49 | lyarwood | johanssone / brinzhang_ ; sorry just as I started replying here I had to jump offline and help with childcare | |
| 09:36:54 | lyarwood | argh | |
| 09:36:59 | lyarwood | Jorhson: sorry | |
| 09:37:26 | lyarwood | Jorhson / brinzhang_ ; so my main issue still with the spec is the change of behaviour in the API that it's introducing | |
| 09:38:18 | lyarwood | Jorhson / brinzhang_ ; before multiple requests would be queued within the computes, with this change callers will have to retry if there's already a request in-flight against the instance | |
| 09:38:47 | lyarwood | Jorhson / brinzhang_ ; that's at least a microversion IMHO | |
| 09:39:05 | Jorhson | lyarwood: if in this way, should we add the task state for this option? | |
| 09:39:27 | lyarwood | Jorhson: this is if you add a task state | |
| 09:39:51 | lyarwood | Jorhson: n-api will start rejecting attach volume requests if the task_state is already ATTACHING, instead of allowing n-cpu to queue things | |
| 09:40:26 | Jorhson | lyarwood: yeah, this is conflicting | |
| 09:40:54 | lyarwood | Jorhson: this all comes down to the RPC call to reservice_block_device_name | |
| 09:41:38 | lyarwood | Jorhson: and I honestly think we can just drop device_name from the APIs and remove this, again with a microversion | |
| 09:42:03 | Jorhson | lyarwood: but i also think it's need for this options, because attaching volume is independent option | |
| 09:42:25 | lyarwood | Jorhson: true but that's why we queue things in the compute with the various locks | |
| 09:43:01 | Jorhson | lyarwood: there will be conflicting if we do others task while the attching task is not over | |
| 09:43:02 | lyarwood | Jorhson: we'd be changing the semantics of the API here pretty drastically and forcing the callers to retry by introducing these task_states | |
| 09:43:22 | brinzhang_ | jorhson: agree with lyarwood, add a microversion to remove the device, and in order to avoid the conflict, we also need to add the task_state as current description in your spec | |
| 09:43:47 | lyarwood | well we have the instance.uuid lock to avoid conflicts | |
| 09:44:03 | lyarwood | most compute operations should be taking that IMHO | |
| 09:44:15 | lyarwood | I know it's not the case for everything at the moment | |
| 09:46:33 | lyarwood | so what if we didn't reject the request if the task_state was attaching and queued it in the API somehow? | |
| 09:46:39 | lyarwood | Jorhson / brinzhang_ ; ^ | |
| 09:47:46 | Jorhson | lyarwood: yeah, for the users, there are less probability to attach multiple volumes, and i agree with your sheme, but i think it's important to solve the option's conflict | |
| 09:48:18 | lyarwood | Jorhson: it's a pretty common usecase with k8s running on OpenStack IMHO | |
| 09:54:38 | brinzhang_ | lyarwood: ok, I understand what you mean, agree | |
| 09:55:28 | brinzhang_ | lyarwood: Jorhson will update this spec later, this way it's better for k8s on OpenStack scenario | |
| 09:59:36 | lyarwood | brinzhang_: ack many thanks both, I'm going to be AFK again this afternoon UK time but I'll review the spec this evening once I'm back. | |
| 10:00:34 | brinzhang_ | lyarwood: ack, thanks too | |
| 10:01:52 | aarents | Hi nova | |
| 10:02:22 | openstack | Launchpad bug 1832248 in OpenStack Compute (nova) "tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume failing when using the Q35 machine type" [Medium,Triaged] - Assigned to Alexandre arents (aarents) | |
| 10:02:22 | aarents | lyarwood: there is some progress on that q35 bug: https://bugs.launchpad.net/nova/+bug/1832248 , if it is a qemu bug, what is the suitable status for this bug Fixed_in_relsease or invalid? | |
| 10:07:26 | lyarwood | aarents: awesome sleuthing! :) | |
| 10:08:04 | lyarwood | aarents: erm, invalid I guess? gibi ^ any thoughts, looking to close a Nova bug that was actually a QEMU bug that has already been fixed | |
| 10:09:11 | gibi | lyarwood: if it is fixed in QEMU and no need for any nova change then Invliad it is | |
| 10:09:20 | gibi | Invalid even | |
| 10:09:25 | lyarwood | cool thanks | |
| 10:15:38 | aarents | cool, I just hope it will fix also my downstrem bug regarding interface attach! | |
| 11:56:55 | sean-k-mooney | do we need to enable the new qemu from the cloud arcive on ussuri and below to ensure we dont hit that bug | |
| 12:04:23 | openstackgerrit | Wenping Song proposed openstack/nova-specs master: Support vGPU management by Cyborg https://review.opendev.org/750116 | |
| 13:04:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent starting services with older than N-1 computes https://review.opendev.org/738482 | |
| 13:04:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add upgrade check about old computes https://review.opendev.org/760520 | |
| 13:19:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent starting services with older than N-1 computes https://review.opendev.org/738482 | |
| 13:21:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add upgrade check about old computes https://review.opendev.org/760520 | |
| 13:23:41 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use _evacuate_server helper in func test https://review.opendev.org/758210 | |
| 13:27:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ignore PCI devices with 32bit domain https://review.opendev.org/756697 | |
| 13:27:17 | openstack | bug 1897528 in OpenStack Compute (nova) "32bit pci domain number is not supported" [High,In progress] https://launchpad.net/bugs/1897528 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 13:27:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1897528 https://review.opendev.org/756696 | |
| 13:36:47 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Support interface attach with qos ports https://review.opendev.org/755477 | |
| 14:03:43 | openstack | bug 1896463 in OpenStack Compute (nova) rocky "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,In progress] https://launchpad.net/bugs/1896463 | |
| 14:03:43 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1896463 in func env https://review.opendev.org/754100 | |
| 14:05:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Set instance host and drop migration under lock https://review.opendev.org/754815 | |
| 14:07:17 | openstack | bug 1896463 in OpenStack Compute (nova) rocky "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,In progress] https://launchpad.net/bugs/1896463 | |
| 14:07:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1896463 in func env https://review.opendev.org/754100 | |
| 14:08:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Set instance host and drop migration under lock https://review.opendev.org/754815 | |
| 14:51:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add upgrade check about old computes https://review.opendev.org/760520 | |
| 15:02:45 | openstackgerrit | Takashi Natsume proposed openstack/nova stable/victoria: Fix a hacking test https://review.opendev.org/758112 | |
| 15:05:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: [doc]: Fix glance image_metadata link https://review.opendev.org/761423 | |
| 15:07:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/victoria: Use cell targeted context to query BDMs for metadata https://review.opendev.org/761424 | |
| 15:29:19 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Bump the lowest eventlet version to 0.26.1 https://review.opendev.org/761427 | |
| 15:55:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: [trivial]: replace NUMNA with NUMA https://review.opendev.org/761436 | |
| 16:19:43 | openstackgerrit | Merged openstack/nova-specs master: [trivial]: replace NUMNA with NUMA https://review.opendev.org/761436 | |
| 16:26:56 | openstack | Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New] | |
| 16:26:56 | gibi | somebody with connection to the libvirt maintainer should look at this nova bug https://bugs.launchpad.net/nova/+bug/1902276 | |
| 16:28:04 | melwitt | kashyap: ^ | |
| 16:28:26 | kashyap | melwitt: Yeah, familar with it, as I worked with the reporter here the other day | |
| 16:28:39 | melwitt | ah ok, cool | |
| 16:28:43 | kashyap | melwitt: I asked one of the libvirt devs on Friday, but my timing wasn't right | |
| 16:28:52 | gibi | kashyap: thanks! | |
| 16:28:57 | kashyap | I'll check again | |
| 16:29:18 | kashyap | gibi: It looks fishy, as it's not 100% reproducible ... as the reporter says "a few minutes later things go back to normal" | |
| 16:29:26 | kashyap | But we don't know what changed :-( | |
| 16:31:47 | gibi | kashyap: the nova image download take ~ 300 seconds for the VM that then triggers the loop in libvirtd so it might be that the hypervisor host has high load | |
| 16:31:57 | kashyap | gibi: Yeah, just reading your report :) | |
| 16:31:58 | gibi | but I was not able to confirm it from the logs | |
| 16:33:13 | kashyap | I see. Hypervisor load sounds plausible - as we've hit load-related (CI) issues libvirt driver before. But still let me check w/ Dan or someone from upstream libvirt | |
| 16:34:25 | gibi | kashyap: thanks for taking this up with the libvirt maintainers | |
| 16:35:34 | kashyap | gibi: Just posted on #virt, OFTC network. | |
| 16:36:18 | kashyap | gibi: Is this blocking patch merges? | |
| 16:39:05 | melwitt | I've been struggling for a couple of days trying to get an approved patch through the gate, but I'm not sure whether that particular bug is involved. I would need to re-look at the logs to verify | |
| 16:39:25 | kashyap | (I've reposted the looping libvirtd log bits as a plain text, as the pastebins expire) | |
| 16:40:58 | kashyap | melwitt: Noted; Michael, the reporter, was saying on last Friday that it's "intermittent", which makes it a bit more difficult to debug | |
| 16:41:56 | melwitt | yeah, that's been the theme of all of the gate bugs I'm aware of. intermittent and thus hard to troubleshoot :( | |
| 16:42:06 | melwitt | *current gate bugs I'm aware of | |
| 16:43:01 | kashyap | Yeah, matches my past experience | |
| 16:43:53 | kashyap | melwitt: In the same vein as how Twitter (I'm not on it) seems to label Trump's tweets as misleading, wonder we should adapt that text for these intermittent bugs :D | |
| 16:44:26 | kashyap | - Some or all of the content shared in this Tweet is disputed and might be misleading about an election or other civic process. | |
| 16:44:29 | kashyap | + Some or all of the content shared in this bug is disputed and might be misleading due to intermittent failures. | |
| 16:46:02 | kashyap | gibi: melwitt: More seriously, can I "subscribe" (Cc) someone else to a LaunchPad, right? | |
| 16:46:21 | kashyap | IIRC, yes. /me tries | |
| 16:46:26 | melwitt | I think you can | |
| 16:48:11 | kashyap | melwitt: I can't :-( I wanted to Cc Michal from libvirt but it says "No items matched <email ID>" | |