| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-07 | |||
| 13:22:23 | kashyap | lyarwood: I'm looking for the offending guest QEMU command-line and its guest kernel version | |
| 13:23:08 | gibi | gmann: only have a request in the reno https://review.opendev.org/#/c/748059 but overall looks good to me | |
| 13:23:55 | gmann | gibi: thanks. updating. | |
| 13:23:56 | kashyap | lyarwood: The reason for the above details is because one of the QEMU devs say "lack of CPU time doesn't make sense [as a potential cause], as hot[un]plug events should be porcessed sooner or later" | |
| 13:25:46 | kashyap | lyarwood: I think I should find the logs here (for the latest failing -focal logs): https://review.opendev.org/#/c/734029/ | |
| 13:26:29 | lyarwood | kashyap: https://zuul.opendev.org/t/openstack/build/eee0dc94780c4555b376f17c4f50c301 is a recent example | |
| 13:26:36 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Migrate default policy file from JSON to YAML https://review.opendev.org/748059 | |
| 13:27:08 | lyarwood | kashyap: https://zuul.opendev.org/t/openstack/build/eee0dc94780c4555b376f17c4f50c301/log/controller/logs/libvirt/qemu/instance-0000007a_log.txt is the QEMU log for an instance that hit this | |
| 13:27:11 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Migrate default policy file from JSON to YAML https://review.opendev.org/748059 | |
| 13:27:21 | lyarwood | kashyap: 1dec20ff-922e-4bed-a97f-1699f114e74b | |
| 13:27:45 | kashyap | lyarwood: Thank you; do you have the guest kernel version? (Or the CirrOS version - then I can figure out the kernel version) | |
| 13:27:59 | lyarwood | kashyap: pretty sure it's the same as the version I listed earlier | |
| 13:28:06 | lyarwood | 5.3.0-26-generic | |
| 13:28:08 | kashyap | Ah, okay; was about to guess as much. Thank you | |
| 13:28:22 | gmann | gibi: updated, added bug in cmt msg also | |
| 13:28:42 | lyarwood | kashyap: just modified the cirros image in my test env to use debug ACPI btw | |
| 13:28:54 | lyarwood | kashyap: just hacking tempest to dump the console log / dmesg on failure | |
| 13:29:08 | kashyap | Ah, cool | |
| 13:35:34 | gibi | gmann: thanks, +2 | |
| 13:35:57 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: [Trivial] Replace ref of policy.json to policy.yaml https://review.opendev.org/749821 | |
| 13:36:22 | gmann | gibi: thanks. ^^ this is trivial one to replace the ref of policy.json in doc and test | |
| 13:37:23 | gibi | looking | |
| 13:45:34 | gibi | sean-k-mooney: I have a question at https://review.opendev.org/#/c/742180/11 | |
| 13:49:57 | bauzas | sean-k-mooney: sooooo, we build the VIFs once we are in the compute service, right? | |
| 13:51:05 | bauzas | well, answering myself | |
| 13:51:21 | bauzas | right, only when we add the fixed IP to an instance | |
| 13:52:26 | bauzas | which is called either after creating the instance in the compute, or when adding the fixed IP directly to an instance by the API... | |
| 14:00:32 | kashyap | lyarwood: So, I just combed through the libvirtd log surrounding the QMP 'device_del' (which does the detach), and here's the little fragment: https://kashyapc.fedorapeople.org/CirrOS_device_detach_issues/libvirtd-log-surrounding-device_del.txt | |
| 14:00:34 | bauzas | gibi: sean-k-mooney: question, should we look at the segments if someone asks the API to put a port to an existance ? | |
| 14:00:59 | bauzas | if so... | |
| 14:01:16 | kashyap | lyarwood: It all looks "clean" until here to me: | |
| 14:01:16 | kashyap | 2020-09-03 20:01:53.019+0000: 65328: debug : qemuMonitorJSONIOProcessEvent:205 : handle DEVICE_DELETED handler=0x7f0230572840 data=0x55d556edf3c0 | |
| 14:01:19 | kashyap | 2020-09-03 20:01:53.019+0000: 65328: debug : qemuMonitorJSONHandleDeviceDeleted:1287 : missing device in device deleted event | |
| 14:01:20 | gibi | port will be bound and I guess neutron will fail the binding if there is no segment on the given host | |
| 14:01:47 | gibi | as far as I remember interface_attach is a call so the error will propagate back the user | |
| 14:01:57 | gibi | bauzas: ^^ | |
| 14:02:18 | bauzas | gibi: ok, so Neutron will check it ? | |
| 14:02:23 | bauzas | if so, fine | |
| 14:02:27 | gibi | I assume, yes | |
| 14:02:30 | bauzas | cool | |
| 14:02:36 | gibi | as neutron would need to assign an ip | |
| 14:02:40 | gibi | during the binding | |
| 14:02:44 | bauzas | anyway, we could provide a caveat documentation if no | |
| 14:03:08 | bauzas | anyway, today is the last day I'm trying to work on this | |
| 14:03:27 | bauzas | gibi: sean-k-mooneyif you have changes you want to me to review, lemme know | |
| 14:03:38 | bauzas | and then I'll review them tomorrow | |
| 14:03:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM libvirt: Start emitting DeviceRemovedEvent and DeviceRemovalFailedEvent events https://review.opendev.org/749929 | |
| 14:04:31 | gibi | bauzas: sriov attach is ready, sean-k-mooney is alread +1 on it and the bottom has +2s from stephenfin. series starts here https://review.opendev.org/#/c/741436 | |
| 14:04:41 | bauzas | gibi: ack, will look | |
| 14:04:48 | gibi | thanks! | |
| 14:05:28 | bauzas | gibi: now that I work on some network features, I know better the related files ;) | |
| 14:05:39 | gibi | :) | |
| 14:07:27 | lyarwood | kashyap: yeah that's long after tempest has stopped waiting for the volume to be detached | |
| 14:07:35 | lyarwood | kashyap: let me grab some logs in pastebin | |
| 14:07:55 | kashyap | lyarwood: I've got some contextual stuff here: https://kashyapc.fedorapeople.org/CirrOS_device_detach_issues/ | |
| 14:16:29 | lyarwood | kashyap: http://paste.openstack.org/show/797545/ - AFAICT n-cpu stops trying to detach the volume much earlier than the libvirtd logs you've posted | |
| 14:18:12 | kashyap | lyarwood: Okay, I perhaps need to look further up; let me see if I can see this "Unable to detach" thing in the log | |
| 14:19:14 | kashyap | lyarwood: I'm stumped - I don't see why that "Unable to detach ..." isn't captured here: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c3a/734029/2/check/devstack-platform-focal/c3ab542/controller/logs/libvirt/libvirtd_log.txt | |
| 14:19:49 | kashyap | (Beaware the above log size: gzip-compressed - 8.2MB; uncompressed - 118MB) | |
| 14:20:00 | lyarwood | 721862 2020-09-03 19:58:35.443+0000: 65331: debug : qemuDomainDeleteDevice:128 : Detaching of device virtio-disk1 failed and no event arrived | |
| 14:20:14 | lyarwood | ^ kashyap I think that's what we are after | |
| 14:20:28 | kashyap | lyarwood: "Huzzah", that's right | |
| 14:33:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for I67504a37b0fe2ae5da3cba2f3122d9d0e18b9481 https://review.opendev.org/750184 | |
| 14:50:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 14:50:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Expand generic reproducer for bug #1879878 https://review.opendev.org/750186 | |
| 14:50:16 | openstack | bug 1879878 in OpenStack Compute (nova) "VM become Error after confirming resize with Error info CPUUnpinningInvalid on source node " [Medium,In progress] https://launchpad.net/bugs/1879878 - Assigned to Stephen Finucane (stephenfinucane) | |
| 14:50:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Set 'old_flavor', 'new_flavor' on source before resize https://review.opendev.org/750187 | |
| 14:54:11 | gmann | stephenfin: are you planning the xenapi removal for Victoria release? if so i can review your tempest patch on priority (as that will block the nova side change) otherwise after Focal migration work. | |
| 14:54:28 | stephenfin | gmann: Yes, I was hoping to | |
| 14:54:45 | stephenfin | I think it's in merge conflict though | |
| 14:55:42 | gmann | yeah. | |
| 14:56:56 | stephenfin | okay, resolved that. docstring conflict | |
| 14:58:45 | stephenfin | gibi: replied at https://review.opendev.org/#/c/746945/6/nova/tests/functional/libvirt/test_pci_sriov_servers.py@a370 | |
| 14:59:27 | gibi | thanks, looking | |
| 15:52:44 | sean-k-mooney | bauzas: technically yes but the port binding would fail | |
| 15:52:51 | sean-k-mooney | ah gibi aready said that | |
| 15:52:57 | bauzas | all cool then | |
| 15:55:16 | sean-k-mooney | gibi so regarding https://review.opendev.org/#/c/742180/11/nova/tests/functional/regressions/test_bug_1888395.py i was thinking of using stephens seriese eventuly to enable the migration testing | |
| 15:55:48 | sean-k-mooney | gibi: once the sriov migration fuctional test series merges tehn that regression test can be updated | |
| 15:55:51 | gibi | sean-k-mooney: yeah that would be nice | |
| 15:56:19 | gibi | I read through stephenfin's series today and I'm +2 almost all the way | |
| 15:57:20 | sean-k-mooney | im not sure if i need all the patches by the way. i was hopeing to get this merged before his series to avoid conflicts on backport but im hoping both merged in victoria | |
| 15:57:39 | sean-k-mooney | all the patches in stephens series that is | |
| 15:58:03 | gibi | sean-k-mooney: my -1 on https://review.opendev.org/#/c/742180/ is about the question if we break SRIOV live migration if there is no multi portbinding | |
| 15:58:18 | sean-k-mooney | yes so sriov live migration requires multiple port bindings | |
| 15:58:26 | sean-k-mooney | it was only ment to work if the backend supported that | |
| 15:58:45 | gibi | then the question will it fail cleanly? | |
| 15:59:01 | sean-k-mooney | yes it will | |
| 15:59:16 | sean-k-mooney | we check if multiple port bindigns is supproted i nthe conductor | |
| 15:59:21 | sean-k-mooney | and fail the migration if not | |
| 16:00:07 | sean-k-mooney | gibi: https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L250-L255 | |
| 16:00:24 | gibi | cool | |
| 16:00:27 | gibi | I'm droping my -1 then | |
| 16:00:46 | sean-k-mooney | :) any other concerns? | |
| 16:01:28 | gibi | nope | |
| 16:01:39 | sean-k-mooney | should we be worried about all those gate timeouts | |
| 16:01:54 | gibi | I haven't checked the gate this afternoon | |
| 16:02:20 | sean-k-mooney | im seeing sqlalcamy errors in the unit tests which are unrelated | |
| 16:02:58 | gibi | I'm leaving for today... | |
| 16:03:01 | gibi | o/ | |