| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-21 | |||
| 18:21:26 | openstackgerrit | Merged openstack/nova stable/rocky: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/742417 | |
| 18:26:23 | CeeMac | Evening all | |
| 18:27:48 | CeeMac | Anyone ever come across issues with guest operating system corruption following a volume retype operation ? | |
| 18:48:23 | artom | Hrmpf, so it actually would work in nova/tests/functional because that __init__.py imports nova.monkey_patch, which is fine because we set EVENTLET_NO_GREENDNS in there | |
| 18:48:35 | artom | But then we need the env var skip mechanism | |
| 18:48:45 | artom | So perhaps better in nova/tests after all? | |
| 18:48:59 | dansmith | I don't really care either way | |
| 18:49:25 | artom | I guess the latter then, because less code. | |
| 18:49:34 | artom | And I'll explain it all with a proper commit message | |
| 18:49:40 | artom | ... *after* I pick up my kinds | |
| 18:49:42 | artom | *kids | |
| #openstack-nova - 2020-09-22 | |||
| 00:00:13 | brinzhang | gibi: ack, thanks | |
| 00:44:19 | openstackgerrit | Yao wang proposed openstack/nova master: Use VIR_DOMAIN_XML_INACTIVE for detach volume https://review.opendev.org/749489 | |
| 00:57:14 | openstackgerrit | Yingji Sun proposed openstack/nova stable/train: Set different VirtualDevice.key https://review.opendev.org/752843 | |
| 07:19:53 | bauzas | good morning Nova | |
| 08:08:07 | lyarwood | Morning | |
| 08:31:11 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Improve CinderFixtureNewAttachFlow https://review.opendev.org/749228 | |
| 08:31:12 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Robustify attachment tracking in CinderFixtureNewAttachFlow https://review.opendev.org/749229 | |
| 08:31:13 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: compute: Use source_bdms to reset attachment_ids during LM rollback https://review.opendev.org/749230 | |
| 08:31:14 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: compute: refactor volume bdm rollback error handling https://review.opendev.org/749231 | |
| 08:31:17 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Add regression tests for bug #1889108 https://review.opendev.org/749232 | |
| 08:31:17 | openstack | bug 1889108 in OpenStack Compute (nova) queens "failures during driver.pre_live_migration remove source attachments during rollback" [Undecided,In progress] https://launchpad.net/bugs/1889108 - Assigned to Lee Yarwood (lyarwood) | |
| 08:31:18 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: compute: Don't delete the original attachment during pre LM rollback https://review.opendev.org/749233 | |
| 08:34:44 | lyarwood | https://2733627e1a0348401721-fb543e4b225e8fdb51d174f0fee213a6.ssl.cf1.rackcdn.com/746981/8/check/nova-tox-functional-py38/594be66/testr_results.html - hmm I've never seen this failure before in the gate | |
| 08:35:06 | lyarwood | nova.tests.functional.test_cross_cell_migrate.TestMultiCellMigrate.test_delete_while_in_verify_resize_status oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor | |
| 08:55:16 | nightmare_unreal | Hi , where can I find logs in a devstack machine. I tried /var/log/ but there was no logs there | |
| 09:01:19 | CeeMac | morning | |
| 09:01:43 | CeeMac | regarding nova's roll in volume retyping an attached volume on a running instance | |
| 09:02:09 | CeeMac | does the quiesce operation rely on the guest agent being available on the instance, or should it be fully transparent to the instance | |
| 09:02:59 | CeeMac | trying to understand an issue with operating system disk corruption on an instance after a volume retype | |
| 09:07:38 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/queens: hardware: fix memory check usage for small/large pages https://review.opendev.org/726867 | |
| 09:07:39 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/queens: Fix overcommit for NUMA-based instances https://review.opendev.org/726868 | |
| 09:07:40 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/queens: Fix typo https://review.opendev.org/753261 | |
| 09:16:30 | kashyap | stephenfin: --^ That's quick; thanks! | |
| 09:16:35 | frickler | nightmare_unreal: most logs are in the systemd journal, try e.g. "journalctl -u devstack@n-api" | |
| 09:16:49 | kashyap | (Although, I don't need 'em myself, though) | |
| 09:23:15 | nightmare_unreal | thanks frickler | |
| 09:30:07 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | |
| 09:30:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982 | |
| 09:30:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_NATIVE_TLS_VERSION https://review.opendev.org/746983 | |
| 09:30:09 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_BETTER_SIGKILL_HANDLING https://review.opendev.org/746984 | |
| 09:30:09 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985 | |
| 09:30:10 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986 | |
| 09:30:10 | openstackgerrit | Lee Yarwood proposed openstack/nova master: test_evacuate.sh: Stop using libvirt-bin https://review.opendev.org/753267 | |
| 09:30:56 | lyarwood | CeeMac: I don't think we quiesce during a retype | |
| 09:31:09 | lyarwood | CeeMac: the entire operation should be transparent to the instance | |
| 09:32:01 | lyarwood | CeeMac: which version of nova and libvirt is this? | |
| 09:41:13 | CeeMac | Nova v18.1.1, libvirt 4.0.0 | |
| 09:41:26 | CeeMac | lyarwood: ^ | |
| 09:49:31 | CeeMac | that was my understanding too, but I've had a couple of instance corrupt after retyping the volume and I can't understand why | |
| 10:20:46 | luyao | stephenfin, alex_xu: Hi, there are 2 patches for vpmem enhencement missed the feature freeze, hope you could review them again if you get time. Thanks. https://review.opendev.org/#/q/topic:bp/vpmem-enhancement+(status:open) | |
| 10:30:23 | lyarwood | CeeMac: sorry had to go afk, so with libvirt 4.0.0 we should still be using virDomainBlockRebase | |
| 10:30:42 | lyarwood | CeeMac: I'd be interested in looking at some example logs if you have any tbh | |
| 10:31:06 | lyarwood | CeeMac: if you could use the event list to find the request-id and create a bug for this I can review them | |
| 10:31:18 | lyarwood | CeeMac: openstack server event list $instance | |
| 10:31:37 | lyarwood | CeeMac: that should list a swap_volume event and associated request-id | |
| 10:31:54 | lyarwood | CeeMac: we might also need libvirtd logs from the host if available | |
| 10:32:29 | CeeMac | lyarwood: I'll see what I can find, not sure I'll get the exact logs for the broken instance as we didn't realise until alter in the day it was an issue | |
| 10:40:15 | brinzhang0 | stephenfin: I seem not understand what would you like to say of https://review.opendev.org/#/c/675304/21/openstackclient/compute/v2/server.py@1868 | |
| 10:41:31 | brinzhang0 | and the Line1888, the parsed_args.server can get the server name or id, right? and we have always used it in openstackclient | |
| 10:44:17 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: test_evacuate.sh: Support libvirt-bin and libvirtd systemd services https://review.opendev.org/753280 | |
| 10:45:30 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: test_evacuate.sh: Stop using libvirt-bin https://review.opendev.org/753281 | |
| 10:46:48 | lyarwood | elod: ^ if you have time today, https://review.opendev.org/753281 should unblock the nova-live-migration job on stable/ussuri now it's using the train UCA | |
| 10:46:56 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: Ensure source compute is up when confirming a resize https://review.opendev.org/748369 | |
| 10:50:06 | elod | lyarwood: sure, I will look at it soon | |
| 10:54:27 | stephenfin | brinzhang0: You can ignore that. I was suggesting something but then realized it wouldn't work | |
| 10:54:34 | stephenfin | Leaving the comment there for other reviewers | |
| 10:55:10 | brinzhang0 | stephenfin: ack, thanks, I will jump these two comments | |
| 10:55:54 | ygk_12345 | hi all i am using OSA train 20.0.2, but when launching a vm, it is goging into error state with the following erro message | |
| 10:56:17 | ygk_12345 | libvirt.libvirtError: Requested operation is not valid: format of backing image '/var/lib/nova/instances/_base/a580bb66b9bb88e1af8354d582ab15a2afba3785' of image '/var/lib/nova/instances/9dd2ab12-f050-4eab-a887-ffc9c4a67dc3/disk' was not specified in the image me | |
| 10:56:46 | ygk_12345 | Requested operation is not valid: format of backing image '/var/lib/nova/instances/_base/a580bb66b9bb88e1af8354d582ab15a2afba3785' of image '/var/lib/nova/instances/9dd2ab12-f050-4eab-a887-ffc9c4a67dc3/disk' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting) | |
| 10:56:52 | ygk_12345 | is this a known issue ? | |
| 10:57:53 | lyarwood | ygk_12345: yeah with libvirt >= 6.0.0 | |
| 10:57:59 | lyarwood | ygk_12345: are you using the train UCA? | |
| 10:58:29 | ygk_12345 | lyarwood it is OSA train 20.0.2 version | |
| 10:59:11 | lyarwood | ygk_12345: https://review.opendev.org/#/c/710788/ landed in stable train 20.2.0 and should resolve that | |
| 10:59:27 | lyarwood | ygk_12345: to be clear that's Nova 20.2.0 | |
| 10:59:59 | ygk_12345 | lyarwood so is this fixed in 20.2.0 OSA train version ? | |
| 11:00:37 | lyarwood | ygk_12345: it's fixed in the 20.2.0 version of Nova, I have no idea which version of OSA would use that | |
| 11:00:49 | ygk_12345 | oh ok | |
| 11:00:56 | lyarwood | https://github.com/openstack/nova/releases/tag/20.2.0 | |
| 11:02:36 | brinzhang0 | stephenfin: addressed your comments, except above :) thanks | |
| 11:23:11 | elod | lyarwood: sorry, just one question: where can we see that we are already using train UCA? Is there a patch for that? | |
| 11:23:11 | CeeMac | lyarwood: would it just be the nova-compute logs or the others too you would need extracts from? We had to delete / recreate one of the instances so it'll be tricky tracking back to find the correct request ID for that one sadly but I have another one where only 1 of the disks succeeded in retyping, the other didnt and it is also now dead | |
| 11:26:14 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Improve CinderFixtureNewAttachFlow https://review.opendev.org/749228 | |
| 11:26:15 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Robustify attachment tracking in CinderFixtureNewAttachFlow https://review.opendev.org/749229 | |
| 11:26:16 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: compute: Use source_bdms to reset attachment_ids during LM rollback https://review.opendev.org/749230 | |
| 11:26:17 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: compute: refactor volume bdm rollback error handling https://review.opendev.org/749231 | |
| 11:26:19 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Add regression tests for bug #1889108 https://review.opendev.org/749232 | |
| 11:26:19 | openstack | bug 1889108 in OpenStack Compute (nova) queens "failures during driver.pre_live_migration remove source attachments during rollback" [Undecided,In progress] https://launchpad.net/bugs/1889108 - Assigned to Lee Yarwood (lyarwood) | |
| 11:26:20 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: compute: Don't delete the original attachment during pre LM rollback https://review.opendev.org/749233 | |
| 11:26:43 | lyarwood | elod: https://review.opendev.org/#/c/752641/ sorry | |
| 11:26:54 | lyarwood | elod: I can respin the commit to point to that if you'd like | |
| 11:27:17 | lyarwood | CeeMac: nova-compute would be the most useful yes | |
| 11:27:31 | elod | lyarwood: no need for that, I just haven't found that :) | |
| 11:56:49 | CeeMac | lyarwood: where would libvirtd log to by default? | |
| 11:57:28 | sean-k-mooney | /var/log/libivrt | |
| 11:57:39 | sean-k-mooney | unless you are using ooo | |
| 11:58:19 | sean-k-mooney | in which case they are in /var/log/containers/nova/nova_libvirt or something like that | |
| 11:59:04 | CeeMac | using osa, will check | |