| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-22 | |||
| 09:44:18 | bauzas | ralonsoh: we have two topics for your team :) https://etherpad.opendev.org/p/nova-bobcat-ptg#L57 | |
| 09:45:58 | ralonsoh | bauzas, give me 10 mins, in my 1:1 now | |
| 09:46:06 | bauzas | cool cool | |
| 09:57:52 | ralonsoh | bauzas, yeah, I saw them this morning. And I have another one: https://etherpad.opendev.org/p/neutron-bobcat-ptg#L95 | |
| 09:58:05 | ralonsoh | is thursday ok? | |
| 09:58:17 | ralonsoh | I think we'll need 30 mins per topic, what do you think? | |
| 10:01:26 | bauzas | ralonsoh: yup, we can do this on Thursday at 1300UTC if you want | |
| 10:01:35 | bauzas | for something like 2 hours | |
| 10:01:40 | ralonsoh | so from 1300 to 1500, perfect | |
| 10:01:49 | bauzas | cool | |
| 10:01:49 | ralonsoh | Nova or Neutron channel? | |
| 10:01:54 | bauzas | as you want | |
| 10:02:02 | ralonsoh | I'll schedule it in Neutron | |
| 10:02:05 | ralonsoh | and I'll send the mail today | |
| 10:02:14 | bauzas | ok | |
| 10:02:17 | bauzas | lgtm | |
| 10:02:41 | bauzas | ralonsoh: just make sure you add the [nova] tag in your email then ;) | |
| 10:02:45 | ralonsoh | for sure | |
| 10:06:17 | bauzas | ralonsoh: you haven't yet booked a room, right ? | |
| 10:06:29 | ralonsoh | no, I'm doing the schedule now | |
| 10:06:32 | bauzas | ok | |
| 10:08:22 | bauzas | carloss: pinging you too because we have a topic for manila too https://etherpad.opendev.org/p/nova-bobcat-ptg#L52 | |
| 10:09:00 | bauzas | carloss: could we discuss this with the Manila team on Wednesday 1500 UTC ? | |
| 10:09:11 | bauzas | for one hour | |
| 10:59:37 | carloss | bauzas: o/ - yes, Wednesday 15UTC sounds good | |
| 10:59:42 | carloss | thanks | |
| 11:00:16 | bauzas | carloss: cool, I'll add it | |
| 11:00:30 | bauzas | carloss: in the nova room works for you ? | |
| 11:01:43 | carloss | bauzas: yes, it works | |
| 11:01:48 | bauzas | cool | |
| 11:02:44 | bauzas | carloss: added in our etherpad then https://etherpad.opendev.org/p/nova-bobcat-ptg#L40 | |
| 11:03:20 | carloss | bauzas++ thanks! | |
| 11:03:41 | bauzas | np | |
| 11:06:53 | bauzas | ralonsoh: can we use the nova etherpad for our cross-project session ? | |
| 11:22:05 | bauzas | ralonsoh: nevermind, let's use your own | |
| 11:22:18 | ralonsoh | I don't mind, there is no problem | |
| 14:04:55 | opendevreview | Merged openstack/nova stable/zed: Reproducer for bug 1951656 https://review.opendev.org/c/openstack/nova/+/866151 | |
| 15:05:24 | opendevreview | René Ribaud proposed openstack/nova master: Reproducers for bug 1869804 https://review.opendev.org/c/openstack/nova/+/877772 | |
| 15:05:24 | opendevreview | René Ribaud proposed openstack/nova master: Fix live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/877773 | |
| 15:33:55 | opendevreview | Merged openstack/nova stable/zed: Handle mdev devices in libvirt 7.7+ https://review.opendev.org/c/openstack/nova/+/866152 | |
| 16:41:32 | opendevreview | Dan Smith proposed openstack/nova master: Make scheduler lazy-load the placement client https://review.opendev.org/c/openstack/nova/+/878238 | |
| 17:47:03 | dansmith | bauzas: do you think we need to do much discussing of this during the ptg? https://review.opendev.org/c/openstack/nova-specs/+/877291 | |
| 17:47:12 | dansmith | or can/should we try to get it approved earlier? | |
| 17:47:27 | dansmith | it's basically what we already have in the backlog spec for the next step | |
| 17:47:47 | bauzas | dansmith: I can do a round of reviews tomorrow | |
| 17:47:53 | dansmith | ack thanks | |
| 17:48:02 | bauzas | I actually *should* do it before the PTG anyway | |
| 17:48:19 | dansmith | yeah, I was thinking it might be good to better surface what actually needs a lot of discussion | |
| 18:55:58 | gmann | dansmith: 1 comment for skip-level-always job change (in case you did not see) https://review.opendev.org/c/openstack/nova/+/875773 | |
| 18:56:13 | dansmith | gmann: ack will look later | |
| 18:56:20 | gmann | sure, thanks | |
| 18:58:25 | opendevreview | René Ribaud proposed openstack/nova master: Reproducers for bug 1869804 https://review.opendev.org/c/openstack/nova/+/877772 | |
| 18:58:26 | opendevreview | René Ribaud proposed openstack/nova master: Fix live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/877773 | |
| #openstack-nova - 2023-03-23 | |||
| 09:16:02 | auniyal | Hi sean-k-mooney | |
| 09:16:09 | auniyal | can you please review this - https://review.opendev.org/c/openstack/nova/+/790447/ | |
| 09:44:53 | zigo | Hi there! | |
| 09:44:53 | zigo | I'm currently doing routine upgrade of compute nodes in a cluster (running Victoria), and I'm getting live-migration errors of VMs like this one: | |
| 09:44:53 | zigo | https://paste.opendev.org/show/bYlTfz7fxnQtzVhpVf91/ | |
| 09:44:53 | zigo | Is it known? Is there a way to fix? Is it related to the version of libvirt or qemu? | |
| 09:49:53 | bauzas | looking | |
| 09:52:47 | bauzas | zigo: good question I guess you've seen the libvirt error | |
| 09:52:49 | bauzas | 2023-03-23 09:37:52.682 3209246 INFO nova.compute.manager [req-5fda88d8-510a-4943-9703-9b47e865a89f - - - - -] [instance: 17672112-c416-494a-88f8-fd7cfa85453b] VM Resumed (Lifecycle Event) 2023-03-23 09:37:52.694 3209246 ERROR nova.virt.libvirt.driver [-] [instance: 17672112-c416-494a-88f8-fd7cfa85453b] Live Migration failure: internal error: qemu unexpectedly closed the monitor: 2023-03-23T09:37:52.215715Z qemu-system-x86_64: VQ | |
| 09:52:51 | bauzas | 0 size 0x80 < last_avail_idx 0x0 - used_idx 0x44 2023-03-23T09:37:52.215742Z qemu-system-x86_64: Failed to load virtio-balloon:virtio 2023-03-23T09:37:52.215745Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:05.0/virtio-balloon' | |
| 09:53:06 | zigo | Yeah, I do. But then, where do I look? | |
| 09:53:09 | zigo | Libvirt logs? | |
| 09:54:07 | bauzas | it reminds me this bug report https://bugs.launchpad.net/cloud-archive/+bug/1848497 | |
| 09:55:19 | zigo | I haven't see anything doing a tail of /var/log/libvirt/qemu/*.log | |
| 09:55:58 | zigo | On Bullseye, I'm running with qemu 1:5.2+dfsg-11+deb11u2 | |
| 09:57:45 | bauzas | and you're not seeing anything with qemu logs ? | |
| 09:58:07 | bauzas | the error is reported by qemu process, not by libvirtd | |
| 09:58:18 | bauzas | so I'd say check the qemu logs | |
| 10:03:23 | bauzas | zigo: so it seems a qemu migration *to* a node with a version of 4.0 or higher is problematic | |
| 10:03:34 | bauzas | which qemu version the source is running ? | |
| 10:03:51 | bauzas | I assume you're not mixing releases | |
| 10:03:58 | bauzas | but I wanted to double-check | |
| 10:04:53 | zigo | Same version of qemu and libvirt in both source and dest. | |
| 10:05:27 | zigo | It's a plain Bullseye, so I use whatever is in Debian Stable (minus the security upgrades that I'm trying to perform). | |
| 10:06:01 | bauzas | and what migration flags are you using ? | |
| 10:06:25 | bauzas | are the vms paused ? | |
| 10:06:39 | bauzas | or suspended ? | |
| 10:08:45 | zigo | They are ACTIVE. | |
| 10:08:56 | zigo | Is there migration flags I can set?!? :) | |
| 10:08:59 | zigo | Where do I look? | |
| 10:09:41 | zigo | I've just done "nova host-evacuate-live <hostname>" ... | |
| 10:10:05 | zigo | (not sure if there's a way to do this with python-openstackclient from Victoria...) | |
| 10:11:11 | zigo | What's weird, is that MANY VMs on the same host are live-migrating without a glitch. Then on average, 2 VMs on each compute can't live-migrate ... | |
| 10:11:49 | sean-k-mooney | zigo: there isnet and that intentional | |
| 10:12:05 | sean-k-mooney | host-evacuate-live is not somthing we recomend operators use | |
| 10:12:14 | zigo | sean-k-mooney: What should I use then? | |
| 10:12:15 | sean-k-mooney | we are intentioally not supporting it in osc | |
| 10:12:33 | bauzas | wait | |
| 10:12:41 | bauzas | evacuate or live-migrate ? | |
| 10:12:45 | bauzas | I'm lost here | |
| 10:12:46 | sean-k-mooney | zigo: you should write yoru won code to live migrate all the vms forma host that actully has error handeling | |
| 10:12:53 | bauzas | oh | |
| 10:12:58 | bauzas | host-evacuate-live | |
| 10:13:08 | bauzas | damn old unspported CLIs | |
| 10:13:24 | sean-k-mooney | technially deprecated rather then unsuppported | |
| 10:13:35 | sean-k-mooney | until we can remvoe it in C/D | |
| 10:13:47 | sean-k-mooney | we need everyone ot use the sdk first | |