| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-23 | |||
| 10:03:51 | bauzas | I assume you're not mixing releases | |
| 10:03:58 | bauzas | but I wanted to double-check | |
| 10:04:53 | zigo | Same version of qemu and libvirt in both source and dest. | |
| 10:05:27 | zigo | It's a plain Bullseye, so I use whatever is in Debian Stable (minus the security upgrades that I'm trying to perform). | |
| 10:06:01 | bauzas | and what migration flags are you using ? | |
| 10:06:25 | bauzas | are the vms paused ? | |
| 10:06:39 | bauzas | or suspended ? | |
| 10:08:45 | zigo | They are ACTIVE. | |
| 10:08:56 | zigo | Is there migration flags I can set?!? :) | |
| 10:08:59 | zigo | Where do I look? | |
| 10:09:41 | zigo | I've just done "nova host-evacuate-live <hostname>" ... | |
| 10:10:05 | zigo | (not sure if there's a way to do this with python-openstackclient from Victoria...) | |
| 10:11:11 | zigo | What's weird, is that MANY VMs on the same host are live-migrating without a glitch. Then on average, 2 VMs on each compute can't live-migrate ... | |
| 10:11:49 | sean-k-mooney | zigo: there isnet and that intentional | |
| 10:12:05 | sean-k-mooney | host-evacuate-live is not somthing we recomend operators use | |
| 10:12:14 | zigo | sean-k-mooney: What should I use then? | |
| 10:12:15 | sean-k-mooney | we are intentioally not supporting it in osc | |
| 10:12:33 | bauzas | wait | |
| 10:12:41 | bauzas | evacuate or live-migrate ? | |
| 10:12:45 | bauzas | I'm lost here | |
| 10:12:46 | sean-k-mooney | zigo: you should write yoru won code to live migrate all the vms forma host that actully has error handeling | |
| 10:12:53 | bauzas | oh | |
| 10:12:58 | bauzas | host-evacuate-live | |
| 10:13:08 | bauzas | damn old unspported CLIs | |
| 10:13:24 | sean-k-mooney | technially deprecated rather then unsuppported | |
| 10:13:35 | sean-k-mooney | until we can remvoe it in C/D | |
| 10:13:47 | sean-k-mooney | we need everyone ot use the sdk first | |
| 10:15:11 | sean-k-mooney | zigo: it wont help now but you used to be able to set https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.mem_stats_period_seconds to 0 to disable the memory ballon | |
| 10:15:48 | sean-k-mooney | can you confirm if that is set to 0 on either host and that the vm has a memory ballon | |
| 10:16:26 | zigo | It's set to default on that host (ie: 10 ...). | |
| 10:16:31 | sean-k-mooney | ack | |
| 10:16:48 | zigo | Should I set it to zero and try again then? | |
| 10:16:50 | sean-k-mooney | i was wondering if having it enabel and disable on differnt hsot coudl cause issues | |
| 10:17:26 | sean-k-mooney | i think this is one of those thigns that you cant change with runnign vms | |
| 10:17:40 | sean-k-mooney | zigo: i assume you cant just cold migrate them | |
| 10:17:43 | zigo | Oh ... :/ | |
| 10:17:43 | bauzas | yup, or it would require a vm recycle | |
| 10:17:59 | sean-k-mooney | /recycle/restart/ | |
| 10:18:24 | zigo | sean-k-mooney: Well, to cold-migrate, I must get in touch with customers to at least warn them about the operation, and let them know their VM will reboot. | |
| 10:18:35 | zigo | That's kind of very annoying with 50 computes and 2k+ VMs ... | |
| 10:18:39 | sean-k-mooney | to answer your orgianl question no im not aware of live migration issues related to memory baloons | |
| 10:20:53 | bauzas | sean-k-mooney: I remember we had some old qemu-4 live migration issues with the qemu balloons | |
| 10:21:02 | bauzas | but zigo isn't impacted | |
| 10:27:50 | sean-k-mooney | zigo: i assume this is persistent | |
| 10:28:03 | sean-k-mooney | i.e. a secodn live migration of the vm has the same error | |
| 10:28:23 | zigo | Right. | |
| 10:28:35 | sean-k-mooney | have you tried migratign to a differnt host? | |
| 10:29:11 | zigo | I didn't try to specify the dest host, but I can try. I'll let you know... | |
| 10:29:24 | sean-k-mooney | im trying to fiture out is the a thing tha that is speciic to the vm, the destion host ectra | |
| 10:32:03 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=1923881 | |
| 10:32:06 | sean-k-mooney | that sound like it | |
| 10:34:13 | sean-k-mooney | similar for virtio-blk https://bugs.launchpad.net/nova/+bug/1737625 | |
| 10:34:55 | sean-k-mooney | ""Dave notes that we get this "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted.""" | |
| 10:36:13 | sean-k-mooney | zigo: are you using post-copy or autoconverge by the way | |
| 10:37:19 | sean-k-mooney | bauzas: this is the qemu 4.0 issue right https://lore.kernel.org/all/156517411102.26464.1302440989654328620.launchpad@gac.canonical.com/T/ | |
| 10:37:42 | sean-k-mooney | https://bugs.launchpad.net/qemu/+bug/1838569 | |
| 10:38:36 | zigo | Not using post-copy (it's set to default, ie: false) | |
| 10:39:11 | zigo | Same for live_migration_permit_auto_converge (set to default: false) | |
| 10:39:22 | zigo | I'm using TLS though... | |
| 10:39:31 | zigo | libvirt over TLS. | |
| 10:40:34 | zigo | I probably should set live_migration_permit_auto_converge to true though, as sometimes, I have to manualy do a live-migration-force-complete ... | |
| 10:40:53 | bauzas | sean-k-mooney: unrelated, was it you who wrote https://etherpad.opendev.org/p/nova-bobcat-ptg#L75 ? | |
| 10:44:07 | sean-k-mooney | i dont rememebr that but i am interested in that | |
| 10:44:15 | sean-k-mooney | maybe dansmith | |
| 10:44:54 | sean-k-mooney | or artom i certenlly ask that question or one like it in our intenal meetings a few week ago | |
| 10:45:41 | sean-k-mooney | my understandign is the glance folks were going to also work on the nova patches | |
| 10:45:43 | bauzas | I'm just mentioning glance will discuss this with cinder | |
| 10:45:47 | sean-k-mooney | and we just needed to reveiew | |
| 10:45:55 | bauzas | and people will join | |
| 10:48:24 | sean-k-mooney | ok are you suggesting we also join that at teh same time | |
| 10:52:46 | bauzas | yeah, if interested | |
| 10:52:53 | bauzas | I wrote that in our etherpad | |
| 10:53:13 | bauzas | but we can ask for a specific glance/nova session if after that some questions remain | |
| 10:53:25 | bauzas | I just arranged this with pdeore | |
| 11:15:26 | artom | sean-k-mooney, bauzas, no, not me | |
| 11:15:47 | bauzas | k | |
| 12:28:22 | stephenfin | Could I get another reviewer on that series to remove sqlalchemy-migrate, please? https://review.opendev.org/q/topic:sqlalchemy-20+project:openstack/nova+is:open | |
| 12:54:12 | sean-k-mooney | so you have two diffent patch changes there | |
| 12:54:41 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/860829 andhttps://review.opendev.org/c/openstack/nova/+/872428 | |
| 12:54:55 | sean-k-mooney | i reviewd the later before and was ok to proceed with it | |
| 12:55:00 | sean-k-mooney | i have not looked at teh former | |
| 13:27:05 | opendevreview | Elod Illes proposed openstack/nova stable/victoria: DNM: gate test https://review.opendev.org/c/openstack/nova/+/878386 | |
| 13:34:16 | opendevreview | Merged openstack/nova stable/victoria: [stable-only][cve] Check VMDK create-type against an allowed list https://review.opendev.org/c/openstack/nova/+/871699 | |
| 13:34:22 | opendevreview | Merged openstack/nova stable/yoga: Reproducer for bug 1951656 https://review.opendev.org/c/openstack/nova/+/866153 | |
| 13:38:13 | dansmith | bauzas: sean-k-mooney wasn't me | |
| 13:38:42 | bauzas | anyway, we'll see this next Thursday then | |
| 13:38:59 | bauzas | elodilles: awesome \o/ https://review.opendev.org/c/openstack/nova/+/871699 | |
| 13:39:31 | bauzas | dansmith: gibi or sean-k-mooney: we need to merge this one now https://review.opendev.org/c/openstack/nova/+/875621 given grenade was modified | |
| 13:41:52 | dansmith | bauzas: yep, got it | |
| 13:43:28 | bauzas | ++ | |
| 13:43:52 | elodilles | bauzas: wow! finally... \o/ | |
| 14:12:54 | dansmith | gmann: around yet? | |
| 14:13:38 | dansmith | gmann: I'm not sure I understand the skip-level-always comment.. you made it gate/voting in the template and thus it needs to be in the gate pipeline in nova's zuul as well is that right? | |
| 14:13:47 | dansmith | just confused by the irrelevant-files part I guess | |
| 14:14:33 | sean-k-mooney | you can override it in nova .zuul.yaml | |
| 14:14:43 | sean-k-mooney | if we wanted too | |
| 14:14:51 | sean-k-mooney | the in repo options take precidnece | |
| 14:15:57 | sean-k-mooney | you would just need to add the job by name to check/gate and set voting false. but i think your really wondiering if ti shoudl be voting? | |
| 14:16:12 | sean-k-mooney | or was this a zuul mechanics question | |
| 14:16:57 | opendevreview | Dan Smith proposed openstack/nova master: Add grenade-skip-level-always to nova https://review.opendev.org/c/openstack/nova/+/875773 | |