| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-29 | |||
| 15:01:53 | bauzas | do you want more time ? | |
| 15:02:12 | whoami-rajat | bauzas, yes we did, no not more time but doing it half an hour before | |
| 15:02:19 | whoami-rajat | at 1530 | |
| 15:05:59 | bauzas | whoami-rajat: sorry, you mean you want to move the session half a hour before ? | |
| 15:06:17 | whoami-rajat | bauzas, yes correct, at 1530 UTC | |
| 15:06:27 | bauzas | whoami-rajat: ok, and for how many time ? | |
| 15:06:30 | bauzas | 1 hour ? | |
| 15:06:33 | whoami-rajat | yep | |
| 15:06:38 | whoami-rajat | we've 2 topics so 1 hour should do | |
| 15:06:38 | bauzas | ie. 1530-1630 ? | |
| 15:06:42 | bauzas | ok | |
| 15:06:55 | bauzas | moving it then on the nova etherpad :) | |
| 15:07:48 | bauzas | whoami-rajat: done https://etherpad.opendev.org/p/nova-bobcat-ptg#L59 | |
| 15:08:16 | whoami-rajat | bauzas, great, thanks for the changing on a short notice :) | |
| 16:41:44 | opendevreview | Amit Uniyal proposed openstack/placement master: Bugtracker link update https://review.opendev.org/c/openstack/placement/+/876768 | |
| 18:39:56 | melwitt | frickler: re: the launchpad page, it appears to be sorting non-active-development series in reverse alphabetical order (I notice that "trunk" is located between "ussuri" and "train"). I haven't been able to guess a reason it sorts alphabetically instead of by the milestone dates set on each series though | |
| 18:40:39 | melwitt | and I don't see a way to change it | |
| 20:56:29 | opendevreview | Sylvain Bauza proposed openstack/nova master: Verify a move operation for cross_az_attach=False https://review.opendev.org/c/openstack/nova/+/878948 | |
| #openstack-nova - 2023-03-30 | |||
| 05:38:52 | opendevreview | Amit Uniyal proposed openstack/placement master: Bugtracker link update https://review.opendev.org/c/openstack/placement/+/876768 | |
| 06:03:36 | frickler | melwitt: that at least sounds plausible, thanks for checking | |
| 06:18:08 | opendevreview | Tobias Urdin proposed openstack/nova master: Remove libvirt tunnelled migration https://review.opendev.org/c/openstack/nova/+/879021 | |
| 09:34:42 | opendevreview | Tobias Urdin proposed openstack/nova master: Remove libvirt tunnelled migration https://review.opendev.org/c/openstack/nova/+/879021 | |
| 09:35:18 | bauzas | if people are OK, I'd like to add a functest for testing cross_az_attach https://review.opendev.org/c/openstack/nova/+/878948 | |
| 09:41:19 | gibi | bauzas: I will be absent from the vPTG from 15:00 UTC hopefully only for half an hour due to a downstream call | |
| 09:44:18 | bauzas | ack | |
| 09:44:30 | bauzas | we'll discuss with glance | |
| 09:44:41 | bauzas | see the agenda I posted on the ML | |
| 09:44:57 | bauzas | we'll discuss with glance *at that time* | |
| 10:15:23 | gibi | ack | |
| 10:27:15 | opendevreview | yatin proposed openstack/nova master: [DNM] Test lower tb cache https://review.opendev.org/c/openstack/nova/+/868419 | |
| 10:41:36 | tobias-urdin | i have a conundrum that i cannot wrap my head around, i've been looking at the possibility of removing the need for remotefs (rsync/scp over ssh) when doing live migrations, that should be possible but requires some RPC changes to get rid of testing if instance dir is on shared storage and some other stuff, just out of curiosity i checked the | |
| 10:41:37 | tobias-urdin | remotefs usage all around and we also use it for config drive migrations (as qemu (libvirt blocks us) does not allow migrating read-only ISO source file), fetching kernel and ramdisk, copying vtpm data, copying cached images from other compute nodes(?) etc | |
| 10:42:38 | tobias-urdin | so while what I want to do, to remove the need for SSH key distribution for live migrations is possible, getting it completely removed in libvirt driver seems hard, because I cannot wrap my head around what we could replace that logic in terms of copying files around | |
| 10:43:54 | tobias-urdin | if only libvirt had a copy file implementation in their protocol we wouldn't need anything else for remote access and could get tls etc, theoretically could be done with block devices using storage pools but that would be even worse imo | |
| 11:13:34 | jrosser | tobias-urdin: key distribution can be avoided by using signed keys | |
| 11:21:21 | zigo | - upgrading both source and destination from Qemu 5.2 to 7.2 (from bullseye-backports) fixes the issue ! \o/ | |
| 11:21:21 | zigo | - It only happens with SOME images, like Rocky Linux 8.7 | |
| 11:21:21 | zigo | bauzas: You remember I couldn't do some live-migration with some of my VMs? It turns out that: | |
| 11:23:19 | tobias-urdin | jrosser: yeah, but it's also more to it than that, what about lateral movement between compute nodes; being able to essentially wipe instance disks of another compute node, or what about a operating system with no ssh running, sure could run ssh in a container and expose nova dir but then point #1 still applies, hard problem | |
| 11:38:30 | sean-k-mooney1 | zigo: glad its fixed but odd that it depeneded on the image | |
| 11:39:05 | zigo | sean-k-mooney1: According to the people from #qemu, it's likely that the image is using some new feature of the virtio stuff. | |
| 11:39:16 | zigo | What's weird is that with Roky Linux 9, I didn't have the issue... | |
| 11:39:51 | sean-k-mooney1 | its proably using the transitional virtio device or something like that | |
| 11:40:14 | sean-k-mooney1 | as it the driver is proably negocating an older feature set | |
| 11:40:34 | zigo | sean-k-mooney1: I'll upgrade all my cluster to the newer version of Qemu and will migrate (non-live) the problematic instances ... | |
| 11:41:17 | zigo | Still very annoying, but lucky, only very few instances are affected. | |
| 12:29:56 | kashyap | zigo: Ahh, so upgrading the source and dest QEMU to 7.2 fixes -- did you test it? | |
| 12:31:00 | kashyap | Well, you did test it, otherwise, you wouldn't put that "\o/" | |
| 12:36:39 | bauzas | zigo: sorry was taking a bit of time off before the vPTG | |
| 12:36:58 | bauzas | (after the usual child taxi for lunch :p ) | |
| 12:39:52 | bauzas | as I said to the PTGbot, we start at 1pm UTC with the neutron x-p session in the Neutron room (juno, ie. https://www.openstack.org/ptg/rooms/juno ) | |
| 12:47:42 | artom | sean-k-mooney, I'll be late for that (son's dentist appointment), can you cover the delete_on_termination stuff? | |
| 12:47:53 | artom | IIRC you're the only other person with the context | |
| 12:49:12 | sean-k-mooney | am yes i can | |
| 12:49:23 | artom | Cheers! | |
| 13:30:49 | bauzas | dvo-plv: are you in the neutron room ? | |
| 13:30:57 | bauzas | dvo-plv: we're discussing your topic now | |
| 13:37:10 | dvo-plv | yes, thank you | |
| 14:42:32 | bauzas | break now until 3pm UTC and then please join the cinder room : https://bluejeans.com/556681290 | |
| 14:42:54 | bauzas | also, I had a topic about Xena EM, we'll discuss this at next meeting | |
| 14:43:10 | bauzas | (unless people have concerns by now, fer sur) | |
| 14:43:39 | elodilles | bauzas: ack, we can have a couple of words about it o/ | |
| 14:44:02 | bauzas | elodilles: tbh, I need to look at the current open changfes | |
| 14:44:29 | sean-k-mooney | bauzas: what room should we be in next | |
| 14:44:34 | sean-k-mooney | well now | |
| 14:45:06 | bauzas | sean-k-mooney: cinder room, 3pm UTC | |
| 14:45:09 | bauzas | we have a break now | |
| 14:45:19 | elodilles | bauzas: yes, there are plenty of open patches: https://review.opendev.org/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/placement+OR+project:openstack/nova)+branch:stable/xena | |
| 14:45:26 | bauzas | that was my ask before we left | |
| 14:46:02 | elodilles | the question is though whether anyone see anything that should be part of the 'final-before-em' release of stable/xena | |
| 14:46:04 | dansmith | sean-k-mooney: what is the qemu security issue that started causing detach issues that you mentioned on the list? | |
| 14:46:37 | sean-k-mooney | dansmith: the orgianl motivation for the change in qemu was related to a secuity issue i belvie | |
| 14:46:46 | sean-k-mooney | i dont actully know the details | |
| 14:46:52 | dansmith | sean-k-mooney: but what's the change? I wasn't aware anything changed (intentionally) | |
| 14:46:58 | bauzas | was in libvirt 8, right? | |
| 14:47:16 | sean-k-mooney | it was undefiend behaivor if you could retry detach while it was in progress | |
| 14:47:29 | sean-k-mooney | they intentionally made it an error and have it abort the inprogress detach | |
| 14:47:33 | bauzas | elodilles: I can raise the question to the nova team by email | |
| 14:47:52 | elodilles | bauzas: yepp, that is perfectly OK i think | |
| 14:47:55 | bauzas | elodilles: and we could conclude on the next weekly meeting | |
| 14:47:56 | sean-k-mooney | in old version fo qemu it would actully try detaching again | |
| 14:48:03 | elodilles | bauzas: ++ | |
| 14:48:07 | dansmith | sean-k-mooney: that's not actually causing us trouble though right? if we needed to retry the detach it probably wasn't working anyway, right? | |
| 14:48:16 | bauzas | elodilles: or the next one, I've seen the deadline for approving | |
| 14:48:31 | dansmith | sean-k-mooney: ah meaning we're never sending the acpi event anymore after the first one? | |
| 14:49:05 | sean-k-mooney | correct after the first one it never gets sent again but also our retry mechaium would stop the detach form proceeding | |
| 14:49:22 | sean-k-mooney | i.e. if was just slow detachign and we retried it would abort the detach | |
| 14:49:51 | elodilles | bauzas: though we should not postpone the release close to the transition, otherwise if we hurry and merge things and something will be broken then we cannot fix it anymore ;) | |
| 14:49:53 | dansmith | hmm, okay | |
| 14:49:58 | sean-k-mooney | dansmith: gibi swapped use form blind retryes on a timeout/interval oto trying to use qemu events | |
| 14:50:16 | sean-k-mooney | or maybe that was lee | |
| 14:50:29 | sean-k-mooney | in either case that was not enough to resolve this issue | |
| 14:50:46 | dansmith | okay, I guess I didn't know about this other detail | |
| 14:50:47 | sean-k-mooney | it just seam like we need to kick the vms several times to get the detach to work | |
| 14:51:06 | bauzas | elodilles: yeah that's understandable, we shall not be lazy | |
| 14:51:09 | dansmith | if it's a matter of the first event getting missed or something, that definitely *could* support the "use a real distro" argument I guess | |
| 14:51:32 | dansmith | if the first one fails, is there some way the "in progress"-ness gets reset such that allowing a retry *ever* works? | |
| 14:51:39 | bauzas | elodilles: despite (tbh), I'm like ~0% interested about the Xena branch :) | |
| 14:51:55 | bauzas | actually, EM is helping my work :) | |
| 14:52:02 | sean-k-mooney | dansmith: currently i belive there is no way to rest the state without restarting the vm | |