| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-02 | |||
| 20:14:49 | sean-k-mooney | not via nova-comptue in the api | |
| 20:15:01 | sean-k-mooney | when we wait for the call | |
| 20:15:21 | sean-k-mooney | i think there are better places to store it however | |
| 20:15:22 | melwitt | if nova-compute needs to rebuild the info cache from nothing, like the db row update example in the bug | |
| 20:16:05 | sean-k-mooney | ya it should be able too | |
| 20:16:30 | sean-k-mooney | we have had cases where we lost ports in the cache due to buggy neturon backend or neutron policy issues | |
| 20:16:38 | sean-k-mooney | e.g. where neutorn returned an empty port list | |
| 20:16:44 | melwitt | nova-compute can't read it from request_specs without it being an upcall. am I missing something? | |
| 20:17:01 | sean-k-mooney | the heal logic will recreate the info cache entries form the neutron data if that happens | |
| 20:17:12 | sean-k-mooney | melwitt: correct it cant | |
| 20:17:32 | melwitt | so storing it in request spec doesn't help afaict | |
| 20:17:41 | sean-k-mooney | not really no | |
| 20:17:43 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/db/main/models.py#L784= | |
| 20:18:08 | sean-k-mooney | the virtual interfaces tabel shoudl store it but it has no feield we can abuse to store it without a db change | |
| 20:18:11 | melwitt | just saying it sounded like a good place to store it initially but if nova-compute can't read it, it doesn't solve this issue | |
| 20:18:47 | sean-k-mooney | instance_system_metadata can store it since it just a set of key value pairs | |
| 20:19:01 | sean-k-mooney | and thats in the cell db | |
| 20:19:07 | sean-k-mooney | so that is proably where i woudl stash it | |
| 20:19:09 | melwitt | yeah, that would work | |
| 20:20:02 | sean-k-mooney | so we jsut have the key be the <neutron port uuid>_preserve_on_delete | |
| 20:20:16 | sean-k-mooney | or store the list as a single key | |
| 20:20:42 | sean-k-mooney | that is proably better since its indexed by the instance_id anyway | |
| 20:20:56 | sean-k-mooney | it denormaises the db technially | |
| 20:21:45 | sean-k-mooney | but a preserve_on_delete_list key that we lookup by "select preserve_on_delete from instance_system_metadata where instance_id = xyz" | |
| 20:22:00 | sean-k-mooney | is much simpler to lookup | |
| 20:22:09 | sean-k-mooney | but either would work | |
| 21:10:21 | opendevreview | Merged openstack/placement stable/ussuri: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840773 | |
| 22:40:37 | opendevreview | melanie witt proposed openstack/nova stable/train: DNM Testing for ceph setup gate fail https://review.opendev.org/c/openstack/nova/+/844530 | |
| #openstack-nova - 2022-06-03 | |||
| 07:05:28 | gibi | o/ | |
| 07:27:25 | bauzas | \o | |
| 08:02:14 | opendevreview | Balazs Gibizer proposed openstack/nova master: Unparent PciDeviceSpec from PciAddressSpec https://review.opendev.org/c/openstack/nova/+/844491 | |
| 08:33:40 | gibi | elodilles, melwitt: what is the stable view on squashes? https://review.opendev.org/c/openstack/nova/+/843680/6#message-f9be680631a00541158243c55609a5804cc7777d | |
| 08:39:43 | gibi | stephenfin: do you have some context on https://review.opendev.org/c/openstack/hacking/+/816676 ? the hacking repo is green now with flake 4 but in nova we have a bunch of new findings. Was there any previous trial moving nova to the newer hacking? | |
| 08:42:14 | songwenping | bauzas:we donnot delete vgpu mdev when delete vm? | |
| 08:42:51 | sean-k-mooney | **cough** black **cough** | |
| 08:43:10 | sean-k-mooney | songwenping: ya our down stream qe may have also found that yesterday | |
| 08:43:18 | sean-k-mooney | songwenping: we are still confirming | |
| 08:44:11 | sean-k-mooney | songwenping: its unclear if something has changed recently i.e. if this only happens if you have mig mode enabled | |
| 08:44:24 | sean-k-mooney | or if this was always an oversight | |
| 08:44:27 | songwenping | sean-k-mooney: from this line https://github.com/openstack/nova/blob/8260979b71b29ce2666d37b3adc7c256482aa16d/nova/virt/libvirt/driver.py#L6485 we can reuse the mdev | |
| 08:44:42 | sean-k-mooney | songwenping: we should not be reusing the mdev | |
| 08:45:10 | kashyap | sean-k-mooney: I think you have Ubuntu box handy, can you please tell the QEMU binary path on it? (Not the /usr/bin/qemu-kvm, but the actual binary) | |
| 08:45:43 | sean-k-mooney | sure that my home server. its not in libexec like its on rhel related distros | |
| 08:45:57 | songwenping | so we should delete the mdev with vm. | |
| 08:45:57 | sean-k-mooney | the fact its different is waht prevent live migration form happening one sec while i check | |
| 08:45:59 | kashyap | sean-k-mooney: I know it's not in /usr/libexec - that's a RHEL/CentOS thing | |
| 08:46:13 | sean-k-mooney | kashyap: yep just sayign they used to be the same | |
| 08:46:33 | kashyap | sean-k-mooney: The thing is on Fedora the binary is this: /usr/bin/qemu-system-x86_64 | |
| 08:46:34 | sean-k-mooney | kashyap: but when centos change to that it broke live migration between centos and ubunutu | |
| 08:47:05 | bauzas | songwenping: no, we leave them | |
| 08:47:07 | kashyap | (And IIRC, Ubuntu uses a different binary name and a different path) | |
| 08:47:13 | sean-k-mooney | /usr/bin/qemu-system-x86_64 | |
| 08:47:30 | sean-k-mooney | its qemu im not seeing qemu-kvm on 22.04 | |
| 08:47:30 | kashyap | Ah, same as Fedora then. Thanks | |
| 08:47:36 | sean-k-mooney | let me double check | |
| 08:47:51 | bauzas | songwenping: you can even precreate the mdevs before starting the nova-compute service, then the n-cpu service will use them instead of creating them | |
| 08:47:51 | sean-k-mooney | sean@cloud:~$ qemu-kvm | |
| 08:47:53 | sean-k-mooney | qemu-kvm: command not found | |
| 08:47:58 | sean-k-mooney | so its not on my path at least | |
| 08:48:04 | kashyap | sean-k-mooney: I don't want that; it's an alias to the QEMU binary on Fedora | |
| 08:48:18 | kashyap | sean-k-mooney: So, "/usr/bin/qemu-system-x86_64" exists on Ubuntu, right? | |
| 08:48:29 | sean-k-mooney | yes | |
| 08:48:48 | songwenping | bauzas:then there will has race problem, https://bugs.launchpad.net/nova/+bug/1836204 | |
| 08:48:58 | sean-k-mooney | sean@cloud:~$ file /usr/bin/qemu-system-x86_64 | |
| 08:49:00 | sean-k-mooney | /usr/bin/qemu-system-x86_64: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=f28deae95b4b0f19a8c8be87c45d1be213cf8b6b, for GNU/Linux 3.2.0, stripped | |
| 08:49:12 | kashyap | sean-k-mooney: Thanks! | |
| 08:49:16 | sean-k-mooney | kashyap: it exiss and is not a symlink its the binary | |
| 08:49:22 | kashyap | Okay, I see that I could've checked it here: https://packages.ubuntu.com/bionic/i386/qemu-system-x86/filelist | |
| 08:49:32 | kashyap | sean-k-mooney: Yeah, I know; if that's the name, then it's the real binary | |
| 08:49:43 | kashyap | 'qemu-kvm' is the symlink on Fedora/CentOS | |
| 08:50:00 | sean-k-mooney | kashyap: not quite | |
| 08:50:12 | sean-k-mooney | well it might be a symlink but it was a script on ubuntu | |
| 08:50:22 | kashyap | /usr/bin/qemu-kvm: symbolic link to qemu-system-x86_64 | |
| 08:50:22 | kashyap | $> file /usr/bin/qemu-kvm | |
| 08:50:28 | songwenping | bauzas:if we create new mdev when alloc, there will be no this problem. | |
| 08:50:33 | kashyap | sean-k-mooney: Yeah, it used to be a wrapper script | |
| 08:50:40 | sean-k-mooney | yep | |
| 08:50:45 | kashyap | (Which just pointed to the real binary. So they removed that) | |
| 08:50:53 | sean-k-mooney | it was a wrapper on ubuntu in 20.04 | |
| 08:51:00 | sean-k-mooney | in 22.04 they droped it | |
| 08:51:44 | bauzas | songwenping: I don't see how it could help | |
| 08:51:59 | sean-k-mooney | kashyap: looks like ti snow a virtual package/metapackage https://packages.ubuntu.com/jammy/qemu-kvm | |
| 08:52:31 | sean-k-mooney | likely for upgrade reasons | |
| 08:53:05 | kashyap | Yeah, same on Fedora | |
| 08:53:19 | sean-k-mooney | kashyap: if you want to poke around i can give you access to the server just let me know | |
| 08:53:19 | kashyap | (contains no files) | |
| 08:53:19 | kashyap | $> rpm -ql qemu-kvm | |
| 08:53:55 | elodilles | gibi: for the sake of easier reviewing i'd say squash is better to avoid. some rare cases though we can have squash: e.g. in case otherwise we could not unblock the gate. or in case we can avoid some introduced bug that the follow-up fixes. | |
| 08:54:04 | kashyap | sean-k-mooney: Thank you, it's not needed at this time. I just wanted to be sure of the path - so that we could supply it to `coredumpctl` :) | |
| 08:54:27 | sean-k-mooney | hum never had to use that | |
| 08:54:39 | sean-k-mooney | does that parse them or cause them :) | |
| 08:55:05 | songwenping | bauzas: in this function https://github.com/openstack/nova/blob/8260979b71b29ce2666d37b3adc7c256482aa16d/nova/virt/libvirt/driver.py#L6416, we use _create_new_mediated_device for every vm instead of select a mdev from mdevs_available. | |
| 08:57:40 | sean-k-mooney | songwenping: we inteded to not reuse mdevs for new vms | |
| 08:57:54 | sean-k-mooney | the other path for intedd for agent restart if i rememebr correctly | |
| 08:58:11 | sean-k-mooney | songwenping: so we shoudl be delete the mdevs if we delete the vm | |
| 08:58:27 | songwenping | sean-k-mooney: cool | |
| 08:58:51 | sean-k-mooney | songwenping: this become more important for cards that can support more then one mdev at a time with generic mdev support | |
| 08:59:15 | sean-k-mooney | i dont think nvidia do with any of there gpus btu other vendors might in the future | |
| 09:00:22 | songwenping | sean-k-mooney: ack, agree to delete the mdevs if we delete the vm | |
| 09:00:22 | sean-k-mooney | bauzas: precrating does not work and i dont think it was part of the sepc so if that was added its a bug | |