| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-02 | |||
| 12:55:38 | sean-k-mooney | ahsen: for me it only became a proable for images over about 5-8Gs | |
| 12:56:53 | ahsen | sean-k-mooney Image is not large and we are using SSD also bandwith is not low | |
| 12:57:44 | lyarwood | ahsen: kk, you should see a request-id logged by Nova from Cinder that you can use to grep through your logs | |
| 12:57:54 | sean-k-mooney | ok increasing the interval to 10 might help but you should look at the cidner logs and try and determin why its taking so long in that case | |
| 12:58:12 | sean-k-mooney | ahsen: what cinder backend are you using by the way | |
| 12:58:38 | ahsen | sean-k-mooney I will look at them | |
| 12:58:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: requirements.txt: Bump os-brick to 4.2.0 https://review.opendev.org/c/openstack/nova/+/778177 | |
| 12:58:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: hyper-v rbd volume support https://review.opendev.org/c/openstack/nova/+/763550 | |
| 12:58:56 | lyarwood | okay lets try this again | |
| 13:00:28 | ahsen | lyarwood and sean-k-mooney I will try what you said. Thank you both | |
| 13:11:52 | openstackgerrit | Merged openstack/nova master: libvirt: add AsyncDeviceEventsHandler https://review.opendev.org/c/openstack/nova/+/772381 | |
| 13:26:44 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/673790/14/nova/virt/libvirt/host.py@1244 - going to grab some lunch but let me know if that concern isn't clear still. | |
| 13:43:34 | openstackgerrit | Elod Illes proposed openstack/nova stable/ussuri: Fallback to same-cell resize with qos ports https://review.opendev.org/c/openstack/nova/+/773932 | |
| 13:48:52 | gmann | brinzhang0: sorry i was away yesterday. replied on https://review.opendev.org/c/openstack/nova/+/766726/ | |
| 13:57:53 | bauzas | gibi: dansmith: huzzah \o/ the Compute RPC API bump patch eventually got a +1 from Zuul https://review.opendev.org/c/openstack/nova/+/761452 | |
| 14:03:28 | bauzas | fwiw, the grenade multinode job helped to find an issue | |
| 14:03:41 | bauzas | + some functest related to numa live migration | |
| 14:07:08 | gibi | bauzas: added the RPC bump to my review queue | |
| 14:11:51 | bauzas | gibi: I could split it by having two changes, one for the compute service manager and one for the compute client, but the existing change is quite simple to be reviewed | |
| 14:12:05 | gibi | thanks I will dig into it | |
| 14:15:47 | bauzas | gibi: I can help you to understand how this works | |
| 14:15:58 | sean-k-mooney | FYI we may need to rework how we do memory tracking to fix a previously unknow aspect of pci passhtough | |
| 14:16:16 | bauzas | gibi: once you begin to look at the RPC API, ping me and I'll explain | |
| 14:16:34 | sean-k-mooney | ill try and file a bug for it when i have time but basically memory oversubsciption cant be done if you have pci passthough/sriov | |
| 14:16:36 | bauzas | gibi: tl;dr: I'm providing a 5.x proxy for supporting old clients | |
| 14:16:44 | sean-k-mooney | it might also affect vgpu | |
| 14:16:59 | bauzas | sean-k-mooney: ack | |
| 14:17:06 | bauzas | sean-k-mooney: vgpu or gpu ? | |
| 14:17:10 | sean-k-mooney | both | |
| 14:17:17 | bauzas | why ? | |
| 14:17:18 | sean-k-mooney | we will have to verify it | |
| 14:17:33 | bauzas | b/c vgpu is different from pci passthrough and sriov | |
| 14:17:39 | sean-k-mooney | bauzas: libivrt is locking the guest memory pages whenever we use pci passthoug | |
| 14:17:50 | sean-k-mooney | it may or may not be doing the same for vgpu | |
| 14:17:55 | bauzas | ah | |
| 14:18:05 | bauzas | test it then, yes | |
| 14:18:39 | bauzas | and like I discussed with you, I'd also like to work on pci attach/detach | |
| 14:18:50 | sean-k-mooney | generic attach ya | |
| 14:19:22 | sean-k-mooney | this came up in a call i had this morining for vdpa when i was trying to dig into why it needed the memory to be locked explictly | |
| 14:19:43 | sean-k-mooney | for vdpa libvirt does not do it implcitly which is why i got dma error | |
| 14:20:02 | sean-k-mooney | so i have not actully avalidated it myself yet | |
| 14:20:22 | sean-k-mooney | once i do ill try and write up my findings | |
| 14:21:04 | sean-k-mooney | im not sure ill have time to try and validate vgpu but when i fiture out how to check if teh vm memory is locked maybe you can try? | |
| 14:28:15 | bauzas | sean-k-mooney: sure, I can try to get some vgpu environment (hopefully) | |
| 15:00:08 | sean-k-mooney | vgpus dont work with ubuntu hosts correct at leat not the nvida version? | |
| 15:00:50 | sean-k-mooney | actully im using teh mainline 5.11 kernel so never mind it wont install on my host anyway | |
| 15:01:06 | sean-k-mooney | the vdpa host im using also has a t4 but i dont really have a way to test it there | |
| 15:02:25 | bauzas | sean-k-mooney: no, ubuntu is not supported by nvidia drivers but you can use intel gvt-g if you really want to test vgpus | |
| 15:02:44 | bauzas | and a i915 host if you have one by hand | |
| 15:03:09 | sean-k-mooney | i was just wondering if the host i have would work or not. am maybe my laptop i could test with libvirt directly i guess | |
| 15:03:21 | bauzas | that's what I did for i915 | |
| 15:03:28 | bauzas | I just used an old laptop for testing | |
| 15:03:37 | sean-k-mooney | it wont happen in the next 2 weeks so we can figure it out later. | |
| 15:04:09 | bauzas | that's actually the problem with gvt-g, you can't find good intel cards that are for production | |
| 15:04:28 | bauzas | all the i915 cards are for desktop lines | |
| 15:05:02 | bauzas | but hopefully, ROCI will help us | |
| 15:05:05 | sean-k-mooney | yes they removed integrated grapshic for the server line when they added it | |
| 15:05:27 | sean-k-mooney | im hoping that the new descreet intel gpus will support it | |
| 15:05:35 | bauzas | fwiw, devstack on rhel works okay to me | |
| 15:05:48 | bauzas | but you'd need to run another env | |
| 15:06:04 | sean-k-mooney | i need a newer kernel for the vdpa work so rhel was not an option | |
| 15:06:13 | sean-k-mooney | it does not have the vendor driver | |
| 15:06:33 | sean-k-mooney | 8.4 might but not 8.3 or even centos stream | |
| 15:07:06 | sean-k-mooney | thats why im using the mainline 5.11 kernel for development | |
| 15:08:15 | bauzas | ha | |
| 15:08:23 | bauzas | and what about fedora ? | |
| 15:08:51 | bauzas | in theory, nvidia RPMs should work but I never tested them | |
| 15:08:54 | sean-k-mooney | it has python 3.9 which breaks eventlests and openstack in general | |
| 15:09:35 | bauzas | the dependency saga. | |
| 15:09:54 | sean-k-mooney | i had to use ubuntu 20.10 as it was the only os i could fined with precomipled libvit/qemu new enough and had access to the mainlin kernel and python 3.8 | |
| 15:10:17 | sean-k-mooney | although i ended up compiling qemu form souce anyway due to a driver bug | |
| 15:11:35 | sean-k-mooney | but ya dependicies for really new hardware features is a pain. | |
| 15:13:24 | sean-k-mooney | * happy | |
| 15:41:28 | sean-k-mooney | bauzas: it look like they might have been wrong. libvirt is provide qemu the capablity to lock memory but it does not look like its using it | |
| 15:42:15 | kashyap | gibi: artom: When you get a min, I think I've addressed all the pressing concerns: https://review.opendev.org/c/openstack/nova/+/774240 | |
| 15:43:24 | gibi | kashyap: ack, I will check back | |
| 15:44:21 | kashyap | Thanks; see the small change log for PS10 ["Mar 01 12:54 PM"] -- the only diff in PS11 is to put back the line that I accidentally removed during rebase. | |
| 16:34:46 | gibi | bauzas: I left some questions in the RPC bump series https://review.opendev.org/c/openstack/nova/+/761452 but overall the patch looks good to me | |
| 16:35:02 | bauzas | thanks for the review, /me looks | |
| 16:35:18 | bauzas | gibi: tbc, we need to hold this change until FF | |
| 16:35:36 | gibi | bauzas: is there any RPC impacting change open I should be aware of? | |
| 16:35:39 | bauzas | but the more we review, the quickier we could merge it just after m-3 | |
| 16:35:57 | bauzas | gibi: some compute service bumps AFAIK | |
| 16:36:14 | gibi | true, service version will be in conflict if we merge those | |
| 16:36:29 | bauzas | https://review.opendev.org/c/openstack/nova/+/761452 has a lot of merge conflicts | |
| 16:36:44 | bauzas | so I'm prepared to rebase this change | |
| 16:37:09 | gibi | bauzas: ack. I will prioritize to land this after FF | |
| 16:37:25 | gibi | but we might get FFE requests | |
| 16:37:36 | gibi | with service bumps | |
| 16:37:45 | gibi | so it will be a tricky balance | |
| 16:38:08 | gibi | anyhow we have to be strict with FFEs due to sortness of time til RC1 | |
| 16:38:37 | gibi | brinzhang0: could you please check what would be the good accel_uuid paramter here https://review.opendev.org/c/openstack/nova/+/761452/9/nova/compute/manager.py#9481 ? | |
| 16:40:25 | bauzas | gibi: what we could discuss is whether this parameter should be mandatory | |
| 16:40:56 | sean-k-mooney | gibi: ya with one week i would say we likely dont have time in most case | |
| 16:40:58 | bauzas | same for all the other methods like rebuild | |
| 16:41:57 | gibi | bauzas: I think it should be mandatory as the impl uses accel_uuids to support cyborg devices, so if accel_uuids param is not passed then we loose cyborg supprot | |
| 16:42:00 | gibi | suport | |
| 16:42:01 | gibi | support even | |
| 16:42:12 | gibi | sean-k-mooney: yeah | |
| 16:42:55 | gibi | sean-k-mooney: we have two week between FF (Mar12) and the lat possible date of RC1 Mar26 | |