| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-04-21 | |||
| 16:13:35 | bauzas | mnaser: you're not the first one to ask for reviews on that stephenfin's old patch, will mark it as review priority for the team | |
| 16:14:10 | mnaser | bauzas: cool thanks, the other one might have been ricolin but thats coming from the same side, so not sure if thats super fair ;) haha | |
| 16:14:34 | bauzas | that being said, the bp isn't validated yet | |
| 16:15:14 | bauzas | and not sure stephenfin will unghost himself :) | |
| 16:15:39 | stephenfin | bauzas: I think mnaser and ricolin are taking care of it now? | |
| 16:16:00 | bauzas | good question | |
| 16:16:14 | mnaser | yeah it's ready (imho) from a code perspective, but if there's something else we have to do, we can take care of | |
| 16:17:20 | bauzas | mnaser: this is just a paperwork question | |
| 16:17:33 | bauzas | since a new extraspec is added + the api validation | |
| 16:17:42 | bauzas | this has to be tracked correctly | |
| 16:17:55 | bauzas | and we need to balance in a meeting whether we need a spec or not | |
| 16:18:40 | bauzas | mnaser: that being said, I'm more than glad to welcome you as a new feature contributor ! :p | |
| 16:19:03 | bauzas | we don't have badges but I can create one for Berlin :p | |
| 16:19:23 | stephenfin | I can't say if it needs a spec, but it definitely needs a blueprint and some discussion in the meeting | |
| 16:19:36 | stephenfin | given there's now an extra spec (I think? | |
| 16:19:38 | stephenfin | ) | |
| 16:20:34 | bauzas | yeah and yeah | |
| 16:20:45 | bauzas | (needs a blueprint and probably don't need a spec) | |
| 16:21:26 | bauzas | stephenfin: have you just added some fancy gerrit topic for some non-existing related blueprint ? | |
| 16:21:43 | bauzas | if so, you're bragging. | |
| 16:22:00 | bauzas | :p | |
| 16:37:19 | opendevreview | Kashyap Chamarthy proposed openstack/nova master: libvirt: Add workaround to remove compareCPU() check on the destination https://review.opendev.org/c/openstack/nova/+/838926 | |
| 16:40:37 | opendevreview | Kashyap Chamarthy proposed openstack/nova master: libvirt: Add workaround to skip compareCPU() check on the destination https://review.opendev.org/c/openstack/nova/+/838926 | |
| 17:08:48 | sean-k-mooney | mnaser: viommu is not useful without a lot of extra work | |
| 17:08:57 | sean-k-mooney | stephenfin: that definetly needs a spec | |
| 17:09:24 | sean-k-mooney | viommu is trivial to enable i have implemeted that locally before | |
| 17:09:43 | sean-k-mooney | mnaser: the issue is all devices will be in the same iommu group | |
| 17:10:20 | sean-k-mooney | so either you disable the memory isolation in the guest vfio-pci module | |
| 17:10:35 | sean-k-mooney | or nova has to take full contol of the pcie toplogy | |
| 17:10:52 | sean-k-mooney | mnaser: stephenfin did some poc work related to this | |
| 17:11:03 | sean-k-mooney | but i dont think its complete | |
| 17:11:15 | sean-k-mooney | and it had upgrade impacts potentially | |
| 17:11:21 | stephenfin | far from complete | |
| 17:11:35 | sean-k-mooney | so it needs a spec to figure that out and more expirece playing with the poc | |
| 17:12:00 | sean-k-mooney | mnaser: by the way apprently the prefomcne of the viommu is really bad with passthough devices | |
| 17:12:13 | sean-k-mooney | im not sure why but that is what the virt team told me in the past | |
| 17:12:58 | sean-k-mooney | hopefully that has changed mnaser have you done any tests locally with livbirt? | |
| 17:13:27 | mnaser | sean-k-mooney: so, my knowledge level to this is that we have a company which ships pci cards which are hardware accelerators | |
| 17:13:58 | mnaser | sean-k-mooney: as we tried to get them to use native openstack, we figured out the extra qemu parameters they had and it included the iommu stuff | |
| 17:14:10 | mnaser | Supposedly, the cards won’t work without them in PCI pass through in the guests. | |
| 17:14:22 | sean-k-mooney | ack | |
| 17:14:31 | sean-k-mooney | what driver to you use in the guest | |
| 17:14:41 | sean-k-mooney | is it vfio-pci or a normal kernel dirver | |
| 17:14:58 | sean-k-mooney | if they dont need to be in there own iommu group the stephenfin patch would enable your usecase | |
| 17:15:04 | mnaser | I think you do bring up a good point that we’re adding it by default to q35 machine types, so I think making another flag to make it opt in might be good | |
| 17:15:17 | sean-k-mooney | so we could perhaps start with the simpel turn this one feature | |
| 17:15:44 | sean-k-mooney | we could possble add it by default | |
| 17:16:01 | sean-k-mooney | but what we cant do by defualt is change the pci layout to put each device in its own iommu group | |
| 17:16:11 | sean-k-mooney | at least not without a lot of testing | |
| 17:16:18 | mnaser | Yeah I don’t think this is what the device needs in this case | |
| 17:16:32 | sean-k-mooney | havign a hw:viommu=on|off extra spec if simple | |
| 17:17:14 | sean-k-mooney | ok stephenfin i would be ok with a specless bluepint for just adding the extraspec/image propety for this | |
| 17:17:38 | sean-k-mooney | but if we were to do the iommu group split out that would need the spec | |
| 17:18:38 | sean-k-mooney | mnaser: tldr if we want each device to be in its own iommu group we need to create a pcie expantion bridge per device. and to do that we need to assign virtual pci devivie adress to every device in the xml | |
| 17:19:10 | sean-k-mooney | mnaser: so that is the bit that has upgade concerns as basically every device in the vm would change its address potentially | |
| 17:21:04 | sean-k-mooney | mnaser: have you tried stephenfin's patch with that card? | |
| 17:22:02 | sean-k-mooney | mnaser: im not sure im comfortable with exposeing the adress with by the way | |
| 17:22:08 | sean-k-mooney | that feel a bit too low level | |
| 17:30:50 | mnaser | sean-k-mooney: i haven't yet, but it's kindof a reimplementation of how they've done things with flat libvirt | |
| 17:31:05 | mnaser | but i am thinknig hw:viommu=on|off is a good idea | |
| 17:31:11 | sean-k-mooney | im about to -1 the patch | |
| 17:31:21 | mnaser | in terms of controlling the address, i'm thinking since it's a flavor extra spec and not a image extra spec, it's an operator-level decision | |
| 17:31:25 | mnaser | okay great | |
| 17:31:32 | sean-k-mooney | no | |
| 17:31:39 | mnaser | we'll discuss further there and ill try to poc something as well | |
| 17:31:47 | sean-k-mooney | so image extra specs are ment to be used for contoleing emulated hardware | |
| 17:31:49 | sean-k-mooney | not extra specs | |
| 17:32:01 | sean-k-mooney | so in generaly this shoudl only be exposed as an image property | |
| 17:32:17 | sean-k-mooney | but if we are to expose it in the image i would be open to having it in both | |
| 17:33:02 | sean-k-mooney | so i think we should have somehtin like hw_viommu_modle=none|intel|smmuv3|virtio in the image to contol turing this on | |
| 17:33:22 | sean-k-mooney | and i woudl be open to also haveing hw:viommu_modle=none|intel|smmuv3|virtio for partiy in the flavor | |
| 17:33:49 | sean-k-mooney | the adress space unless its somethign we are going to schdule on i am not conviced we shoudl expose as a configurable | |
| 17:35:30 | mnaser | sean-k-mooney: you know this stuff better than we do, as long as at the end of the day, we can get the xml to show up on those virt guests, we're good to go | |
| 17:35:36 | mnaser | but i agree on the image extra specs controlling hw_viommu_module too | |
| 17:36:13 | sean-k-mooney | going forwad i think virtio is what we will want but that is very new | |
| 17:36:33 | sean-k-mooney | intel only works on x86 and smmuv3 only works on arm | |
| 17:36:36 | sean-k-mooney | virtio works on both | |
| 18:04:00 | sean-k-mooney | mnaser: stephenfin comment in line | |
| 18:04:02 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/830646 | |
| 18:20:11 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Run against performance.json patch https://review.opendev.org/c/openstack/nova/+/838934 | |
| 18:20:15 | dansmith | gmann: ^ | |
| 18:20:53 | sean-k-mooney | dansmith: what is that mesuring by the way | |
| 18:21:05 | gmann | dansmith: ack | |
| 18:21:08 | dansmith | sean-k-mooney: https://85682c22f75746955b38-998a6cfec97762292b03290ad6103366.ssl.cf5.rackcdn.com/837139/20/check/nova-ceph-multistore/d3c4ea2/controller/logs/performance.json | |
| 18:21:22 | dansmith | sean-k-mooney: a random comparison: https://termbin.com/rls0 | |
| 18:22:31 | sean-k-mooney | so ram usage, db queiss and api request for differnt services? | |
| 18:22:41 | dansmith | for the moment | |
| 18:23:07 | sean-k-mooney | ok what is the overall goal? | |
| 18:23:35 | dansmith | ideally to get some flag when a patch increases one of these values substantially | |
| 18:23:40 | sean-k-mooney | by the way we had weird behavior in a downstream case where there was very high memory usage if you disabeld swap entirely | |
| 18:24:10 | dansmith | like, c-bak uses 1GiB of ram for one tempest run, which is pretty dang high.. not sure when that happened (maybe always) but.. | |
| 18:24:36 | sean-k-mooney | does it have swap enabeld on the vm? | |
| 18:24:47 | clarkb | yes we enable swap on all of our test nodes | |
| 18:24:58 | sean-k-mooney | clarkb: i think some jobs turn it back off | |
| 18:25:04 | clarkb | thoug hamybe that happens at a job level but it is definitely there for devstack + tempest beacuse you OOM otherwise | |
| 18:25:13 | clarkb | and ya swapoff is always an option | |
| 18:25:17 | dansmith | yeah we're flying really close to the sun right now | |
| 18:25:26 | sean-k-mooney | dansmith: we saw nova-compute and neutron l2 agents both taking over 2G each downstream when swap was off | |
| 18:25:40 | sean-k-mooney | and it went back to normal when it was turn on | |
| 18:25:45 | dansmith | not sure why that would be, but interesting | |