| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-04-21 | |||
| 17:08:48 | sean-k-mooney | mnaser: viommu is not useful without a lot of extra work | |
| 17:08:57 | sean-k-mooney | stephenfin: that definetly needs a spec | |
| 17:09:24 | sean-k-mooney | viommu is trivial to enable i have implemeted that locally before | |
| 17:09:43 | sean-k-mooney | mnaser: the issue is all devices will be in the same iommu group | |
| 17:10:20 | sean-k-mooney | so either you disable the memory isolation in the guest vfio-pci module | |
| 17:10:35 | sean-k-mooney | or nova has to take full contol of the pcie toplogy | |
| 17:10:52 | sean-k-mooney | mnaser: stephenfin did some poc work related to this | |
| 17:11:03 | sean-k-mooney | but i dont think its complete | |
| 17:11:15 | sean-k-mooney | and it had upgrade impacts potentially | |
| 17:11:21 | stephenfin | far from complete | |
| 17:11:35 | sean-k-mooney | so it needs a spec to figure that out and more expirece playing with the poc | |
| 17:12:00 | sean-k-mooney | mnaser: by the way apprently the prefomcne of the viommu is really bad with passthough devices | |
| 17:12:13 | sean-k-mooney | im not sure why but that is what the virt team told me in the past | |
| 17:12:58 | sean-k-mooney | hopefully that has changed mnaser have you done any tests locally with livbirt? | |
| 17:13:27 | mnaser | sean-k-mooney: so, my knowledge level to this is that we have a company which ships pci cards which are hardware accelerators | |
| 17:13:58 | mnaser | sean-k-mooney: as we tried to get them to use native openstack, we figured out the extra qemu parameters they had and it included the iommu stuff | |
| 17:14:10 | mnaser | Supposedly, the cards won’t work without them in PCI pass through in the guests. | |
| 17:14:22 | sean-k-mooney | ack | |
| 17:14:31 | sean-k-mooney | what driver to you use in the guest | |
| 17:14:41 | sean-k-mooney | is it vfio-pci or a normal kernel dirver | |
| 17:14:58 | sean-k-mooney | if they dont need to be in there own iommu group the stephenfin patch would enable your usecase | |
| 17:15:04 | mnaser | I think you do bring up a good point that we’re adding it by default to q35 machine types, so I think making another flag to make it opt in might be good | |
| 17:15:17 | sean-k-mooney | so we could perhaps start with the simpel turn this one feature | |
| 17:15:44 | sean-k-mooney | we could possble add it by default | |
| 17:16:01 | sean-k-mooney | but what we cant do by defualt is change the pci layout to put each device in its own iommu group | |
| 17:16:11 | sean-k-mooney | at least not without a lot of testing | |
| 17:16:18 | mnaser | Yeah I don’t think this is what the device needs in this case | |
| 17:16:32 | sean-k-mooney | havign a hw:viommu=on|off extra spec if simple | |
| 17:17:14 | sean-k-mooney | ok stephenfin i would be ok with a specless bluepint for just adding the extraspec/image propety for this | |
| 17:17:38 | sean-k-mooney | but if we were to do the iommu group split out that would need the spec | |
| 17:18:38 | sean-k-mooney | mnaser: tldr if we want each device to be in its own iommu group we need to create a pcie expantion bridge per device. and to do that we need to assign virtual pci devivie adress to every device in the xml | |
| 17:19:10 | sean-k-mooney | mnaser: so that is the bit that has upgade concerns as basically every device in the vm would change its address potentially | |
| 17:21:04 | sean-k-mooney | mnaser: have you tried stephenfin's patch with that card? | |
| 17:22:02 | sean-k-mooney | mnaser: im not sure im comfortable with exposeing the adress with by the way | |
| 17:22:08 | sean-k-mooney | that feel a bit too low level | |
| 17:30:50 | mnaser | sean-k-mooney: i haven't yet, but it's kindof a reimplementation of how they've done things with flat libvirt | |
| 17:31:05 | mnaser | but i am thinknig hw:viommu=on|off is a good idea | |
| 17:31:11 | sean-k-mooney | im about to -1 the patch | |
| 17:31:21 | mnaser | in terms of controlling the address, i'm thinking since it's a flavor extra spec and not a image extra spec, it's an operator-level decision | |
| 17:31:25 | mnaser | okay great | |
| 17:31:32 | sean-k-mooney | no | |
| 17:31:39 | mnaser | we'll discuss further there and ill try to poc something as well | |
| 17:31:47 | sean-k-mooney | so image extra specs are ment to be used for contoleing emulated hardware | |
| 17:31:49 | sean-k-mooney | not extra specs | |
| 17:32:01 | sean-k-mooney | so in generaly this shoudl only be exposed as an image property | |
| 17:32:17 | sean-k-mooney | but if we are to expose it in the image i would be open to having it in both | |
| 17:33:02 | sean-k-mooney | so i think we should have somehtin like hw_viommu_modle=none|intel|smmuv3|virtio in the image to contol turing this on | |
| 17:33:22 | sean-k-mooney | and i woudl be open to also haveing hw:viommu_modle=none|intel|smmuv3|virtio for partiy in the flavor | |
| 17:33:49 | sean-k-mooney | the adress space unless its somethign we are going to schdule on i am not conviced we shoudl expose as a configurable | |
| 17:35:30 | mnaser | sean-k-mooney: you know this stuff better than we do, as long as at the end of the day, we can get the xml to show up on those virt guests, we're good to go | |
| 17:35:36 | mnaser | but i agree on the image extra specs controlling hw_viommu_module too | |
| 17:36:13 | sean-k-mooney | going forwad i think virtio is what we will want but that is very new | |
| 17:36:33 | sean-k-mooney | intel only works on x86 and smmuv3 only works on arm | |
| 17:36:36 | sean-k-mooney | virtio works on both | |
| 18:04:00 | sean-k-mooney | mnaser: stephenfin comment in line | |
| 18:04:02 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/830646 | |
| 18:20:11 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Run against performance.json patch https://review.opendev.org/c/openstack/nova/+/838934 | |
| 18:20:15 | dansmith | gmann: ^ | |
| 18:20:53 | sean-k-mooney | dansmith: what is that mesuring by the way | |
| 18:21:05 | gmann | dansmith: ack | |
| 18:21:08 | dansmith | sean-k-mooney: https://85682c22f75746955b38-998a6cfec97762292b03290ad6103366.ssl.cf5.rackcdn.com/837139/20/check/nova-ceph-multistore/d3c4ea2/controller/logs/performance.json | |
| 18:21:22 | dansmith | sean-k-mooney: a random comparison: https://termbin.com/rls0 | |
| 18:22:31 | sean-k-mooney | so ram usage, db queiss and api request for differnt services? | |
| 18:22:41 | dansmith | for the moment | |
| 18:23:07 | sean-k-mooney | ok what is the overall goal? | |
| 18:23:35 | dansmith | ideally to get some flag when a patch increases one of these values substantially | |
| 18:23:40 | sean-k-mooney | by the way we had weird behavior in a downstream case where there was very high memory usage if you disabeld swap entirely | |
| 18:24:10 | dansmith | like, c-bak uses 1GiB of ram for one tempest run, which is pretty dang high.. not sure when that happened (maybe always) but.. | |
| 18:24:36 | sean-k-mooney | does it have swap enabeld on the vm? | |
| 18:24:47 | clarkb | yes we enable swap on all of our test nodes | |
| 18:24:58 | sean-k-mooney | clarkb: i think some jobs turn it back off | |
| 18:25:04 | clarkb | thoug hamybe that happens at a job level but it is definitely there for devstack + tempest beacuse you OOM otherwise | |
| 18:25:13 | clarkb | and ya swapoff is always an option | |
| 18:25:17 | dansmith | yeah we're flying really close to the sun right now | |
| 18:25:26 | sean-k-mooney | dansmith: we saw nova-compute and neutron l2 agents both taking over 2G each downstream when swap was off | |
| 18:25:40 | sean-k-mooney | and it went back to normal when it was turn on | |
| 18:25:45 | dansmith | not sure why that would be, but interesting | |
| 18:25:48 | sean-k-mooney | ... with no swap acutlly being used | |
| 18:25:51 | clarkb | one of the things that happened that led to this was the rbac work drastically increased the db queries/cost and this sort of thing could catch that? | |
| 18:26:09 | dansmith | clarkb: for ironic | |
| 18:26:17 | sean-k-mooney | dansmith: it seamed like when swap was disabele the virtual memory and resident memory was the same | |
| 18:26:29 | sean-k-mooney | but when it was enabel there was a large deleta between the two | |
| 18:26:31 | dansmith | sean-k-mooney: well, makes sense right? | |
| 18:26:46 | dansmith | maybe that's just an artifact then.. | |
| 18:27:13 | dansmith | if you're only looking at resident, then you see the working set, but if you have no other option, then rss=total | |
| 18:27:13 | sean-k-mooney | with swap on the virt memroy was the same but the prviate resent memroy was less and no swap was being used | |
| 18:28:02 | dansmith | right, but it's probably because it has overcomitted to the process and until it's CoWd it can lie with swap but not if there's no swap? | |
| 18:28:33 | sean-k-mooney | i dont realy know but it did nto appre to be paging it out to swap space | |
| 18:28:50 | dansmith | right, but until it needs to it won't, | |
| 18:28:55 | dansmith | but it can lie in that case | |
| 18:28:56 | sean-k-mooney | but perhaps there was some cow sematics at play | |
| 18:29:21 | dansmith | what I mean by cow is, you can expand your heap but until you touch a page, the kernel doesn't have to actually allocate you one | |
| 18:29:58 | sean-k-mooney | yes | |
| 18:30:02 | dansmith | I dunno what python's behavior is really, other than "use a crapload of memory all the time" | |
| 18:30:09 | sean-k-mooney | you can grow the adress space but not page it to phsyical ram | |
| 18:30:19 | sean-k-mooney | but it looked liek without swap python was always commiting | |
| 18:30:33 | dansmith | right, so without swap, you don't want to overcommit because the process thinks it already has that memory | |
| 18:31:05 | dansmith | but with swap, you can lie to it and offload something else during a fault | |
| 18:31:09 | sean-k-mooney | yep so just something to be aware of | |
| 18:31:24 | sean-k-mooney | if you see big difference between jobs | |