| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-04-21 | |||
| 17:30:50 | mnaser | sean-k-mooney: i haven't yet, but it's kindof a reimplementation of how they've done things with flat libvirt | |
| 17:31:05 | mnaser | but i am thinknig hw:viommu=on|off is a good idea | |
| 17:31:11 | sean-k-mooney | im about to -1 the patch | |
| 17:31:21 | mnaser | in terms of controlling the address, i'm thinking since it's a flavor extra spec and not a image extra spec, it's an operator-level decision | |
| 17:31:25 | mnaser | okay great | |
| 17:31:32 | sean-k-mooney | no | |
| 17:31:39 | mnaser | we'll discuss further there and ill try to poc something as well | |
| 17:31:47 | sean-k-mooney | so image extra specs are ment to be used for contoleing emulated hardware | |
| 17:31:49 | sean-k-mooney | not extra specs | |
| 17:32:01 | sean-k-mooney | so in generaly this shoudl only be exposed as an image property | |
| 17:32:17 | sean-k-mooney | but if we are to expose it in the image i would be open to having it in both | |
| 17:33:02 | sean-k-mooney | so i think we should have somehtin like hw_viommu_modle=none|intel|smmuv3|virtio in the image to contol turing this on | |
| 17:33:22 | sean-k-mooney | and i woudl be open to also haveing hw:viommu_modle=none|intel|smmuv3|virtio for partiy in the flavor | |
| 17:33:49 | sean-k-mooney | the adress space unless its somethign we are going to schdule on i am not conviced we shoudl expose as a configurable | |
| 17:35:30 | mnaser | sean-k-mooney: you know this stuff better than we do, as long as at the end of the day, we can get the xml to show up on those virt guests, we're good to go | |
| 17:35:36 | mnaser | but i agree on the image extra specs controlling hw_viommu_module too | |
| 17:36:13 | sean-k-mooney | going forwad i think virtio is what we will want but that is very new | |
| 17:36:33 | sean-k-mooney | intel only works on x86 and smmuv3 only works on arm | |
| 17:36:36 | sean-k-mooney | virtio works on both | |
| 18:04:00 | sean-k-mooney | mnaser: stephenfin comment in line | |
| 18:04:02 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/830646 | |
| 18:20:11 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Run against performance.json patch https://review.opendev.org/c/openstack/nova/+/838934 | |
| 18:20:15 | dansmith | gmann: ^ | |
| 18:20:53 | sean-k-mooney | dansmith: what is that mesuring by the way | |
| 18:21:05 | gmann | dansmith: ack | |
| 18:21:08 | dansmith | sean-k-mooney: https://85682c22f75746955b38-998a6cfec97762292b03290ad6103366.ssl.cf5.rackcdn.com/837139/20/check/nova-ceph-multistore/d3c4ea2/controller/logs/performance.json | |
| 18:21:22 | dansmith | sean-k-mooney: a random comparison: https://termbin.com/rls0 | |
| 18:22:31 | sean-k-mooney | so ram usage, db queiss and api request for differnt services? | |
| 18:22:41 | dansmith | for the moment | |
| 18:23:07 | sean-k-mooney | ok what is the overall goal? | |
| 18:23:35 | dansmith | ideally to get some flag when a patch increases one of these values substantially | |
| 18:23:40 | sean-k-mooney | by the way we had weird behavior in a downstream case where there was very high memory usage if you disabeld swap entirely | |
| 18:24:10 | dansmith | like, c-bak uses 1GiB of ram for one tempest run, which is pretty dang high.. not sure when that happened (maybe always) but.. | |
| 18:24:36 | sean-k-mooney | does it have swap enabeld on the vm? | |
| 18:24:47 | clarkb | yes we enable swap on all of our test nodes | |
| 18:24:58 | sean-k-mooney | clarkb: i think some jobs turn it back off | |
| 18:25:04 | clarkb | thoug hamybe that happens at a job level but it is definitely there for devstack + tempest beacuse you OOM otherwise | |
| 18:25:13 | clarkb | and ya swapoff is always an option | |
| 18:25:17 | dansmith | yeah we're flying really close to the sun right now | |
| 18:25:26 | sean-k-mooney | dansmith: we saw nova-compute and neutron l2 agents both taking over 2G each downstream when swap was off | |
| 18:25:40 | sean-k-mooney | and it went back to normal when it was turn on | |
| 18:25:45 | dansmith | not sure why that would be, but interesting | |
| 18:25:48 | sean-k-mooney | ... with no swap acutlly being used | |
| 18:25:51 | clarkb | one of the things that happened that led to this was the rbac work drastically increased the db queries/cost and this sort of thing could catch that? | |
| 18:26:09 | dansmith | clarkb: for ironic | |
| 18:26:17 | sean-k-mooney | dansmith: it seamed like when swap was disabele the virtual memory and resident memory was the same | |
| 18:26:29 | sean-k-mooney | but when it was enabel there was a large deleta between the two | |
| 18:26:31 | dansmith | sean-k-mooney: well, makes sense right? | |
| 18:26:46 | dansmith | maybe that's just an artifact then.. | |
| 18:27:13 | sean-k-mooney | with swap on the virt memroy was the same but the prviate resent memroy was less and no swap was being used | |
| 18:27:13 | dansmith | if you're only looking at resident, then you see the working set, but if you have no other option, then rss=total | |
| 18:28:02 | dansmith | right, but it's probably because it has overcomitted to the process and until it's CoWd it can lie with swap but not if there's no swap? | |
| 18:28:33 | sean-k-mooney | i dont realy know but it did nto appre to be paging it out to swap space | |
| 18:28:50 | dansmith | right, but until it needs to it won't, | |
| 18:28:55 | dansmith | but it can lie in that case | |
| 18:28:56 | sean-k-mooney | but perhaps there was some cow sematics at play | |
| 18:29:21 | dansmith | what I mean by cow is, you can expand your heap but until you touch a page, the kernel doesn't have to actually allocate you one | |
| 18:29:58 | sean-k-mooney | yes | |
| 18:30:02 | dansmith | I dunno what python's behavior is really, other than "use a crapload of memory all the time" | |
| 18:30:09 | sean-k-mooney | you can grow the adress space but not page it to phsyical ram | |
| 18:30:19 | sean-k-mooney | but it looked liek without swap python was always commiting | |
| 18:30:33 | dansmith | right, so without swap, you don't want to overcommit because the process thinks it already has that memory | |
| 18:31:05 | dansmith | but with swap, you can lie to it and offload something else during a fault | |
| 18:31:09 | sean-k-mooney | yep so just something to be aware of | |
| 18:31:24 | sean-k-mooney | if you see big difference between jobs | |
| 18:31:32 | dansmith | I think turning off swap on any of our workers would likely be a big fail | |
| 18:31:33 | sean-k-mooney | we shoudl just ensure the both have the same swap config | |
| 18:31:40 | dansmith | they do, afaik | |
| 18:31:48 | dansmith | but yeah, good to know | |
| 18:32:08 | sean-k-mooney | am i think they all have 2G by default unless the providers flavor add extra swap space | |
| 18:32:16 | sean-k-mooney | some jobs increase it to 8G | |
| 18:32:20 | clarkb | none of our providers give us swap | |
| 18:32:25 | sean-k-mooney | ack | |
| 18:32:26 | clarkb | we create it in the jobs | |
| 18:32:32 | sean-k-mooney | yep | |
| 18:32:36 | clarkb | there is a difference though. Some use swapfiles and some use swap partitions | |
| 18:32:47 | clarkb | we prefer to partition but can only do that when we have a second device | |
| 18:32:59 | sean-k-mooney | yep i am reusing the upstream swap role in my ansible stuff | |
| 18:33:10 | sean-k-mooney | so i read over it 2 weeks ago | |
| 18:33:58 | sean-k-mooney | clarkb: actully since your here is there a reason thats in openstakc-zuul-jobs i was considering porting it to zuul-jobs but just said i woudl ask | |
| 18:34:00 | sean-k-mooney | https://github.com/openstack/openstack-zuul-jobs/tree/master/roles/configure-swap | |
| 18:34:23 | clarkb | I think because its very specific to our cloud providers and the devices they expose? | |
| 18:34:32 | sean-k-mooney | https://github.com/openstack/openstack-zuul-jobs/blob/master/roles/configure-swap/tasks/main.yaml | |
| 18:34:38 | clarkb | you can probably make it more generic, possibly by dropping the partitioned swap option | |
| 18:34:42 | sean-k-mooney | it has some stuff for ephemeral0 | |
| 18:35:00 | sean-k-mooney | but it work to just use a swap file on root | |
| 18:35:11 | clarkb | also when it was first written we didn't use dd to preallicate which was fine until linux decided to break that | |
| 18:35:23 | clarkb | but now that we use dd it should work on all the filesystems that can host a swapfile | |
| 18:35:47 | sean-k-mooney | this https://github.com/openstack/openstack-zuul-jobs/blob/master/roles/configure-swap/tasks/main.yaml#L3-L29= i think is really the only non generic part | |
| 18:36:01 | sean-k-mooney | but ya no worries | |
| 18:36:29 | clarkb | specifically swapfile on ext4 created with fallocate worked until very recently on linux. But recent changes broke that and made ext4 much more like xfs and others | |
| 18:36:56 | sean-k-mooney | hum ok | |
| 18:37:08 | sean-k-mooney | we use fallocate in nova to preallocate | |
| 18:37:12 | sean-k-mooney | what exactly changed | |
| 18:37:28 | clarkb | I'd have to go find the commit that removed fallocate to remember | |
| 18:37:33 | sean-k-mooney | i think we check for odirect | |
| 18:37:37 | clarkb | note docuemntation specifically says fallocate is fine but it isn't anymore | |
| 18:37:50 | clarkb | but it is specifically for how swapfiles are managed by the kernel | |
| 18:38:50 | sean-k-mooney | ok | |
| 18:38:53 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/imagebackend.py#L283-L301= | |