| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-04-21 | |||
| 17:36:33 | sean-k-mooney | intel only works on x86 and smmuv3 only works on arm | |
| 17:36:36 | sean-k-mooney | virtio works on both | |
| 18:04:00 | sean-k-mooney | mnaser: stephenfin comment in line | |
| 18:04:02 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/830646 | |
| 18:20:11 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Run against performance.json patch https://review.opendev.org/c/openstack/nova/+/838934 | |
| 18:20:15 | dansmith | gmann: ^ | |
| 18:20:53 | sean-k-mooney | dansmith: what is that mesuring by the way | |
| 18:21:05 | gmann | dansmith: ack | |
| 18:21:08 | dansmith | sean-k-mooney: https://85682c22f75746955b38-998a6cfec97762292b03290ad6103366.ssl.cf5.rackcdn.com/837139/20/check/nova-ceph-multistore/d3c4ea2/controller/logs/performance.json | |
| 18:21:22 | dansmith | sean-k-mooney: a random comparison: https://termbin.com/rls0 | |
| 18:22:31 | sean-k-mooney | so ram usage, db queiss and api request for differnt services? | |
| 18:22:41 | dansmith | for the moment | |
| 18:23:07 | sean-k-mooney | ok what is the overall goal? | |
| 18:23:35 | dansmith | ideally to get some flag when a patch increases one of these values substantially | |
| 18:23:40 | sean-k-mooney | by the way we had weird behavior in a downstream case where there was very high memory usage if you disabeld swap entirely | |
| 18:24:10 | dansmith | like, c-bak uses 1GiB of ram for one tempest run, which is pretty dang high.. not sure when that happened (maybe always) but.. | |
| 18:24:36 | sean-k-mooney | does it have swap enabeld on the vm? | |
| 18:24:47 | clarkb | yes we enable swap on all of our test nodes | |
| 18:24:58 | sean-k-mooney | clarkb: i think some jobs turn it back off | |
| 18:25:04 | clarkb | thoug hamybe that happens at a job level but it is definitely there for devstack + tempest beacuse you OOM otherwise | |
| 18:25:13 | clarkb | and ya swapoff is always an option | |
| 18:25:17 | dansmith | yeah we're flying really close to the sun right now | |
| 18:25:26 | sean-k-mooney | dansmith: we saw nova-compute and neutron l2 agents both taking over 2G each downstream when swap was off | |
| 18:25:40 | sean-k-mooney | and it went back to normal when it was turn on | |
| 18:25:45 | dansmith | not sure why that would be, but interesting | |
| 18:25:48 | sean-k-mooney | ... with no swap acutlly being used | |
| 18:25:51 | clarkb | one of the things that happened that led to this was the rbac work drastically increased the db queries/cost and this sort of thing could catch that? | |
| 18:26:09 | dansmith | clarkb: for ironic | |
| 18:26:17 | sean-k-mooney | dansmith: it seamed like when swap was disabele the virtual memory and resident memory was the same | |
| 18:26:29 | sean-k-mooney | but when it was enabel there was a large deleta between the two | |
| 18:26:31 | dansmith | sean-k-mooney: well, makes sense right? | |
| 18:26:46 | dansmith | maybe that's just an artifact then.. | |
| 18:27:13 | sean-k-mooney | with swap on the virt memroy was the same but the prviate resent memroy was less and no swap was being used | |
| 18:27:13 | dansmith | if you're only looking at resident, then you see the working set, but if you have no other option, then rss=total | |
| 18:28:02 | dansmith | right, but it's probably because it has overcomitted to the process and until it's CoWd it can lie with swap but not if there's no swap? | |
| 18:28:33 | sean-k-mooney | i dont realy know but it did nto appre to be paging it out to swap space | |
| 18:28:50 | dansmith | right, but until it needs to it won't, | |
| 18:28:55 | dansmith | but it can lie in that case | |
| 18:28:56 | sean-k-mooney | but perhaps there was some cow sematics at play | |
| 18:29:21 | dansmith | what I mean by cow is, you can expand your heap but until you touch a page, the kernel doesn't have to actually allocate you one | |
| 18:29:58 | sean-k-mooney | yes | |
| 18:30:02 | dansmith | I dunno what python's behavior is really, other than "use a crapload of memory all the time" | |
| 18:30:09 | sean-k-mooney | you can grow the adress space but not page it to phsyical ram | |
| 18:30:19 | sean-k-mooney | but it looked liek without swap python was always commiting | |
| 18:30:33 | dansmith | right, so without swap, you don't want to overcommit because the process thinks it already has that memory | |
| 18:31:05 | dansmith | but with swap, you can lie to it and offload something else during a fault | |
| 18:31:09 | sean-k-mooney | yep so just something to be aware of | |
| 18:31:24 | sean-k-mooney | if you see big difference between jobs | |
| 18:31:32 | dansmith | I think turning off swap on any of our workers would likely be a big fail | |
| 18:31:33 | sean-k-mooney | we shoudl just ensure the both have the same swap config | |
| 18:31:40 | dansmith | they do, afaik | |
| 18:31:48 | dansmith | but yeah, good to know | |
| 18:32:08 | sean-k-mooney | am i think they all have 2G by default unless the providers flavor add extra swap space | |
| 18:32:16 | sean-k-mooney | some jobs increase it to 8G | |
| 18:32:20 | clarkb | none of our providers give us swap | |
| 18:32:25 | sean-k-mooney | ack | |
| 18:32:26 | clarkb | we create it in the jobs | |
| 18:32:32 | sean-k-mooney | yep | |
| 18:32:36 | clarkb | there is a difference though. Some use swapfiles and some use swap partitions | |
| 18:32:47 | clarkb | we prefer to partition but can only do that when we have a second device | |
| 18:32:59 | sean-k-mooney | yep i am reusing the upstream swap role in my ansible stuff | |
| 18:33:10 | sean-k-mooney | so i read over it 2 weeks ago | |
| 18:33:58 | sean-k-mooney | clarkb: actully since your here is there a reason thats in openstakc-zuul-jobs i was considering porting it to zuul-jobs but just said i woudl ask | |
| 18:34:00 | sean-k-mooney | https://github.com/openstack/openstack-zuul-jobs/tree/master/roles/configure-swap | |
| 18:34:23 | clarkb | I think because its very specific to our cloud providers and the devices they expose? | |
| 18:34:32 | sean-k-mooney | https://github.com/openstack/openstack-zuul-jobs/blob/master/roles/configure-swap/tasks/main.yaml | |
| 18:34:38 | clarkb | you can probably make it more generic, possibly by dropping the partitioned swap option | |
| 18:34:42 | sean-k-mooney | it has some stuff for ephemeral0 | |
| 18:35:00 | sean-k-mooney | but it work to just use a swap file on root | |
| 18:35:11 | clarkb | also when it was first written we didn't use dd to preallicate which was fine until linux decided to break that | |
| 18:35:23 | clarkb | but now that we use dd it should work on all the filesystems that can host a swapfile | |
| 18:35:47 | sean-k-mooney | this https://github.com/openstack/openstack-zuul-jobs/blob/master/roles/configure-swap/tasks/main.yaml#L3-L29= i think is really the only non generic part | |
| 18:36:01 | sean-k-mooney | but ya no worries | |
| 18:36:29 | clarkb | specifically swapfile on ext4 created with fallocate worked until very recently on linux. But recent changes broke that and made ext4 much more like xfs and others | |
| 18:36:56 | sean-k-mooney | hum ok | |
| 18:37:08 | sean-k-mooney | we use fallocate in nova to preallocate | |
| 18:37:12 | sean-k-mooney | what exactly changed | |
| 18:37:28 | clarkb | I'd have to go find the commit that removed fallocate to remember | |
| 18:37:33 | sean-k-mooney | i think we check for odirect | |
| 18:37:37 | clarkb | note docuemntation specifically says fallocate is fine but it isn't anymore | |
| 18:37:50 | clarkb | but it is specifically for how swapfiles are managed by the kernel | |
| 18:38:50 | sean-k-mooney | ok | |
| 18:38:53 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/imagebackend.py#L283-L301= | |
| 18:38:59 | sean-k-mooney | we do this but perhaps that is still ok | |
| 18:39:38 | clarkb | https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/750941 is the change there is a red hat bz link that I can't open | |
| 18:40:56 | sean-k-mooney | thats bad form we are ment to make them public if we are using them in commit or upstream bugs i wonder if there is a reason that lee did not in this case | |
| 18:41:15 | sean-k-mooney | oh because it was rhel9 proably in this cases | |
| 18:41:50 | sean-k-mooney | ya that was commit before cenots 9 or rhel 9 were a thing publicly | |
| 18:53:04 | sean-k-mooney | clarkb: not much in it but you shoudl be able to see it now https://bugzilla.redhat.com/show_bug.cgi?id=1827115 | |
| 18:53:40 | sean-k-mooney | tldr fallocate -z shoud be the same as dd and result in no holes | |
| 18:54:59 | clarkb | I wonder if it is any faster at that point | |
| 18:55:07 | clarkb | might be worth testing that as creating the file can be slow at time | |
| 18:55:52 | sean-k-mooney | ya not really sure to be honest | |
| 18:58:57 | clarkb | sean-k-mooney: https://lkml.org/lkml/2020/9/24/810 | |
| 18:59:21 | clarkb | maybe it is all happy now again | |
| 19:00:51 | sean-k-mooney | i cant view the last message but the previous ones in that thread look promising at least | |
| 19:07:41 | sean-k-mooney | clarkb: if there was a performace hit we could just revert lees change. shall i open a revert and see if it works? | |
| 19:08:32 | sean-k-mooney | i guess we would need a DNM test patch agaisnt devstack to test all the distros that depending on the revert | |
| 19:09:23 | sean-k-mooney | actully we proably should talk about this in #openstack-qa isntead | |
| 19:16:33 | opendevreview | sean mooney proposed openstack/nova master: enable locking test fixture https://review.opendev.org/c/openstack/nova/+/838942 | |