| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-04-21 | |||
| 18:26:09 | dansmith | clarkb: for ironic | |
| 18:26:17 | sean-k-mooney | dansmith: it seamed like when swap was disabele the virtual memory and resident memory was the same | |
| 18:26:29 | sean-k-mooney | but when it was enabel there was a large deleta between the two | |
| 18:26:31 | dansmith | sean-k-mooney: well, makes sense right? | |
| 18:26:46 | dansmith | maybe that's just an artifact then.. | |
| 18:27:13 | dansmith | if you're only looking at resident, then you see the working set, but if you have no other option, then rss=total | |
| 18:27:13 | sean-k-mooney | with swap on the virt memroy was the same but the prviate resent memroy was less and no swap was being used | |
| 18:28:02 | dansmith | right, but it's probably because it has overcomitted to the process and until it's CoWd it can lie with swap but not if there's no swap? | |
| 18:28:33 | sean-k-mooney | i dont realy know but it did nto appre to be paging it out to swap space | |
| 18:28:50 | dansmith | right, but until it needs to it won't, | |
| 18:28:55 | dansmith | but it can lie in that case | |
| 18:28:56 | sean-k-mooney | but perhaps there was some cow sematics at play | |
| 18:29:21 | dansmith | what I mean by cow is, you can expand your heap but until you touch a page, the kernel doesn't have to actually allocate you one | |
| 18:29:58 | sean-k-mooney | yes | |
| 18:30:02 | dansmith | I dunno what python's behavior is really, other than "use a crapload of memory all the time" | |
| 18:30:09 | sean-k-mooney | you can grow the adress space but not page it to phsyical ram | |
| 18:30:19 | sean-k-mooney | but it looked liek without swap python was always commiting | |
| 18:30:33 | dansmith | right, so without swap, you don't want to overcommit because the process thinks it already has that memory | |
| 18:31:05 | dansmith | but with swap, you can lie to it and offload something else during a fault | |
| 18:31:09 | sean-k-mooney | yep so just something to be aware of | |
| 18:31:24 | sean-k-mooney | if you see big difference between jobs | |
| 18:31:32 | dansmith | I think turning off swap on any of our workers would likely be a big fail | |
| 18:31:33 | sean-k-mooney | we shoudl just ensure the both have the same swap config | |
| 18:31:40 | dansmith | they do, afaik | |
| 18:31:48 | dansmith | but yeah, good to know | |
| 18:32:08 | sean-k-mooney | am i think they all have 2G by default unless the providers flavor add extra swap space | |
| 18:32:16 | sean-k-mooney | some jobs increase it to 8G | |
| 18:32:20 | clarkb | none of our providers give us swap | |
| 18:32:25 | sean-k-mooney | ack | |
| 18:32:26 | clarkb | we create it in the jobs | |
| 18:32:32 | sean-k-mooney | yep | |
| 18:32:36 | clarkb | there is a difference though. Some use swapfiles and some use swap partitions | |
| 18:32:47 | clarkb | we prefer to partition but can only do that when we have a second device | |
| 18:32:59 | sean-k-mooney | yep i am reusing the upstream swap role in my ansible stuff | |
| 18:33:10 | sean-k-mooney | so i read over it 2 weeks ago | |
| 18:33:58 | sean-k-mooney | clarkb: actully since your here is there a reason thats in openstakc-zuul-jobs i was considering porting it to zuul-jobs but just said i woudl ask | |
| 18:34:00 | sean-k-mooney | https://github.com/openstack/openstack-zuul-jobs/tree/master/roles/configure-swap | |
| 18:34:23 | clarkb | I think because its very specific to our cloud providers and the devices they expose? | |
| 18:34:32 | sean-k-mooney | https://github.com/openstack/openstack-zuul-jobs/blob/master/roles/configure-swap/tasks/main.yaml | |
| 18:34:38 | clarkb | you can probably make it more generic, possibly by dropping the partitioned swap option | |
| 18:34:42 | sean-k-mooney | it has some stuff for ephemeral0 | |
| 18:35:00 | sean-k-mooney | but it work to just use a swap file on root | |
| 18:35:11 | clarkb | also when it was first written we didn't use dd to preallicate which was fine until linux decided to break that | |
| 18:35:23 | clarkb | but now that we use dd it should work on all the filesystems that can host a swapfile | |
| 18:35:47 | sean-k-mooney | this https://github.com/openstack/openstack-zuul-jobs/blob/master/roles/configure-swap/tasks/main.yaml#L3-L29= i think is really the only non generic part | |
| 18:36:01 | sean-k-mooney | but ya no worries | |
| 18:36:29 | clarkb | specifically swapfile on ext4 created with fallocate worked until very recently on linux. But recent changes broke that and made ext4 much more like xfs and others | |
| 18:36:56 | sean-k-mooney | hum ok | |
| 18:37:08 | sean-k-mooney | we use fallocate in nova to preallocate | |
| 18:37:12 | sean-k-mooney | what exactly changed | |
| 18:37:28 | clarkb | I'd have to go find the commit that removed fallocate to remember | |
| 18:37:33 | sean-k-mooney | i think we check for odirect | |
| 18:37:37 | clarkb | note docuemntation specifically says fallocate is fine but it isn't anymore | |
| 18:37:50 | clarkb | but it is specifically for how swapfiles are managed by the kernel | |
| 18:38:50 | sean-k-mooney | ok | |
| 18:38:53 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/imagebackend.py#L283-L301= | |
| 18:38:59 | sean-k-mooney | we do this but perhaps that is still ok | |
| 18:39:38 | clarkb | https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/750941 is the change there is a red hat bz link that I can't open | |
| 18:40:56 | sean-k-mooney | thats bad form we are ment to make them public if we are using them in commit or upstream bugs i wonder if there is a reason that lee did not in this case | |
| 18:41:15 | sean-k-mooney | oh because it was rhel9 proably in this cases | |
| 18:41:50 | sean-k-mooney | ya that was commit before cenots 9 or rhel 9 were a thing publicly | |
| 18:53:04 | sean-k-mooney | clarkb: not much in it but you shoudl be able to see it now https://bugzilla.redhat.com/show_bug.cgi?id=1827115 | |
| 18:53:40 | sean-k-mooney | tldr fallocate -z shoud be the same as dd and result in no holes | |
| 18:54:59 | clarkb | I wonder if it is any faster at that point | |
| 18:55:07 | clarkb | might be worth testing that as creating the file can be slow at time | |
| 18:55:52 | sean-k-mooney | ya not really sure to be honest | |
| 18:58:57 | clarkb | sean-k-mooney: https://lkml.org/lkml/2020/9/24/810 | |
| 18:59:21 | clarkb | maybe it is all happy now again | |
| 19:00:51 | sean-k-mooney | i cant view the last message but the previous ones in that thread look promising at least | |
| 19:07:41 | sean-k-mooney | clarkb: if there was a performace hit we could just revert lees change. shall i open a revert and see if it works? | |
| 19:08:32 | sean-k-mooney | i guess we would need a DNM test patch agaisnt devstack to test all the distros that depending on the revert | |
| 19:09:23 | sean-k-mooney | actully we proably should talk about this in #openstack-qa isntead | |
| 19:16:33 | opendevreview | sean mooney proposed openstack/nova master: enable locking test fixture https://review.opendev.org/c/openstack/nova/+/838942 | |
| 19:16:44 | sean-k-mooney | artom: gibi ^ that shoudl fix it | |
| 19:19:32 | lyarwood | sean-k-mooney: FWIW the bug was public when I linked it, it was made private by someone else a few months later. | |
| 19:19:54 | lyarwood | sean-k-mooney: also looks like that switch for fallocate landed after my original change | |
| 19:20:37 | opendevreview | Ghanshyam proposed openstack/nova master: Update python classifier as per testing runtime https://review.opendev.org/c/openstack/nova/+/838943 | |
| 19:20:47 | sean-k-mooney | lyarwood: ack not blaming you by the way i assumed you had a good reason to keep it private ot that it was public when you filed it but didnt check who made it private | |
| 19:21:00 | lyarwood | Yeah no issues | |
| 19:22:21 | sean-k-mooney | i was going to submit a revirt but i see were are not actully passing -z | |
| 19:22:36 | artom | sean-k-mooney, that make sense, but why did we not hit more often upstream> | |
| 19:22:37 | artom | ? | |
| 19:22:39 | sean-k-mooney | i might try updating it to use -z and regert the size ot 8192 | |
| 19:22:42 | artom | In downstream CI it's 100% | |
| 19:23:00 | sean-k-mooney | artom: the upper constrait bump only happend 2 months ago | |
| 19:23:16 | sean-k-mooney | so i guess we only started seeing it recently | |
| 19:23:24 | sean-k-mooney | it could be realted to mirrors having old versions | |
| 19:23:29 | sean-k-mooney | but not really sure | |
| 19:24:02 | opendevreview | Ghanshyam proposed openstack/python-novaclient master: Update python classifier as per testing runtime https://review.opendev.org/c/openstack/python-novaclient/+/838944 | |
| 19:27:08 | artom | sean-k-mooney, ah, so actually, lemme test that with my downstream DNM patch | |
| 19:31:32 | sean-k-mooney | that shoudl be a clean cherry pick to wallby since i actuly did it on wallayb first and then stashed it checkout masters and applied it | |
| 19:31:52 | sean-k-mooney | so i can chery pick it upstream and then do it downstream if you like | |
| 19:52:10 | opendevreview | sean mooney proposed openstack/nova stable/yoga: enable locking test fixture https://review.opendev.org/c/openstack/nova/+/838877 | |
| 19:52:36 | sean-k-mooney | ... damit | |
| 19:52:47 | sean-k-mooney | gerrit nolonger addes the cherry-picked form lines by default | |
| 19:53:07 | sean-k-mooney | oh i bet its because its not merged. | |
| 19:54:36 | opendevreview | sean mooney proposed openstack/nova stable/yoga: enable locking test fixture https://review.opendev.org/c/openstack/nova/+/838877 | |
| 19:55:11 | opendevreview | sean mooney proposed openstack/nova stable/xena: enable locking test fixture https://review.opendev.org/c/openstack/nova/+/838945 | |
| 19:55:49 | opendevreview | sean mooney proposed openstack/nova stable/wallaby: enable locking test fixture https://review.opendev.org/c/openstack/nova/+/838946 | |
| 20:02:37 | melwitt | sean-k-mooney: correct, it will add the lines only if merged | |