| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-16 | |||
| 17:27:09 | sean-k-mooney | going to 32 gb feels excessinve but based on the low point (and concurnace 8) we were using about 10G on the contoler | |
| 17:27:53 | sean-k-mooney | so if we had a 16G flavor or even a 11G flavor we woudl entirly elimiate our usage of swap form the job | |
| 17:28:21 | clarkb | fwiw we do have larger flavors they are just limited in quantity as only a few (maybe one?) cloud currently provide them | |
| 17:28:36 | sean-k-mooney | ya i was looking at that | |
| 17:29:03 | sean-k-mooney | we have 32G instance form one vexhost cloud and 16G instnace form another | |
| 17:29:19 | sean-k-mooney | and then 8G instnace in general | |
| 17:29:27 | clarkb | but also I think any "make nodes bigger" effort should coincide with a "understand where the memory use is coming from and whether or not it represents bugs/leaks" effort | |
| 17:30:32 | sean-k-mooney | ya so it looks like if we were able to move to 16G as standard we would signifcatly increase the performance of the job (tempest full was about an hour) | |
| 17:30:46 | clarkb | for example privsep uses tremendous amounts of memory for what it is | |
| 17:30:47 | sean-k-mooney | but if we were to move to that we would likely expand to fill it | |
| 17:30:57 | clarkb | with the little info I have I would classify that as a bug | |
| 17:32:16 | sean-k-mooney | clarkb: frickler dansmith while ye are aroudn could ye look at my zram change when ye have time https://review.opendev.org/c/openstack/devstack/+/890693 | |
| 17:33:20 | sean-k-mooney | clarkb: on the privsep topic, i have not looked at it in detail before but do you knwo roughly how much ram its using per process | |
| 17:33:35 | dansmith | sean-k-mooney: yeah I'm for that for sure, but not sure the qa people will want it on by default | |
| 17:33:35 | sean-k-mooney | i assume it partly depend on the service? i.e. nova vs neutron | |
| 17:33:54 | frickler | sean-k-mooney: tbh I've stayed away from that patch for now since I'm wary of making the CI setup even more complex | |
| 17:33:57 | dansmith | gmann: kopecmartin ^ | |
| 17:34:08 | sean-k-mooney | ack i can change the defaults os if people prefer it to be off by default ill refactor for that | |
| 17:34:26 | JayF | sean-k-mooney: /me points an ironic change at 890693 for science | |
| 17:34:27 | sean-k-mooney | its off by default locally but on by default in ci currently | |
| 17:34:30 | dansmith | sean-k-mooney: if you disable it by default I'll +2 now, but if not, I want a read from gmann at least | |
| 17:35:09 | sean-k-mooney | ack ill do that i need to deploy a devstack vm for my healthcheck stuff anyway so ill upated it then | |
| 17:35:24 | clarkb | sean-k-mooney: the memory usage varies by the service talking to the privsep instance (which implies to me that workload probably impacts buffer sizes). In total I think it was about half a gig for privsep | |
| 17:35:58 | sean-k-mooney | clarkb: i think tis more related to the size of our oslo.config global + the code that is imported | |
| 17:36:54 | clarkb | sean-k-mooney: oh ya I guess everything is a regex? Those shouldn't be terribly large but if you have enough of them... might also be worth checking if libre2 is more efficient maybe | |
| 17:37:09 | sean-k-mooney | we have some chaced module level varables for example that might be related to this | |
| 17:37:42 | clarkb | sean-k-mooney: for zswap I notice you set swappiness to 100. I think devstack sets swappiness elsewhere you might need to reconcile the two to ensure they don't fight each other | |
| 17:38:25 | clarkb | sean-k-mooney: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap/tasks/main.yaml#L46-L61 this is the code that does it and I think devstack jobs call that first so you would override which I guess is fine | |
| 17:38:41 | sean-k-mooney | clarkb: i was thinking fo https://github.com/openstack/nova/blob/a72f7eaac78927892b937d451cbacb24a83c05ac/nova/api/validation/parameter_types.py#L106 | |
| 17:38:54 | sean-k-mooney | although privsep shoudl not be using the api stuff | |
| 17:39:00 | sean-k-mooney | so hopefully that wont matter | |
| 17:39:19 | clarkb | I don't have any other comments on the zswap change. Seems like it may be worth experimenting with | |
| 17:39:32 | dansmith | yep | |
| 17:40:15 | sean-k-mooney | i know people have seen a lot of success with desktop responsiveness on rassbery pi with it | |
| 17:40:23 | sean-k-mooney | so i was hoping it woudl help us in a simialr way | |
| 17:40:38 | sean-k-mooney | ok ill go remove the default change in teh patch | |
| 17:40:47 | sean-k-mooney | and we can then enabel it on a per job basises in nova | |
| 17:44:18 | sean-k-mooney | JayF: did you see https://review.opendev.org/c/openstack/nova/+/905406 by the way | |
| 17:44:41 | sean-k-mooney | JayF: that extract the common instnace metadata code up to the driver level and refactors libvirt to use it | |
| 17:44:56 | JayF | https://review.opendev.org/c/openstack/nova/+/900831 I really want this to land before I push up more patches | |
| 17:45:09 | JayF | I've been burned in the past by stacking a bunch of patches and being in rebase hell | |
| 17:45:21 | sean-k-mooney | ah yes well want that to land to make backport simpler | |
| 17:45:23 | JayF | So I have that in my back pocket as a "next thing" but waiting for 900831 to land first | |
| 17:48:43 | clarkb | JayF: this may or may not be useful but the way I try to deal with those is to "squash back" I make all of my edits on the tip of my dev branch and then commit will be something like "squash fix for commit foo" then I git rebase -i HEAD~X where X is the number of commits I need to go back to and squash the new commit back into the old one. Then git review the whole thing. I | |
| 17:48:44 | clarkb | find it simplifies things because I can focus on the necessary fixes and then focus on the accounting and usually I only need a single rebase to update a whole stack with any number of fixes | |
| 17:49:12 | JayF | clarkb: it's more that I have enough things on my list that I can action and will move more quickly so I just prioritize those over things which will languish for longer | |
| 17:49:34 | bauzas | git reflog FTW | |
| 17:49:40 | JayF | I can't remember the last time I was just like "there's nothing to do today" :D | |
| 17:55:45 | jrosser | clarkb: a colleague of mine demoed this recently https://github.com/tummychow/git-absorb | |
| 17:56:02 | jrosser | which is some kind of magic | |
| 17:56:44 | clarkb | that looks similar to what I do with the pairing of changes to squash done automatically based on file heuristics | |
| 17:57:13 | clarkb | There is also git restack which corvus wrote so that you don't have to manually determine the value of X when rebasing | |
| 19:05:23 | sean-k-mooney | johnthetubaguy: are you able to be the second core to review https://review.opendev.org/c/openstack/nova/+/900831 its JayF fix for https://launchpad.net/bugs/2043036 | |
| 19:05:39 | sean-k-mooney | otherwizse melwitt dansmith ^ perhasp ye can review | |
| 19:33:09 | opendevreview | Merged openstack/osc-placement master: Bump hacking https://review.opendev.org/c/openstack/osc-placement/+/905715 | |
| 20:19:36 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 20:19:36 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 20:19:37 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 20:38:01 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 20:44:41 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 20:51:58 | opendevreview | Merged openstack/python-novaclient master: Bump hacking https://review.opendev.org/c/openstack/python-novaclient/+/905707 | |
| 21:06:20 | opendevreview | Merged openstack/placement master: Bump hacking https://review.opendev.org/c/openstack/placement/+/905706 | |
| #openstack-nova - 2024-01-17 | |||
| 02:00:49 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 02:56:31 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: make |
|
| 02:56:31 | opendevreview | melanie witt proposed openstack/nova master: WIP Suppport migration with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/905512 | |
| 02:56:32 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 03:19:58 | songwenping | hi team, the latest nova code raise exception for 'nova list' with the error info TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig', what's wrong? | |
| 04:48:56 | opendevreview | Amit Uniyal proposed openstack/nova master: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/905684 | |
| 04:48:57 | opendevreview | Amit Uniyal proposed openstack/nova master: Fixes bug 2048184 https://review.opendev.org/c/openstack/nova/+/904817 | |
| 05:20:27 | opendevreview | Amit Uniyal proposed openstack/nova master: Fixes bug 2048184 https://review.opendev.org/c/openstack/nova/+/904817 | |
| 06:49:17 | opendevreview | Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824 | |
| 10:27:34 | bauzas | humpf, upgraded to F39 and now tox and python3.12 are creating me problems :( | |
| 10:27:59 | bauzas | tox==4.12.1 | |
| 10:30:11 | bauzas | good news is that I can still run functional tests if I target by the python version (like -e functional-py39) | |
| 10:42:04 | bauzas | ok, behavioral change with tox, no worriezsq | |
| 11:15:06 | kashyap | Good to know; /me is still to ugprade to F39. Was planning to do this week | |
| 12:53:33 | opendevreview | Fabian Wiesel proposed openstack/nova master: Add more password generation options https://review.opendev.org/c/openstack/nova/+/865669 | |
| 14:47:41 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 14:47:41 | opendevreview | Sylvain Bauza proposed openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 14:47:42 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 14:47:42 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 14:55:18 | opendevreview | Sylvain Bauza proposed openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 14:55:18 | opendevreview | Sylvain Bauza proposed openstack/nova master: Augment the LibvirtLiveMigrateData object https://review.opendev.org/c/openstack/nova/+/904175 | |
| 14:55:19 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 14:55:19 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 14:55:20 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 15:17:15 | opendevreview | Hervé Beraud proposed openstack/nova master: bump eventlet to latest version that support python 3.12 https://review.opendev.org/c/openstack/nova/+/905928 | |
| 15:17:15 | opendevreview | Hervé Beraud proposed openstack/nova master: bump eventlet to latest version that support python 3.12 https://review.opendev.org/c/openstack/nova/+/905928 | |
| 17:35:15 | bauzas | argh, I'm hitting py312 issues on my laptop and I can't generate a relnote as we don't support venv target with py versions | |
| 18:09:29 | opendevreview | Sylvain Bauza proposed openstack/nova master: Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 18:09:29 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 18:13:12 | _colby | Hey Guys. We upgraded to oenstack yoga recently and now all our vGPU stuff does not work any more. We had to patch Wallaby, but Xena worked without issues. Now on Yoga we can spin up a single vGPU on a host (has 2 x A40 gpu (SR-IOV with Virtual Functions). But no more than 1 vgpu instance. We are specifying enabled_vgpu_types and device_Addresses for each type. Are we doing this incorrectly now with yoga? | |
| 18:14:45 | bauzas | _colby: I need to leave litterally now, but what's the stack when you ask for more | |
| 18:14:47 | bauzas | ? | |
| 18:15:24 | bauzas | I guess you know all the quirks with SRIOV GPUs | |
| 18:15:24 | _colby | bauzas: it says not valid host | |
| 18:15:47 | bauzas | you may have residues that lead to wrong placement decisions | |
| 18:15:48 | _colby | yes just wondering what changed between xena and yoga. Things worked fine before. | |
| 18:16:05 | bauzas | first thing is to doublecheck your inventories and allocations | |
| 18:16:21 | bauzas | and compare them with the existing mdevs and instances | |