| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-16 | |||
| 17:04:10 | dansmith | frickler: tbf, I hadn't even looked at the patch, I was going based on sean-k-mooney, melwitt saying nova-next was the only one still doing image boot | |
| 17:05:13 | dansmith | the original bug we opened for this GPF was before we even had the resize-volume-backed test, FWIW :) | |
| 17:20:51 | tkajinam | o/ I'd appreciate it if https://review.opendev.org/c/openstack/nova/+/905314 can get some attention because it's now blocking requirement bump. | |
| 17:22:51 | dansmith | frickler: yeah, that instance in question is doing preloaded kernel boot: | |
| 17:22:52 | dansmith | Jan 16 11:45:59.955345 np0036421623 nova-compute[74855]: <kernel>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/kernel</kernel> | |
| 17:22:56 | dansmith | Jan 16 11:45:59.955345 np0036421623 nova-compute[74855]: <initrd>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/ramdisk</initrd> | |
| 17:23:20 | dansmith | so, shouldn't have loaded the ramdisk from disk | |
| 17:23:57 | sean-k-mooney | dansmith: sorry was drinking coffee reading back | |
| 17:24:34 | sean-k-mooney | frickler: dansmith https://review.opendev.org/c/openstack/nova/+/902809/1 | |
| 17:24:51 | sean-k-mooney | we had a folow up patch to overried the images for jobs defiend out of repo | |
| 17:25:20 | sean-k-mooney | so tempest-integrated-compute was updated here https://review.opendev.org/c/openstack/nova/+/902809/1/.zuul.yaml#944 | |
| 17:26:34 | sean-k-mooney | dansmith: i quickly tried enabling the larger vms here by the way | |
| 17:27:09 | sean-k-mooney | going to 32 gb feels excessinve but based on the low point (and concurnace 8) we were using about 10G on the contoler | |
| 17:27:53 | sean-k-mooney | so if we had a 16G flavor or even a 11G flavor we woudl entirly elimiate our usage of swap form the job | |
| 17:28:21 | clarkb | fwiw we do have larger flavors they are just limited in quantity as only a few (maybe one?) cloud currently provide them | |
| 17:28:36 | sean-k-mooney | ya i was looking at that | |
| 17:29:03 | sean-k-mooney | we have 32G instance form one vexhost cloud and 16G instnace form another | |
| 17:29:19 | sean-k-mooney | and then 8G instnace in general | |
| 17:29:27 | clarkb | but also I think any "make nodes bigger" effort should coincide with a "understand where the memory use is coming from and whether or not it represents bugs/leaks" effort | |
| 17:30:32 | sean-k-mooney | ya so it looks like if we were able to move to 16G as standard we would signifcatly increase the performance of the job (tempest full was about an hour) | |
| 17:30:46 | clarkb | for example privsep uses tremendous amounts of memory for what it is | |
| 17:30:47 | sean-k-mooney | but if we were to move to that we would likely expand to fill it | |
| 17:30:57 | clarkb | with the little info I have I would classify that as a bug | |
| 17:32:16 | sean-k-mooney | clarkb: frickler dansmith while ye are aroudn could ye look at my zram change when ye have time https://review.opendev.org/c/openstack/devstack/+/890693 | |
| 17:33:20 | sean-k-mooney | clarkb: on the privsep topic, i have not looked at it in detail before but do you knwo roughly how much ram its using per process | |
| 17:33:35 | sean-k-mooney | i assume it partly depend on the service? i.e. nova vs neutron | |
| 17:33:35 | dansmith | sean-k-mooney: yeah I'm for that for sure, but not sure the qa people will want it on by default | |
| 17:33:54 | frickler | sean-k-mooney: tbh I've stayed away from that patch for now since I'm wary of making the CI setup even more complex | |
| 17:33:57 | dansmith | gmann: kopecmartin ^ | |
| 17:34:08 | sean-k-mooney | ack i can change the defaults os if people prefer it to be off by default ill refactor for that | |
| 17:34:26 | JayF | sean-k-mooney: /me points an ironic change at 890693 for science | |
| 17:34:27 | sean-k-mooney | its off by default locally but on by default in ci currently | |
| 17:34:30 | dansmith | sean-k-mooney: if you disable it by default I'll +2 now, but if not, I want a read from gmann at least | |
| 17:35:09 | sean-k-mooney | ack ill do that i need to deploy a devstack vm for my healthcheck stuff anyway so ill upated it then | |
| 17:35:24 | clarkb | sean-k-mooney: the memory usage varies by the service talking to the privsep instance (which implies to me that workload probably impacts buffer sizes). In total I think it was about half a gig for privsep | |
| 17:35:58 | sean-k-mooney | clarkb: i think tis more related to the size of our oslo.config global + the code that is imported | |
| 17:36:54 | clarkb | sean-k-mooney: oh ya I guess everything is a regex? Those shouldn't be terribly large but if you have enough of them... might also be worth checking if libre2 is more efficient maybe | |
| 17:37:09 | sean-k-mooney | we have some chaced module level varables for example that might be related to this | |
| 17:37:42 | clarkb | sean-k-mooney: for zswap I notice you set swappiness to 100. I think devstack sets swappiness elsewhere you might need to reconcile the two to ensure they don't fight each other | |
| 17:38:25 | clarkb | sean-k-mooney: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap/tasks/main.yaml#L46-L61 this is the code that does it and I think devstack jobs call that first so you would override which I guess is fine | |
| 17:38:41 | sean-k-mooney | clarkb: i was thinking fo https://github.com/openstack/nova/blob/a72f7eaac78927892b937d451cbacb24a83c05ac/nova/api/validation/parameter_types.py#L106 | |
| 17:38:54 | sean-k-mooney | although privsep shoudl not be using the api stuff | |
| 17:39:00 | sean-k-mooney | so hopefully that wont matter | |
| 17:39:19 | clarkb | I don't have any other comments on the zswap change. Seems like it may be worth experimenting with | |
| 17:39:32 | dansmith | yep | |
| 17:40:15 | sean-k-mooney | i know people have seen a lot of success with desktop responsiveness on rassbery pi with it | |
| 17:40:23 | sean-k-mooney | so i was hoping it woudl help us in a simialr way | |
| 17:40:38 | sean-k-mooney | ok ill go remove the default change in teh patch | |
| 17:40:47 | sean-k-mooney | and we can then enabel it on a per job basises in nova | |
| 17:44:18 | sean-k-mooney | JayF: did you see https://review.opendev.org/c/openstack/nova/+/905406 by the way | |
| 17:44:41 | sean-k-mooney | JayF: that extract the common instnace metadata code up to the driver level and refactors libvirt to use it | |
| 17:44:56 | JayF | https://review.opendev.org/c/openstack/nova/+/900831 I really want this to land before I push up more patches | |
| 17:45:09 | JayF | I've been burned in the past by stacking a bunch of patches and being in rebase hell | |
| 17:45:21 | sean-k-mooney | ah yes well want that to land to make backport simpler | |
| 17:45:23 | JayF | So I have that in my back pocket as a "next thing" but waiting for 900831 to land first | |
| 17:48:43 | clarkb | JayF: this may or may not be useful but the way I try to deal with those is to "squash back" I make all of my edits on the tip of my dev branch and then commit will be something like "squash fix for commit foo" then I git rebase -i HEAD~X where X is the number of commits I need to go back to and squash the new commit back into the old one. Then git review the whole thing. I | |
| 17:48:44 | clarkb | find it simplifies things because I can focus on the necessary fixes and then focus on the accounting and usually I only need a single rebase to update a whole stack with any number of fixes | |
| 17:49:12 | JayF | clarkb: it's more that I have enough things on my list that I can action and will move more quickly so I just prioritize those over things which will languish for longer | |
| 17:49:34 | bauzas | git reflog FTW | |
| 17:49:40 | JayF | I can't remember the last time I was just like "there's nothing to do today" :D | |
| 17:55:45 | jrosser | clarkb: a colleague of mine demoed this recently https://github.com/tummychow/git-absorb | |
| 17:56:02 | jrosser | which is some kind of magic | |
| 17:56:44 | clarkb | that looks similar to what I do with the pairing of changes to squash done automatically based on file heuristics | |
| 17:57:13 | clarkb | There is also git restack which corvus wrote so that you don't have to manually determine the value of X when rebasing | |
| 19:05:23 | sean-k-mooney | johnthetubaguy: are you able to be the second core to review https://review.opendev.org/c/openstack/nova/+/900831 its JayF fix for https://launchpad.net/bugs/2043036 | |
| 19:05:39 | sean-k-mooney | otherwizse melwitt dansmith ^ perhasp ye can review | |
| 19:33:09 | opendevreview | Merged openstack/osc-placement master: Bump hacking https://review.opendev.org/c/openstack/osc-placement/+/905715 | |
| 20:19:36 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 20:19:36 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 20:19:37 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 20:38:01 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 20:44:41 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 20:51:58 | opendevreview | Merged openstack/python-novaclient master: Bump hacking https://review.opendev.org/c/openstack/python-novaclient/+/905707 | |
| 21:06:20 | opendevreview | Merged openstack/placement master: Bump hacking https://review.opendev.org/c/openstack/placement/+/905706 | |
| #openstack-nova - 2024-01-17 | |||
| 02:00:49 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 02:56:31 | opendevreview | melanie witt proposed openstack/nova master: WIP Suppport migration with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/905512 | |
| 02:56:31 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: make |
|
| 02:56:32 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 03:19:58 | songwenping | hi team, the latest nova code raise exception for 'nova list' with the error info TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig', what's wrong? | |
| 04:48:56 | opendevreview | Amit Uniyal proposed openstack/nova master: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/905684 | |
| 04:48:57 | opendevreview | Amit Uniyal proposed openstack/nova master: Fixes bug 2048184 https://review.opendev.org/c/openstack/nova/+/904817 | |
| 05:20:27 | opendevreview | Amit Uniyal proposed openstack/nova master: Fixes bug 2048184 https://review.opendev.org/c/openstack/nova/+/904817 | |
| 06:49:17 | opendevreview | Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824 | |
| 10:27:34 | bauzas | humpf, upgraded to F39 and now tox and python3.12 are creating me problems :( | |
| 10:27:59 | bauzas | tox==4.12.1 | |
| 10:30:11 | bauzas | good news is that I can still run functional tests if I target by the python version (like -e functional-py39) | |
| 10:42:04 | bauzas | ok, behavioral change with tox, no worriezsq | |
| 11:15:06 | kashyap | Good to know; /me is still to ugprade to F39. Was planning to do this week | |
| 12:53:33 | opendevreview | Fabian Wiesel proposed openstack/nova master: Add more password generation options https://review.opendev.org/c/openstack/nova/+/865669 | |
| 14:47:41 | opendevreview | Sylvain Bauza proposed openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 14:47:41 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 14:47:42 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 14:47:42 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 14:55:18 | opendevreview | Sylvain Bauza proposed openstack/nova master: Augment the LibvirtLiveMigrateData object https://review.opendev.org/c/openstack/nova/+/904175 | |
| 14:55:18 | opendevreview | Sylvain Bauza proposed openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 14:55:19 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 14:55:19 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 14:55:20 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 15:17:15 | opendevreview | Hervé Beraud proposed openstack/nova master: bump eventlet to latest version that support python 3.12 https://review.opendev.org/c/openstack/nova/+/905928 | |
| 15:17:15 | opendevreview | Hervé Beraud proposed openstack/nova master: bump eventlet to latest version that support python 3.12 https://review.opendev.org/c/openstack/nova/+/905928 | |