| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-16 | |||
| 16:29:06 | opendevreview | Takashi Kajinami proposed openstack/placement master: Bump hacking https://review.opendev.org/c/openstack/placement/+/905706 | |
| 16:29:08 | Uggla | I think https://bugs.launchpad.net/nova/+bug/2048154 is important to be fixed. I hope @Amit could have a look. | |
| 16:30:26 | opendevreview | Takashi Kajinami proposed openstack/python-novaclient master: Bump hacking https://review.opendev.org/c/openstack/python-novaclient/+/905707 | |
| 16:41:58 | opendevreview | Takashi Kajinami proposed openstack/osc-placement master: Bump hacking https://review.opendev.org/c/openstack/osc-placement/+/905715 | |
| 16:44:24 | frickler | kashyap: dansmith: added some comments on the etherpad. given that the issue (afaict) only happens for volume resize tests, I'm wondering how we could verify that no actual data corruption is happening on the volume in that scenario | |
| 16:44:42 | frickler | because if it does happen, chasing possible kernel bugs is kind of moot | |
| 16:45:14 | kashyap | frickler: Hi, yeah, I've just seen your comment on the Etherpad. Good question. (I'm in a meeting, and will be slow here) | |
| 16:45:31 | dansmith | frickler: these test instances haven't even made it out of initial kernel boot, so no real opportunity to corrupt anything yet | |
| 16:46:05 | frickler | is this happening on the initial boot, not after resize? | |
| 16:46:06 | dansmith | frickler: and we're not booting from the volumes either, since we're using the preloaded kernel boot method, which means corruption of the on-disk data causing the crash wouldn't be a thing | |
| 16:47:34 | dansmith | before the recent switch to kernel preload that could have been a possible option, but with kernel preload, disk corruption due to the resize should be ruled out AFAIK | |
| 16:49:09 | dansmith | I'm also not sure these are only happening on resize tests | |
| 16:57:37 | frickler | well the errors seem to be happening around the time when the switch to the actual root-fs would be happening. and in the traceback I was looking at, it was happening after the resize. do you happen to have a pointer to the preloaded kernel change? | |
| 16:59:00 | dansmith | frickler: I think it's still loading modules from the initramfs when it crashes | |
| 17:00:03 | dansmith | frickler: https://review.opendev.org/c/openstack/nova/+/902217 | |
| 17:00:07 | dansmith | (happened while I was out) | |
| 17:01:41 | dansmith | frickler: I think what you're seeing there is that it tried to load all the modules from the initramfs that it needed to have access to storage, which all crashed, and then it kept going trying to do the switch_root and failed to do so | |
| 17:02:26 | dansmith | it did get vda from the virtio driver so I would think it would have worked even, but with that level of broken having happened who knows | |
| 17:03:23 | frickler | from a quick scan, that patch only changes some specific nova jobs, not tempest-integrated-compute? | |
| 17:04:10 | dansmith | frickler: tbf, I hadn't even looked at the patch, I was going based on sean-k-mooney, melwitt saying nova-next was the only one still doing image boot | |
| 17:05:13 | dansmith | the original bug we opened for this GPF was before we even had the resize-volume-backed test, FWIW :) | |
| 17:20:51 | tkajinam | o/ I'd appreciate it if https://review.opendev.org/c/openstack/nova/+/905314 can get some attention because it's now blocking requirement bump. | |
| 17:22:51 | dansmith | frickler: yeah, that instance in question is doing preloaded kernel boot: | |
| 17:22:52 | dansmith | Jan 16 11:45:59.955345 np0036421623 nova-compute[74855]: <kernel>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/kernel</kernel> | |
| 17:22:56 | dansmith | Jan 16 11:45:59.955345 np0036421623 nova-compute[74855]: <initrd>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/ramdisk</initrd> | |
| 17:23:20 | dansmith | so, shouldn't have loaded the ramdisk from disk | |
| 17:23:57 | sean-k-mooney | dansmith: sorry was drinking coffee reading back | |
| 17:24:34 | sean-k-mooney | frickler: dansmith https://review.opendev.org/c/openstack/nova/+/902809/1 | |
| 17:24:51 | sean-k-mooney | we had a folow up patch to overried the images for jobs defiend out of repo | |
| 17:25:20 | sean-k-mooney | so tempest-integrated-compute was updated here https://review.opendev.org/c/openstack/nova/+/902809/1/.zuul.yaml#944 | |
| 17:26:34 | sean-k-mooney | dansmith: i quickly tried enabling the larger vms here by the way | |
| 17:27:09 | sean-k-mooney | going to 32 gb feels excessinve but based on the low point (and concurnace 8) we were using about 10G on the contoler | |
| 17:27:53 | sean-k-mooney | so if we had a 16G flavor or even a 11G flavor we woudl entirly elimiate our usage of swap form the job | |
| 17:28:21 | clarkb | fwiw we do have larger flavors they are just limited in quantity as only a few (maybe one?) cloud currently provide them | |
| 17:28:36 | sean-k-mooney | ya i was looking at that | |
| 17:29:03 | sean-k-mooney | we have 32G instance form one vexhost cloud and 16G instnace form another | |
| 17:29:19 | sean-k-mooney | and then 8G instnace in general | |
| 17:29:27 | clarkb | but also I think any "make nodes bigger" effort should coincide with a "understand where the memory use is coming from and whether or not it represents bugs/leaks" effort | |
| 17:30:32 | sean-k-mooney | ya so it looks like if we were able to move to 16G as standard we would signifcatly increase the performance of the job (tempest full was about an hour) | |
| 17:30:46 | clarkb | for example privsep uses tremendous amounts of memory for what it is | |
| 17:30:47 | sean-k-mooney | but if we were to move to that we would likely expand to fill it | |
| 17:30:57 | clarkb | with the little info I have I would classify that as a bug | |
| 17:32:16 | sean-k-mooney | clarkb: frickler dansmith while ye are aroudn could ye look at my zram change when ye have time https://review.opendev.org/c/openstack/devstack/+/890693 | |
| 17:33:20 | sean-k-mooney | clarkb: on the privsep topic, i have not looked at it in detail before but do you knwo roughly how much ram its using per process | |
| 17:33:35 | sean-k-mooney | i assume it partly depend on the service? i.e. nova vs neutron | |
| 17:33:35 | dansmith | sean-k-mooney: yeah I'm for that for sure, but not sure the qa people will want it on by default | |
| 17:33:54 | frickler | sean-k-mooney: tbh I've stayed away from that patch for now since I'm wary of making the CI setup even more complex | |
| 17:33:57 | dansmith | gmann: kopecmartin ^ | |
| 17:34:08 | sean-k-mooney | ack i can change the defaults os if people prefer it to be off by default ill refactor for that | |
| 17:34:26 | JayF | sean-k-mooney: /me points an ironic change at 890693 for science | |
| 17:34:27 | sean-k-mooney | its off by default locally but on by default in ci currently | |
| 17:34:30 | dansmith | sean-k-mooney: if you disable it by default I'll +2 now, but if not, I want a read from gmann at least | |
| 17:35:09 | sean-k-mooney | ack ill do that i need to deploy a devstack vm for my healthcheck stuff anyway so ill upated it then | |
| 17:35:24 | clarkb | sean-k-mooney: the memory usage varies by the service talking to the privsep instance (which implies to me that workload probably impacts buffer sizes). In total I think it was about half a gig for privsep | |
| 17:35:58 | sean-k-mooney | clarkb: i think tis more related to the size of our oslo.config global + the code that is imported | |
| 17:36:54 | clarkb | sean-k-mooney: oh ya I guess everything is a regex? Those shouldn't be terribly large but if you have enough of them... might also be worth checking if libre2 is more efficient maybe | |
| 17:37:09 | sean-k-mooney | we have some chaced module level varables for example that might be related to this | |
| 17:37:42 | clarkb | sean-k-mooney: for zswap I notice you set swappiness to 100. I think devstack sets swappiness elsewhere you might need to reconcile the two to ensure they don't fight each other | |
| 17:38:25 | clarkb | sean-k-mooney: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap/tasks/main.yaml#L46-L61 this is the code that does it and I think devstack jobs call that first so you would override which I guess is fine | |
| 17:38:41 | sean-k-mooney | clarkb: i was thinking fo https://github.com/openstack/nova/blob/a72f7eaac78927892b937d451cbacb24a83c05ac/nova/api/validation/parameter_types.py#L106 | |
| 17:38:54 | sean-k-mooney | although privsep shoudl not be using the api stuff | |
| 17:39:00 | sean-k-mooney | so hopefully that wont matter | |
| 17:39:19 | clarkb | I don't have any other comments on the zswap change. Seems like it may be worth experimenting with | |
| 17:39:32 | dansmith | yep | |
| 17:40:15 | sean-k-mooney | i know people have seen a lot of success with desktop responsiveness on rassbery pi with it | |
| 17:40:23 | sean-k-mooney | so i was hoping it woudl help us in a simialr way | |
| 17:40:38 | sean-k-mooney | ok ill go remove the default change in teh patch | |
| 17:40:47 | sean-k-mooney | and we can then enabel it on a per job basises in nova | |
| 17:44:18 | sean-k-mooney | JayF: did you see https://review.opendev.org/c/openstack/nova/+/905406 by the way | |
| 17:44:41 | sean-k-mooney | JayF: that extract the common instnace metadata code up to the driver level and refactors libvirt to use it | |
| 17:44:56 | JayF | https://review.opendev.org/c/openstack/nova/+/900831 I really want this to land before I push up more patches | |
| 17:45:09 | JayF | I've been burned in the past by stacking a bunch of patches and being in rebase hell | |
| 17:45:21 | sean-k-mooney | ah yes well want that to land to make backport simpler | |
| 17:45:23 | JayF | So I have that in my back pocket as a "next thing" but waiting for 900831 to land first | |
| 17:48:43 | clarkb | JayF: this may or may not be useful but the way I try to deal with those is to "squash back" I make all of my edits on the tip of my dev branch and then commit will be something like "squash fix for commit foo" then I git rebase -i HEAD~X where X is the number of commits I need to go back to and squash the new commit back into the old one. Then git review the whole thing. I | |
| 17:48:44 | clarkb | find it simplifies things because I can focus on the necessary fixes and then focus on the accounting and usually I only need a single rebase to update a whole stack with any number of fixes | |
| 17:49:12 | JayF | clarkb: it's more that I have enough things on my list that I can action and will move more quickly so I just prioritize those over things which will languish for longer | |
| 17:49:34 | bauzas | git reflog FTW | |
| 17:49:40 | JayF | I can't remember the last time I was just like "there's nothing to do today" :D | |
| 17:55:45 | jrosser | clarkb: a colleague of mine demoed this recently https://github.com/tummychow/git-absorb | |
| 17:56:02 | jrosser | which is some kind of magic | |
| 17:56:44 | clarkb | that looks similar to what I do with the pairing of changes to squash done automatically based on file heuristics | |
| 17:57:13 | clarkb | There is also git restack which corvus wrote so that you don't have to manually determine the value of X when rebasing | |
| 19:05:23 | sean-k-mooney | johnthetubaguy: are you able to be the second core to review https://review.opendev.org/c/openstack/nova/+/900831 its JayF fix for https://launchpad.net/bugs/2043036 | |
| 19:05:39 | sean-k-mooney | otherwizse melwitt dansmith ^ perhasp ye can review | |
| 19:33:09 | opendevreview | Merged openstack/osc-placement master: Bump hacking https://review.opendev.org/c/openstack/osc-placement/+/905715 | |
| 20:19:36 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 20:19:36 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 20:19:37 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 20:38:01 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 20:44:41 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 20:51:58 | opendevreview | Merged openstack/python-novaclient master: Bump hacking https://review.opendev.org/c/openstack/python-novaclient/+/905707 | |
| 21:06:20 | opendevreview | Merged openstack/placement master: Bump hacking https://review.opendev.org/c/openstack/placement/+/905706 | |
| #openstack-nova - 2024-01-17 | |||
| 02:00:49 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 02:56:31 | opendevreview | melanie witt proposed openstack/nova master: WIP Suppport migration with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/905512 | |
| 02:56:31 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: make |
|
| 02:56:32 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 03:19:58 | songwenping | hi team, the latest nova code raise exception for 'nova list' with the error info TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig', what's wrong? | |
| 04:48:56 | opendevreview | Amit Uniyal proposed openstack/nova master: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/905684 | |
| 04:48:57 | opendevreview | Amit Uniyal proposed openstack/nova master: Fixes bug 2048184 https://review.opendev.org/c/openstack/nova/+/904817 | |