Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-01-16
16:07:32 elodilles nothing to report actually
16:07:37 elodilles state is the same as last week
16:07:57 elodilles (i've pinged release cores to approve Nova's Zed release)
16:08:02 elodilles that's all
16:08:02 bauzas ++
16:08:13 bauzas I think I did +1 for the zed release
16:08:21 elodilles yepp, you did
16:08:25 bauzas cool
16:08:36 bauzas elodilles: thanks
16:08:39 elodilles thanks too :)
16:08:40 bauzas #topic vmwareapi 3rd-party CI efforts Highlights
16:08:50 bauzas grandchild: fwiesel: your time :)
16:09:01 fwiesel #Info ETA on exemption for public access by end of week. Hopefully under DNS name: openstack-ci-logs.global.cloud.sap
16:09:12 bauzas <3
16:09:14 fwiesel So, not much happened. Still wrapped in red tape. But at least I have a dns name.
16:09:35 bauzas so we'll wait :)
16:09:36 fwiesel That's it from my side. Any questions?
16:09:41 bauzas thanks a lot
16:09:44 bauzas nope from me
16:09:50 fwiesel You're welcome.
16:10:27 bauzas I think we said we should wait until milestone-2 for seeing whether we would remove vmwareapi, but given what you did and what you continue, of course we won't
16:10:45 bauzas so let's continue to see what things happen
16:11:01 fwiesel Thanks. Much appreciated.
16:11:08 bauzas thanks folks for the work, definitely nice
16:11:21 bauzas #topic Open discussion
16:11:32 bauzas nothing in the agenda, anything anyone ?
16:12:18 bauzas looks not,
16:12:23 bauzas thanks all, was a quick one
16:12:27 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-01-16-16.00.log.html
16:12:27 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-01-16-16.00.txt
16:12:27 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-01-16-16.00.html
16:12:27 opendevmeet Meeting ended Tue Jan 16 16:12:27 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:12:27 bauzas #endmeeting
16:13:02 fwiesel Thanks chat with you next week, hopefully with more news
16:14:39 bauzas ++
16:26:16 Uggla @bauzas, sorry I was discussing with Artom, and I did not managed to answer.
16:27:47 Uggla Yep I updated the status of the bugs. I read a lot of them, but did not manage to triage a lot of them.
16:29:06 opendevreview Takashi Kajinami proposed openstack/placement master: Bump hacking https://review.opendev.org/c/openstack/placement/+/905706
16:29:08 Uggla I think https://bugs.launchpad.net/nova/+bug/2048154 is important to be fixed. I hope @Amit could have a look.
16:30:26 opendevreview Takashi Kajinami proposed openstack/python-novaclient master: Bump hacking https://review.opendev.org/c/openstack/python-novaclient/+/905707
16:41:58 opendevreview Takashi Kajinami proposed openstack/osc-placement master: Bump hacking https://review.opendev.org/c/openstack/osc-placement/+/905715
16:44:24 frickler kashyap: dansmith: added some comments on the etherpad. given that the issue (afaict) only happens for volume resize tests, I'm wondering how we could verify that no actual data corruption is happening on the volume in that scenario
16:44:42 frickler because if it does happen, chasing possible kernel bugs is kind of moot
16:45:14 kashyap frickler: Hi, yeah, I've just seen your comment on the Etherpad. Good question. (I'm in a meeting, and will be slow here)
16:45:31 dansmith frickler: these test instances haven't even made it out of initial kernel boot, so no real opportunity to corrupt anything yet
16:46:05 frickler is this happening on the initial boot, not after resize?
16:46:06 dansmith frickler: and we're not booting from the volumes either, since we're using the preloaded kernel boot method, which means corruption of the on-disk data causing the crash wouldn't be a thing
16:47:34 dansmith before the recent switch to kernel preload that could have been a possible option, but with kernel preload, disk corruption due to the resize should be ruled out AFAIK
16:49:09 dansmith I'm also not sure these are only happening on resize tests
16:57:37 frickler well the errors seem to be happening around the time when the switch to the actual root-fs would be happening. and in the traceback I was looking at, it was happening after the resize. do you happen to have a pointer to the preloaded kernel change?
16:59:00 dansmith frickler: I think it's still loading modules from the initramfs when it crashes
17:00:03 dansmith frickler: https://review.opendev.org/c/openstack/nova/+/902217
17:00:07 dansmith (happened while I was out)
17:01:41 dansmith frickler: I think what you're seeing there is that it tried to load all the modules from the initramfs that it needed to have access to storage, which all crashed, and then it kept going trying to do the switch_root and failed to do so
17:02:26 dansmith it did get vda from the virtio driver so I would think it would have worked even, but with that level of broken having happened who knows
17:03:23 frickler from a quick scan, that patch only changes some specific nova jobs, not tempest-integrated-compute?
17:04:10 dansmith frickler: tbf, I hadn't even looked at the patch, I was going based on sean-k-mooney, melwitt saying nova-next was the only one still doing image boot
17:05:13 dansmith the original bug we opened for this GPF was before we even had the resize-volume-backed test, FWIW :)
17:20:51 tkajinam o/ I'd appreciate it if https://review.opendev.org/c/openstack/nova/+/905314 can get some attention because it's now blocking requirement bump.
17:22:51 dansmith frickler: yeah, that instance in question is doing preloaded kernel boot:
17:22:52 dansmith Jan 16 11:45:59.955345 np0036421623 nova-compute[74855]: <kernel>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/kernel</kernel>
17:22:56 dansmith Jan 16 11:45:59.955345 np0036421623 nova-compute[74855]: <initrd>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/ramdisk</initrd>
17:23:20 dansmith so, shouldn't have loaded the ramdisk from disk
17:23:57 sean-k-mooney dansmith: sorry was drinking coffee reading back
17:24:34 sean-k-mooney frickler: dansmith https://review.opendev.org/c/openstack/nova/+/902809/1
17:24:51 sean-k-mooney we had a folow up patch to overried the images for jobs defiend out of repo
17:25:20 sean-k-mooney so tempest-integrated-compute was updated here https://review.opendev.org/c/openstack/nova/+/902809/1/.zuul.yaml#944
17:26:34 sean-k-mooney dansmith: i quickly tried enabling the larger vms here by the way
17:27:09 sean-k-mooney going to 32 gb feels excessinve but based on the low point (and concurnace 8) we were using about 10G on the contoler
17:27:53 sean-k-mooney so if we had a 16G flavor or even a 11G flavor we woudl entirly elimiate our usage of swap form the job
17:28:21 clarkb fwiw we do have larger flavors they are just limited in quantity as only a few (maybe one?) cloud currently provide them
17:28:36 sean-k-mooney ya i was looking at that
17:29:03 sean-k-mooney we have 32G instance form one vexhost cloud and 16G instnace form another
17:29:19 sean-k-mooney and then 8G instnace in general
17:29:27 clarkb but also I think any "make nodes bigger" effort should coincide with a "understand where the memory use is coming from and whether or not it represents bugs/leaks" effort
17:30:32 sean-k-mooney ya so it looks like if we were able to move to 16G as standard we would signifcatly increase the performance of the job (tempest full was about an hour)
17:30:46 clarkb for example privsep uses tremendous amounts of memory for what it is
17:30:47 sean-k-mooney but if we were to move to that we would likely expand to fill it
17:30:57 clarkb with the little info I have I would classify that as a bug
17:32:16 sean-k-mooney clarkb: frickler dansmith while ye are aroudn could ye look at my zram change when ye have time https://review.opendev.org/c/openstack/devstack/+/890693
17:33:20 sean-k-mooney clarkb: on the privsep topic, i have not looked at it in detail before but do you knwo roughly how much ram its using per process
17:33:35 dansmith sean-k-mooney: yeah I'm for that for sure, but not sure the qa people will want it on by default
17:33:35 sean-k-mooney i assume it partly depend on the service? i.e. nova vs neutron
17:33:54 frickler sean-k-mooney: tbh I've stayed away from that patch for now since I'm wary of making the CI setup even more complex
17:33:57 dansmith gmann: kopecmartin ^
17:34:08 sean-k-mooney ack i can change the defaults os if people prefer it to be off by default ill refactor for that
17:34:26 JayF sean-k-mooney: /me points an ironic change at 890693 for science
17:34:27 sean-k-mooney its off by default locally but on by default in ci currently
17:34:30 dansmith sean-k-mooney: if you disable it by default I'll +2 now, but if not, I want a read from gmann at least
17:35:09 sean-k-mooney ack ill do that i need to deploy a devstack vm for my healthcheck stuff anyway so ill upated it then
17:35:24 clarkb sean-k-mooney: the memory usage varies by the service talking to the privsep instance (which implies to me that workload probably impacts buffer sizes). In total I think it was about half a gig for privsep
17:35:58 sean-k-mooney clarkb: i think tis more related to the size of our oslo.config global + the code that is imported
17:36:54 clarkb sean-k-mooney: oh ya I guess everything is a regex? Those shouldn't be terribly large but if you have enough of them... might also be worth checking if libre2 is more efficient maybe
17:37:09 sean-k-mooney we have some chaced module level varables for example that might be related to this
17:37:42 clarkb sean-k-mooney: for zswap I notice you set swappiness to 100. I think devstack sets swappiness elsewhere you might need to reconcile the two to ensure they don't fight each other
17:38:25 clarkb sean-k-mooney: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap/tasks/main.yaml#L46-L61 this is the code that does it and I think devstack jobs call that first so you would override which I guess is fine
17:38:41 sean-k-mooney clarkb: i was thinking fo https://github.com/openstack/nova/blob/a72f7eaac78927892b937d451cbacb24a83c05ac/nova/api/validation/parameter_types.py#L106
17:38:54 sean-k-mooney although privsep shoudl not be using the api stuff
17:39:00 sean-k-mooney so hopefully that wont matter
17:39:19 clarkb I don't have any other comments on the zswap change. Seems like it may be worth experimenting with

Earlier   Later