Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-11-29
01:05:11 melwitt sean-k-mooney[m]: thanks for helping with this, have a good night o/
01:20:18 melwitt (leaving a message for tomorrow) bauzas, gibi, sean-k-mooney: I made a list of some of the CI failures I saw on one of dan's patches to help me keep track and linked to this ^ discussion ... https://etherpad.opendev.org/p/nova-ci-failures-minimal in case anyone might be interested
01:23:18 melwitt I made a new etherpad bc I have trouble parsing etherpads with larger amounts of text
08:20:59 tkajinam wondering if that "please report a bug" message is beneficial these days . I agree it was in the past when OpenStack was not yet common but I doubt it still is considering its current status
08:22:20 tkajinam recent reports are mostly caused by unrelated problems (wrong configuration, broken rabbitmq, etc) and personally I've not seen any reports related to actual logic bug in nova for a while
08:33:00 frickler clarkb: sean-k-mooney[m] is right, cirros is using a stock ubuntu kernel, which is why that makes up about half of the complete image size. but that's also why I'm not convinced that this is the source for the issues. did someone check for possible general OOM situations for these failures?
08:48:45 jkulik [ 13.568826] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00001000
08:48:45 jkulik to me that kernel-panic topic sounds like something is wrong with the image or with loading it. I'd interprete it as /sbin/init running into an error and the kernel thus panicing. /sbin/init: can't load library 'libtirpc.so.3'
08:49:43 jkulik this bug report seems related https://bugs.launchpad.net/tempest/+bug/1888224 when ends in "It looks like rootfs image is corrupted."
08:51:05 jkulik I'm not sure how the test infra works, but if it only happens sporadically - could it be that some workers have a corrupt image and others don't?
09:52:34 frickler the cirros image should be baked into the cloud image we boot our test instances from, so if there really is some data corruption happening, it would need to be at a very low level and would likely lead to a more diverse spread of error patterns
10:00:28 bauzas when looking at the kernel panics, yeah I'm not sure other OSes would better work
10:27:02 sean-k-mooney[m] frickler: i am thinking its a general OOM issue which is why i was suggesting reducint the pcie ports we allocate and or adding swap to the tempest flavor to avoid continually adding more ram to the flavors. say add 64mb of swap in the flavor
10:42:11 sean-k-mooney[m] *reduceing
11:01:47 sean-k-mooney[m] frickler: by the way how would you feel about building busybox in statically linked mode so we can aovid dynmaiclly loading these shared objects in general? https://github.com/cirros-dev/cirros/blob/main/conf/busybox.config#L43
11:40:41 jkulik hm ... yeah, could be OOM since the kernel-panic is in `exc_page_fault` ... maybe the kernel tries to load the shared library into memory but doesn't have the space?
12:01:06 sean-k-mooney[m] we have 128m total the kernel ramdisk was using 32mb of that and we allocated 24 pcie root devices each of which uses about 1mb or ram so we are using about half our ram just in does two. the initramfs can be unloaded once we complete the switch_root to the root block device but thats that failed. so ya it not entirely cleared if it OOM’d or not partly due to how early this is in boot but its worth exporing
12:37:51 elodilles bauzas: hi, you've asked me on the meeting to ping you today about the nova stable releases o:) ( https://review.opendev.org/q/project:openstack/releases+is:open+intopic:nova )
12:39:27 elodilles bauzas: let me know if for any reason any of the release patches needs an update (hopefully the version bumps are correct, and hashes should be the latest)
12:40:19 opendevreview Danylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added. https://review.opendev.org/c/openstack/nova/+/876075
13:03:32 frickler sean-k-mooney[m]: we can test that, would have to check how the effect on image size looks like
13:13:33 sean-k-mooney frickler: ya im not sure if it woudl go up or down (both on disk and at runtime)
13:14:29 sean-k-mooney frickler: are there any videos or blog posts on how cirros is built and its design goals?
13:15:36 sean-k-mooney i have too many other pulls on my time right now but kind of interested in either improving cirros to adress these probelems ro replacing it eventually
13:17:06 sean-k-mooney that way i was looking at alpine as a replacement, tinycore ala ironic ipa style would be fien too but if we can tweak cirros to make it more stable in this regrad and continue to use it then that fine in my book too
13:38:54 dvo-plv Hello, sean-k-mooney Are you here ?
13:39:32 sean-k-mooney im listenting to an internal call but yep
13:40:06 frickler sean-k-mooney: I'm not aware of any kind of docs except for what is inside the repo itself. if you want to talk to smoser directly, you can find us in #cirros on liberachat. iiuc the design goal is to have a minimal image for testing purposes, which matches pretty much what openstack CI does with it
13:41:23 dvo-plv I'm currently has a time to work with nova patch
13:41:38 dvo-plv I've resolve some of your comments
13:41:46 dvo-plv https://review.opendev.org/c/openstack/nova/+/876075/28/nova/virt/libvirt/config.py#1815
13:41:46 dvo-plv but I have an open question
13:44:26 sean-k-mooney frickler: ack i skimed the docs breifly last night but have not actully looked at how the build system works or the current content of the image in any dept
13:44:50 sean-k-mooney frickler: so i was conisderign trying the static build myself
13:46:06 sean-k-mooney dvo-plv: so "self.driver_packed is True" is a common python mistake
13:46:22 sean-k-mooney you shoudl not use "is" to compare if something is True
13:47:12 sean-k-mooney is shoudl only be used to test agains monostate types like NONE or to check the adress of two objects
13:47:55 sean-k-mooney so " self.driver_packed is True or" shoudl just be "self.driver_packed or"
13:48:01 dvo-plv oh, I see what you mean, you would like to rewrite it to the just "if self.driver_packed"
13:48:15 sean-k-mooney yep
13:48:24 sean-k-mooney jsut remove "is True"
13:48:52 dvo-plv I thought that you want some another statment here, okay, I will do it asap
13:49:39 sean-k-mooney cool
13:50:01 sean-k-mooney i normally give a code example when i ask for changes like this but i didnt last year
13:50:05 sean-k-mooney *night
14:07:00 opendevreview Danylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added. https://review.opendev.org/c/openstack/nova/+/876075
14:09:57 dvo-plv its okay, I just get your comment wrong from my side
14:20:03 opendevreview Elod Illes proposed openstack/nova stable/zed: add a regression test for all compute RPCAPI 6.x pinnings for rebuild https://review.opendev.org/c/openstack/nova/+/900307
14:20:04 opendevreview Elod Illes proposed openstack/nova stable/zed: Fix rebuild compute RPC API exception for rolling-upgrades https://review.opendev.org/c/openstack/nova/+/900341
14:20:05 opendevreview Elod Illes proposed openstack/nova stable/zed: Adding server actions tests to grenade-multinode https://review.opendev.org/c/openstack/nova/+/900342
14:29:01 elodilles bauzas: i've updated the zed version of the 'RPC backports', please review if you want them included in the zed release ^^^
14:29:26 bauzas sure
14:29:45 bauzas gibi: do you want https://review.opendev.org/c/openstack/nova/+/901656 to be in the first Bobcat z release ?
14:35:21 gibi bauzas: ohh we can try a recheck but I don't want to hold the release
14:37:03 bauzas just rechecked
14:37:07 gibi me too :D
14:37:15 gibi thanks for the ping
14:37:29 bauzas gibi: looks not me wasn't due to a ssh issue
14:38:03 bauzas oh, you looked at the other job failure
14:38:08 bauzas my bad
14:48:18 gibi no worries :)
14:56:12 dvo-plv gibi, Maybe you will have a hance to review this one ?
14:56:13 dvo-plv https://review.opendev.org/c/openstack/nova-specs/+/895924
14:56:19 dvo-plv https://review.opendev.org/c/openstack/nova/+/876075
14:56:19 dvo-plv and this
16:54:07 bauzas gibi: sean-k-mooney: before I leave, so about https://review.opendev.org/c/openstack/nova/+/902084/1
16:54:16 bauzas we have two possibilities :
16:54:24 bauzas 1/ use a star like I did
16:54:42 bauzas 2/ let device_addresses be optional
16:55:08 bauzas this way,
16:55:23 bauzas https://paste.opendev.org/show/bbw8uYTcDqasoSSD54WD/ would be acceptable
16:55:24 sean-k-mooney if we have fxed https://review.opendev.org/c/openstack/nova/+/899406/2
16:55:31 sean-k-mooney then i think it can be optional
16:56:04 bauzas that's a bit differenbt
16:56:21 sean-k-mooney that is fixing the fact that if you have just enabled_mdev_types = nvidia-35
16:56:27 sean-k-mooney device_addresses
16:56:32 bauzas pas-ha[m] change is accepting to only use one type but with a section
16:56:37 bauzas yeah
16:56:41 sean-k-mooney ya so
16:57:00 sean-k-mooney without that i dont think device_adderss being optional is really a good approch
16:57:04 bauzas so, my WIP (if we make dev_add optional) would change the existing behaviour
16:57:06 sean-k-mooney btu if we fix that then im fine with it
16:57:39 bauzas which is that if you use two types but only one got a section, then none of them use any section
16:58:08 bauzas not sure we could backport that
16:58:13 sean-k-mooney thats a bug right
16:58:22 bauzas not really
16:58:29 sean-k-mooney it kind of is
16:58:45 sean-k-mooney im aware of the legacy behavior for the first type
16:58:53 bauzas https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7938-L7951
16:59:00 sean-k-mooney but the second adn subsquent type are ment ot have a section
16:59:23 sean-k-mooney and we shoudl not ignore them if they are defiend just because the first type does not ahve one
16:59:23 bauzas this conditional makes all types needing to have groups for each of them
16:59:48 bauzas sure, I'm just changing this is a bit of a behavioural change
16:59:59 bauzas which couldn't be acceptable for a backport
17:00:05 sean-k-mooney so i do think all types shoudl have a section in general
17:00:15 sean-k-mooney however that only makes snese if there is somethign in it
17:00:18 bauzas you do or don't ?
17:00:55 sean-k-mooney i think we shoudl requrie a section for each mdev type unconditonally
17:00:59 sean-k-mooney personlaly
17:01:11 sean-k-mooney but wiether device_adresss is required or not is a sperate question

Earlier   Later