Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-02
14:46:02 dansmith oh it does fail for me on master,
14:46:09 dansmith I ran preivously before the tests were added
14:46:11 dansmith I will fix
14:47:07 opendevreview Dan Smith proposed openstack/nova master: Fix vmdk_allowed_types checking https://review.opendev.org/c/openstack/nova/+/923258
14:47:54 sean-k-mooney reappoved
14:50:02 opendevreview Dan Smith proposed openstack/nova stable/2024.1: Fix vmdk_allowed_types checking https://review.opendev.org/c/openstack/nova/+/923276
14:55:26 opendevreview Dan Smith proposed openstack/nova stable/2023.2: Fix vmdk_allowed_types checking https://review.opendev.org/c/openstack/nova/+/923287
14:58:43 dansmith mnaser: tkajinam sorry about that, thanks for the heads up 2023.1 coming in a sec
15:00:59 opendevreview Dan Smith proposed openstack/nova stable/2023.1: Fix vmdk_allowed_types checking https://review.opendev.org/c/openstack/nova/+/923291
15:09:09 JayF melwitt: sean-k-mooney: re: the lock greening errors in python 3.12: https://github.com/eventlet/eventlet/issues/969 is what itamarst tracked it to.
15:12:11 frickler sean-k-mooney: dansmith: do you want us to promote any of the approved patches into gate or do you prefer for all check jobs to finish first?
15:13:03 dansmith frickler: definitely want to see the tests first
15:19:47 sean-k-mooney frickler: based on the quick review we have done we belive the patchs shoudl be correct however we need the wider coverage form check to confirm we are not regressing any edgecases. if we see a timeout ectra or other infa issues we can consier promotiign or requeing
15:22:17 frickler well I'm not talking about force-merging (yet), you'll have the same jobs running in the gate pipeline again (except n-v), but anyway, the decision is up to you
15:27:44 opendevreview Fabian Wiesel proposed openstack/nova-specs master: Lazy Metadata Loading in order to Reduce Server Load https://review.opendev.org/c/openstack/nova-specs/+/922201
15:32:40 tkajinam dansmith, no problem, and thank you !
15:54:00 dansmith definitely want to see all the scenarios run against this since its so much change and it's nearly impossible to do that ahead of time
15:54:16 dansmith if we hit obviously unrelated things I think promote/force is the tool we can use to expedite
15:54:22 dansmith thanks frickler for checking
15:56:26 sean-k-mooney im checkign now but the nova-live-migration job fails on the second patch
15:56:28 sean-k-mooney https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7fc/923256/1/check/nova-live-migration/7fc79c6/
15:56:29 opendevreview Fabian Wiesel proposed openstack/nova-specs master: Lazy Metadata Loading in order to Reduce Server Load https://review.opendev.org/c/openstack/nova-specs/+/922201
15:56:31 sean-k-mooney the ceph one passed
15:56:52 dansmith hmm
15:57:07 dansmith that is odd or sure
15:57:09 dansmith *for
15:57:20 dansmith I have to do a medial thing in a few minutes, so I will be out a bit around there
15:57:24 sean-k-mooney looks like only the blockmigation tests
15:57:38 dansmith right but perhaps something is upset about the disk image?
15:57:56 dansmith Jul 02 15:17:20.783035 np0037859393 nova-compute[41928]: ERROR oslo_messaging.rpc.server nova.exception.ImageUnacceptable: Image 02acc766-7c4b-4ffb-a72d-080ea50527d4 is unacceptable: Image not in a supported format
15:57:57 dansmith eff
15:58:19 sean-k-mooney yep
15:58:57 sean-k-mooney AttributeError: 'NoneType' object has no attribute 'from_file'
15:59:03 sean-k-mooney but i tought we caught that
16:01:15 sean-k-mooney so this is raising https://review.opendev.org/c/openstack/nova/+/923256/1/nova/virt/images.py#152
16:01:32 sean-k-mooney and we take this else https://review.opendev.org/c/openstack/nova/+/923256/1/nova/virt/images.py#174
16:02:07 sean-k-mooney oh ok i need to look at the ohte rhost to see it properly
16:04:12 sean-k-mooney Jul 02 15:19:58.679517 np0037859379 nova-compute[85178]: WARNING nova.virt.images [None req-3f27d1d1-b0a1-46a9-ac4b-6b869e49ea15 tempest-LiveMigrationTest-599041496 tempest-LiveMigrationTest-599041496-project-admin] Unable to perform deep image inspection on type 'aki'
16:04:28 sean-k-mooney so we have special cased for ami image but not aki
16:05:17 sean-k-mooney so i think its passing on the ami iamge and failing on the kernel image
16:06:06 sean-k-mooney dansmith: and we are only seeing it in that job becasue tis the only one using uec images
16:06:08 sean-k-mooney https://github.com/openstack/nova/blob/master/.zuul.yaml#L120-L121
16:07:20 sean-k-mooney so "if disk_format == 'ami':" should be "if disk_format in ('ami','aki'):"
16:07:29 sean-k-mooney here https://review.opendev.org/c/openstack/nova/+/923256/1/nova/virt/images.py#166
16:14:08 dansmith sean-k-mooney: aha, so I think we need to have a serious discussion about AMI in nova because as far as I can tell, the disk image can be anything from a qcow to a raw ext2 or ntfs filesystem and trying to determine safety there is a nightmare
16:14:10 sean-k-mooney dansmith: im carefully going to put a patch on the end to fix 'aki'
16:14:27 sean-k-mooney ya
16:14:32 dansmith perhaps if we fix our own losing track of the format of the base images in nova we could handle it better, but idk
16:14:37 dansmith sean-k-mooney: ack on the fix thanks ++
16:14:40 sean-k-mooney i was also wonderign about the other contaienr formats like ova ovf
16:15:07 sean-k-mooney so we will need to squash the fix into patch 2
16:15:17 sean-k-mooney but i dont want to kick the current set out of the gate
16:15:55 sean-k-mooney hench put it on the end and we can either force merge or pull it back depending on perfernce/urgency
16:15:55 dansmith okay, yeah, let's confirm it fixes it and there's nothing else before we squash but sounds good
16:15:59 dansmith we'll have to fix all the backports too
16:16:33 sean-k-mooney i sould aslo includ ari
16:16:56 sean-k-mooney https://creodias.docs.cloudferro.com/en/latest/cloud/What-Image-Formats-are-available-in-OpenStack-Creodias-Cloud.html
16:20:07 dansmith do we ever run qemu-img cmmands on the ramdisk? I'd guess not
16:20:51 opendevreview sean mooney proposed openstack/nova master: add support for aki and ari images https://review.opendev.org/c/openstack/nova/+/923317
16:21:37 sean-k-mooney dansmith: i dont think we do but if we do its failign before we get to that point
16:22:07 dansmith maybe we just run it on all three pieces for no reason? but okay, no complaints about the patch as-is
16:22:52 sean-k-mooney proably realistically if any of the aobve were qcows i would expect it to trigger the vulnerablity
16:23:02 sean-k-mooney so checking all 3 part is a good thing to do regardless
16:23:23 dansmith ah, that's a good point
16:25:02 sean-k-mooney brb ill keep an eye on the jobs when im back
16:26:40 gibi I'm arond until the top of the hour (already +2 on the Sean's patch)
16:46:04 sean-k-mooney openstacksdk-functional-devstack failed on the 4 patch but is passing on the rest
16:46:07 sean-k-mooney enstack.exceptions.ConflictException: ConflictException: 409
16:46:13 sean-k-mooney in the TestAutoAllocatedTopology test
16:46:26 sean-k-mooney i have seen that fail on othe rpatches and it seams to just be flaky
16:46:47 sean-k-mooney so for now ill leave the curernt set complete
16:47:25 sean-k-mooney it will get rerun once i rebase the patches for the ami/aki/ari fix anyway
16:47:46 dansmith ack
16:48:31 sean-k-mooney im assumign that merged reletivly recently because i have only started to see it fail like that in the last 2-3 weeks
16:48:41 sean-k-mooney i dont recall that job failing previously to that
16:50:43 sean-k-mooney hum no apprently that was added 8 years ago
16:51:05 sean-k-mooney that makes me feel old given the autoallocate stuff was added to enable people to move form nova-networks...
16:51:53 sean-k-mooney ofcouse everythign pre covide feels ancient at this point
17:40:53 dansmith sean-k-mooney: going to need a very quick approval on a job change in nova's ceph job to unblock glance in a sec
17:45:50 opendevreview Dan Smith proposed openstack/nova master: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923322
17:46:03 dansmith sean-k-mooney: ^
17:48:38 dansmith or any other nova cores
17:49:16 dansmith frickler: that one will take a long time in the current queue to merge and glance can't merge their stuff without it, so if you have thoughts... :)
17:52:15 gmann dansmith: one question, this is tempest default, that is what you mean in commit msg https://github.com/openstack/tempest/blob/6b8ff89c8fffcc231d9e5b639d57d6e4a8bd47d3/tempest/config.py#L657
17:52:30 dansmith gmann: yes
17:52:32 dansmith abhishek_: ^
17:52:50 abhishek_ yes
17:52:50 dansmith gmann: this is basically the problem: https://github.com/openstack/tempest/blob/master/tempest/api/image/base.py#L103
17:56:06 gmann dansmith: ohk, +2 but it will be good if we can also fix test to fetch the required image instead of replying on list order in config option
17:56:34 gmann *relying
17:56:39 dansmith replied agree of course
17:57:03 gmann dansmith: thanks
17:57:38 abhishek_ gmann: agree to fix the test as well
17:57:54 gmann +1
18:17:38 frickler dansmith: the only chance to speed up the CI in the current situation likely would be to temporarily abandon all the stable branch changes that eat up resources but fill anyway, and restore them once all fixes have merged in master
18:17:53 dansmith seems like a bad plan
18:18:00 frickler s/fill/will fail/
18:18:32 frickler other then that, just wait until zuul has worked through the queue
18:18:41 dansmith roger that
18:20:16 JayF sean-k-mooney: https://github.com/eventlet/eventlet/pull/970/files is the fix for that eventlet "couldn't green locks" issue we saw popup in python 3.12. I'd expect it to merge+be in next eventlet release. TL;DR: eventlet monkey-patching was happening after we greened locks + as a side effect of running the lock greening, we'd create more locks to be greened AIUI

Earlier   Later