Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-02
15:22:17 frickler well I'm not talking about force-merging (yet), you'll have the same jobs running in the gate pipeline again (except n-v), but anyway, the decision is up to you
15:27:44 opendevreview Fabian Wiesel proposed openstack/nova-specs master: Lazy Metadata Loading in order to Reduce Server Load https://review.opendev.org/c/openstack/nova-specs/+/922201
15:32:40 tkajinam dansmith, no problem, and thank you !
15:54:00 dansmith definitely want to see all the scenarios run against this since its so much change and it's nearly impossible to do that ahead of time
15:54:16 dansmith if we hit obviously unrelated things I think promote/force is the tool we can use to expedite
15:54:22 dansmith thanks frickler for checking
15:56:26 sean-k-mooney im checkign now but the nova-live-migration job fails on the second patch
15:56:28 sean-k-mooney https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7fc/923256/1/check/nova-live-migration/7fc79c6/
15:56:29 opendevreview Fabian Wiesel proposed openstack/nova-specs master: Lazy Metadata Loading in order to Reduce Server Load https://review.opendev.org/c/openstack/nova-specs/+/922201
15:56:31 sean-k-mooney the ceph one passed
15:56:52 dansmith hmm
15:57:07 dansmith that is odd or sure
15:57:09 dansmith *for
15:57:20 dansmith I have to do a medial thing in a few minutes, so I will be out a bit around there
15:57:24 sean-k-mooney looks like only the blockmigation tests
15:57:38 dansmith right but perhaps something is upset about the disk image?
15:57:56 dansmith Jul 02 15:17:20.783035 np0037859393 nova-compute[41928]: ERROR oslo_messaging.rpc.server nova.exception.ImageUnacceptable: Image 02acc766-7c4b-4ffb-a72d-080ea50527d4 is unacceptable: Image not in a supported format
15:57:57 dansmith eff
15:58:19 sean-k-mooney yep
15:58:57 sean-k-mooney AttributeError: 'NoneType' object has no attribute 'from_file'
15:59:03 sean-k-mooney but i tought we caught that
16:01:15 sean-k-mooney so this is raising https://review.opendev.org/c/openstack/nova/+/923256/1/nova/virt/images.py#152
16:01:32 sean-k-mooney and we take this else https://review.opendev.org/c/openstack/nova/+/923256/1/nova/virt/images.py#174
16:02:07 sean-k-mooney oh ok i need to look at the ohte rhost to see it properly
16:04:12 sean-k-mooney Jul 02 15:19:58.679517 np0037859379 nova-compute[85178]: WARNING nova.virt.images [None req-3f27d1d1-b0a1-46a9-ac4b-6b869e49ea15 tempest-LiveMigrationTest-599041496 tempest-LiveMigrationTest-599041496-project-admin] Unable to perform deep image inspection on type 'aki'
16:04:28 sean-k-mooney so we have special cased for ami image but not aki
16:05:17 sean-k-mooney so i think its passing on the ami iamge and failing on the kernel image
16:06:06 sean-k-mooney dansmith: and we are only seeing it in that job becasue tis the only one using uec images
16:06:08 sean-k-mooney https://github.com/openstack/nova/blob/master/.zuul.yaml#L120-L121
16:07:20 sean-k-mooney so "if disk_format == 'ami':" should be "if disk_format in ('ami','aki'):"
16:07:29 sean-k-mooney here https://review.opendev.org/c/openstack/nova/+/923256/1/nova/virt/images.py#166
16:14:08 dansmith sean-k-mooney: aha, so I think we need to have a serious discussion about AMI in nova because as far as I can tell, the disk image can be anything from a qcow to a raw ext2 or ntfs filesystem and trying to determine safety there is a nightmare
16:14:10 sean-k-mooney dansmith: im carefully going to put a patch on the end to fix 'aki'
16:14:27 sean-k-mooney ya
16:14:32 dansmith perhaps if we fix our own losing track of the format of the base images in nova we could handle it better, but idk
16:14:37 dansmith sean-k-mooney: ack on the fix thanks ++
16:14:40 sean-k-mooney i was also wonderign about the other contaienr formats like ova ovf
16:15:07 sean-k-mooney so we will need to squash the fix into patch 2
16:15:17 sean-k-mooney but i dont want to kick the current set out of the gate
16:15:55 sean-k-mooney hench put it on the end and we can either force merge or pull it back depending on perfernce/urgency
16:15:55 dansmith okay, yeah, let's confirm it fixes it and there's nothing else before we squash but sounds good
16:15:59 dansmith we'll have to fix all the backports too
16:16:33 sean-k-mooney i sould aslo includ ari
16:16:56 sean-k-mooney https://creodias.docs.cloudferro.com/en/latest/cloud/What-Image-Formats-are-available-in-OpenStack-Creodias-Cloud.html
16:20:07 dansmith do we ever run qemu-img cmmands on the ramdisk? I'd guess not
16:20:51 opendevreview sean mooney proposed openstack/nova master: add support for aki and ari images https://review.opendev.org/c/openstack/nova/+/923317
16:21:37 sean-k-mooney dansmith: i dont think we do but if we do its failign before we get to that point
16:22:07 dansmith maybe we just run it on all three pieces for no reason? but okay, no complaints about the patch as-is
16:22:52 sean-k-mooney proably realistically if any of the aobve were qcows i would expect it to trigger the vulnerablity
16:23:02 sean-k-mooney so checking all 3 part is a good thing to do regardless
16:23:23 dansmith ah, that's a good point
16:25:02 sean-k-mooney brb ill keep an eye on the jobs when im back
16:26:40 gibi I'm arond until the top of the hour (already +2 on the Sean's patch)
16:46:04 sean-k-mooney openstacksdk-functional-devstack failed on the 4 patch but is passing on the rest
16:46:07 sean-k-mooney enstack.exceptions.ConflictException: ConflictException: 409
16:46:13 sean-k-mooney in the TestAutoAllocatedTopology test
16:46:26 sean-k-mooney i have seen that fail on othe rpatches and it seams to just be flaky
16:46:47 sean-k-mooney so for now ill leave the curernt set complete
16:47:25 sean-k-mooney it will get rerun once i rebase the patches for the ami/aki/ari fix anyway
16:47:46 dansmith ack
16:48:31 sean-k-mooney im assumign that merged reletivly recently because i have only started to see it fail like that in the last 2-3 weeks
16:48:41 sean-k-mooney i dont recall that job failing previously to that
16:50:43 sean-k-mooney hum no apprently that was added 8 years ago
16:51:05 sean-k-mooney that makes me feel old given the autoallocate stuff was added to enable people to move form nova-networks...
16:51:53 sean-k-mooney ofcouse everythign pre covide feels ancient at this point
17:40:53 dansmith sean-k-mooney: going to need a very quick approval on a job change in nova's ceph job to unblock glance in a sec
17:45:50 opendevreview Dan Smith proposed openstack/nova master: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923322
17:46:03 dansmith sean-k-mooney: ^
17:48:38 dansmith or any other nova cores
17:49:16 dansmith frickler: that one will take a long time in the current queue to merge and glance can't merge their stuff without it, so if you have thoughts... :)
17:52:15 gmann dansmith: one question, this is tempest default, that is what you mean in commit msg https://github.com/openstack/tempest/blob/6b8ff89c8fffcc231d9e5b639d57d6e4a8bd47d3/tempest/config.py#L657
17:52:30 dansmith gmann: yes
17:52:32 dansmith abhishek_: ^
17:52:50 abhishek_ yes
17:52:50 dansmith gmann: this is basically the problem: https://github.com/openstack/tempest/blob/master/tempest/api/image/base.py#L103
17:56:06 gmann dansmith: ohk, +2 but it will be good if we can also fix test to fetch the required image instead of replying on list order in config option
17:56:34 gmann *relying
17:56:39 dansmith replied agree of course
17:57:03 gmann dansmith: thanks
17:57:38 abhishek_ gmann: agree to fix the test as well
17:57:54 gmann +1
18:17:38 frickler dansmith: the only chance to speed up the CI in the current situation likely would be to temporarily abandon all the stable branch changes that eat up resources but fill anyway, and restore them once all fixes have merged in master
18:17:53 dansmith seems like a bad plan
18:18:00 frickler s/fill/will fail/
18:18:32 frickler other then that, just wait until zuul has worked through the queue
18:18:41 dansmith roger that
18:20:16 JayF sean-k-mooney: https://github.com/eventlet/eventlet/pull/970/files is the fix for that eventlet "couldn't green locks" issue we saw popup in python 3.12. I'd expect it to merge+be in next eventlet release. TL;DR: eventlet monkey-patching was happening after we greened locks + as a side effect of running the lock greening, we'd create more locks to be greened AIUI
18:38:43 sean-k-mooney JayF: ack ill try and take a look
18:39:02 JayF sean-k-mooney: consider it just an FYI and notice that I made sure it was taken care of
18:39:06 JayF not sure you have an AI
18:40:11 sean-k-mooney JayF: thanks it more i was on calls for the last hour or so and i want to check back in on the status fo the cve
18:40:26 sean-k-mooney so ill read over that tomorrow
18:40:33 JayF yeah that is more important, only reason I'm letting you know is to get it off your list if it was ever on it
18:40:47 sean-k-mooney :)
19:10:49 fungi with regard to speeding up fixes, we *can* enqueue the more urgent changes directly into the gate pipeline without waiting for them to clear check. the gate pipeline automatically gets resource priority over the check pipeline
19:11:20 dansmith fungi: up to you really... we have the nova and glance patches that both need to go
19:11:24 fungi once they're in the gate, we can also reorder the queue to put them ahead of the rest
19:11:46 dansmith they're just tempest/zuul changes and gmann is +2, so... might be good
19:12:25 fungi they'll need workflow +1 votes before i can enqueue them, but they'll still have to pass the same tests in the gate before they can merge anyway so that should be safe
19:12:48 dansmith the nova one should be good

Earlier   Later