Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-05
15:01:19 frickler yes, it is well possible that it is all unrelated and just unstable tests, I just wanted someone to check it
15:03:46 dansmith yep, definitely doesn't look related
15:19:52 fungi did 923288 get reenqueued into the gate accidentally? it's already merged
15:20:21 fungi if we're reshuffling for the current failures anyway, should probably eject that one at the same time
15:21:14 dansmith fungi: maybe someone wanted to be really really sure it landed?
15:21:23 fungi double-merged!
15:21:42 fungi a change so nice we'll merge it twice
15:21:43 dansmith there were definitely things in check and forced into gate at the same time yesterday, perhaps a zuul bug or race?
15:24:09 frickler hmm, according to my bash history, I didn't reenqueue 923288, but zuul has been doing some additional requeues of patches on top of others
15:25:00 frickler maybe a race condition when I did 923289 and zuul wasn't aware yet of 923288 having been merged
15:25:20 fungi normally only if they're not yet merged and meet criteria to be enqueued and depend on a change you've enqueued (either explicitly via depends-on or implicitly through git commit parentage)
15:26:08 fungi but yeah, can't rule out bugs either
15:26:19 frickler but likely if that buildset passes, it will result in a merge failure and cause a gate reset anyway, so we can as well abandon it now IMO
15:27:12 fungi i agree we should kick it out, i'd give 50/50 odds on whether it would actually cause a gate reset or just no-op (not sure what gerrit's behavior is if you call the submit api on an already-merged change)
15:27:22 frickler and I'm reaching EOD quickly (time to watch some soccer), so if you could take over, fungi, that'd be nice
15:27:32 fungi glad to!
15:28:06 fungi enjoy the fußball!
15:28:34 frickler there's also a new set of deferred patches in https://etherpad.opendev.org/p/cve-gate-reshuffling in case the gate gets lets crowded, else I'll handle these tomorrow
15:29:11 fungi thanks, i saw that too when i was looking back at the pad to see if i had maybe been the one to accidentally reenqueue the already-merged change
15:29:14 opendevreview sean mooney proposed openstack/nova master: add iso file format inspector https://review.opendev.org/c/openstack/nova/+/923533
15:30:39 sean-k-mooney frickler: is that the saem one i looked at eariler
15:31:08 sean-k-mooney i commented on the binding failed failure and recheck one fo the jobs for it when i was reviewing 2023.1
15:31:58 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/923289/2#message-348bc1fded02d0ec6cd19807bcc2aec0245f606f
15:32:43 fungi okay, so going to reshuffle the gate putting these first: 923287,3 923289,2 923290,2 923291,3 (and drop 923288,2 since it's already merged)
15:33:05 fungi any other requests while i'm causing a full gate reset anyway?
15:35:12 fungi taking your silence as a resounding "no"
16:51:44 mnaser has anyone pushed/proposed the idea of rack-level failure domains in openstack?
16:51:55 mnaser I know a cloud is a cloud, but VMware people really feel like they see the need to get there
16:52:44 sean-k-mooney failure domaisn are really not a thing in openstack
16:52:51 JayF FYI; I know it's pretty low on you all's list right now, but the fix for the un-upgraded rlocks in eventlet for py3.12 is in the next release
16:53:07 sean-k-mooney the only way to really do that is using seperate keystone regions and isntalls of all the service per rack
16:53:55 sean-k-mooney mnaser: looking only at nova netiern cells or AZ map to failure domains
16:54:15 sean-k-mooney i.e. if the host running your schduer or api dies that affacts all hosts in that region
16:54:37 sean-k-mooney neutron also spanc all cells/AZs
16:54:49 fungi pretty sure the same can be said for the host running your vcenter api in a vmware environment
16:55:16 sean-k-mooney yep so it depens on your defintion of failure domains and if your inlcuding the contol plane in that
16:55:38 sean-k-mooney i.e. workload failrue domains vs netwrok vs storage vs contolplane
16:55:51 sean-k-mooney cinder backend are stroage level failure domains
16:56:39 sean-k-mooney of couse that is only true if you access them over seperate stoage networks
16:57:07 fungi or have some sort of high availability for the storage network
16:57:23 sean-k-mooney yep
16:57:45 mnaser sean-k-mooney: workload failure domains
16:57:46 sean-k-mooney mlag or similar with two diffent top of rack switch ectra
16:57:53 mnaser im thinking a scheduler hint maybe of some sorts + a filter?
16:57:57 fungi like cross-chassis lag
16:58:30 sean-k-mooney mnaser: we brifly discussed having somethign like cephs cluster map
16:58:37 mnaser yeah
16:59:11 sean-k-mooney basically building a tree of host aggrates and grouping things in a heriachy then allowing affintiy/anti affinity ectra
16:59:17 mnaser exactly
16:59:27 mnaser im thinking server group + an extra scheduler hint other than `group` to get that
16:59:39 mnaser but im wondering if im walking a path someone tried before and I can pick up work from or
16:59:56 sean-k-mooney we said it was proably too complex and out of scope of nova but that was also like 6 years ago so thngs have changes somewhat
17:00:18 sean-k-mooney the last time we really talks about this properly was pre/start of the pandmeic
17:00:51 sean-k-mooney mnaser: there is a request ot do az anti affinty this/last cycle
17:01:28 mnaser working in this case, it feels odd to have 1 az per rack
17:01:28 sean-k-mooney in generally i woul dnot be agsint this in principal. it just would not be high on my list of thing to do next
17:01:50 mnaser like that feels like its brute forcing the az feature if you're on rack level
17:01:52 sean-k-mooney az's are just named aggreates
17:02:37 mnaser yeah but I feel like in the perspective of a consumer of the cloud they might be like "oh isn't this like a azure/aws/etc az?"
17:03:08 sean-k-mooney and the answer to hell no
17:03:20 sean-k-mooney but i get that persective
17:03:42 sean-k-mooney so what i would say is we could and might want to make nova eand other services better able to supprot that usecase
17:03:57 mnaser cause like maybe it would be nice to be like.. this is an az .. but also you can have different fault domains
17:04:06 sean-k-mooney but i think we would need some feature in placment to do it effeictly
17:04:24 sean-k-mooney doing it in a filter or weigher would likely be slow an maybe buggy
17:05:40 sean-k-mooney im not saying we could not do it in nova alone but we would need to becareful with how we design it
17:05:44 fungi in retrospect it was probably a mistake to call them availability zones when the implementation wasn't equivalent to aws's az concept, because we've had to fight that confusion ever since, but it'd be hard to change now (i still refer to projects as tenants, after all)
17:06:35 sean-k-mooney nova still has instance_type and flavor depending on what your looking at
17:06:44 fungi indeed
17:07:20 sean-k-mooney so i think an aggerate groups feature is definlty doable and i like cpehs model
17:07:38 fungi we just missed rounding out the suite of renames with "color" and "texture" and maybe "odor"
17:07:48 sean-k-mooney i just feel like crambing that into the existing filters/weither might be sub optimal
17:09:04 sean-k-mooney im sure if we renamed flavors to Terroir it would confuse noone
17:23:45 mnaser sean-k-mooney: I think the path of least resistance is a filter that checks if there is a server group assigned, and a scheduler hint called `different_failure_domain` .. and that goes over host aggregates which have a `failure_domain` attribute and ensures that the hosts in the server group dont end up in that same host aggregate
17:24:51 sean-k-mooney mnaser: i really dont like schduler hits
17:24:58 sean-k-mooney btu thats an option
17:25:22 mnaser sean-k-mooney: could also use `rules` for server groups to have `different_failure_domain` too
17:25:29 opendevreview Dan Smith proposed openstack/nova master: Add iso file format inspector https://review.opendev.org/c/openstack/nova/+/923533
17:25:29 opendevreview Dan Smith proposed openstack/nova master: Reproduce iso regression with deep format inspection https://review.opendev.org/c/openstack/nova/+/923507
17:25:31 mnaser but that would mean an api change
17:25:34 sean-k-mooney server group/instance wawere filteres are some of the most ram/compute intesitve you can create
17:27:29 mnaser yeah
17:27:37 mnaser that is for sure going to be a busy one
17:28:17 mnaser im not sure if there's another approach
17:43:44 fungi 923287 failed out on nova-next and nova-multi-cell: https://zuul.opendev.org/t/openstack/buildset/0a50e0745e3e4759bf743b39208ed1dc
17:44:07 fungi i've reenqueued it and repromoted
18:02:10 opendevreview Dan Smith proposed openstack/nova master: Add iso file format inspector https://review.opendev.org/c/openstack/nova/+/923533
18:27:44 dansmith fungi: confirmed at least one of those was a guest kernel panic so obviously unrelated
18:28:03 fungi exciting!
18:28:25 fungi and good to know, thanks for taking a look
18:28:26 dansmith not really, we see guest kernel panics way too often :(
18:28:42 fungi yeah, my excitement was understated sarcasm really, sorry
18:28:49 dansmith heh, okay
18:29:48 fungi i've about worn out all other tools at my disposal, so sarcasm is what's left at this point
18:30:27 dansmith yeah, you don't have to apologize to me for that.. I'm beyond exhausted.. I dunno what's after that before death, maybe "acceptance" ?
18:51:31 opendevreview Pavlo Shchelokovskyy proposed openstack/nova master: Fix device type when booting from ISO image https://review.opendev.org/c/openstack/nova/+/909611
19:33:15 opendevreview Dan Smith proposed openstack/nova master: Add iso file format inspector https://review.opendev.org/c/openstack/nova/+/923533
19:54:58 opendevreview Merged openstack/nova stable/2023.2: Fix vmdk_allowed_types checking https://review.opendev.org/c/openstack/nova/+/923287
19:55:04 opendevreview Merged openstack/nova stable/2023.1: Check images with format_inspector for safety https://review.opendev.org/c/openstack/nova/+/923289
20:11:30 fungi 923290 hit a package install error, and that knocked out 923291 as a child commit. i'll get them both back into play but they're the only changes left from the ossa

Earlier   Later