Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-06-24
12:44:16 LarsErik1 sean-k-mooney: lol, my irc-vm kernel paniced :P
12:44:24 LarsErik1 might have lost a few messages there :P
12:44:53 sean-k-mooney no worries i was speculatting that perhaps eyou did not hae the 2 patches we emrged in 2023.1
12:45:36 LarsErik1 if that's the case, then UCA does not include the patches...
12:46:40 LarsErikP I think they should be included. UCA ships 27.1.0
12:48:12 sean-k-mooney we have had 27.2.0 and 27.3.0 since then
12:49:05 sean-k-mooney it does have https://review.opendev.org/c/openstack/nova/+/870794
12:49:20 sean-k-mooney so you have the workaround optio nto disabel the startup check
12:49:50 sean-k-mooney it also has https://review.opendev.org/c/openstack/nova/+/869950
12:50:23 sean-k-mooney so it is usign the newer api that is not ment to have this proablem at least for the startup check
12:52:50 sean-k-mooney LarsErikP: if you dont mind could you add the output of `virsh capabilities` and `domcapabilities --arch x86_64 --machine q35` to https://bugs.launchpad.net/nova/+bug/2039803
12:53:27 sean-k-mooney that the data we are parsing so it would help to have data for a system where it actully fails
12:54:47 sean-k-mooney LarsErikP: looking at my laptop which is runign debian testing i do not see a noTSK variant of SapphireRapids by the way
12:55:14 sean-k-mooney im using libvirt 10.0.0
12:56:10 sean-k-mooney so SapphireRapids-noTSX might be a downstream only cpu model added by cannonical
12:56:45 sean-k-mooney if so then it may be a reall earror and the relevent host might not ahve that patched verison of libvirt?
13:01:13 LarsErikP I can add that info to the bug. no worries =)
13:04:21 LarsErikP done
13:04:42 LarsErikP I have to leave work now... Thanks for your help!
13:48:58 opendevreview Takashi Kajinami proposed openstack/nova master: Migrate MEM_ENCRYPTION_CONTEXT from root provider https://review.opendev.org/c/openstack/nova/+/921814
14:44:10 stephenfin gmann: Thanks for all the reviews on the OpenAPI series last week :) Much appreciated. I think I have addressed your comments but lmk if I missed anything. I'm hoping sean-k-mooney will get a chance to take a look again this week or next so we can start get more of that merged
14:57:52 sean-k-mooney LarsErikP: your SapphireRapids-noTSX virsh capablities shows <feature name='tsx-ctrl'/>
14:59:22 sean-k-mooney and lookig at virsh domain stats SapphireRapids-noTSX is not a supproted model
14:59:50 sean-k-mooney LarsErikP: this feels like a bug in the SapphireRapids-noTSX definition in your libvirt package in ubuntu
15:00:14 sean-k-mooney based on domstats it thinkg the cloase model is Icelake-Server
16:01:24 bauzas doh, spent a while for debugging *why* my mtty change was failing
16:01:39 bauzas sean-k-mooney: fwiw, I need to enable the nova plugins https://review.opendev.org/c/openstack/nova/+/897708/13/.zuul.yaml#490 :)
16:02:08 bauzas call me stupid
16:02:53 sean-k-mooney hehe yep that would help
16:03:00 sean-k-mooney we enebale it in nova-next i think by default
16:03:09 sean-k-mooney that is where we wanted to eventually test the mtty stuff
16:03:14 bauzas nope, was added by dan in its mtty patch :)
16:03:33 bauzas but yeah, given I created another job, I need also to use it :)
16:03:44 bauzas the mtty modules were then not created :)
16:03:56 sean-k-mooney that at least explains why no rp inventories were created
16:04:06 bauzas correct
16:08:51 sean-k-mooney did you push an updated review with it enabeld
16:09:59 bauzas not yet, will do tomorrow morning
16:10:06 bauzas just testing it on my noble instance
16:10:24 bauzas (restacking by now)
16:32:07 jangutter Hi folks, a newbie question here: we've got a high-concurrency environment with a lot of instances booting at the same time. We have custom placement resources (but we also see this with things like VCPU), and we see hypervisors sometimes overcommitted.
16:33:10 jangutter I'm assuming that this is because the hypervisor doesn't hard-enforce the placement resources, and that this is because of a collision at scheduling time?
16:58:04 sean-k-mooney jangutter: so placement is enforcing atomic allcoations
16:58:15 sean-k-mooney so the api will not allow an over commit to happen
16:58:46 sean-k-mooney so that implies that the allcoation ratio or other config options like reseved count are being modifed with instnace on the hosts
16:59:28 sean-k-mooney the compute agent and schduler do not ned to hard enforce the placement allocation claims becasue placement does
17:00:16 sean-k-mooney i.e. if two schdluer process both have an allcoation candiate for n vcpu and only n are aviable only one of the two schduler proces will be able to take the allcoation candate and turn it into an allocation
17:00:46 sean-k-mooney the other will get a generation confclict and/or an allcoation error
17:01:12 sean-k-mooney the converation of an allocation candiate to an alloction happens in the conductor
17:01:19 sean-k-mooney before we call the compute.
17:01:49 sean-k-mooney so the conductor will just try the next alternate host in the allocation candate set until it can atomicaly claim
17:02:01 sean-k-mooney by default the schdler will return 3 alternate host
17:02:14 sean-k-mooney the primary host and 2 addtional hosts
17:03:02 jangutter Oho, so something's hinky on our conductor!
17:03:13 sean-k-mooney placement allows the allcoation ratio or reseved amount to be updated such that the inventory is oversubscibed but does not allow a new allcoation to be made in that state
17:03:48 sean-k-mooney jangutter: that or your runing placment with an active active galeara and the db is not consitent
17:04:04 jangutter Bingo on the second one!
17:04:11 sean-k-mooney ... we recently found out that is not safe
17:04:29 sean-k-mooney jangutter: oh your using A/A galea?
17:04:42 jangutter indeed!
17:04:53 sean-k-mooney ok there is a workaround you can try
17:04:56 sean-k-mooney once sec
17:05:55 sean-k-mooney https://github.com/openstack-k8s-operators/placement-operator/commit/a5b0bf43b21f495db52eac58c44e52261744605c
17:06:00 sean-k-mooney mysql_wsrep_sync_wait = 1
17:06:26 sean-k-mooney jangutter: so we were thinking of deployign with a/a for our new installer
17:06:39 sean-k-mooney then we found that most service dont work properly id you do
17:06:41 sean-k-mooney mysql_wsrep_sync_wait = 1
17:06:47 jangutter thanks, that makes absolute sense now.
17:07:08 sean-k-mooney might work as that forces galera to ensure that it sync on every read
17:07:28 sean-k-mooney or rather read form the most up to date write set whatever its called internally
17:07:48 jangutter (we'll have to check our performance stats on that of course, but you can't argue against CAP)
17:08:08 sean-k-mooney looking at nova code while we were debuging it i dont think we have the code to proerly handal A/A galare with delayed consitency in nova
17:08:47 sean-k-mooney ya so even if its a few seconds of a replciation dely it make the db nolonger atomic
17:09:11 sean-k-mooney and placemnt which is designed to rely on that for the allcoations really cant tollerate that
17:10:14 sean-k-mooney jangutter: still at workday?
17:10:27 jangutter what gave it away :-p
17:10:44 sean-k-mooney nothing we just have not chatted in a while
17:10:57 sean-k-mooney i was wondering if you were still in dublin and working with them
17:11:44 jangutter correct on both counts, also somehow the amount of board games available has grown....
17:12:13 sean-k-mooney does it ever decrease i tought they just started taking up space in fried houes instead :)
17:12:21 sean-k-mooney *friends
17:13:04 sean-k-mooney jangutter: in case its of use what we ended up doing in our new installer is this https://github.com/openstack-k8s-operators/mariadb-operator/pull/229/files#diff-deca2c41d0839d5052bb1da5ac7ad924f126ef4adcf10fea54d00a1a737f668b
17:13:33 sean-k-mooney jangutter: we reverted to active passive with a script that is invoked by galera when teh leader changes
17:13:48 sean-k-mooney that updated the endpoint in the kubernets service to point only to the new leader
17:15:43 jangutter oh man, I have PTSD about "leader election" and etcd.
17:16:21 jangutter It makes sense, 99% of the time you have a leader, but you want a smooth handover during updates.
17:16:52 sean-k-mooney jangutter: yep and its using a feature of galera that allows it to execute a user script when the leader changes
17:17:42 sean-k-mooney so we are now just providing a script for it to execute and leveragin the fact that k8s adds servie user tokens/config into the pods to allow it to auth and do the service update
17:18:23 sean-k-mooney it kind of sucks that gelera does not just proxy the connection ot the current leader internally but if the hack works...
17:19:00 jangutter we used an analogous way to backup the db by selecting a non-active node and transferring the state from it. the scary thing is that a chunk of this is load-bearing bash scripts.
17:19:32 sean-k-mooney https://imgs.xkcd.com/comics/dependency.png
17:19:48 jangutter 100% true.
17:20:19 sean-k-mooney i wish there was an image macro that would just replace the right arrow
17:31:51 gmann stephenfin: ack, will do 2nd round of reviews.
#openstack-nova - 2024-06-25
09:53:59 opendevreview Takashi Kajinami proposed openstack/nova master: Migrate MEM_ENCRYPTION_CONTEXT from root provider https://review.opendev.org/c/openstack/nova/+/921814
10:05:30 opendevreview Michael Still proposed openstack/nova-specs master: Propose a new API microversion for a spice-direct console. https://review.opendev.org/c/openstack/nova-specs/+/915190
10:06:57 mikal sean-k-mooney: thanks again for your comments, I've replied. I'll take some time in the next couple of days to prototype a flow where Nova requests a connection URL from Kerbside.
10:19:51 sean-k-mooney mikal: ill loop back ot them shortly
14:22:38 bauzas gibi: others, I absolutely forgot I had an appointement at 5:45pm CET today that clashes with the nova meeting
14:22:51 bauzas anyone fancy to run the meeting or I shall I cancel it ?

Earlier   Later