Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-06-24
16:10:06 bauzas just testing it on my noble instance
16:10:24 bauzas (restacking by now)
16:32:07 jangutter Hi folks, a newbie question here: we've got a high-concurrency environment with a lot of instances booting at the same time. We have custom placement resources (but we also see this with things like VCPU), and we see hypervisors sometimes overcommitted.
16:33:10 jangutter I'm assuming that this is because the hypervisor doesn't hard-enforce the placement resources, and that this is because of a collision at scheduling time?
16:58:04 sean-k-mooney jangutter: so placement is enforcing atomic allcoations
16:58:15 sean-k-mooney so the api will not allow an over commit to happen
16:58:46 sean-k-mooney so that implies that the allcoation ratio or other config options like reseved count are being modifed with instnace on the hosts
16:59:28 sean-k-mooney the compute agent and schduler do not ned to hard enforce the placement allocation claims becasue placement does
17:00:16 sean-k-mooney i.e. if two schdluer process both have an allcoation candiate for n vcpu and only n are aviable only one of the two schduler proces will be able to take the allcoation candate and turn it into an allocation
17:00:46 sean-k-mooney the other will get a generation confclict and/or an allcoation error
17:01:12 sean-k-mooney the converation of an allocation candiate to an alloction happens in the conductor
17:01:19 sean-k-mooney before we call the compute.
17:01:49 sean-k-mooney so the conductor will just try the next alternate host in the allocation candate set until it can atomicaly claim
17:02:01 sean-k-mooney by default the schdler will return 3 alternate host
17:02:14 sean-k-mooney the primary host and 2 addtional hosts
17:03:02 jangutter Oho, so something's hinky on our conductor!
17:03:13 sean-k-mooney placement allows the allcoation ratio or reseved amount to be updated such that the inventory is oversubscibed but does not allow a new allcoation to be made in that state
17:03:48 sean-k-mooney jangutter: that or your runing placment with an active active galeara and the db is not consitent
17:04:04 jangutter Bingo on the second one!
17:04:11 sean-k-mooney ... we recently found out that is not safe
17:04:29 sean-k-mooney jangutter: oh your using A/A galea?
17:04:42 jangutter indeed!
17:04:53 sean-k-mooney ok there is a workaround you can try
17:04:56 sean-k-mooney once sec
17:05:55 sean-k-mooney https://github.com/openstack-k8s-operators/placement-operator/commit/a5b0bf43b21f495db52eac58c44e52261744605c
17:06:00 sean-k-mooney mysql_wsrep_sync_wait = 1
17:06:26 sean-k-mooney jangutter: so we were thinking of deployign with a/a for our new installer
17:06:39 sean-k-mooney then we found that most service dont work properly id you do
17:06:41 sean-k-mooney mysql_wsrep_sync_wait = 1
17:06:47 jangutter thanks, that makes absolute sense now.
17:07:08 sean-k-mooney might work as that forces galera to ensure that it sync on every read
17:07:28 sean-k-mooney or rather read form the most up to date write set whatever its called internally
17:07:48 jangutter (we'll have to check our performance stats on that of course, but you can't argue against CAP)
17:08:08 sean-k-mooney looking at nova code while we were debuging it i dont think we have the code to proerly handal A/A galare with delayed consitency in nova
17:08:47 sean-k-mooney ya so even if its a few seconds of a replciation dely it make the db nolonger atomic
17:09:11 sean-k-mooney and placemnt which is designed to rely on that for the allcoations really cant tollerate that
17:10:14 sean-k-mooney jangutter: still at workday?
17:10:27 jangutter what gave it away :-p
17:10:44 sean-k-mooney nothing we just have not chatted in a while
17:10:57 sean-k-mooney i was wondering if you were still in dublin and working with them
17:11:44 jangutter correct on both counts, also somehow the amount of board games available has grown....
17:12:13 sean-k-mooney does it ever decrease i tought they just started taking up space in fried houes instead :)
17:12:21 sean-k-mooney *friends
17:13:04 sean-k-mooney jangutter: in case its of use what we ended up doing in our new installer is this https://github.com/openstack-k8s-operators/mariadb-operator/pull/229/files#diff-deca2c41d0839d5052bb1da5ac7ad924f126ef4adcf10fea54d00a1a737f668b
17:13:33 sean-k-mooney jangutter: we reverted to active passive with a script that is invoked by galera when teh leader changes
17:13:48 sean-k-mooney that updated the endpoint in the kubernets service to point only to the new leader
17:15:43 jangutter oh man, I have PTSD about "leader election" and etcd.
17:16:21 jangutter It makes sense, 99% of the time you have a leader, but you want a smooth handover during updates.
17:16:52 sean-k-mooney jangutter: yep and its using a feature of galera that allows it to execute a user script when the leader changes
17:17:42 sean-k-mooney so we are now just providing a script for it to execute and leveragin the fact that k8s adds servie user tokens/config into the pods to allow it to auth and do the service update
17:18:23 sean-k-mooney it kind of sucks that gelera does not just proxy the connection ot the current leader internally but if the hack works...
17:19:00 jangutter we used an analogous way to backup the db by selecting a non-active node and transferring the state from it. the scary thing is that a chunk of this is load-bearing bash scripts.
17:19:32 sean-k-mooney https://imgs.xkcd.com/comics/dependency.png
17:19:48 jangutter 100% true.
17:20:19 sean-k-mooney i wish there was an image macro that would just replace the right arrow
17:31:51 gmann stephenfin: ack, will do 2nd round of reviews.
#openstack-nova - 2024-06-25
09:53:59 opendevreview Takashi Kajinami proposed openstack/nova master: Migrate MEM_ENCRYPTION_CONTEXT from root provider https://review.opendev.org/c/openstack/nova/+/921814
10:05:30 opendevreview Michael Still proposed openstack/nova-specs master: Propose a new API microversion for a spice-direct console. https://review.opendev.org/c/openstack/nova-specs/+/915190
10:06:57 mikal sean-k-mooney: thanks again for your comments, I've replied. I'll take some time in the next couple of days to prototype a flow where Nova requests a connection URL from Kerbside.
10:19:51 sean-k-mooney mikal: ill loop back ot them shortly
14:22:38 bauzas gibi: others, I absolutely forgot I had an appointement at 5:45pm CET today that clashes with the nova meeting
14:22:51 bauzas anyone fancy to run the meeting or I shall I cancel it ?
14:23:50 sean-k-mooney i only had one topic which is i would like to move nova-next to run on noble i ahve a patch up to do that so we can always disucss that in gerrit
14:24:09 sean-k-mooney i didnt have anything else to raise today
14:25:31 bauzas I eventually found the problem with my own mtty patch, it was requiring the devstack plugin to accept noble (we were blindly setting sources.list to jammy)
14:26:06 bauzas shit, I also won't be able to run the meeting on next two tuesdays
14:31:28 bauzas apologies for not having planned it before :(
14:52:51 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420
14:52:51 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140
15:02:00 gibi I can run a quick / reduced nova meeting today
15:02:08 gibi bauzas: ^^
15:02:39 bauzas gibi: thanks, I just wrote an email to -discuss, could you please reply to it ?
15:04:54 gibi done
15:50:20 gibi nova meeting starts in 10 minute
15:50:21 gibi s
16:01:40 opendevmeet The meeting name has been set to 'nova'
16:01:40 opendevmeet Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:40 opendevmeet Meeting started Tue Jun 25 16:01:40 2024 UTC and is due to finish in 60 minutes. The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:40 gibi #startmeeting nova
16:01:54 fwiesel \o
16:02:07 tkajinam o/
16:02:11 elodilles o/
16:02:32 gibi hi folks. I will be your uprepared host today
16:03:48 gibi #topic Bugs (stuck/critical)
16:03:48 Uggla o/
16:03:53 gibi #info No Critical bug
16:03:57 gibi #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster
16:04:16 gibi any bug that feels critical to discuss here?
16:05:40 gibi if not then
16:05:41 gibi #topic Gate status
16:05:45 gibi #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:05:48 gibi #link https://etherpad.opendev.org/p/nova-ci-failures-minimal
16:05:52 gibi #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:05:55 gibi #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:05:58 gibi #info Please try to provide meaningful comment when you recheck
16:06:25 gibi is there any gate issue that needs to be raised?
16:07:40 gibi then
16:07:41 gibi #topic Release Planning
16:07:45 gibi #link https://releases.openstack.org/dalmatian/schedule.html
16:07:53 gibi #info Dalmatian-2 is next week

Earlier   Later