| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-24 | |||
| 12:29:48 | sean-k-mooney | historically this has not been a problem but with the microcode chagne intel has been making for the vulnerablities it has started to be | |
| 12:30:35 | sean-k-mooney | we have a bug and attempeted fix to track this but its semi stalled mainly be cause of dev/review time | |
| 12:31:33 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/2039803 | |
| 12:31:59 | sean-k-mooney | and i think https://review.opendev.org/c/openstack/nova/+/899185 is the initall attempted fix but its not complete/correct | |
| 12:33:17 | sean-k-mooney | LarsErikP: our downsteam products are based on train and wallaby and we have had similar issue with saphire rappids in both cases. | |
| 12:33:50 | sean-k-mooney | in our case its compounted by the fact oru tain based product uses a kernel/qemu that predate teh reelase of saphire rapids silicon | |
| 12:34:13 | sean-k-mooney | so there is no SapphireRapids cpu model aviaable in our train based release | |
| 12:34:41 | sean-k-mooney | our wallaby baed reslese is new enough to actully know what SapphireRapids is but there we see the issue you hit | |
| 12:35:06 | LarsErikP | Understand. For me, this behaviour did not occur before I upgraded from Zed to 2023.1.. | |
| 12:35:39 | sean-k-mooney | ya w had a partical fix in 2023.1 for a related issue | |
| 12:36:01 | sean-k-mooney | ill see if i can fidn that quickly. its why we added the workarounds | |
| 12:36:50 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/870794 and https://review.opendev.org/c/openstack/nova/+/869950 | |
| 12:38:23 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with stateless firmware https://review.opendev.org/c/openstack/nova/+/908890 | |
| 12:40:15 | sean-k-mooney | LarsErikP: it actully looks like you may not have https://review.opendev.org/c/openstack/nova/+/869950 | |
| 12:40:36 | sean-k-mooney | based on your stack trace | |
| 12:40:37 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/869950/9/nova/virt/libvirt/driver.py | |
| 12:41:01 | sean-k-mooney | its calling File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10015, in _compare_cpu | |
| 12:43:15 | sean-k-mooney | perhaps not | |
| 12:44:11 | LarsErik1 | sean-k-mooney: lol, my irc-vm kernel paniced :P | |
| 12:44:16 | LarsErik1 | sean-k-mooney: lol, my irc-vm kernel paniced :P | |
| 12:44:24 | LarsErik1 | might have lost a few messages there :P | |
| 12:44:53 | sean-k-mooney | no worries i was speculatting that perhaps eyou did not hae the 2 patches we emrged in 2023.1 | |
| 12:45:36 | LarsErik1 | if that's the case, then UCA does not include the patches... | |
| 12:46:40 | LarsErikP | I think they should be included. UCA ships 27.1.0 | |
| 12:48:12 | sean-k-mooney | we have had 27.2.0 and 27.3.0 since then | |
| 12:49:05 | sean-k-mooney | it does have https://review.opendev.org/c/openstack/nova/+/870794 | |
| 12:49:20 | sean-k-mooney | so you have the workaround optio nto disabel the startup check | |
| 12:49:50 | sean-k-mooney | it also has https://review.opendev.org/c/openstack/nova/+/869950 | |
| 12:50:23 | sean-k-mooney | so it is usign the newer api that is not ment to have this proablem at least for the startup check | |
| 12:52:50 | sean-k-mooney | LarsErikP: if you dont mind could you add the output of `virsh capabilities` and `domcapabilities --arch x86_64 --machine q35` to https://bugs.launchpad.net/nova/+bug/2039803 | |
| 12:53:27 | sean-k-mooney | that the data we are parsing so it would help to have data for a system where it actully fails | |
| 12:54:47 | sean-k-mooney | LarsErikP: looking at my laptop which is runign debian testing i do not see a noTSK variant of SapphireRapids by the way | |
| 12:55:14 | sean-k-mooney | im using libvirt 10.0.0 | |
| 12:56:10 | sean-k-mooney | so SapphireRapids-noTSX might be a downstream only cpu model added by cannonical | |
| 12:56:45 | sean-k-mooney | if so then it may be a reall earror and the relevent host might not ahve that patched verison of libvirt? | |
| 13:01:13 | LarsErikP | I can add that info to the bug. no worries =) | |
| 13:04:21 | LarsErikP | done | |
| 13:04:42 | LarsErikP | I have to leave work now... Thanks for your help! | |
| 13:48:58 | opendevreview | Takashi Kajinami proposed openstack/nova master: Migrate MEM_ENCRYPTION_CONTEXT from root provider https://review.opendev.org/c/openstack/nova/+/921814 | |
| 14:44:10 | stephenfin | gmann: Thanks for all the reviews on the OpenAPI series last week :) Much appreciated. I think I have addressed your comments but lmk if I missed anything. I'm hoping sean-k-mooney will get a chance to take a look again this week or next so we can start get more of that merged | |
| 14:57:52 | sean-k-mooney | LarsErikP: your SapphireRapids-noTSX virsh capablities shows <feature name='tsx-ctrl'/> | |
| 14:59:22 | sean-k-mooney | and lookig at virsh domain stats SapphireRapids-noTSX is not a supproted model | |
| 14:59:50 | sean-k-mooney | LarsErikP: this feels like a bug in the SapphireRapids-noTSX definition in your libvirt package in ubuntu | |
| 15:00:14 | sean-k-mooney | based on domstats it thinkg the cloase model is Icelake-Server | |
| 16:01:24 | bauzas | doh, spent a while for debugging *why* my mtty change was failing | |
| 16:01:39 | bauzas | sean-k-mooney: fwiw, I need to enable the nova plugins https://review.opendev.org/c/openstack/nova/+/897708/13/.zuul.yaml#490 :) | |
| 16:02:08 | bauzas | call me stupid | |
| 16:02:53 | sean-k-mooney | hehe yep that would help | |
| 16:03:00 | sean-k-mooney | we enebale it in nova-next i think by default | |
| 16:03:09 | sean-k-mooney | that is where we wanted to eventually test the mtty stuff | |
| 16:03:14 | bauzas | nope, was added by dan in its mtty patch :) | |
| 16:03:33 | bauzas | but yeah, given I created another job, I need also to use it :) | |
| 16:03:44 | bauzas | the mtty modules were then not created :) | |
| 16:03:56 | sean-k-mooney | that at least explains why no rp inventories were created | |
| 16:04:06 | bauzas | correct | |
| 16:08:51 | sean-k-mooney | did you push an updated review with it enabeld | |
| 16:09:59 | bauzas | not yet, will do tomorrow morning | |
| 16:10:06 | bauzas | just testing it on my noble instance | |
| 16:10:24 | bauzas | (restacking by now) | |
| 16:32:07 | jangutter | Hi folks, a newbie question here: we've got a high-concurrency environment with a lot of instances booting at the same time. We have custom placement resources (but we also see this with things like VCPU), and we see hypervisors sometimes overcommitted. | |
| 16:33:10 | jangutter | I'm assuming that this is because the hypervisor doesn't hard-enforce the placement resources, and that this is because of a collision at scheduling time? | |
| 16:58:04 | sean-k-mooney | jangutter: so placement is enforcing atomic allcoations | |
| 16:58:15 | sean-k-mooney | so the api will not allow an over commit to happen | |
| 16:58:46 | sean-k-mooney | so that implies that the allcoation ratio or other config options like reseved count are being modifed with instnace on the hosts | |
| 16:59:28 | sean-k-mooney | the compute agent and schduler do not ned to hard enforce the placement allocation claims becasue placement does | |
| 17:00:16 | sean-k-mooney | i.e. if two schdluer process both have an allcoation candiate for n vcpu and only n are aviable only one of the two schduler proces will be able to take the allcoation candate and turn it into an allocation | |
| 17:00:46 | sean-k-mooney | the other will get a generation confclict and/or an allcoation error | |
| 17:01:12 | sean-k-mooney | the converation of an allocation candiate to an alloction happens in the conductor | |
| 17:01:19 | sean-k-mooney | before we call the compute. | |
| 17:01:49 | sean-k-mooney | so the conductor will just try the next alternate host in the allocation candate set until it can atomicaly claim | |
| 17:02:01 | sean-k-mooney | by default the schdler will return 3 alternate host | |
| 17:02:14 | sean-k-mooney | the primary host and 2 addtional hosts | |
| 17:03:02 | jangutter | Oho, so something's hinky on our conductor! | |
| 17:03:13 | sean-k-mooney | placement allows the allcoation ratio or reseved amount to be updated such that the inventory is oversubscibed but does not allow a new allcoation to be made in that state | |
| 17:03:48 | sean-k-mooney | jangutter: that or your runing placment with an active active galeara and the db is not consitent | |
| 17:04:04 | jangutter | Bingo on the second one! | |
| 17:04:11 | sean-k-mooney | ... we recently found out that is not safe | |
| 17:04:29 | sean-k-mooney | jangutter: oh your using A/A galea? | |
| 17:04:42 | jangutter | indeed! | |
| 17:04:53 | sean-k-mooney | ok there is a workaround you can try | |
| 17:04:56 | sean-k-mooney | once sec | |
| 17:05:55 | sean-k-mooney | https://github.com/openstack-k8s-operators/placement-operator/commit/a5b0bf43b21f495db52eac58c44e52261744605c | |
| 17:06:00 | sean-k-mooney | mysql_wsrep_sync_wait = 1 | |
| 17:06:26 | sean-k-mooney | jangutter: so we were thinking of deployign with a/a for our new installer | |
| 17:06:39 | sean-k-mooney | then we found that most service dont work properly id you do | |
| 17:06:41 | sean-k-mooney | mysql_wsrep_sync_wait = 1 | |
| 17:06:47 | jangutter | thanks, that makes absolute sense now. | |
| 17:07:08 | sean-k-mooney | might work as that forces galera to ensure that it sync on every read | |
| 17:07:28 | sean-k-mooney | or rather read form the most up to date write set whatever its called internally | |
| 17:07:48 | jangutter | (we'll have to check our performance stats on that of course, but you can't argue against CAP) | |
| 17:08:08 | sean-k-mooney | looking at nova code while we were debuging it i dont think we have the code to proerly handal A/A galare with delayed consitency in nova | |
| 17:08:47 | sean-k-mooney | ya so even if its a few seconds of a replciation dely it make the db nolonger atomic | |
| 17:09:11 | sean-k-mooney | and placemnt which is designed to rely on that for the allcoations really cant tollerate that | |
| 17:10:14 | sean-k-mooney | jangutter: still at workday? | |
| 17:10:27 | jangutter | what gave it away :-p | |
| 17:10:44 | sean-k-mooney | nothing we just have not chatted in a while | |
| 17:10:57 | sean-k-mooney | i was wondering if you were still in dublin and working with them | |
| 17:11:44 | jangutter | correct on both counts, also somehow the amount of board games available has grown.... | |
| 17:12:13 | sean-k-mooney | does it ever decrease i tought they just started taking up space in fried houes instead :) | |
| 17:12:21 | sean-k-mooney | *friends | |