Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-01-16
11:19:17 gibi auniyal: no, if you have one host with an instance in an anti-affinity group and you try to schedule the second instance to the same group then it will not trigger a reschedule
11:19:36 gibi it will simply fail the scheduling with NoValidHost
11:19:45 auniyal yes
11:20:07 bauzas yes you need a test with 2 nodes, don't disagree
11:20:26 gibi bauzas: two nodes will not help either as the scheduler will pick the other host
11:20:35 bauzas gibi: not if you trick it :)
11:20:38 gibi to trigger a late affinity check failure (and hence the build failure counter increase) you need to have two parallel scheduling request
11:21:10 bauzas gibi: my proposal is simplier with a functest, we have code snippets for tricking the scheduler
11:22:01 bauzas or you could use the az hack
11:22:08 bauzas it will skip the scheduler
11:23:18 gibi (alternatively we could try to remove the anti-affinity filter from the config then the scheduler will allow both VMs to the same host, and the second will fail the late affinity check there)
11:26:58 bauzas gibi: ah, you missed then my point
11:27:06 bauzas (12:09:29) bauzas: sean-k-mooney: if we have a functest that creates a group with anti-affinity policy without having the filter, then we can create two instances asking for the same host
11:27:30 bauzas we indeed need a functest that *doesn't* use the AntiAffinityFilter
11:27:48 bauzas faking the scheduler is just for making sure we land instances on the same host
11:28:55 gibi bauzas: ack, then we thought about the same thing. cool :)
11:29:17 auniyal can we reproduce it manually
11:31:28 auniyal I understand reschdule is correct but its should be counted
11:32:04 auniyal so we just need to verify this before sauing buld failed at https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/manager.py#L2265
11:32:33 auniyal *it should NOT be counted
11:37:10 bauzas auniyal: you can reproduce it with devstack
11:37:19 bauzas auniyal: make sure the filter is disabled
11:37:34 bauzas and force to create two instances with the same group on the same host
11:37:43 bauzas that should work
11:37:58 sean-k-mooney bauzas: you can contol what filters are used in the fucntest so that should not be a problem
11:38:07 bauzas introspecting the stats field would be a bit trickier but I think we log the stats with the DEBUG level
11:38:32 sean-k-mooney ya we likely do but if you really need to you can jsut go driect to the db
11:38:44 bauzas sean-k-mooney: yeah, and I even think we have a funtest fake filter for ensuring all instances go the same host
11:38:54 sean-k-mooney often w will use a spy function to intercept and recored such thigns
11:39:07 sean-k-mooney bauzas: we do yes
11:39:08 bauzas anyway, /me goes off for lunch
11:42:40 kashyap Hmm, on this bz: https://bugzilla.redhat.com/show_bug.cgi?id=2138381 (on CPU compatibility). A Red Hat customer-facing person says using the new CPU API still throws the same error. Actually removing the check is what works correctly:
11:42:44 kashyap https://review.opendev.org/c/openstack/nova/+/869587 -- libvirt: Remove compareCPU() check in _check_cpu_compatibility()
11:43:48 kashyap About debuggability concerns (if you remove the compareCPU() check: the same guy confirms you'd get the same error from libvirt. And it is still debuggable (which is what I said before)
11:44:33 sean-k-mooney kashyap: its much much much less debugable as you now need to boot a vm to triger it
11:46:16 sean-k-mooney since you have https://review.opendev.org/c/openstack/nova/+/869950 i think i would prefer if you abandoned https://review.opendev.org/c/openstack/nova/+/869587 and we proceded with the replacment instead
11:47:26 kashyap sean-k-mooney: Sure, I would also prefer the replacement
11:47:41 kashyap We can agree to disagree on "much much much"
11:48:05 kashyap I don't have the energy to argue much anyway; /me is still recovering from a bike accident
11:48:55 sean-k-mooney when they tested this did they use teh cpu falgs to remove teh flag
11:49:45 sean-k-mooney as in did they set cpu_extra_flags=-whatever
11:49:55 kashyap I don't think they have specified it; I'll ask 'em on th ebz
11:50:33 sean-k-mooney ok its also kind fo sucks that the errror message does not tell you want is incompatible
11:50:56 sean-k-mooney it woudl be nice if it said this set of features are unavaible
11:51:34 kashyap Yeah
11:52:24 sean-k-mooney looks like they found a beaker node internally with icelake to test on
11:52:40 sean-k-mooney maybe we can do the same or get access to test it there
11:53:36 sean-k-mooney it might save some back and forth if you can get direct ssh access to see whats happening or get a dev env yourself that you can deploy devstack on
11:58:05 kashyap sean-k-mooney: Also, we should still keep the removing the check option open absolutely. Please don't insist on keeping it w/o good reasons.
11:58:35 kashyap The tests pass, and DanPB also once said that code is wrong and should be even removed.
11:59:11 kashyap ("tests pass" is not the full reason; but it is not causing problems/troubles. And it was also properly tested by the same RHT person)
11:59:23 sean-k-mooney kashyap: i have given yuou a good reason it will regress novas functionality to remove it and i have explained why
12:00:06 sean-k-mooney if i was insiting i would be using my -2 rights on the patch. i am not
12:01:27 kashyap sean-k-mooney: Sigh; the definition of "regression" is not serious here. We're going in circles. I also want other people's take here.
12:02:08 kashyap (You have to see the _effect_ of the patch: it is changing _where_ it is failing. Yes, it's a kind of a "regression"; but functionally users are better off)
12:02:33 kashyap Anyway. Let's explore the replacement patch in fuller too.
12:07:32 kashyap sean-k-mooney: I'm in agreement with you on definitely using the newer API, as that's a net-benefit. (I was not debating that one.)
12:11:57 sahid o/ guys, do we have a process to convert an option from bool to int?
12:17:57 sean-k-mooney sahid: we do it called not doing it. basically if your changing the type you have to rename the option and deprecat the old one. in this case your going form bool which in our congi is based on string to int
12:18:42 sean-k-mooney you can do that in plcaee befause we accpet true/yes|false/no not just 1|0
12:19:16 sean-k-mooney sahid: what config option do you want to modify
12:19:52 sean-k-mooney you will basically have to deprecate the old one and add a new one in the new format and support both in the A cycle.
12:20:44 sean-k-mooney supporting both formats is required becasue you are not allowd to requrie config change to upgrade
12:25:26 sahid yes that the point I don't want to break things.
12:25:58 sahid sean-k-mooney: I'm not sure to understand you mean we can update from bool to int transparently as this is using a string to int?
12:27:17 sean-k-mooney we cant do it transparently because its string to int
12:27:23 sahid oh.. but the pb in our case will be that, a True will not be converted to a int
12:27:28 sean-k-mooney in c it would be int to int
12:28:06 sean-k-mooney we still need to accpet yes/y/True ectra in the config
12:28:18 sean-k-mooney and that woudl have to be converted to 1 i guess
12:28:37 sean-k-mooney but you would also have to accpet 1 and any other values you are supproting
12:29:32 sean-k-mooney so ya after your change you still need to be able to handel a config with True in it as valid if it was to be transparent
12:29:39 sean-k-mooney so thats the problem in this case
12:33:17 sahid sean-k-mooney: is related to this one, if you have a moment to take a look https://review.opendev.org/c/openstack/nova/+/867324
12:33:40 sahid basically it's to add ability to set number of retry
12:34:36 sahid originaly the option is Bool, and used to activated or desactive announces
12:34:43 sean-k-mooney ah that patch i saw that breifly fly by
12:34:58 sean-k-mooney honestly i would just add a second config option for the retry
12:35:04 sahid it's now needed to specify a number of retries and i wamted to avoid that we introduce a new option
12:35:32 sean-k-mooney yep but if we do this i think its just clean to add a new option and default to 1 or 3
12:35:36 sahid yes... as it turn now it's basically what we will have to do
12:36:44 sean-k-mooney ya so workaround options still are treatd like normal config options so the same rules apply
12:36:46 sahid you mean we could harcored the number of retry instead,
12:37:01 sean-k-mooney in this case i would jsu tkeep the enable as a bool and add a retry option
12:37:17 sean-k-mooney well we coudl but im ok with a config option for the reties
12:37:41 sean-k-mooney ill just comment on the patch one sec.
12:37:50 sahid so one option to enable, one option to set the number of retries, and one option to specify the interval
12:37:59 sahid cool thank you
12:38:11 sean-k-mooney yep exactly
12:38:30 sean-k-mooney and we can set teh retires and interval to whatever we think is a good default
12:39:06 sahid ok fairenough :)
12:43:25 sean-k-mooney ok done i was suggesting 1 or 3 because 1 i sthe current behavior and 3 is what qemu defaults too when it sends them
13:37:58 kashyap sean-k-mooney: BTW a small data point on that "mpx" saga: if Nova doesn't break at the first CPU compare in check_cpu_compatibility(), then using "cpu_model_extra_flags=-mpx" works
13:38:26 kashyap (That gives a hint too that the first compare is wrong)
13:39:34 sean-k-mooney the way it should be working is we should be removing the mpx flag form all the modles listed in cpu_models and if any of them pass then we proceed as normal
13:40:09 sean-k-mooney so as long as any of the listed modeles work with the cpu_model_extra_flags option applied then we shoudl boot
13:40:19 sean-k-mooney /boot/start the agent/
13:41:00 sean-k-mooney although really if any of them are invlied with that combination we shoudl reject it as an error
16:33:27 opendevreview Aaron S proposed openstack/nova master: Add further workaround features for qemu_monitor_announce_self https://review.opendev.org/c/openstack/nova/+/867324
16:56:35 opendevreview Artom Lifshitz proposed openstack/nova master: Microversion 2.94: FQDN in hostname https://review.opendev.org/c/openstack/nova/+/869812

Earlier   Later