Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-01-16
11:06:27 viks__ sean-k-mooney: thanks for the suggestions
11:09:29 bauzas sean-k-mooney: if we have a functest that creates a group with anti-affinity policy without having the filter, then we can create two instances asking for the same host
11:09:34 auniyal sean-k-mooney, bauzas, suppose we have only one host hostA, an instance with server group having a anti-affinity is present hostA, so if user create anothere instance of same group, that time it will go for reschedule, and hence increase the counter of build faild
11:09:40 auniyal is this a valid scenario
11:09:44 bauzas sean-k-mooney: and then we could verify the stats value
11:09:44 auniyal to test
11:10:20 bauzas auniyal: yeah
11:10:31 bauzas auniyal: but then you need to verify the counter
11:10:46 bauzas https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/resource_tracker.py#L1978-L1980
11:11:20 bauzas which is a defaultdict set here https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/resource_tracker.py#L99
11:12:08 bauzas so the stats is updated here https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/stats.py#L143
11:12:47 bauzas so, as you'll see, this is a keyed dict
11:13:08 bauzas with a functional test, you can introspect that compute.stats dict
11:13:47 bauzas and see that the value of that dict for the key 'failed_builds' is incremented by 1
11:13:57 bauzas after creating inst2
11:14:09 bauzas and that's what we want to change
11:14:26 bauzas auniyal: ^
11:16:32 auniyal ack, this will tell us isntance failed again and agian
11:19:17 gibi auniyal: no, if you have one host with an instance in an anti-affinity group and you try to schedule the second instance to the same group then it will not trigger a reschedule
11:19:36 gibi it will simply fail the scheduling with NoValidHost
11:19:45 auniyal yes
11:20:07 bauzas yes you need a test with 2 nodes, don't disagree
11:20:26 gibi bauzas: two nodes will not help either as the scheduler will pick the other host
11:20:35 bauzas gibi: not if you trick it :)
11:20:38 gibi to trigger a late affinity check failure (and hence the build failure counter increase) you need to have two parallel scheduling request
11:21:10 bauzas gibi: my proposal is simplier with a functest, we have code snippets for tricking the scheduler
11:22:01 bauzas or you could use the az hack
11:22:08 bauzas it will skip the scheduler
11:23:18 gibi (alternatively we could try to remove the anti-affinity filter from the config then the scheduler will allow both VMs to the same host, and the second will fail the late affinity check there)
11:26:58 bauzas gibi: ah, you missed then my point
11:27:06 bauzas (12:09:29) bauzas: sean-k-mooney: if we have a functest that creates a group with anti-affinity policy without having the filter, then we can create two instances asking for the same host
11:27:30 bauzas we indeed need a functest that *doesn't* use the AntiAffinityFilter
11:27:48 bauzas faking the scheduler is just for making sure we land instances on the same host
11:28:55 gibi bauzas: ack, then we thought about the same thing. cool :)
11:29:17 auniyal can we reproduce it manually
11:31:28 auniyal I understand reschdule is correct but its should be counted
11:32:04 auniyal so we just need to verify this before sauing buld failed at https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/manager.py#L2265
11:32:33 auniyal *it should NOT be counted
11:37:10 bauzas auniyal: you can reproduce it with devstack
11:37:19 bauzas auniyal: make sure the filter is disabled
11:37:34 bauzas and force to create two instances with the same group on the same host
11:37:43 bauzas that should work
11:37:58 sean-k-mooney bauzas: you can contol what filters are used in the fucntest so that should not be a problem
11:38:07 bauzas introspecting the stats field would be a bit trickier but I think we log the stats with the DEBUG level
11:38:32 sean-k-mooney ya we likely do but if you really need to you can jsut go driect to the db
11:38:44 bauzas sean-k-mooney: yeah, and I even think we have a funtest fake filter for ensuring all instances go the same host
11:38:54 sean-k-mooney often w will use a spy function to intercept and recored such thigns
11:39:07 sean-k-mooney bauzas: we do yes
11:39:08 bauzas anyway, /me goes off for lunch
11:42:40 kashyap Hmm, on this bz: https://bugzilla.redhat.com/show_bug.cgi?id=2138381 (on CPU compatibility). A Red Hat customer-facing person says using the new CPU API still throws the same error. Actually removing the check is what works correctly:
11:42:44 kashyap https://review.opendev.org/c/openstack/nova/+/869587 -- libvirt: Remove compareCPU() check in _check_cpu_compatibility()
11:43:48 kashyap About debuggability concerns (if you remove the compareCPU() check: the same guy confirms you'd get the same error from libvirt. And it is still debuggable (which is what I said before)
11:44:33 sean-k-mooney kashyap: its much much much less debugable as you now need to boot a vm to triger it
11:46:16 sean-k-mooney since you have https://review.opendev.org/c/openstack/nova/+/869950 i think i would prefer if you abandoned https://review.opendev.org/c/openstack/nova/+/869587 and we proceded with the replacment instead
11:47:26 kashyap sean-k-mooney: Sure, I would also prefer the replacement
11:47:41 kashyap We can agree to disagree on "much much much"
11:48:05 kashyap I don't have the energy to argue much anyway; /me is still recovering from a bike accident
11:48:55 sean-k-mooney when they tested this did they use teh cpu falgs to remove teh flag
11:49:45 sean-k-mooney as in did they set cpu_extra_flags=-whatever
11:49:55 kashyap I don't think they have specified it; I'll ask 'em on th ebz
11:50:33 sean-k-mooney ok its also kind fo sucks that the errror message does not tell you want is incompatible
11:50:56 sean-k-mooney it woudl be nice if it said this set of features are unavaible
11:51:34 kashyap Yeah
11:52:24 sean-k-mooney looks like they found a beaker node internally with icelake to test on
11:52:40 sean-k-mooney maybe we can do the same or get access to test it there
11:53:36 sean-k-mooney it might save some back and forth if you can get direct ssh access to see whats happening or get a dev env yourself that you can deploy devstack on
11:58:05 kashyap sean-k-mooney: Also, we should still keep the removing the check option open absolutely. Please don't insist on keeping it w/o good reasons.
11:58:35 kashyap The tests pass, and DanPB also once said that code is wrong and should be even removed.
11:59:11 kashyap ("tests pass" is not the full reason; but it is not causing problems/troubles. And it was also properly tested by the same RHT person)
11:59:23 sean-k-mooney kashyap: i have given yuou a good reason it will regress novas functionality to remove it and i have explained why
12:00:06 sean-k-mooney if i was insiting i would be using my -2 rights on the patch. i am not
12:01:27 kashyap sean-k-mooney: Sigh; the definition of "regression" is not serious here. We're going in circles. I also want other people's take here.
12:02:08 kashyap (You have to see the _effect_ of the patch: it is changing _where_ it is failing. Yes, it's a kind of a "regression"; but functionally users are better off)
12:02:33 kashyap Anyway. Let's explore the replacement patch in fuller too.
12:07:32 kashyap sean-k-mooney: I'm in agreement with you on definitely using the newer API, as that's a net-benefit. (I was not debating that one.)
12:11:57 sahid o/ guys, do we have a process to convert an option from bool to int?
12:17:57 sean-k-mooney sahid: we do it called not doing it. basically if your changing the type you have to rename the option and deprecat the old one. in this case your going form bool which in our congi is based on string to int
12:18:42 sean-k-mooney you can do that in plcaee befause we accpet true/yes|false/no not just 1|0
12:19:16 sean-k-mooney sahid: what config option do you want to modify
12:19:52 sean-k-mooney you will basically have to deprecate the old one and add a new one in the new format and support both in the A cycle.
12:20:44 sean-k-mooney supporting both formats is required becasue you are not allowd to requrie config change to upgrade
12:25:26 sahid yes that the point I don't want to break things.
12:25:58 sahid sean-k-mooney: I'm not sure to understand you mean we can update from bool to int transparently as this is using a string to int?
12:27:17 sean-k-mooney we cant do it transparently because its string to int
12:27:23 sahid oh.. but the pb in our case will be that, a True will not be converted to a int
12:27:28 sean-k-mooney in c it would be int to int
12:28:06 sean-k-mooney we still need to accpet yes/y/True ectra in the config
12:28:18 sean-k-mooney and that woudl have to be converted to 1 i guess
12:28:37 sean-k-mooney but you would also have to accpet 1 and any other values you are supproting
12:29:32 sean-k-mooney so ya after your change you still need to be able to handel a config with True in it as valid if it was to be transparent
12:29:39 sean-k-mooney so thats the problem in this case
12:33:17 sahid sean-k-mooney: is related to this one, if you have a moment to take a look https://review.opendev.org/c/openstack/nova/+/867324
12:33:40 sahid basically it's to add ability to set number of retry
12:34:36 sahid originaly the option is Bool, and used to activated or desactive announces
12:34:43 sean-k-mooney ah that patch i saw that breifly fly by
12:34:58 sean-k-mooney honestly i would just add a second config option for the retry
12:35:04 sahid it's now needed to specify a number of retries and i wamted to avoid that we introduce a new option
12:35:32 sean-k-mooney yep but if we do this i think its just clean to add a new option and default to 1 or 3
12:35:36 sahid yes... as it turn now it's basically what we will have to do
12:36:44 sean-k-mooney ya so workaround options still are treatd like normal config options so the same rules apply

Earlier   Later