Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-06-28
13:07:55 sean-k-mooney they just need to use an older microverion
13:08:12 bauzas the default policy is admin
13:08:20 sean-k-mooney bauzas: is it?
13:08:26 bauzas for the az hack ? yes
13:08:41 sean-k-mooney i tought we did not have a sepreate policy for it
13:08:42 stephenfin I must admit I don't understand the issue
13:08:43 bauzas (and fortunately)
13:08:45 sean-k-mooney just the az one
13:08:56 stephenfin why wouldn't a simple "does this host belong to this AZ" check make sense?
13:09:03 stephenfin it's not too expensive fwict
13:09:09 gibi bauzas: do you suggest to keep allowing calling --availability_zone my-az:host-not-in-my-az and succeed in old microversions?
13:09:13 bauzas sean-k-mooney: I'm 100% sure about the different policy
13:09:14 stephenfin a simple lookup in the API DB
13:09:53 bauzas gibi: we *could* fix this for old versions only, but then I have another concern
13:10:12 sean-k-mooney stephenfin: my main issue is that you are doing it in a different location to the other az check
13:10:15 bauzas gibi: my other concern is that I know some environments that don't use the AZfilter
13:10:19 sean-k-mooney stephenfin: whihc is doen in the schduler i belive
13:11:03 bauzas gibi: and previously, you were able to use the az hack without the AZFilter
13:11:22 gibi bauzas: yes, but that hack resulted in an inconsistent system
13:11:28 gibi as described in the bug
13:11:35 bauzas gibi: not if you don't use the filter
13:11:49 bauzas see the problem ?
13:12:19 sean-k-mooney gibi: well its basically forcing the host
13:12:26 sean-k-mooney same as a forced migration
13:12:33 gibi so if you dont use teh AzFilter then no AZ recorded in the instance or in the request_spec?
13:12:38 bauzas gibi: and again, I remember ourselves saying A LOT 'well, if you force a host, then meh"
13:12:56 sean-k-mooney gibi: the az should be recoreded in the isntacne regaradless of the filter
13:13:20 bauzas gibi: now, we become super picky about forcing hosts and we want to verify them
13:13:37 gibi I think the base issue is that if you use the hack then you end up having inconsistent az recorded in the instance and in the request_spec
13:13:39 bauzas but again, you *SHOULDN'T* force a destination
13:13:52 gibi we cannot remove the hack from old microversions
13:13:56 bauzas that's why we added 2.74 version
13:14:12 bauzas to have a way to propose a target without forcing it
13:14:19 sean-k-mooney gibi: you wont always
13:14:29 bauzas and we said as a consensus that we should stop supporting to force move
13:15:08 bauzas so, if operators wanna move (because again, you need to be ADMIN in order to use the AZ hack), then your dog
13:15:13 sean-k-mooney bauzas: this is the policy yes https://github.com/openstack/nova/blob/master/nova/policies/servers.py#L204-L225
13:15:35 bauzas again, I was 100% sure about it
13:15:39 sean-k-mooney oh no that the new one
13:15:40 gibi bauzas: even if we fix the bug in the hack the admin still can move to any host just need to specy the proper az name of the host
13:15:48 gibi bauzas: so no functionality is lost
13:15:53 sean-k-mooney this is the old one https://github.com/openstack/nova/blob/master/nova/policies/servers.py#L177-L196
13:16:00 bauzas gibi: admins can move to bad targets anyway
13:16:15 sean-k-mooney gibi: what would the fix be
13:16:28 bauzas gibi: admins can force migrate to hosts without verifying other attributes of the host
13:16:33 gibi bauzas: it is not a bad target, the host is valid, nova just record a wrong az name during the move as it trustes admin input
13:16:43 sean-k-mooney gibi: just include the AZ and not the host in the request spec?
13:17:07 gibi sean-k-mooney: the fix is to make sure admin provide an az name for the host that is valid for the host, then nova will record a valid az name
13:17:12 bauzas again, we're breaking existing behaviours if we change things
13:17:23 gibi bauzas: we are fixing a bug
13:17:29 bauzas because again, some operators opt-out the AZfilter
13:17:31 gibi and such we can break old buggy behaviro
13:17:45 bauzas gibi: it's not a bug, it's a 40x
13:17:56 gibi bauzas: the db inconsistency is the bug
13:17:59 bauzas no
13:18:00 sean-k-mooney gibi: stephenfin if we add this check we shoudl also move the az exits check to the api also
13:18:09 bauzas you asked for a target you can't succeed
13:18:23 sean-k-mooney stephenfin: you added that to your check but did you remove the check later
13:18:29 bauzas gibi: you can create an instance on AZ1, then force migrate to AZ2
13:18:30 gibi bauzas: I still think that if nova creates an inconsistent db record then we should fix that
13:18:41 bauzas gibi: and then, good luck with resizing the instance
13:19:08 sean-k-mooney bauzas: well a resize in that case will resize back to AZ1
13:19:08 gibi so I can accept any fix that result in a consistent db data.
13:19:31 bauzas again, this is a forced operation and we made a clear statement on the fact broken migrations are not nova's fault
13:20:28 bauzas gibi: if we really want to fix this thing
13:20:40 bauzas gibi: I'd then suggest two things
13:20:51 bauzas gibi: 1/ remove the call by a new microversion
13:21:24 bauzas 2/ change the az value to None or to the host AZ in the az hack method
13:21:38 bauzas the az value is meaningless when you use the force hosts
13:21:59 bauzas but I wouldn't hardstop on the call
13:22:06 bauzas eg.
13:22:36 bauzas nova boot --az az1:host_in_az2 would consist into getting the tuple (None, host, node)
13:22:36 gibi 1/ is totally OK to me. So remove the hack in future version.
13:22:39 sean-k-mooney setting it to none would be consitent with using --host
13:23:02 bauzas or actually (schedule_default_az, host, node)
13:23:18 bauzas I mean, setting the returned az to be the default AZ from the option
13:23:26 bauzas (which defaults to None)
13:23:33 gibi 2/ if we can simulate --host when --az was given with bad az name, and log a warning, then I can accep that as well
13:24:15 gibi so keep the existing bad (but used) behavior but avoid incosistent db data
13:24:22 gibi in old microversin
13:24:23 bauzas gibi: the crucial distinction between --host and the az hack is the fact we call out the scheduler on the former, not on the latter
13:25:11 bauzas gibi: honestly, again, ops are using the az hack not for the az, but for providing a target
13:25:49 bauzas gibi: so agreed, we should log a warning (after all, this is an op who did this) and just propose the default AZ as a returned AZ
13:26:39 bauzas if people really want to both force to a target *AND* stick on this AZ, then they can use --host and --az (without the az hack)
13:27:26 gibi yepp
13:27:42 bauzas I'll log my thoughts in the review
13:28:54 gibi bauzas: thanks
13:29:11 gibi let's see how stephenfin feels about it after his lunch
13:33:31 bauzas sure
13:34:23 sean-k-mooney gibi: here is a patch to update teh neutron doc by the way https://review.opendev.org/c/openstack/neutron/+/798302
13:34:54 sean-k-mooney git distracted by the previous conversation
13:35:02 bauzas gosh, eavesdrop is soooo slow to update the
13:35:45 sean-k-mooney bauzas: i think its a cron job or similar
13:35:56 bauzas last updated bits are from more than 20 mins
13:35:58 sean-k-mooney it often pretty quick but sometimes its delayed
13:36:19 sean-k-mooney ya that sometimes happens
13:36:35 sean-k-mooney ususally its only a minute or so behind at most
13:37:37 bauzas still lagging
13:39:55 ganso lyarwood: hi! could you please take one quick look at https://review.opendev.org/c/openstack/nova/+/795432 ? The other reviewers said they are waiting for your feedback. Thanks in advance!
13:40:21 gibi sean-k-mooney: ups, I also pushed a doc patch https://review.opendev.org/c/openstack/neutron/+/798294

Earlier   Later