Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-01-16
09:43:08 gibi viks__: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instances_path and https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.state_path are the way I think
09:48:43 viks__ gibi: thanks... got it.. actually i was searching for `/var/lib/nova` in sample conf so i did not find it before... anyway can i set multiple values for it say for eg: `/var/lib/nova,/var/lib/nova1` where they are differnt 2 mount points ?
09:50:54 gibi viks__: you can only set a single path
09:52:48 viks__ gibi: ok.. thanks.. one more thing.. when to use `instances_path` if `state_path` itself will do the job? any suggestions?
09:55:18 gibi viks__: if you want to store the instance local disks in a different place then the nova lock files then instances_path will let you separate the instance disk from the lock files
10:04:32 viks__ gibi: ok.. thanks
10:14:46 opendevreview Sahid Orentino Ferdjaoui proposed openstack/nova master: compute: enhance compute evacuate instance to support target state https://review.opendev.org/c/openstack/nova/+/858383
10:14:46 opendevreview Sahid Orentino Ferdjaoui proposed openstack/nova master: api: extend evacuate instance to support target state https://review.opendev.org/c/openstack/nova/+/858384
10:32:43 gibi dansmith: opened the bug for the OOM https://bugs.launchpad.net/nova/+bug/2002951
10:54:17 sean-k-mooney viks__: if you cant use a single mount path for soem reason you could fake it using lvm voluems to combine multipel disk into one or do it at thte file system level instead of block level using mergerfs https://manpages.ubuntu.com/manpages/impish/man1/mergerfs.1.html
10:55:38 sean-k-mooney nova need everything to be in a single directory on the file system but we dont really care where that folder comes form or how you created it
10:56:31 bauzas auniyal: so, about what we discussed for https://bugs.launchpad.net/nova/+bug/1996732
10:57:11 bauzas auniyal: what you need to do first is to check how to look at the host_state.failed_builds value
10:58:16 sean-k-mooney all you need to do is add a new exception that inherits form the existign one and then where we incremente the value skip it if its the new excpetion
10:58:36 sean-k-mooney the late affinity failure will use the new expction and not be counted
10:58:54 sean-k-mooney but because it inherits form the orginal any clean up that was previousl done will still be done
10:59:36 sean-k-mooney so ya one of the first steps is find out where we modify the build failure value
10:59:51 sean-k-mooney and also where we raise the current excetption for the affinity check
11:00:29 sean-k-mooney after that you can add the new expction and modify both to use it
11:00:34 bauzas sean-k-mooney: the concern of auniyal was how to test iut
11:04:06 sean-k-mooney out side of functional tests it would be tricky to do end to end but unit test for the excption raising and functional test for the end to end interaction. there is no point doing tepest type testing since fault inject will be needed and that is not somethign tempest is good for
11:06:07 bauzas sean-k-mooney: I think it's possible to have a functest for it
11:06:19 bauzas but we need to be able to verify the stats
11:06:27 viks__ sean-k-mooney: thanks for the suggestions
11:09:29 bauzas sean-k-mooney: if we have a functest that creates a group with anti-affinity policy without having the filter, then we can create two instances asking for the same host
11:09:34 auniyal sean-k-mooney, bauzas, suppose we have only one host hostA, an instance with server group having a anti-affinity is present hostA, so if user create anothere instance of same group, that time it will go for reschedule, and hence increase the counter of build faild
11:09:40 auniyal is this a valid scenario
11:09:44 auniyal to test
11:09:44 bauzas sean-k-mooney: and then we could verify the stats value
11:10:20 bauzas auniyal: yeah
11:10:31 bauzas auniyal: but then you need to verify the counter
11:10:46 bauzas https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/resource_tracker.py#L1978-L1980
11:11:20 bauzas which is a defaultdict set here https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/resource_tracker.py#L99
11:12:08 bauzas so the stats is updated here https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/stats.py#L143
11:12:47 bauzas so, as you'll see, this is a keyed dict
11:13:08 bauzas with a functional test, you can introspect that compute.stats dict
11:13:47 bauzas and see that the value of that dict for the key 'failed_builds' is incremented by 1
11:13:57 bauzas after creating inst2
11:14:09 bauzas and that's what we want to change
11:14:26 bauzas auniyal: ^
11:16:32 auniyal ack, this will tell us isntance failed again and agian
11:19:17 gibi auniyal: no, if you have one host with an instance in an anti-affinity group and you try to schedule the second instance to the same group then it will not trigger a reschedule
11:19:36 gibi it will simply fail the scheduling with NoValidHost
11:19:45 auniyal yes
11:20:07 bauzas yes you need a test with 2 nodes, don't disagree
11:20:26 gibi bauzas: two nodes will not help either as the scheduler will pick the other host
11:20:35 bauzas gibi: not if you trick it :)
11:20:38 gibi to trigger a late affinity check failure (and hence the build failure counter increase) you need to have two parallel scheduling request
11:21:10 bauzas gibi: my proposal is simplier with a functest, we have code snippets for tricking the scheduler
11:22:01 bauzas or you could use the az hack
11:22:08 bauzas it will skip the scheduler
11:23:18 gibi (alternatively we could try to remove the anti-affinity filter from the config then the scheduler will allow both VMs to the same host, and the second will fail the late affinity check there)
11:26:58 bauzas gibi: ah, you missed then my point
11:27:06 bauzas (12:09:29) bauzas: sean-k-mooney: if we have a functest that creates a group with anti-affinity policy without having the filter, then we can create two instances asking for the same host
11:27:30 bauzas we indeed need a functest that *doesn't* use the AntiAffinityFilter
11:27:48 bauzas faking the scheduler is just for making sure we land instances on the same host
11:28:55 gibi bauzas: ack, then we thought about the same thing. cool :)
11:29:17 auniyal can we reproduce it manually
11:31:28 auniyal I understand reschdule is correct but its should be counted
11:32:04 auniyal so we just need to verify this before sauing buld failed at https://github.com/openstack/nova/blob/2eb358cdcec36fcfe5388ce6982d2961ca949d0a/nova/compute/manager.py#L2265
11:32:33 auniyal *it should NOT be counted
11:37:10 bauzas auniyal: you can reproduce it with devstack
11:37:19 bauzas auniyal: make sure the filter is disabled
11:37:34 bauzas and force to create two instances with the same group on the same host
11:37:43 bauzas that should work
11:37:58 sean-k-mooney bauzas: you can contol what filters are used in the fucntest so that should not be a problem
11:38:07 bauzas introspecting the stats field would be a bit trickier but I think we log the stats with the DEBUG level
11:38:32 sean-k-mooney ya we likely do but if you really need to you can jsut go driect to the db
11:38:44 bauzas sean-k-mooney: yeah, and I even think we have a funtest fake filter for ensuring all instances go the same host
11:38:54 sean-k-mooney often w will use a spy function to intercept and recored such thigns
11:39:07 sean-k-mooney bauzas: we do yes
11:39:08 bauzas anyway, /me goes off for lunch
11:42:40 kashyap Hmm, on this bz: https://bugzilla.redhat.com/show_bug.cgi?id=2138381 (on CPU compatibility). A Red Hat customer-facing person says using the new CPU API still throws the same error. Actually removing the check is what works correctly:
11:42:44 kashyap https://review.opendev.org/c/openstack/nova/+/869587 -- libvirt: Remove compareCPU() check in _check_cpu_compatibility()
11:43:48 kashyap About debuggability concerns (if you remove the compareCPU() check: the same guy confirms you'd get the same error from libvirt. And it is still debuggable (which is what I said before)
11:44:33 sean-k-mooney kashyap: its much much much less debugable as you now need to boot a vm to triger it
11:46:16 sean-k-mooney since you have https://review.opendev.org/c/openstack/nova/+/869950 i think i would prefer if you abandoned https://review.opendev.org/c/openstack/nova/+/869587 and we proceded with the replacment instead
11:47:26 kashyap sean-k-mooney: Sure, I would also prefer the replacement
11:47:41 kashyap We can agree to disagree on "much much much"
11:48:05 kashyap I don't have the energy to argue much anyway; /me is still recovering from a bike accident
11:48:55 sean-k-mooney when they tested this did they use teh cpu falgs to remove teh flag
11:49:45 sean-k-mooney as in did they set cpu_extra_flags=-whatever
11:49:55 kashyap I don't think they have specified it; I'll ask 'em on th ebz
11:50:33 sean-k-mooney ok its also kind fo sucks that the errror message does not tell you want is incompatible
11:50:56 sean-k-mooney it woudl be nice if it said this set of features are unavaible
11:51:34 kashyap Yeah
11:52:24 sean-k-mooney looks like they found a beaker node internally with icelake to test on
11:52:40 sean-k-mooney maybe we can do the same or get access to test it there
11:53:36 sean-k-mooney it might save some back and forth if you can get direct ssh access to see whats happening or get a dev env yourself that you can deploy devstack on
11:58:05 kashyap sean-k-mooney: Also, we should still keep the removing the check option open absolutely. Please don't insist on keeping it w/o good reasons.
11:58:35 kashyap The tests pass, and DanPB also once said that code is wrong and should be even removed.
11:59:11 kashyap ("tests pass" is not the full reason; but it is not causing problems/troubles. And it was also properly tested by the same RHT person)
11:59:23 sean-k-mooney kashyap: i have given yuou a good reason it will regress novas functionality to remove it and i have explained why
12:00:06 sean-k-mooney if i was insiting i would be using my -2 rights on the patch. i am not
12:01:27 kashyap sean-k-mooney: Sigh; the definition of "regression" is not serious here. We're going in circles. I also want other people's take here.
12:02:08 kashyap (You have to see the _effect_ of the patch: it is changing _where_ it is failing. Yes, it's a kind of a "regression"; but functionally users are better off)
12:02:33 kashyap Anyway. Let's explore the replacement patch in fuller too.
12:07:32 kashyap sean-k-mooney: I'm in agreement with you on definitely using the newer API, as that's a net-benefit. (I was not debating that one.)
12:11:57 sahid o/ guys, do we have a process to convert an option from bool to int?
12:17:57 sean-k-mooney sahid: we do it called not doing it. basically if your changing the type you have to rename the option and deprecat the old one. in this case your going form bool which in our congi is based on string to int

Earlier   Later