Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-18
20:05:09 sean-k-mooney thats the thing. the schedule is selecting host1 for both because of the weigher the the scond instance appears to be reschduled to host2 and you end up with anti affinity
20:06:04 mriedem OH
20:06:13 mriedem ok yeah then just set max_attempts=0 and that should kill that
20:06:42 mriedem we don't get the reschedule in devstack b/c we disabled the up-calls to the api db for that late check in the compute
20:07:05 mriedem that's something else you could do is disable the late-check up-call to mimic dsvm setup
20:07:07 mriedem that might be better
20:07:22 mriedem because honestly we shouldn't get alternatives for either of these instances in a 2 host setup
20:07:26 mriedem since there would be no alternatives
20:07:56 sean-k-mooney setting the max seams quicker to test. if that cases it to fail i can try disableing the up call
20:11:34 jlk oh wow, hrm. Testing out domain.destroy() seems to NOT send a soft shutdown signal. INTERESTING.
20:12:47 sean-k-mooney mriedem: yep adding self.flags(max_attempts=0, group='scheduler') makes the regression fail
20:16:04 mriedem sean-k-mooney: sweet, can you try with disabling the up-calls?
20:16:26 mriedem sean-k-mooney: https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_group_policy_check_upcall
20:16:39 sean-k-mooney actully it failed because max attempts must be >= 1
20:16:44 sean-k-mooney testing again
20:16:56 mriedem then if that introduces the regression you could fix up the test and push that up,
20:17:01 mriedem then i could rebase the bug fix on top of it
20:20:26 sean-k-mooney ok with it set to max_attempts=1 i get Build of instance cbf4b2dd-8301-4d62-97f5-d74b00b21d64 was re-scheduled: Anti-affinity instance group policy was violated.
20:20:36 sean-k-mooney so ya ill try disableing that check now
20:20:37 mriedem yup as expected
20:21:50 mriedem cfriesen_: re: your reply in the ML thread about this bug, i think we can do both fixes actually for this same thing - because we ran into it this time in the antiaffinity filter, but the fact we pick the first request spec and send that to the scheduler with a list of other instance uuids and send that reqspec into each filter with what could be the wrong instance_uuid is still a thing worth fixing,
20:21:55 sean-k-mooney what config group do i set disable_group_policy_check_upcall in? is workarounds its own group
20:22:00 mriedem and num_instances being set per-scheduler request is something else we could fix
20:22:06 mriedem sean-k-mooney: [workarounds]
20:22:08 mriedem yes
20:22:26 mriedem honestly num_instances shouldn't even be needed anymore, but it predates sending the list of instance uuids to select_destinations
20:22:54 mriedem we kind of sorely need to drop scheduler rpcapi 4.x compatibility and move to 5.0
20:23:09 mriedem so we can drop a lot of this garbage compat code in the scheduler
20:24:27 sean-k-mooney is that something we can do in stien. im guessing we need 4.x to live as long as cellsv1 or something else is still around?
20:24:49 mriedem it's usually sometihng we do before we cut a release,
20:24:58 mriedem we add a compat shim and then drop the shim once we are in n+1 master
20:25:13 mriedem dansmith is the one that's always done it
20:25:28 sean-k-mooney ah. ok so with disableing the upcall i new get to your assertion and it fails with 2!=1
20:25:37 mriedem sean-k-mooney: awesome
20:25:51 mriedem want to fix the assertion and add a FIXME note in there about cleaning up the assertion once the bug is fixed?
20:26:00 mriedem probably need to update the docstring in the test also
20:26:30 sean-k-mooney sure will do.
20:26:43 sean-k-mooney this would have been a nightmare to find without a debugger
20:27:12 mriedem if i would have been thinking clearly about the devstack env, i would have eventually realized that we might be hitting a reschedule which devstack doesn't
20:27:18 mriedem but, brain is mush
20:27:32 mriedem i would realize it at 3am most likely
20:44:48 openstackgerrit sean mooney proposed openstack/nova master: Add regression test for bug 1781710 https://review.openstack.org/583339
20:44:50 openstack bug 1781710 in OpenStack Compute (nova) "ServersOnMultiNodesTest.test_create_server_with_scheduler_hint_group_anti_affinity failing with "Servers are on the same host"" [High,Fix released] https://launchpad.net/bugs/1781710 - Assigned to Matt Riedemann (mriedem)
20:47:34 sean-k-mooney by the way have people hit this.
20:47:36 sean-k-mooney RuntimeError: On Python 3, zvm sdk supports to Python 3.5
20:48:21 sean-k-mooney im currently on centos 7.5 and i install python 3.6. we dont support 3.4 upstream right
20:49:16 sean-k-mooney centos only ship 3.4 and 3.6 in the package manager
20:50:33 sean-k-mooney efried: do you know what are the changes of getting 3.6 support in the zvm sdk
20:50:59 efried sean-k-mooney: You mean so you can tox with the effin thing?
20:51:12 sean-k-mooney yes
20:51:29 sean-k-mooney right now i cand run pep8 or any of the python3 based envs
20:51:46 sean-k-mooney unless in install 3.5 that is or downgrade to 3.4
20:51:49 efried sean-k-mooney: Sorry, no idea. I got around it by sudo pip installing zVMCloudConnector on my system (not the venv)
20:52:04 efried sean-k-mooney: Oh, maybe 3.6 support is a different issue.
20:52:50 sean-k-mooney there is a check for the python 3 version. i symlinked python3 to 3.6 inst normally 3.4
20:52:57 sean-k-mooney on centos anyway.
20:53:12 sean-k-mooney i can just swap the symlink back to 3.4 for now
20:55:07 sean-k-mooney are the gates still running ubuntu 16.04?
20:55:25 sean-k-mooney ubuntu 18.04 also ships 3.6 by defualt i think now
20:57:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
20:57:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739
20:57:15 mriedem we're still testing against 16.04 in the gate for rocky so that's what i use
20:58:15 sean-k-mooney well untill the zvm sdk is drop as a depency or upgraded to support 3.6 we wont be able to move nova gate jobs to 18.04
20:58:44 sean-k-mooney at least given the python3 first goal
20:58:54 mriedem alert the TC!
20:59:30 mriedem that's this thing right? https://github.com/mfcloud/python-zvm-sdk/blob/master/setup.py
20:59:31 efried sean-k-mooney: I wish dansmith or mikal or someone had warned us about problems like this. http://lists.openstack.org/pipermail/openstack-dev/2018-April/129294.html
20:59:57 mriedem https://github.com/mfcloud/python-zvm-sdk/blob/master/setup.py#L39
21:00:13 efried http://lists.openstack.org/pipermail/openstack-dev/2018-April/129350.html
21:00:50 mriedem i guess this is where we'd report a bug that it doesn't work with 3.6 https://bugs.launchpad.net/python-zvm-sdk/+bugs
21:01:00 mriedem and then harass the shit out of jichenjc to get that fixed and released
21:01:29 efried support 3.7 while we're at it, so we don't get behind immediately.
21:01:45 sean-k-mooney mriedem: ya the weird thing is the error message seams to indicate it support 3.5 but only declares 2.7 support in that file
21:01:57 mriedem yeah it definitely supports 3.5 https://github.com/mfcloud/python-zvm-sdk/blob/master/tox.ini#L27
21:02:04 mriedem what blows up with 3.6?
21:02:09 mriedem it just doesn't install/
21:02:10 mriedem ?
21:02:23 sean-k-mooney yep tox fails while pip installing it
21:02:30 mriedem what's the error?
21:02:56 sean-k-mooney one sec
21:04:29 sean-k-mooney http://paste.openstack.org/show/726237/
21:06:04 mriedem bingo https://github.com/mfcloud/python-zvm-sdk/blob/master/zvmsdk/version.py#L28
21:07:02 sean-k-mooney you know if you replace that 5 with a 6 it will proably work
21:07:28 mriedem try it
21:07:55 mriedem or just try doing that and add a tox36 target and run it's unit tests
21:08:08 mriedem this could be your first (and only ever) zvmcloudconnector patch :)
21:09:31 sean-k-mooney some might say its one too many.
21:09:54 sean-k-mooney that said i just downloaded and ran its py27 unitests and one of them still failed
21:09:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
21:09:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739
21:10:28 mriedem let me state one more time, for the record, for this week, that i hate all of the request spec compat code in conductor
21:11:12 sean-k-mooney mriedem: duely noted.
21:11:57 sean-k-mooney i just found another bug in there test code... i am going to go down a rabbit whole with this i can tell
21:12:31 _ix Afternoon folks. I realize this channel is less support oriented, but there's no response in #openstack. I'm having an issue in our pike deployment with resource allocations still existing on one compute, but the vm itself resides on another. It's not reflecting additional usage on the original VM, but the scheduler still won't allow me to instantiate there.
21:12:39 _ix How can I clear the allocation from the original hypervisor?
21:13:05 _ix > There are allocations remaining against the source host that might need to be removed
21:14:21 sean-k-mooney _ix: did you migrate or resize the vm at some point? if you have not completed the migrations then it will have both allocations if i rembere correctly
21:14:39 _ix I did, but the migration failed.
21:15:01 _ix I believe it was the case where a live migration was ongoing, and the targeted compute's services were unavailable or became unavailable.
21:15:24 _ix From a user's perspective, there doesn't seem to be a way to back out of the migration.

Earlier   Later