Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-18
20:20:26 sean-k-mooney ok with it set to max_attempts=1 i get Build of instance cbf4b2dd-8301-4d62-97f5-d74b00b21d64 was re-scheduled: Anti-affinity instance group policy was violated.
20:20:36 sean-k-mooney so ya ill try disableing that check now
20:20:37 mriedem yup as expected
20:21:50 mriedem cfriesen_: re: your reply in the ML thread about this bug, i think we can do both fixes actually for this same thing - because we ran into it this time in the antiaffinity filter, but the fact we pick the first request spec and send that to the scheduler with a list of other instance uuids and send that reqspec into each filter with what could be the wrong instance_uuid is still a thing worth fixing,
20:21:55 sean-k-mooney what config group do i set disable_group_policy_check_upcall in? is workarounds its own group
20:22:00 mriedem and num_instances being set per-scheduler request is something else we could fix
20:22:06 mriedem sean-k-mooney: [workarounds]
20:22:08 mriedem yes
20:22:26 mriedem honestly num_instances shouldn't even be needed anymore, but it predates sending the list of instance uuids to select_destinations
20:22:54 mriedem we kind of sorely need to drop scheduler rpcapi 4.x compatibility and move to 5.0
20:23:09 mriedem so we can drop a lot of this garbage compat code in the scheduler
20:24:27 sean-k-mooney is that something we can do in stien. im guessing we need 4.x to live as long as cellsv1 or something else is still around?
20:24:49 mriedem it's usually sometihng we do before we cut a release,
20:24:58 mriedem we add a compat shim and then drop the shim once we are in n+1 master
20:25:13 mriedem dansmith is the one that's always done it
20:25:28 sean-k-mooney ah. ok so with disableing the upcall i new get to your assertion and it fails with 2!=1
20:25:37 mriedem sean-k-mooney: awesome
20:25:51 mriedem want to fix the assertion and add a FIXME note in there about cleaning up the assertion once the bug is fixed?
20:26:00 mriedem probably need to update the docstring in the test also
20:26:30 sean-k-mooney sure will do.
20:26:43 sean-k-mooney this would have been a nightmare to find without a debugger
20:27:12 mriedem if i would have been thinking clearly about the devstack env, i would have eventually realized that we might be hitting a reschedule which devstack doesn't
20:27:18 mriedem but, brain is mush
20:27:32 mriedem i would realize it at 3am most likely
20:44:48 openstackgerrit sean mooney proposed openstack/nova master: Add regression test for bug 1781710 https://review.openstack.org/583339
20:44:50 openstack bug 1781710 in OpenStack Compute (nova) "ServersOnMultiNodesTest.test_create_server_with_scheduler_hint_group_anti_affinity failing with "Servers are on the same host"" [High,Fix released] https://launchpad.net/bugs/1781710 - Assigned to Matt Riedemann (mriedem)
20:47:34 sean-k-mooney by the way have people hit this.
20:47:36 sean-k-mooney RuntimeError: On Python 3, zvm sdk supports to Python 3.5
20:48:21 sean-k-mooney im currently on centos 7.5 and i install python 3.6. we dont support 3.4 upstream right
20:49:16 sean-k-mooney centos only ship 3.4 and 3.6 in the package manager
20:50:33 sean-k-mooney efried: do you know what are the changes of getting 3.6 support in the zvm sdk
20:50:59 efried sean-k-mooney: You mean so you can tox with the effin thing?
20:51:12 sean-k-mooney yes
20:51:29 sean-k-mooney right now i cand run pep8 or any of the python3 based envs
20:51:46 sean-k-mooney unless in install 3.5 that is or downgrade to 3.4
20:51:49 efried sean-k-mooney: Sorry, no idea. I got around it by sudo pip installing zVMCloudConnector on my system (not the venv)
20:52:04 efried sean-k-mooney: Oh, maybe 3.6 support is a different issue.
20:52:50 sean-k-mooney there is a check for the python 3 version. i symlinked python3 to 3.6 inst normally 3.4
20:52:57 sean-k-mooney on centos anyway.
20:53:12 sean-k-mooney i can just swap the symlink back to 3.4 for now
20:55:07 sean-k-mooney are the gates still running ubuntu 16.04?
20:55:25 sean-k-mooney ubuntu 18.04 also ships 3.6 by defualt i think now
20:57:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
20:57:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739
20:57:15 mriedem we're still testing against 16.04 in the gate for rocky so that's what i use
20:58:15 sean-k-mooney well untill the zvm sdk is drop as a depency or upgraded to support 3.6 we wont be able to move nova gate jobs to 18.04
20:58:44 sean-k-mooney at least given the python3 first goal
20:58:54 mriedem alert the TC!
20:59:30 mriedem that's this thing right? https://github.com/mfcloud/python-zvm-sdk/blob/master/setup.py
20:59:31 efried sean-k-mooney: I wish dansmith or mikal or someone had warned us about problems like this. http://lists.openstack.org/pipermail/openstack-dev/2018-April/129294.html
20:59:57 mriedem https://github.com/mfcloud/python-zvm-sdk/blob/master/setup.py#L39
21:00:13 efried http://lists.openstack.org/pipermail/openstack-dev/2018-April/129350.html
21:00:50 mriedem i guess this is where we'd report a bug that it doesn't work with 3.6 https://bugs.launchpad.net/python-zvm-sdk/+bugs
21:01:00 mriedem and then harass the shit out of jichenjc to get that fixed and released
21:01:29 efried support 3.7 while we're at it, so we don't get behind immediately.
21:01:45 sean-k-mooney mriedem: ya the weird thing is the error message seams to indicate it support 3.5 but only declares 2.7 support in that file
21:01:57 mriedem yeah it definitely supports 3.5 https://github.com/mfcloud/python-zvm-sdk/blob/master/tox.ini#L27
21:02:04 mriedem what blows up with 3.6?
21:02:09 mriedem it just doesn't install/
21:02:10 mriedem ?
21:02:23 sean-k-mooney yep tox fails while pip installing it
21:02:30 mriedem what's the error?
21:02:56 sean-k-mooney one sec
21:04:29 sean-k-mooney http://paste.openstack.org/show/726237/
21:06:04 mriedem bingo https://github.com/mfcloud/python-zvm-sdk/blob/master/zvmsdk/version.py#L28
21:07:02 sean-k-mooney you know if you replace that 5 with a 6 it will proably work
21:07:28 mriedem try it
21:07:55 mriedem or just try doing that and add a tox36 target and run it's unit tests
21:08:08 mriedem this could be your first (and only ever) zvmcloudconnector patch :)
21:09:31 sean-k-mooney some might say its one too many.
21:09:54 sean-k-mooney that said i just downloaded and ran its py27 unitests and one of them still failed
21:09:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
21:09:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739
21:10:28 mriedem let me state one more time, for the record, for this week, that i hate all of the request spec compat code in conductor
21:11:12 sean-k-mooney mriedem: duely noted.
21:11:57 sean-k-mooney i just found another bug in there test code... i am going to go down a rabbit whole with this i can tell
21:12:31 _ix Afternoon folks. I realize this channel is less support oriented, but there's no response in #openstack. I'm having an issue in our pike deployment with resource allocations still existing on one compute, but the vm itself resides on another. It's not reflecting additional usage on the original VM, but the scheduler still won't allow me to instantiate there.
21:12:39 _ix How can I clear the allocation from the original hypervisor?
21:13:05 _ix > There are allocations remaining against the source host that might need to be removed
21:14:21 sean-k-mooney _ix: did you migrate or resize the vm at some point? if you have not completed the migrations then it will have both allocations if i rembere correctly
21:14:39 _ix I did, but the migration failed.
21:15:01 _ix I believe it was the case where a live migration was ongoing, and the targeted compute's services were unavailable or became unavailable.
21:15:24 _ix From a user's perspective, there doesn't seem to be a way to back out of the migration.
21:15:40 sean-k-mooney _ix so the vm migrated but the migration timeded out becase the node became unableale?
21:16:23 _ix The vm stayed on the source but received allocation on the target.
21:16:24 sean-k-mooney _ix: well migration is an admin action so "end users" should not be aware of them in general
21:16:33 _ix That's a fair point.
21:17:10 sean-k-mooney mriedem: jaypipes efried do you know how to clean up the stale allocation?
21:17:29 _ix How can I force the issue? I've manually added allocations records to reflect reality in the nova_api.allocations table and marked the failed migration as deleted in nova.migrations.
21:17:51 efried sean-k-mooney: I haven't been following the conversation. You want to manually remove an allocation from placement?
21:18:02 _ix Yes please.
21:18:15 sean-k-mooney efried: live migration failed but allocation on dest remain
21:18:32 efried mriedem is the osc-placement expert. I can only tell you how to do it programmatically :)
21:19:48 openstack Launchpad bug 1753676 in OpenStack Compute (nova) "Live migration not working as Expected when Restarting nova-compute service while migration from source node" [Undecided,Confirmed]
21:19:48 _ix It's an open bug, but it's such an edge case, I totally understand why this might lead to something out-of-state: https://bugs.launchpad.net/nova/+bug/1753676
21:20:23 _ix I'm pleased to report live migrations tend to work quite well and fast, I might add.
21:20:34 sean-k-mooney _ix: ya migration in pike was kind of buggy. dansmith was working on fixing that i think
21:20:56 mriedem yeah osc-placement
21:21:16 mriedem https://docs.openstack.org/osc-placement/latest/index.html
21:21:29 mriedem https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-set specifically

Earlier   Later