Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-18
19:30:01 jlk Hey all. I'd like some help confirming my investigation. I'm digging into how Nova does a hard reboot of an instance (libvirt, kvm), and from what I can tell, it looks like it relies on a libvirt API call to 'destroy()', which from my reading will by default ASK the OS to shutdown cleanly, with a "reasonable" timeout before SIGKILL.
19:30:33 jlk It doesn't look like there is a way w/ Nova <-> libvirt to skip the SIGTERM and just go directly to SIGKILL
19:30:42 jlk nor can I tell what the "reasonable timeout" is.
19:48:39 openstackgerrit Eric Fried proposed openstack/nova master: Avoid requesting DISK_GB allocation for root_gb on BFV instances https://review.openstack.org/580720
19:49:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
19:49:59 efried mriedem: https://review.openstack.org/#/c/580720/ needed a UT test - that 'or False' thing apparently matters in a weird way that other non-booleans don't.
19:50:03 efried UT fix*
19:50:22 openstackgerrit Eric Fried proposed openstack/nova master: Func test for improper cn local DISK_GB reporting https://review.openstack.org/583646
19:52:47 mriedem by lucifers beard
19:54:13 mriedem oh yeah root can be None
19:54:15 mriedem derp
19:56:08 sean-k-mooney mriedem: is there a way in the funtional tests to override what filters are enabled or turn off retries in the conductor
19:57:38 sean-k-mooney mriedem: actully i think i have found the filters at least
19:58:21 mriedem yes you can configure the filters prior to starting the scheduler service
19:58:59 sean-k-mooney mriedem: im pretty sure you regression test is passing becasue a retry is happening and the second instance it being schduled to an alternate host
19:59:20 efried mriedem: Yeah, but why (oh, why) is it okay to assign a BooleanField other non-boolean values but not None?
19:59:58 sean-k-mooney efried: because python. or in this case proably because ovo
20:00:58 efried Totally ovo. Python loves treating None as False.
20:02:57 mriedem sean-k-mooney: hmm, yeah, could also just set CONF.scheduler.max_attempts=0
20:03:02 mriedem that would disable retries
20:03:05 mriedem *reschedules
20:03:33 sean-k-mooney ill try both
20:03:36 mriedem also, even if we did reschedule, the affinity check in compute should kick that out b/c the other instance is already on that host
20:05:09 sean-k-mooney thats the thing. the schedule is selecting host1 for both because of the weigher the the scond instance appears to be reschduled to host2 and you end up with anti affinity
20:06:04 mriedem OH
20:06:13 mriedem ok yeah then just set max_attempts=0 and that should kill that
20:06:42 mriedem we don't get the reschedule in devstack b/c we disabled the up-calls to the api db for that late check in the compute
20:07:05 mriedem that's something else you could do is disable the late-check up-call to mimic dsvm setup
20:07:07 mriedem that might be better
20:07:22 mriedem because honestly we shouldn't get alternatives for either of these instances in a 2 host setup
20:07:26 mriedem since there would be no alternatives
20:07:56 sean-k-mooney setting the max seams quicker to test. if that cases it to fail i can try disableing the up call
20:11:34 jlk oh wow, hrm. Testing out domain.destroy() seems to NOT send a soft shutdown signal. INTERESTING.
20:12:47 sean-k-mooney mriedem: yep adding self.flags(max_attempts=0, group='scheduler') makes the regression fail
20:16:04 mriedem sean-k-mooney: sweet, can you try with disabling the up-calls?
20:16:26 mriedem sean-k-mooney: https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_group_policy_check_upcall
20:16:39 sean-k-mooney actully it failed because max attempts must be >= 1
20:16:44 sean-k-mooney testing again
20:16:56 mriedem then if that introduces the regression you could fix up the test and push that up,
20:17:01 mriedem then i could rebase the bug fix on top of it
20:20:26 sean-k-mooney ok with it set to max_attempts=1 i get Build of instance cbf4b2dd-8301-4d62-97f5-d74b00b21d64 was re-scheduled: Anti-affinity instance group policy was violated.
20:20:36 sean-k-mooney so ya ill try disableing that check now
20:20:37 mriedem yup as expected
20:21:50 mriedem cfriesen_: re: your reply in the ML thread about this bug, i think we can do both fixes actually for this same thing - because we ran into it this time in the antiaffinity filter, but the fact we pick the first request spec and send that to the scheduler with a list of other instance uuids and send that reqspec into each filter with what could be the wrong instance_uuid is still a thing worth fixing,
20:21:55 sean-k-mooney what config group do i set disable_group_policy_check_upcall in? is workarounds its own group
20:22:00 mriedem and num_instances being set per-scheduler request is something else we could fix
20:22:06 mriedem sean-k-mooney: [workarounds]
20:22:08 mriedem yes
20:22:26 mriedem honestly num_instances shouldn't even be needed anymore, but it predates sending the list of instance uuids to select_destinations
20:22:54 mriedem we kind of sorely need to drop scheduler rpcapi 4.x compatibility and move to 5.0
20:23:09 mriedem so we can drop a lot of this garbage compat code in the scheduler
20:24:27 sean-k-mooney is that something we can do in stien. im guessing we need 4.x to live as long as cellsv1 or something else is still around?
20:24:49 mriedem it's usually sometihng we do before we cut a release,
20:24:58 mriedem we add a compat shim and then drop the shim once we are in n+1 master
20:25:13 mriedem dansmith is the one that's always done it
20:25:28 sean-k-mooney ah. ok so with disableing the upcall i new get to your assertion and it fails with 2!=1
20:25:37 mriedem sean-k-mooney: awesome
20:25:51 mriedem want to fix the assertion and add a FIXME note in there about cleaning up the assertion once the bug is fixed?
20:26:00 mriedem probably need to update the docstring in the test also
20:26:30 sean-k-mooney sure will do.
20:26:43 sean-k-mooney this would have been a nightmare to find without a debugger
20:27:12 mriedem if i would have been thinking clearly about the devstack env, i would have eventually realized that we might be hitting a reschedule which devstack doesn't
20:27:18 mriedem but, brain is mush
20:27:32 mriedem i would realize it at 3am most likely
20:44:48 openstackgerrit sean mooney proposed openstack/nova master: Add regression test for bug 1781710 https://review.openstack.org/583339
20:44:50 openstack bug 1781710 in OpenStack Compute (nova) "ServersOnMultiNodesTest.test_create_server_with_scheduler_hint_group_anti_affinity failing with "Servers are on the same host"" [High,Fix released] https://launchpad.net/bugs/1781710 - Assigned to Matt Riedemann (mriedem)
20:47:34 sean-k-mooney by the way have people hit this.
20:47:36 sean-k-mooney RuntimeError: On Python 3, zvm sdk supports to Python 3.5
20:48:21 sean-k-mooney im currently on centos 7.5 and i install python 3.6. we dont support 3.4 upstream right
20:49:16 sean-k-mooney centos only ship 3.4 and 3.6 in the package manager
20:50:33 sean-k-mooney efried: do you know what are the changes of getting 3.6 support in the zvm sdk
20:50:59 efried sean-k-mooney: You mean so you can tox with the effin thing?
20:51:12 sean-k-mooney yes
20:51:29 sean-k-mooney right now i cand run pep8 or any of the python3 based envs
20:51:46 sean-k-mooney unless in install 3.5 that is or downgrade to 3.4
20:51:49 efried sean-k-mooney: Sorry, no idea. I got around it by sudo pip installing zVMCloudConnector on my system (not the venv)
20:52:04 efried sean-k-mooney: Oh, maybe 3.6 support is a different issue.
20:52:50 sean-k-mooney there is a check for the python 3 version. i symlinked python3 to 3.6 inst normally 3.4
20:52:57 sean-k-mooney on centos anyway.
20:53:12 sean-k-mooney i can just swap the symlink back to 3.4 for now
20:55:07 sean-k-mooney are the gates still running ubuntu 16.04?
20:55:25 sean-k-mooney ubuntu 18.04 also ships 3.6 by defualt i think now
20:57:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
20:57:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739
20:57:15 mriedem we're still testing against 16.04 in the gate for rocky so that's what i use
20:58:15 sean-k-mooney well untill the zvm sdk is drop as a depency or upgraded to support 3.6 we wont be able to move nova gate jobs to 18.04
20:58:44 sean-k-mooney at least given the python3 first goal
20:58:54 mriedem alert the TC!
20:59:30 mriedem that's this thing right? https://github.com/mfcloud/python-zvm-sdk/blob/master/setup.py
20:59:31 efried sean-k-mooney: I wish dansmith or mikal or someone had warned us about problems like this. http://lists.openstack.org/pipermail/openstack-dev/2018-April/129294.html
20:59:57 mriedem https://github.com/mfcloud/python-zvm-sdk/blob/master/setup.py#L39
21:00:13 efried http://lists.openstack.org/pipermail/openstack-dev/2018-April/129350.html
21:00:50 mriedem i guess this is where we'd report a bug that it doesn't work with 3.6 https://bugs.launchpad.net/python-zvm-sdk/+bugs
21:01:00 mriedem and then harass the shit out of jichenjc to get that fixed and released
21:01:29 efried support 3.7 while we're at it, so we don't get behind immediately.
21:01:45 sean-k-mooney mriedem: ya the weird thing is the error message seams to indicate it support 3.5 but only declares 2.7 support in that file
21:01:57 mriedem yeah it definitely supports 3.5 https://github.com/mfcloud/python-zvm-sdk/blob/master/tox.ini#L27
21:02:04 mriedem what blows up with 3.6?
21:02:09 mriedem it just doesn't install/
21:02:10 mriedem ?

Earlier   Later