| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-18 | |||
| 18:25:39 | sean-k-mooney | flwang1: in that case it should still be using rootwrap | |
| 18:25:50 | sean-k-mooney | flwang1: so you should not need privsep-helper | |
| 18:26:02 | flwang1 | oslo rootwrap? | |
| 18:26:13 | flwang1 | or the rootwrap of nova itself | |
| 18:26:21 | sean-k-mooney | flwang1: also that means your running pre kilo? | |
| 18:26:29 | sean-k-mooney | flwang1: good question | |
| 18:26:55 | flwang1 | sean-k-mooney: a little bit better than kilo | |
| 18:30:03 | sean-k-mooney | flwang1: anyway back to your question nova's rootwapper delegates to oslo | |
| 18:30:49 | flwang1 | but in oslo.rootwrap, i still can't find the privsep-helper command :( | |
| 18:31:17 | flwang1 | sean-k-mooney: and you're right, i can see nova-rootwrap = oslo_rootwrap.cmd:main | |
| 18:31:50 | sean-k-mooney | am what version of nova are you trying to deploy? | |
| 18:33:29 | sean-k-mooney | flwang1: or what error are your actully hitting? you mention os-brick earlier is that throwing an error for you | |
| 18:35:12 | flwang1 | sean-k-mooney: http://paste.openstack.org/show/726222/ | |
| 18:35:36 | flwang1 | in our debian install code, we have something like above | |
| 18:36:05 | flwang1 | and it failed because can't find the privsep-helper under /opt/cat/openstack/nova/bin/ | |
| 18:37:17 | sean-k-mooney | right well for a start line 9 should never execute | |
| 18:38:43 | flwang1 | sean-k-mooney: sorry, what does that mean? | |
| 18:39:37 | sean-k-mooney | line 8 you check if /usr/bin/privsep-helper does not exisits then on line 9 if that is true you check is /usr/bin/privsep-helper a file but line 8 asserts it does not exisit | |
| 18:41:17 | sean-k-mooney | flwang1: so rm /usr/bin/privsep-helper can never run | |
| 18:42:03 | flwang1 | sean-k-mooney: yep, so hence why line 10 will try to copy it from /opt/cat/openstack/nova/bin/ to /usr/bin | |
| 18:42:05 | flwang1 | and failed | |
| 18:43:07 | sean-k-mooney | well as written it will always try to create the symlink. line 9 can be removed without changeing the behavior. | |
| 18:43:15 | sean-k-mooney | but back to your issue | |
| 18:45:50 | sean-k-mooney | nova does not provide privsep-helper ever. it depends on oslo.prevsep to do that. so the nova-compute deb package should depend on oslo.prevsep and the post install of oslo.privsep should copy it to /usr/bin/privsep-helper | |
| 18:47:05 | flwang1 | but depend on the new oslo.privsep for a old nova version is not safe, right? | |
| 18:47:18 | flwang1 | because it may cause some version conflicts | |
| 18:47:40 | sean-k-mooney | old nova did not use privsep it used rootwrapper instead. | |
| 18:48:04 | flwang1 | ok, i can see your point now. | |
| 18:48:04 | sean-k-mooney | so unless you have backported privsep support then your nova does not need the privsep-helper | |
| 18:48:48 | flwang1 | so does that mean, if the nova version doesn't need oslo.privsep, then it probably mean we don't need this privsep-helper? | |
| 18:49:39 | sean-k-mooney | yep unless it uses a lib like os-bricks or os-vif that needit but its not the nova package that should be installing it in that case | |
| 18:50:11 | sean-k-mooney | so basically if the nova you are using dose not use privsep you can proably delete that postint scripts or at least that part of it | |
| 18:51:48 | flwang1 | sean-k-mooney: thanks a lot, that makes my day | |
| 18:52:01 | flwang1 | now it's 6:51AM here :) | |
| 18:52:53 | sean-k-mooney | almost 8PM here. you should proably get some sleep either that or you are an early riser | |
| 18:54:29 | flwang1 | sean-k-mooney: many thanks, mate. I will probably say hi if I can see you at Berlin summit ;) | |
| 19:19:02 | sean-k-mooney | mriedem: im still stepping throught the regression test by the way. we do actully allocate both instance to the first host after weighing | |
| 19:20:01 | sean-k-mooney | mriedem: so before we call self._ensure_sufficent_hosts here https://github.com/openstack/nova/blob/c54428cfe1aab8ef9c83df2f9a2718cae864f862/nova/scheduler/filter_scheduler.py#L245 both instance are scheduled to host1 | |
| 19:30:01 | jlk | Hey all. I'd like some help confirming my investigation. I'm digging into how Nova does a hard reboot of an instance (libvirt, kvm), and from what I can tell, it looks like it relies on a libvirt API call to 'destroy()', which from my reading will by default ASK the OS to shutdown cleanly, with a "reasonable" timeout before SIGKILL. | |
| 19:30:33 | jlk | It doesn't look like there is a way w/ Nova <-> libvirt to skip the SIGTERM and just go directly to SIGKILL | |
| 19:30:42 | jlk | nor can I tell what the "reasonable timeout" is. | |
| 19:48:39 | openstackgerrit | Eric Fried proposed openstack/nova master: Avoid requesting DISK_GB allocation for root_gb on BFV instances https://review.openstack.org/580720 | |
| 19:49:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715 | |
| 19:49:59 | efried | mriedem: https://review.openstack.org/#/c/580720/ needed a UT test - that 'or False' thing apparently matters in a weird way that other non-booleans don't. | |
| 19:50:03 | efried | UT fix* | |
| 19:50:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Func test for improper cn local DISK_GB reporting https://review.openstack.org/583646 | |
| 19:52:47 | mriedem | by lucifers beard | |
| 19:54:13 | mriedem | oh yeah root can be None | |
| 19:54:15 | mriedem | derp | |
| 19:56:08 | sean-k-mooney | mriedem: is there a way in the funtional tests to override what filters are enabled or turn off retries in the conductor | |
| 19:57:38 | sean-k-mooney | mriedem: actully i think i have found the filters at least | |
| 19:58:21 | mriedem | yes you can configure the filters prior to starting the scheduler service | |
| 19:58:59 | sean-k-mooney | mriedem: im pretty sure you regression test is passing becasue a retry is happening and the second instance it being schduled to an alternate host | |
| 19:59:20 | efried | mriedem: Yeah, but why (oh, why) is it okay to assign a BooleanField other non-boolean values but not None? | |
| 19:59:58 | sean-k-mooney | efried: because python. or in this case proably because ovo | |
| 20:00:58 | efried | Totally ovo. Python loves treating None as False. | |
| 20:02:57 | mriedem | sean-k-mooney: hmm, yeah, could also just set CONF.scheduler.max_attempts=0 | |
| 20:03:02 | mriedem | that would disable retries | |
| 20:03:05 | mriedem | *reschedules | |
| 20:03:33 | sean-k-mooney | ill try both | |
| 20:03:36 | mriedem | also, even if we did reschedule, the affinity check in compute should kick that out b/c the other instance is already on that host | |
| 20:05:09 | sean-k-mooney | thats the thing. the schedule is selecting host1 for both because of the weigher the the scond instance appears to be reschduled to host2 and you end up with anti affinity | |
| 20:06:04 | mriedem | OH | |
| 20:06:13 | mriedem | ok yeah then just set max_attempts=0 and that should kill that | |
| 20:06:42 | mriedem | we don't get the reschedule in devstack b/c we disabled the up-calls to the api db for that late check in the compute | |
| 20:07:05 | mriedem | that's something else you could do is disable the late-check up-call to mimic dsvm setup | |
| 20:07:07 | mriedem | that might be better | |
| 20:07:22 | mriedem | because honestly we shouldn't get alternatives for either of these instances in a 2 host setup | |
| 20:07:26 | mriedem | since there would be no alternatives | |
| 20:07:56 | sean-k-mooney | setting the max seams quicker to test. if that cases it to fail i can try disableing the up call | |
| 20:11:34 | jlk | oh wow, hrm. Testing out domain.destroy() seems to NOT send a soft shutdown signal. INTERESTING. | |
| 20:12:47 | sean-k-mooney | mriedem: yep adding self.flags(max_attempts=0, group='scheduler') makes the regression fail | |
| 20:16:04 | mriedem | sean-k-mooney: sweet, can you try with disabling the up-calls? | |
| 20:16:26 | mriedem | sean-k-mooney: https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_group_policy_check_upcall | |
| 20:16:39 | sean-k-mooney | actully it failed because max attempts must be >= 1 | |
| 20:16:44 | sean-k-mooney | testing again | |
| 20:16:56 | mriedem | then if that introduces the regression you could fix up the test and push that up, | |
| 20:17:01 | mriedem | then i could rebase the bug fix on top of it | |
| 20:20:26 | sean-k-mooney | ok with it set to max_attempts=1 i get Build of instance cbf4b2dd-8301-4d62-97f5-d74b00b21d64 was re-scheduled: Anti-affinity instance group policy was violated. | |
| 20:20:36 | sean-k-mooney | so ya ill try disableing that check now | |
| 20:20:37 | mriedem | yup as expected | |
| 20:21:50 | mriedem | cfriesen_: re: your reply in the ML thread about this bug, i think we can do both fixes actually for this same thing - because we ran into it this time in the antiaffinity filter, but the fact we pick the first request spec and send that to the scheduler with a list of other instance uuids and send that reqspec into each filter with what could be the wrong instance_uuid is still a thing worth fixing, | |
| 20:21:55 | sean-k-mooney | what config group do i set disable_group_policy_check_upcall in? is workarounds its own group | |
| 20:22:00 | mriedem | and num_instances being set per-scheduler request is something else we could fix | |
| 20:22:06 | mriedem | sean-k-mooney: [workarounds] | |
| 20:22:08 | mriedem | yes | |
| 20:22:26 | mriedem | honestly num_instances shouldn't even be needed anymore, but it predates sending the list of instance uuids to select_destinations | |
| 20:22:54 | mriedem | we kind of sorely need to drop scheduler rpcapi 4.x compatibility and move to 5.0 | |
| 20:23:09 | mriedem | so we can drop a lot of this garbage compat code in the scheduler | |
| 20:24:27 | sean-k-mooney | is that something we can do in stien. im guessing we need 4.x to live as long as cellsv1 or something else is still around? | |
| 20:24:49 | mriedem | it's usually sometihng we do before we cut a release, | |
| 20:24:58 | mriedem | we add a compat shim and then drop the shim once we are in n+1 master | |
| 20:25:13 | mriedem | dansmith is the one that's always done it | |
| 20:25:28 | sean-k-mooney | ah. ok so with disableing the upcall i new get to your assertion and it fails with 2!=1 | |
| 20:25:37 | mriedem | sean-k-mooney: awesome | |
| 20:25:51 | mriedem | want to fix the assertion and add a FIXME note in there about cleaning up the assertion once the bug is fixed? | |
| 20:26:00 | mriedem | probably need to update the docstring in the test also | |
| 20:26:30 | sean-k-mooney | sure will do. | |
| 20:26:43 | sean-k-mooney | this would have been a nightmare to find without a debugger | |
| 20:27:12 | mriedem | if i would have been thinking clearly about the devstack env, i would have eventually realized that we might be hitting a reschedule which devstack doesn't | |