| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-18 | |||
| 18:08:42 | mriedem | for resize to same host | |
| 18:09:11 | sean-k-mooney | right. so we want to allow resize to same host | |
| 18:09:14 | mriedem | yes | |
| 18:09:14 | sean-k-mooney | ? | |
| 18:09:19 | mriedem | which is what that was added for | |
| 18:09:19 | sean-k-mooney | ok | |
| 18:09:23 | mriedem | and there is a related functional test somewhere for it | |
| 18:10:32 | sean-k-mooney | well in the multi create case this cases the second instance to pass on the first host because we use the same request spec for all instances and there for the same uuid | |
| 18:11:08 | sean-k-mooney | so for both instance both hosts pass | |
| 18:11:12 | flwang1 | mriedem: hello, i have a question about this nova/bin/privsep-helper | |
| 18:11:43 | flwang1 | mriedem: could you please help me understand if it's still used? | |
| 18:11:45 | flwang1 | thanks | |
| 18:13:34 | sean-k-mooney | flwang1: as far as i am aware teh privsep-helper is till used to spawn the privesep deamon | |
| 18:13:51 | flwang1 | can anybody help me understand where can I get the privsep-helper? we're using an old version of Nova, and recently after a re packaging, we can't find the file | |
| 18:14:14 | flwang1 | sean-k-mooney: thanks, do you know how is it installed? by os-brick? | |
| 18:15:36 | mriedem | sean-k-mooney: yes that's the bug | |
| 18:16:00 | openstackgerrit | Hongbin Lu proposed openstack/nova master: [DNM] Test neutron change https://review.openstack.org/583687 | |
| 18:16:17 | mriedem | sean-k-mooney: but i'm not sure why my functional test wouldn't recreate that issue if the first weighed host is going to be picked for both instances | |
| 18:17:07 | mriedem | flwang1: i'm not sure what that is, but it sounds like something you'd have to find from the source rpm in the old package | |
| 18:17:43 | sean-k-mooney | mriedem: in the debugger selected_hosts==set('host2','host1') i need to step through the weighing to figure out why however | |
| 18:18:02 | flwang1 | mriedem: yep, i know, i'm trying to figure out where does it come from so that I can include that debian pkg or build it with source code | |
| 18:19:06 | mriedem | flwang1: http://git.openstack.org/cgit/openstack/oslo.privsep/tree/setup.cfg#n40 | |
| 18:19:16 | mriedem | it's a binary in oslo.privsep | |
| 18:19:20 | sean-k-mooney | flwang1: its created by installing oslo privsep https://github.com/openstack/oslo.privsep/blob/master/setup.cfg#L40 | |
| 18:19:25 | mriedem | I WIN | |
| 18:19:32 | sean-k-mooney | :) | |
| 18:19:40 | flwang1 | sean-k-mooney: mriedem: hah, many thanks, folks | |
| 18:19:56 | flwang1 | mriedem: hope you're doing well, man | |
| 18:20:29 | flwang1 | time flys | |
| 18:22:09 | mriedem | huh, wtf, why does unshelve even call populate_retry - unshelve failures aren't rescheduled.... | |
| 18:22:17 | mriedem | flwang1: just fixing bugs from grizzly | |
| 18:22:38 | flwang1 | mriedem: yep | |
| 18:23:12 | flwang1 | sean-k-mooney: mriedem: damn it, we're using a nova version at that moment, there is no oslo.privsep in the world :( | |
| 18:25:39 | sean-k-mooney | flwang1: in that case it should still be using rootwrap | |
| 18:25:50 | sean-k-mooney | flwang1: so you should not need privsep-helper | |
| 18:26:02 | flwang1 | oslo rootwrap? | |
| 18:26:13 | flwang1 | or the rootwrap of nova itself | |
| 18:26:21 | sean-k-mooney | flwang1: also that means your running pre kilo? | |
| 18:26:29 | sean-k-mooney | flwang1: good question | |
| 18:26:55 | flwang1 | sean-k-mooney: a little bit better than kilo | |
| 18:30:03 | sean-k-mooney | flwang1: anyway back to your question nova's rootwapper delegates to oslo | |
| 18:30:49 | flwang1 | but in oslo.rootwrap, i still can't find the privsep-helper command :( | |
| 18:31:17 | flwang1 | sean-k-mooney: and you're right, i can see nova-rootwrap = oslo_rootwrap.cmd:main | |
| 18:31:50 | sean-k-mooney | am what version of nova are you trying to deploy? | |
| 18:33:29 | sean-k-mooney | flwang1: or what error are your actully hitting? you mention os-brick earlier is that throwing an error for you | |
| 18:35:12 | flwang1 | sean-k-mooney: http://paste.openstack.org/show/726222/ | |
| 18:35:36 | flwang1 | in our debian install code, we have something like above | |
| 18:36:05 | flwang1 | and it failed because can't find the privsep-helper under /opt/cat/openstack/nova/bin/ | |
| 18:37:17 | sean-k-mooney | right well for a start line 9 should never execute | |
| 18:38:43 | flwang1 | sean-k-mooney: sorry, what does that mean? | |
| 18:39:37 | sean-k-mooney | line 8 you check if /usr/bin/privsep-helper does not exisits then on line 9 if that is true you check is /usr/bin/privsep-helper a file but line 8 asserts it does not exisit | |
| 18:41:17 | sean-k-mooney | flwang1: so rm /usr/bin/privsep-helper can never run | |
| 18:42:03 | flwang1 | sean-k-mooney: yep, so hence why line 10 will try to copy it from /opt/cat/openstack/nova/bin/ to /usr/bin | |
| 18:42:05 | flwang1 | and failed | |
| 18:43:07 | sean-k-mooney | well as written it will always try to create the symlink. line 9 can be removed without changeing the behavior. | |
| 18:43:15 | sean-k-mooney | but back to your issue | |
| 18:45:50 | sean-k-mooney | nova does not provide privsep-helper ever. it depends on oslo.prevsep to do that. so the nova-compute deb package should depend on oslo.prevsep and the post install of oslo.privsep should copy it to /usr/bin/privsep-helper | |
| 18:47:05 | flwang1 | but depend on the new oslo.privsep for a old nova version is not safe, right? | |
| 18:47:18 | flwang1 | because it may cause some version conflicts | |
| 18:47:40 | sean-k-mooney | old nova did not use privsep it used rootwrapper instead. | |
| 18:48:04 | flwang1 | ok, i can see your point now. | |
| 18:48:04 | sean-k-mooney | so unless you have backported privsep support then your nova does not need the privsep-helper | |
| 18:48:48 | flwang1 | so does that mean, if the nova version doesn't need oslo.privsep, then it probably mean we don't need this privsep-helper? | |
| 18:49:39 | sean-k-mooney | yep unless it uses a lib like os-bricks or os-vif that needit but its not the nova package that should be installing it in that case | |
| 18:50:11 | sean-k-mooney | so basically if the nova you are using dose not use privsep you can proably delete that postint scripts or at least that part of it | |
| 18:51:48 | flwang1 | sean-k-mooney: thanks a lot, that makes my day | |
| 18:52:01 | flwang1 | now it's 6:51AM here :) | |
| 18:52:53 | sean-k-mooney | almost 8PM here. you should proably get some sleep either that or you are an early riser | |
| 18:54:29 | flwang1 | sean-k-mooney: many thanks, mate. I will probably say hi if I can see you at Berlin summit ;) | |
| 19:19:02 | sean-k-mooney | mriedem: im still stepping throught the regression test by the way. we do actully allocate both instance to the first host after weighing | |
| 19:20:01 | sean-k-mooney | mriedem: so before we call self._ensure_sufficent_hosts here https://github.com/openstack/nova/blob/c54428cfe1aab8ef9c83df2f9a2718cae864f862/nova/scheduler/filter_scheduler.py#L245 both instance are scheduled to host1 | |
| 19:30:01 | jlk | Hey all. I'd like some help confirming my investigation. I'm digging into how Nova does a hard reboot of an instance (libvirt, kvm), and from what I can tell, it looks like it relies on a libvirt API call to 'destroy()', which from my reading will by default ASK the OS to shutdown cleanly, with a "reasonable" timeout before SIGKILL. | |
| 19:30:33 | jlk | It doesn't look like there is a way w/ Nova <-> libvirt to skip the SIGTERM and just go directly to SIGKILL | |
| 19:30:42 | jlk | nor can I tell what the "reasonable timeout" is. | |
| 19:48:39 | openstackgerrit | Eric Fried proposed openstack/nova master: Avoid requesting DISK_GB allocation for root_gb on BFV instances https://review.openstack.org/580720 | |
| 19:49:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715 | |
| 19:49:59 | efried | mriedem: https://review.openstack.org/#/c/580720/ needed a UT test - that 'or False' thing apparently matters in a weird way that other non-booleans don't. | |
| 19:50:03 | efried | UT fix* | |
| 19:50:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Func test for improper cn local DISK_GB reporting https://review.openstack.org/583646 | |
| 19:52:47 | mriedem | by lucifers beard | |
| 19:54:13 | mriedem | oh yeah root can be None | |
| 19:54:15 | mriedem | derp | |
| 19:56:08 | sean-k-mooney | mriedem: is there a way in the funtional tests to override what filters are enabled or turn off retries in the conductor | |
| 19:57:38 | sean-k-mooney | mriedem: actully i think i have found the filters at least | |
| 19:58:21 | mriedem | yes you can configure the filters prior to starting the scheduler service | |
| 19:58:59 | sean-k-mooney | mriedem: im pretty sure you regression test is passing becasue a retry is happening and the second instance it being schduled to an alternate host | |
| 19:59:20 | efried | mriedem: Yeah, but why (oh, why) is it okay to assign a BooleanField other non-boolean values but not None? | |
| 19:59:58 | sean-k-mooney | efried: because python. or in this case proably because ovo | |
| 20:00:58 | efried | Totally ovo. Python loves treating None as False. | |
| 20:02:57 | mriedem | sean-k-mooney: hmm, yeah, could also just set CONF.scheduler.max_attempts=0 | |
| 20:03:02 | mriedem | that would disable retries | |
| 20:03:05 | mriedem | *reschedules | |
| 20:03:33 | sean-k-mooney | ill try both | |
| 20:03:36 | mriedem | also, even if we did reschedule, the affinity check in compute should kick that out b/c the other instance is already on that host | |
| 20:05:09 | sean-k-mooney | thats the thing. the schedule is selecting host1 for both because of the weigher the the scond instance appears to be reschduled to host2 and you end up with anti affinity | |
| 20:06:04 | mriedem | OH | |
| 20:06:13 | mriedem | ok yeah then just set max_attempts=0 and that should kill that | |
| 20:06:42 | mriedem | we don't get the reschedule in devstack b/c we disabled the up-calls to the api db for that late check in the compute | |
| 20:07:05 | mriedem | that's something else you could do is disable the late-check up-call to mimic dsvm setup | |
| 20:07:07 | mriedem | that might be better | |