| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-18 | |||
| 17:08:39 | melwitt | thinking about the rebuild destroy_disks_on_failure thing again, I have the same question as cfriesen_ from the old review, it seems like we shouldn't destroy disks on any rebuild failure, shared or not. if rebuild didn't complete, shouldn't the disk remain? | |
| 17:19:20 | mdbooth | sean-k-mooney: I tend to use pdb, tbh | |
| 17:20:08 | sean-k-mooney | mdbooth: well apparent the process is now. install pycharm, turn on gevent suport then rightclick the test file you want and select debug | |
| 17:20:39 | sean-k-mooney | not sure how to doe it with vscode but since pycharm is working im happy | |
| 17:21:22 | mdbooth | sean-k-mooney: Interesting. Never even attempted to get a visual debugger working with Nova tests. | |
| 17:21:51 | sean-k-mooney | mdbooth: if you kickout 1 infinity loop in nova compute it works for the main agent too | |
| 17:33:29 | mriedem | melwitt: sure, but w/o changes to the spawn method, you need to check if there is an active evacuate migration record for the instance to know if you're rebuilding or evacuating | |
| 17:33:38 | mriedem | since the task_state only tells you doing one of those | |
| 17:33:51 | mriedem | *you're | |
| 17:34:35 | mriedem | efried_rollin: i've got the move cases handled for dansmith's is_bfv change, and functional test written to show it works, but of course there are a billion unit tests that blow up for unmocked thigns | |
| 17:36:20 | melwitt | mriedem: but does it matter whether we're rebuilding vs evacuating? I was thinking destroy_disks_on_failure=False if rebuild or evac, both | |
| 17:39:20 | melwitt | checking the task_state makes sense, just not seeing why we need to differentiate between rebuild and evac (though I do understand your point that checking the migration record is needed if we do need to differentiate) | |
| 17:45:25 | openstackgerrit | Merged openstack/nova master: API: add support to abort queued live migration in microversion 2.65 https://review.openstack.org/573136 | |
| 17:45:49 | sean-k-mooney | mriedem: for the antiafinity filter... should this be false based on the comment ? https://github.com/openstack/nova/blob/eb4f65a7951e921b1cd8d05713e144e72f2f254f/nova/scheduler/filters/affinity_filter.py#L99 | |
| 17:49:25 | sean-k-mooney | im steping through your regrssion test in pycharm but i think that is a latent bug. | |
| 17:51:50 | sean-k-mooney | also running the regression under the debugger it also very clear we only ever use the uuid of the first instance in the list. | |
| 18:05:10 | openstackgerrit | Merged openstack/nova master: Address nit in 79dac41fee178dabb547f4d7bc10609630767131 https://review.openstack.org/583212 | |
| 18:07:52 | mriedem | melwitt: ok, idk, i'm really half assing trying to understand that problem atm | |
| 18:08:23 | mriedem | sean-k-mooney: it's a double negative | |
| 18:08:29 | mriedem | "must not return the source as a non-possible destination" | |
| 18:08:40 | mriedem | meaning, must return the source as a possible destination | |
| 18:08:42 | mriedem | for resize to same host | |
| 18:09:11 | sean-k-mooney | right. so we want to allow resize to same host | |
| 18:09:14 | sean-k-mooney | ? | |
| 18:09:14 | mriedem | yes | |
| 18:09:19 | sean-k-mooney | ok | |
| 18:09:19 | mriedem | which is what that was added for | |
| 18:09:23 | mriedem | and there is a related functional test somewhere for it | |
| 18:10:32 | sean-k-mooney | well in the multi create case this cases the second instance to pass on the first host because we use the same request spec for all instances and there for the same uuid | |
| 18:11:08 | sean-k-mooney | so for both instance both hosts pass | |
| 18:11:12 | flwang1 | mriedem: hello, i have a question about this nova/bin/privsep-helper | |
| 18:11:43 | flwang1 | mriedem: could you please help me understand if it's still used? | |
| 18:11:45 | flwang1 | thanks | |
| 18:13:34 | sean-k-mooney | flwang1: as far as i am aware teh privsep-helper is till used to spawn the privesep deamon | |
| 18:13:51 | flwang1 | can anybody help me understand where can I get the privsep-helper? we're using an old version of Nova, and recently after a re packaging, we can't find the file | |
| 18:14:14 | flwang1 | sean-k-mooney: thanks, do you know how is it installed? by os-brick? | |
| 18:15:36 | mriedem | sean-k-mooney: yes that's the bug | |
| 18:16:00 | openstackgerrit | Hongbin Lu proposed openstack/nova master: [DNM] Test neutron change https://review.openstack.org/583687 | |
| 18:16:17 | mriedem | sean-k-mooney: but i'm not sure why my functional test wouldn't recreate that issue if the first weighed host is going to be picked for both instances | |
| 18:17:07 | mriedem | flwang1: i'm not sure what that is, but it sounds like something you'd have to find from the source rpm in the old package | |
| 18:17:43 | sean-k-mooney | mriedem: in the debugger selected_hosts==set('host2','host1') i need to step through the weighing to figure out why however | |
| 18:18:02 | flwang1 | mriedem: yep, i know, i'm trying to figure out where does it come from so that I can include that debian pkg or build it with source code | |
| 18:19:06 | mriedem | flwang1: http://git.openstack.org/cgit/openstack/oslo.privsep/tree/setup.cfg#n40 | |
| 18:19:16 | mriedem | it's a binary in oslo.privsep | |
| 18:19:20 | sean-k-mooney | flwang1: its created by installing oslo privsep https://github.com/openstack/oslo.privsep/blob/master/setup.cfg#L40 | |
| 18:19:25 | mriedem | I WIN | |
| 18:19:32 | sean-k-mooney | :) | |
| 18:19:40 | flwang1 | sean-k-mooney: mriedem: hah, many thanks, folks | |
| 18:19:56 | flwang1 | mriedem: hope you're doing well, man | |
| 18:20:29 | flwang1 | time flys | |
| 18:22:09 | mriedem | huh, wtf, why does unshelve even call populate_retry - unshelve failures aren't rescheduled.... | |
| 18:22:17 | mriedem | flwang1: just fixing bugs from grizzly | |
| 18:22:38 | flwang1 | mriedem: yep | |
| 18:23:12 | flwang1 | sean-k-mooney: mriedem: damn it, we're using a nova version at that moment, there is no oslo.privsep in the world :( | |
| 18:25:39 | sean-k-mooney | flwang1: in that case it should still be using rootwrap | |
| 18:25:50 | sean-k-mooney | flwang1: so you should not need privsep-helper | |
| 18:26:02 | flwang1 | oslo rootwrap? | |
| 18:26:13 | flwang1 | or the rootwrap of nova itself | |
| 18:26:21 | sean-k-mooney | flwang1: also that means your running pre kilo? | |
| 18:26:29 | sean-k-mooney | flwang1: good question | |
| 18:26:55 | flwang1 | sean-k-mooney: a little bit better than kilo | |
| 18:30:03 | sean-k-mooney | flwang1: anyway back to your question nova's rootwapper delegates to oslo | |
| 18:30:49 | flwang1 | but in oslo.rootwrap, i still can't find the privsep-helper command :( | |
| 18:31:17 | flwang1 | sean-k-mooney: and you're right, i can see nova-rootwrap = oslo_rootwrap.cmd:main | |
| 18:31:50 | sean-k-mooney | am what version of nova are you trying to deploy? | |
| 18:33:29 | sean-k-mooney | flwang1: or what error are your actully hitting? you mention os-brick earlier is that throwing an error for you | |
| 18:35:12 | flwang1 | sean-k-mooney: http://paste.openstack.org/show/726222/ | |
| 18:35:36 | flwang1 | in our debian install code, we have something like above | |
| 18:36:05 | flwang1 | and it failed because can't find the privsep-helper under /opt/cat/openstack/nova/bin/ | |
| 18:37:17 | sean-k-mooney | right well for a start line 9 should never execute | |
| 18:38:43 | flwang1 | sean-k-mooney: sorry, what does that mean? | |
| 18:39:37 | sean-k-mooney | line 8 you check if /usr/bin/privsep-helper does not exisits then on line 9 if that is true you check is /usr/bin/privsep-helper a file but line 8 asserts it does not exisit | |
| 18:41:17 | sean-k-mooney | flwang1: so rm /usr/bin/privsep-helper can never run | |
| 18:42:03 | flwang1 | sean-k-mooney: yep, so hence why line 10 will try to copy it from /opt/cat/openstack/nova/bin/ to /usr/bin | |
| 18:42:05 | flwang1 | and failed | |
| 18:43:07 | sean-k-mooney | well as written it will always try to create the symlink. line 9 can be removed without changeing the behavior. | |
| 18:43:15 | sean-k-mooney | but back to your issue | |
| 18:45:50 | sean-k-mooney | nova does not provide privsep-helper ever. it depends on oslo.prevsep to do that. so the nova-compute deb package should depend on oslo.prevsep and the post install of oslo.privsep should copy it to /usr/bin/privsep-helper | |
| 18:47:05 | flwang1 | but depend on the new oslo.privsep for a old nova version is not safe, right? | |
| 18:47:18 | flwang1 | because it may cause some version conflicts | |
| 18:47:40 | sean-k-mooney | old nova did not use privsep it used rootwrapper instead. | |
| 18:48:04 | sean-k-mooney | so unless you have backported privsep support then your nova does not need the privsep-helper | |
| 18:48:04 | flwang1 | ok, i can see your point now. | |
| 18:48:48 | flwang1 | so does that mean, if the nova version doesn't need oslo.privsep, then it probably mean we don't need this privsep-helper? | |
| 18:49:39 | sean-k-mooney | yep unless it uses a lib like os-bricks or os-vif that needit but its not the nova package that should be installing it in that case | |
| 18:50:11 | sean-k-mooney | so basically if the nova you are using dose not use privsep you can proably delete that postint scripts or at least that part of it | |
| 18:51:48 | flwang1 | sean-k-mooney: thanks a lot, that makes my day | |
| 18:52:01 | flwang1 | now it's 6:51AM here :) | |
| 18:52:53 | sean-k-mooney | almost 8PM here. you should proably get some sleep either that or you are an early riser | |
| 18:54:29 | flwang1 | sean-k-mooney: many thanks, mate. I will probably say hi if I can see you at Berlin summit ;) | |
| 19:19:02 | sean-k-mooney | mriedem: im still stepping throught the regression test by the way. we do actully allocate both instance to the first host after weighing | |
| 19:20:01 | sean-k-mooney | mriedem: so before we call self._ensure_sufficent_hosts here https://github.com/openstack/nova/blob/c54428cfe1aab8ef9c83df2f9a2718cae864f862/nova/scheduler/filter_scheduler.py#L245 both instance are scheduled to host1 | |
| 19:30:01 | jlk | Hey all. I'd like some help confirming my investigation. I'm digging into how Nova does a hard reboot of an instance (libvirt, kvm), and from what I can tell, it looks like it relies on a libvirt API call to 'destroy()', which from my reading will by default ASK the OS to shutdown cleanly, with a "reasonable" timeout before SIGKILL. | |
| 19:30:33 | jlk | It doesn't look like there is a way w/ Nova <-> libvirt to skip the SIGTERM and just go directly to SIGKILL | |
| 19:30:42 | jlk | nor can I tell what the "reasonable timeout" is. | |
| 19:48:39 | openstackgerrit | Eric Fried proposed openstack/nova master: Avoid requesting DISK_GB allocation for root_gb on BFV instances https://review.openstack.org/580720 | |
| 19:49:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715 | |
| 19:49:59 | efried | mriedem: https://review.openstack.org/#/c/580720/ needed a UT test - that 'or False' thing apparently matters in a weird way that other non-booleans don't. | |
| 19:50:03 | efried | UT fix* | |
| 19:50:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Func test for improper cn local DISK_GB reporting https://review.openstack.org/583646 | |