Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-09
21:20:02 mriedem because this means we'll consume the network-vif-plugged event from pre_live_migration before we call driver.live_migration which does the bw stuff,
21:20:10 mriedem the event won't come for that waiter,
21:20:27 mriedem but the guest transfer is so fast, won't we just finish the operatoin before we ever had a chance to timeout?
21:20:41 mriedem like, do i need a patch that puts a fake sleep in the driver's live migratoin metohd?
21:20:42 mriedem *method
21:21:15 dansmith even at 1MB/s?
21:21:19 mriedem could set the vif_plugging_timeout to like 1 minute, and add a 30 second sleep in the driver
21:21:28 mriedem well,
21:21:39 dansmith should go slower there, but I guess it won't take long enough
21:21:41 mriedem maybe not, but the test will timeout before the 5 minute vif_plugging_timeout i think
21:21:56 dansmith yeah okay so we'll have to force it down I guess
21:22:00 mriedem just wondering if i should set the vif_plugging_timeout to like 1 minute
21:23:15 dansmith 30sec but yeah
21:23:44 jaypipes melwitt, dansmith: regarding https://review.openstack.org/#/c/540258, even if we fix the scheduler/top-level issues around server group affinity and multiple cells, that's still not going to fix the eleventh-hour on-the-compute-node checks that currently run just for affinity groups, though, right? I mean, the computes can't talk cross-cell anyway so there would be no way for those on-compute-node checks to run...
21:23:44 mriedem ok updated; hopefully my zuul fu is strong enough
21:24:58 mriedem jaypipes: yes https://review.openstack.org/#/c/540258/8/nova/scheduler/utils.py@738
21:25:11 mriedem "Also note that we could be racing if we have multiple server create requests for the same affinity group and the scheduler decides to put them each in different cells - the late affinity check in the compute won't resolve that because the upcall check is targeted to the cell the compute is in, and won't see any other hosts for other members in other cells. Separate bug though..."
21:26:42 jaypipes mriedem: ack, ok, just wanted to verify I wasn't crazypants.
21:26:58 mriedem good to have more than just me thinking that
21:27:03 mriedem *that it's an issue..
21:27:13 mriedem not that you're (not?) crazy
21:27:50 dansmith prometheanfire: commented on the bug
21:28:18 melwitt yeah, it makes sense. I was re-thinking about the upcalls described in https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls and it's true that they can only work if you're single cell
21:28:49 melwitt once you're multi-cell/split-MQ I think none of them can work
21:29:09 smcginnis melwitt: Howdy. How are things coming along for the RC?
21:30:11 mriedem ha
21:30:31 dansmith smcginnis: awesome
21:30:34 dansmith they're going awesome
21:30:35 dansmith thanks for asking
21:30:40 melwitt haha ....
21:31:00 smcginnis :P
21:31:00 smcginnis And I know no one here would ever be sarcastic so... that's great!
21:31:42 melwitt smcginnis: we stumbled upon something that we'll need to fix for rc2. but as for rc1, I'm waiting on the RPC version alias patch to land, then will propose the release for rc1
21:32:23 smcginnis melwitt: Cool, sounds good. I know a few others already know they will need to get an RC2, so that's no big deal. Thanks.
21:33:20 mriedem dansmith: so one comment on the bug,
21:33:28 mriedem by default the compute manager won't wait for the event
21:33:35 mriedem the config is false for backward compat
21:33:39 melwitt mriedem: I guess, since we're having a rc2, should I just leave the rpc version alias for then? or get it for rc1?
21:33:55 melwitt smcginnis: we're not alone... :)
21:34:02 dansmith mriedem: oh? I didn't see a config valve
21:34:10 mriedem yes, live_migration_wait_for_vif_plug
21:34:24 mriedem b/c not all network backends send the event for just vif plugging (ODL)
21:34:31 smcginnis melwitt: We can wait a bit. Unless you think it will be several hours yet.
21:35:02 mriedem melwitt: probably fine either way
21:35:31 dansmith mriedem: ah, so we handled that in sahid's patch by looking for the vif_Type
21:35:46 dansmith making people know to opt into proper behavior kidna sucks
21:35:46 melwitt mriedem: thx. probably will just go ahead now and let that be in rc2
21:36:02 melwitt the version alias was recheck at 13:30 so it's got awhile before it will have a chance to merge
21:36:03 mriedem true,
21:36:09 mriedem but ODL shows up as ovs vif_type
21:36:13 dansmith oh
21:36:16 dansmith well, that sucks
21:36:17 mriedem which means we have no idea how to not wait for ODL
21:36:24 mriedem remember the hard reboot wait fiasco?
21:36:28 dansmith well, I'd say let them opt out
21:36:31 dansmith personallt
21:36:34 dansmith but whatever
21:36:42 mriedem that's why i plan on making the option True by default in stein
21:37:06 melwitt smcginnis: the version alias patch was rechecked an hour ago, so it'll be awhile to merge, if the gate doesn't fail on us again. but since we're having a rc2, that patch will be fine to go into rc2, so I can just cut rc1 now
21:37:09 dansmith oh I see, I just read your comment
21:37:11 dansmith okay
21:37:12 mriedem i might have also defaulted to False when i originally thought we'd backport this
21:37:40 smcginnis melwitt: Up to you. I'm fine waiting awhile.
21:37:51 mriedem smcginnis doesn't have to go to china tomorrow
21:37:56 melwitt heh
21:38:23 smcginnis :)
21:40:02 openstackgerrit Dan Smith proposed openstack/nova master: Revert "libvirt: slow live-migration to ensure network is ready" https://review.openstack.org/590538
21:40:09 dansmith that was a super nasty revert, fyi
21:40:15 dansmith so look at it with critical eyes
21:40:33 melwitt mriedem, get out soul crusher #3
21:45:01 melwitt smcginnis: am I to include cycle-highlights in the patch? I see that was done for queens
21:45:08 melwitt *in the release patch
21:46:06 prometheanfire dansmith: thanks
21:48:21 smcginnis melwitt: Ideally, yes. Marketing type folks would love to have that.
21:48:31 smcginnis melwitt: It can be a follow up patch too though.
21:48:31 melwitt ok, I will include them
21:48:36 melwitt thanks
21:56:07 mriedem dansmith: ok done
21:56:17 mriedem the params stuff looks like it made that terrible
21:57:40 dansmith yes, yes it did
21:57:44 dansmith mriedem: you want a reno that says what exactly?
21:58:07 mriedem well we can revert this because the original bug is fixed with the new config option right?
21:58:13 dansmith that bug $orig was solved automatically but because of bug $new you must now enable $conf?
21:58:18 mriedem right
21:58:31 dansmith the original bug was arguably less bad than the current state
21:58:32 dansmith okay
21:59:17 mriedem the chances of anyone even having picked up that fix on stable already and be relying on it are pretty slim, at least for upstream, but you guys sound like you had at least one major customer that needed this
21:59:37 dansmith they did, but it never worked for them.. I think I now know why :)
22:01:15 dansmith mriedem: and are you asking me to actually clean up those tests here or just commenting about later reverts?
22:04:36 melwitt mriedem: rc1 release proposed https://review.openstack.org/590574
22:05:18 melwitt I tried to pick the top highlights, let me know if I should add/remove based on your opinion
22:05:42 mriedem dansmith: just commenting, and that we can clean up that other unused stuff in the later separate revert
22:07:34 openstackgerrit Dan Smith proposed openstack/nova master: Revert "libvirt: slow live-migration to ensure network is ready" https://review.openstack.org/590538
22:08:36 dansmith since we're doing this in rc2, and since melwitt will be up early tomorrow to ping him anyway, I assume we're going to wait for sahid's ack before putting this in?
22:08:47 mriedem melwitt: lgtm
22:09:07 mriedem yeah, also waiting on the recreate in my ci patch
22:09:12 dansmith yeah
22:09:36 melwitt yeah, let's talk to sahid tomorrow
22:15:08 mriedem and,
22:15:20 mriedem just wrapped up my plumbing thing

Earlier   Later