Posted	Nick	Remark
#openstack-nova - 2018-06-01
15:49:32	jgwentworth	superdan: I think he means for a scatter-gather, yeah?
15:49:38	mnaser	aka if the value of having a unified control plane is enough to take that performance hit
15:49:39	SpamapS	Well, worst latency that has instances in the project you're listing I suppose
15:49:40	superdan	SpamapS: if you spread things evenly across cells then it will, but if you have a tenant per cell (as one example) then you'll get normal performance if you're not in one of theother cells
15:50:00	superdan	jgwentworth: yes
15:50:40	superdan	mnaser: right, so relatively static workload, they just want a unified view of it, even if operations take a while, because they're infrequentish?
15:50:48	SpamapS	I didn't mean to say all operations will suffer that much
15:50:55	mnaser	superdan: according to them, correct
15:51:27	superdan	mnaser: yeah, so if that's the preference, that's cool. we're not as good as we should be on tolerating faults in the remote links right now,
15:51:41	superdan	mnaser: which means you're increasing your chances for things not working if one link is down, but that's on the books to improve
15:52:01	mnaser	i guess that's for them to understand as a limitation if we do take this path
15:52:04	mnaser	maybe good feedback to bring back :p
15:52:21	superdan	at least the story is on the path to improve, but yeah
15:52:55	mnaser	i'm suggesting 4 regions instead (na-east, na-west, europe, asia).. sure its not a unified control plane but still a lot better in terms of surprises we'll run into
15:53:11	superdan	yep, that's a better approach if they can tolerate it
15:54:18	hansmoleman	there are projects meant to abstract the multi-region thing https://wiki.openstack.org/wiki/Kingbird
15:54:22	hansmoleman	but no idea how good those are
15:54:29	SpamapS	mnaser: those are the 4 regions we have, and we do not have a unified control plane. It's nice that they're isolated from eachother.
15:54:32	mnaser	yeah i've heard of kingbird before but it's a bit scary as a concept :p
15:54:40	SpamapS	The only thing they share is LDAP.
15:55:00	mnaser	yeah, i was thinking share a single keystone, multiple regions in the catalog
15:55:31	mnaser	if this works out, hopefully we can share the story, what they're doing with nova is really, really cool.
15:56:11	hansmoleman	that would be great
15:56:12	jgwentworth	bpoulos: just fyi cert validation is back in a review runway, so be on the lookout for reviews https://etherpad.openstack.org/p/nova-runways-rocky
15:58:01	mnaser	jgwentworth: superdan SpamapS thanks for the input
16:02:21	openstackgerrit	Lee Yarwood proposed openstack/nova stable/ocata: libvirt: handle DiskNotFound during update_available_resource https://review.openstack.org/571432
16:02:22	openstackgerrit	Lee Yarwood proposed openstack/nova stable/ocata: libvirt: Skip fetching the virtual size of block devices https://review.openstack.org/571433
16:03:44	superdan	mnaser: since you're around, and since you do upgrades a lot, let me ask...
16:04:03	mnaser	i'm always around :p what's up?
16:04:24	superdan	mnaser: when you do an upgrade from say pike to queens, do you move everything off of every compute node before you do anything to that node, or do you just stop compute services, upgrade packages, and then re-start services?
16:04:58	mnaser	we have the ability to live migrate everything because of ceph, but i've never needed to do that, my thought process is -- services are control plane and they don't affect vms
16:05:04	mnaser	so it's always been in place upgrades
16:05:13	superdan	mnaser: right okay, that's what I'd expect
16:05:23	jgwentworth	it was in place upgrades when I was at yahoo too
16:05:35	mnaser	superdan: and that's also a lot of what 99% of deployment tools do as well, afaik
16:06:11	superdan	mnaser: if we forced you to migrate every instance in your cloud to roll to rocky, that would be a big deal right?
16:06:33	superdan	jgwentworth: yeah, I think it's the only case for very large clouds because it's not feasible to move everything, so that's good data
16:06:44	superdan	pretty sure I can say RAX didn't either
16:06:48	jgwentworth	yep, that
16:07:13	mnaser	superdan: uh, very big, and id hope its <blink> bright red in the release notes
16:07:34	mnaser	and also the problem is that it's not possible for some other users which don't have the infrastructure to do migrations
16:07:37	hansmoleman	superdan: i want to say huawei public cloud does a lot of live migrations, but Kevin_Zheng would know better than me on their upgrade strategy
16:07:38	jgwentworth	yeah, I think penick would have a problem with that too
16:07:41	superdan	mnaser: what about when you need to patch hypervisors or kernels? you do full slide puzzle to prevent any instance downtime?
16:08:07	mnaser	yes, hypervisor and kernel patching (as well as ceph for customers who we do hyperconverged deployments) all get emptied out first
16:08:22	mnaser	reasoning is: hypervisor and kernel touches something that affects the actual running vm, nova-compute does not
16:08:28	superdan	mnaser: okay, you never schedule downtime per instance? my cloud provider does it that way
16:08:54	mnaser	nope, since we have ceph everywhere, live migrations are easy and function extremely well
16:09:06	mnaser	10g network + ceph = no problems
16:09:19	superdan	mnaser: yeah, so what if you didn't have that? I know it's a bit theoretical but,
16:09:35	superdan	what if you had to transfer the entire disk and memory across the network for every migration?
16:09:48	hansmoleman	we know that would kill NTT..
16:09:49	hansmoleman	at least
16:09:50	superdan	would that change the calculus?
16:10:02	mnaser	yeah, big time, it would make the upgrade a huge pain
16:10:09	superdan	ack, okay thanks
16:10:10	mnaser	especially in a public cloud, you have to coordinate across so many customers..
16:10:37	hansmoleman	is there something specifically up for review or being debated that would require us to move all instances to migrate to rocky?
16:10:57	superdan	yeah, the NRP allocation conversion thread
16:11:10	hansmoleman	ok i figured
16:11:18	hansmoleman	i've been blissfully ignorant on that so far
16:11:25	mnaser	^^ me too
16:11:31	superdan	I don't think forcing operators to migrate all instances to convert their allocations to nested is reasonable
16:11:34	mnaser	because i don't particularly understand it that much :)
16:11:48	mnaser	but maybe i should chime in, because i don't know all that much about the whole story
16:11:51	SpamapS	We have 20gbit networking for migrations, and we're not ceph based. We aren't going to do a full migration for kernel/kvm updates, we'll schedule downtime. Reboot time is about 8 minutes.
16:12:12	SpamapS	We do AZ's, and make sure to complete one AZ before doing the next.
16:12:14	superdan	SpamapS: ah sweet, that's a data point I was looking for
16:12:21	superdan	my cloud provider does the same
16:12:33	superdan	basically tells me that my instances will reboot on X at Y hour, downtime is minutes
16:12:48	mnaser	i think in subjects like this, it would be helpful if operators can get a much more simple question like the one superdan asked rather than having to understand whats going on with nested resource providers
16:12:50	jgwentworth	my cloud provider does that too
16:12:50	superdan	and lets me do it early if I want
16:13:15	hansmoleman	can't we have an allocation transformer tool or something for NRP?
16:13:22	SpamapS	And when people are sad because their one pet VM was down for 10 minutes causing their service to be down, we definitely pat them on the back and say "there there". https://vignette.wikia.nocookie.net/glee/images/7/7a/Sheldon-leonard-there-there.gif/revision/latest?cb=20140908045609
16:13:22	superdan	mnaser: right, knowing if you're affected by the NRP change, or how, is also something I don't want to mix into the upgrade decision for them either
16:13:34	superdan	hansmoleman: that's what I'm saying we owe to the users
16:13:43	superdan	SpamapS: ++
16:13:51	superdan	oh shit
16:13:54	hansmoleman	nova-manage placement heal_allocations --fix-nrp
16:13:55	superdan	you woke the bear
16:14:12	hansmoleman	--easy-button
16:14:21	superdan	hansmoleman: unfortunately, we can't really do it completely outside with a tool I think, because we need info from the compute node
16:14:34	SpamapS	superdan: the "lets me do it early" is interesting. So you're saying that if you hard reboot your instance, you don't get a downtime during the window?
16:14:53	hansmoleman	superdan: maybe on restart of the compute service then? like we did for ironic instance flavors?
16:15:03	hansmoleman	and we'll be doing for legacy bdm attachments
16:15:03	superdan	SpamapS: I can opt to take the downtime early, which is just them (cold) migrating me to another node that is already fixed, but on my schedule instead of theirs, yeah
16:15:09	SpamapS	AH
16:15:11	superdan	hansmoleman: right, that's what I think we need to do
16:15:12	SpamapS	that's neat
16:15:19	SpamapS	I think I'm going to put that on our todo list.
16:15:33	superdan	SpamapS: sometimes I opt fo that so I can check the health ofmy pet immediately, and sometimes I don't care, depending on which instance it is
16:15:41	jgwentworth	superdan, hansmoleman: ++ cause that also works with FFU
16:15:57	hansmoleman	well,
16:15:58	superdan	jgwentworth: well, actually it doesn't
16:16:01	hansmoleman	right
16:16:08	mnaser	ok so really silly could a rocky nova-compute check if it's running for the first time and do the migrations?
16:16:08	hansmoleman	there was that big stink about pci stuff in one of the upgrades
16:16:09	superdan	jgwentworth: we have to provide them a way to do it outside too
16:16:12	superdan	hansmoleman: right

Earlier   Later