Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-01
15:54:22 hansmoleman but no idea how good those are
15:54:29 SpamapS mnaser: those are the 4 regions we have, and we do not have a unified control plane. It's nice that they're isolated from eachother.
15:54:32 mnaser yeah i've heard of kingbird before but it's a bit scary as a concept :p
15:54:40 SpamapS The only thing they share is LDAP.
15:55:00 mnaser yeah, i was thinking share a single keystone, multiple regions in the catalog
15:55:31 mnaser if this works out, hopefully we can share the story, what they're doing with nova is really, really cool.
15:56:11 hansmoleman that would be great
15:56:12 jgwentworth bpoulos: just fyi cert validation is back in a review runway, so be on the lookout for reviews https://etherpad.openstack.org/p/nova-runways-rocky
15:58:01 mnaser jgwentworth: superdan SpamapS thanks for the input
16:02:21 openstackgerrit Lee Yarwood proposed openstack/nova stable/ocata: libvirt: handle DiskNotFound during update_available_resource https://review.openstack.org/571432
16:02:22 openstackgerrit Lee Yarwood proposed openstack/nova stable/ocata: libvirt: Skip fetching the virtual size of block devices https://review.openstack.org/571433
16:03:44 superdan mnaser: since you're around, and since you do upgrades a lot, let me ask...
16:04:03 mnaser i'm always around :p what's up?
16:04:24 superdan mnaser: when you do an upgrade from say pike to queens, do you move everything off of every compute node before you do anything to that node, or do you just stop compute services, upgrade packages, and then re-start services?
16:04:58 mnaser we have the ability to live migrate everything because of ceph, but i've never needed to do that, my thought process is -- services are control plane and they don't affect vms
16:05:04 mnaser so it's always been in place upgrades
16:05:13 superdan mnaser: right okay, that's what I'd expect
16:05:23 jgwentworth it was in place upgrades when I was at yahoo too
16:05:35 mnaser superdan: and that's also a lot of what 99% of deployment tools do as well, afaik
16:06:11 superdan mnaser: if we forced you to migrate every instance in your cloud to roll to rocky, that would be a big deal right?
16:06:33 superdan jgwentworth: yeah, I think it's the only case for very large clouds because it's not feasible to move everything, so that's good data
16:06:44 superdan pretty sure I can say RAX didn't either
16:06:48 jgwentworth yep, that
16:07:13 mnaser superdan: uh, very big, and id hope its <blink> bright red in the release notes
16:07:34 mnaser and also the problem is that it's not possible for some other users which don't have the infrastructure to do migrations
16:07:37 hansmoleman superdan: i want to say huawei public cloud does a lot of live migrations, but Kevin_Zheng would know better than me on their upgrade strategy
16:07:38 jgwentworth yeah, I think penick would have a problem with that too
16:07:41 superdan mnaser: what about when you need to patch hypervisors or kernels? you do full slide puzzle to prevent any instance downtime?
16:08:07 mnaser yes, hypervisor and kernel patching (as well as ceph for customers who we do hyperconverged deployments) all get emptied out first
16:08:22 mnaser reasoning is: hypervisor and kernel touches something that affects the actual running vm, nova-compute does not
16:08:28 superdan mnaser: okay, you never schedule downtime per instance? my cloud provider does it that way
16:08:54 mnaser nope, since we have ceph everywhere, live migrations are easy and function extremely well
16:09:06 mnaser 10g network + ceph = no problems
16:09:19 superdan mnaser: yeah, so what if you didn't have that? I know it's a bit theoretical but,
16:09:35 superdan what if you had to transfer the entire disk and memory across the network for every migration?
16:09:48 hansmoleman we know that would kill NTT..
16:09:49 hansmoleman at least
16:09:50 superdan would that change the calculus?
16:10:02 mnaser yeah, big time, it would make the upgrade a huge pain
16:10:09 superdan ack, okay thanks
16:10:10 mnaser especially in a public cloud, you have to coordinate across so many customers..
16:10:37 hansmoleman is there something specifically up for review or being debated that would require us to move all instances to migrate to rocky?
16:10:57 superdan yeah, the NRP allocation conversion thread
16:11:10 hansmoleman ok i figured
16:11:18 hansmoleman i've been blissfully ignorant on that so far
16:11:25 mnaser ^^ me too
16:11:31 superdan I don't think forcing operators to migrate all instances to convert their allocations to nested is reasonable
16:11:34 mnaser because i don't particularly understand it that much :)
16:11:48 mnaser but maybe i should chime in, because i don't know all that much about the whole story
16:11:51 SpamapS We have 20gbit networking for migrations, and we're not ceph based. We aren't going to do a full migration for kernel/kvm updates, we'll schedule downtime. Reboot time is about 8 minutes.
16:12:12 SpamapS We do AZ's, and make sure to complete one AZ before doing the next.
16:12:14 superdan SpamapS: ah sweet, that's a data point I was looking for
16:12:21 superdan my cloud provider does the same
16:12:33 superdan basically tells me that my instances will reboot on X at Y hour, downtime is minutes
16:12:48 mnaser i think in subjects like this, it would be helpful if operators can get a much more simple question like the one superdan asked rather than having to understand whats going on with nested resource providers
16:12:50 jgwentworth my cloud provider does that too
16:12:50 superdan and lets me do it early if I want
16:13:15 hansmoleman can't we have an allocation transformer tool or something for NRP?
16:13:22 SpamapS And when people are sad because their one pet VM was down for 10 minutes causing their service to be down, we definitely pat them on the back and say "there there". https://vignette.wikia.nocookie.net/glee/images/7/7a/Sheldon-leonard-there-there.gif/revision/latest?cb=20140908045609
16:13:22 superdan mnaser: right, knowing if you're affected by the NRP change, or how, is also something I don't want to mix into the upgrade decision for them either
16:13:34 superdan hansmoleman: that's what I'm saying we owe to the users
16:13:43 superdan SpamapS: ++
16:13:51 superdan oh shit
16:13:54 hansmoleman nova-manage placement heal_allocations --fix-nrp
16:13:55 superdan you woke the bear
16:14:12 hansmoleman --easy-button
16:14:21 superdan hansmoleman: unfortunately, we can't really do it completely outside with a tool I think, because we need info from the compute node
16:14:34 SpamapS superdan: the "lets me do it early" is interesting. So you're saying that if you hard reboot your instance, you don't get a downtime during the window?
16:14:53 hansmoleman superdan: maybe on restart of the compute service then? like we did for ironic instance flavors?
16:15:03 hansmoleman and we'll be doing for legacy bdm attachments
16:15:03 superdan SpamapS: I can opt to take the downtime early, which is just them (cold) migrating me to another node that is already fixed, but on my schedule instead of theirs, yeah
16:15:09 SpamapS AH
16:15:11 superdan hansmoleman: right, that's what I think we need to do
16:15:12 SpamapS that's neat
16:15:19 SpamapS I think I'm going to put that on our todo list.
16:15:33 superdan SpamapS: sometimes I opt fo that so I can check the health ofmy pet immediately, and sometimes I don't care, depending on which instance it is
16:15:41 jgwentworth superdan, hansmoleman: ++ cause that also works with FFU
16:15:57 hansmoleman well,
16:15:58 superdan jgwentworth: well, actually it doesn't
16:16:01 hansmoleman right
16:16:08 mnaser ok so really silly could a rocky nova-compute check if it's running for the first time and do the migrations?
16:16:08 hansmoleman there was that big stink about pci stuff in one of the upgrades
16:16:09 superdan jgwentworth: we have to provide them a way to do it outside too
16:16:12 superdan hansmoleman: right
16:16:19 superdan mnaser: yes
16:16:37 jgwentworth oh, right... depending on where the startup code is during the fast-forward. I see
16:16:55 hansmoleman which is why there was a migration CLI for the ironic flavor stuff as well
16:17:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Trim the fat on HostState.instances https://review.openstack.org/569247
16:17:40 jgwentworth yeah. as long as there's a way we can, just worried about stuff like the PCI thing that required a pause in a fast forward
16:19:52 bpoulos jgwentworth: thank you for the heads up! I'll keep my eye on the cert validation patches
16:20:01 hansmoleman sob, oom with the fake virt driver in devstack after 26 computes out of 100
16:20:28 jgwentworth bpoulos: awesome, thanks
16:20:53 mnaser well it could be an upgrade note where "if you're doing ffu, run this nova-manage thing, if you're doing a normal upgrade, nova will fix things on start"
16:21:39 jgwentworth yeah, it could be that (and is how we've done previous things). provide a way to do an offline batch migration
16:22:39 superdan mnaser: we _have_ to provide that yeah
16:23:02 jgwentworth being able to do offline batch is useful in other ways too, for example the flavor migration from (was it kilo?) where instead of having flavors migrated on-the-fly while things are running, an operator could choose to do them in a batch via nova-manage during off peak time
16:23:09 superdan mnaser: what I want to avoid is you having to do N manual conversions for N compute nodes, when you're not FFUing and upgrading in place
16:23:34 superdan jgwentworth: yeah that was much easier because it didn't require knowledge of resource topologies
16:23:52 mnaser superdan: well, maybe i'm over simplifying things but if there is some function of fix_nested_resource_providers() and that same one can either be called from nova-manage or on start up
16:24:23 superdan mnaser: yeah, totes, I'm saying we should do it automatically if we can, fall back to manual if you are FFUing or want to do it while stuff is offline

Earlier   Later