| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-01 | |||
| 15:48:47 | jgwentworth | mnaser: the api connects directly to cell databases, so that scenario would be slow | |
| 15:48:51 | superdan | mnaser: if unified control plane is what you care about the most, then... | |
| 15:48:54 | superdan | SpamapS: that's not true | |
| 15:49:20 | mnaser | so the thing is, api operations isn't a big deal for them, but i don't know how much slower they'd be | |
| 15:49:32 | jgwentworth | superdan: I think he means for a scatter-gather, yeah? | |
| 15:49:38 | mnaser | aka if the value of having a unified control plane is enough to take that performance hit | |
| 15:49:39 | SpamapS | Well, worst latency that has instances in the project you're listing I suppose | |
| 15:49:40 | superdan | SpamapS: if you spread things evenly across cells then it will, but if you have a tenant per cell (as one example) then you'll get normal performance if you're not in one of theother cells | |
| 15:50:00 | superdan | jgwentworth: yes | |
| 15:50:40 | superdan | mnaser: right, so relatively static workload, they just want a unified view of it, even if operations take a while, because they're infrequentish? | |
| 15:50:48 | SpamapS | I didn't mean to say all operations will suffer that much | |
| 15:50:55 | mnaser | superdan: according to them, correct | |
| 15:51:27 | superdan | mnaser: yeah, so if that's the preference, that's cool. we're not as good as we should be on tolerating faults in the remote links right now, | |
| 15:51:41 | superdan | mnaser: which means you're increasing your chances for things not working if one link is down, but that's on the books to improve | |
| 15:52:01 | mnaser | i guess that's for them to understand as a limitation if we do take this path | |
| 15:52:04 | mnaser | maybe good feedback to bring back :p | |
| 15:52:21 | superdan | at least the story is on the path to improve, but yeah | |
| 15:52:55 | mnaser | i'm suggesting 4 regions instead (na-east, na-west, europe, asia).. sure its not a unified control plane but still a lot better in terms of surprises we'll run into | |
| 15:53:11 | superdan | yep, that's a better approach if they can tolerate it | |
| 15:54:18 | hansmoleman | there are projects meant to abstract the multi-region thing https://wiki.openstack.org/wiki/Kingbird | |
| 15:54:22 | hansmoleman | but no idea how good those are | |
| 15:54:29 | SpamapS | mnaser: those are the 4 regions we have, and we do not have a unified control plane. It's nice that they're isolated from eachother. | |
| 15:54:32 | mnaser | yeah i've heard of kingbird before but it's a bit scary as a concept :p | |
| 15:54:40 | SpamapS | The only thing they share is LDAP. | |
| 15:55:00 | mnaser | yeah, i was thinking share a single keystone, multiple regions in the catalog | |
| 15:55:31 | mnaser | if this works out, hopefully we can share the story, what they're doing with nova is really, really cool. | |
| 15:56:11 | hansmoleman | that would be great | |
| 15:56:12 | jgwentworth | bpoulos: just fyi cert validation is back in a review runway, so be on the lookout for reviews https://etherpad.openstack.org/p/nova-runways-rocky | |
| 15:58:01 | mnaser | jgwentworth: superdan SpamapS thanks for the input | |
| 16:02:21 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ocata: libvirt: handle DiskNotFound during update_available_resource https://review.openstack.org/571432 | |
| 16:02:22 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ocata: libvirt: Skip fetching the virtual size of block devices https://review.openstack.org/571433 | |
| 16:03:44 | superdan | mnaser: since you're around, and since you do upgrades a lot, let me ask... | |
| 16:04:03 | mnaser | i'm always around :p what's up? | |
| 16:04:24 | superdan | mnaser: when you do an upgrade from say pike to queens, do you move everything off of every compute node before you do anything to that node, or do you just stop compute services, upgrade packages, and then re-start services? | |
| 16:04:58 | mnaser | we have the ability to live migrate everything because of ceph, but i've never needed to do that, my thought process is -- services are control plane and they don't affect vms | |
| 16:05:04 | mnaser | so it's always been in place upgrades | |
| 16:05:13 | superdan | mnaser: right okay, that's what I'd expect | |
| 16:05:23 | jgwentworth | it was in place upgrades when I was at yahoo too | |
| 16:05:35 | mnaser | superdan: and that's also a lot of what 99% of deployment tools do as well, afaik | |
| 16:06:11 | superdan | mnaser: if we forced you to migrate every instance in your cloud to roll to rocky, that would be a big deal right? | |
| 16:06:33 | superdan | jgwentworth: yeah, I think it's the only case for very large clouds because it's not feasible to move everything, so that's good data | |
| 16:06:44 | superdan | pretty sure I can say RAX didn't either | |
| 16:06:48 | jgwentworth | yep, that | |
| 16:07:13 | mnaser | superdan: uh, very big, and id hope its <blink> bright red in the release notes | |
| 16:07:34 | mnaser | and also the problem is that it's not possible for some other users which don't have the infrastructure to do migrations | |
| 16:07:37 | hansmoleman | superdan: i want to say huawei public cloud does a lot of live migrations, but Kevin_Zheng would know better than me on their upgrade strategy | |
| 16:07:38 | jgwentworth | yeah, I think penick would have a problem with that too | |
| 16:07:41 | superdan | mnaser: what about when you need to patch hypervisors or kernels? you do full slide puzzle to prevent any instance downtime? | |
| 16:08:07 | mnaser | yes, hypervisor and kernel patching (as well as ceph for customers who we do hyperconverged deployments) all get emptied out first | |
| 16:08:22 | mnaser | reasoning is: hypervisor and kernel touches something that affects the actual running vm, nova-compute does not | |
| 16:08:28 | superdan | mnaser: okay, you never schedule downtime per instance? my cloud provider does it that way | |
| 16:08:54 | mnaser | nope, since we have ceph everywhere, live migrations are easy and function extremely well | |
| 16:09:06 | mnaser | 10g network + ceph = no problems | |
| 16:09:19 | superdan | mnaser: yeah, so what if you didn't have that? I know it's a bit theoretical but, | |
| 16:09:35 | superdan | what if you had to transfer the entire disk and memory across the network for every migration? | |
| 16:09:48 | hansmoleman | we know that would kill NTT.. | |
| 16:09:49 | hansmoleman | at least | |
| 16:09:50 | superdan | would that change the calculus? | |
| 16:10:02 | mnaser | yeah, big time, it would make the upgrade a huge pain | |
| 16:10:09 | superdan | ack, okay thanks | |
| 16:10:10 | mnaser | especially in a public cloud, you have to coordinate across so many customers.. | |
| 16:10:37 | hansmoleman | is there something specifically up for review or being debated that would require us to move all instances to migrate to rocky? | |
| 16:10:57 | superdan | yeah, the NRP allocation conversion thread | |
| 16:11:10 | hansmoleman | ok i figured | |
| 16:11:18 | hansmoleman | i've been blissfully ignorant on that so far | |
| 16:11:25 | mnaser | ^^ me too | |
| 16:11:31 | superdan | I don't think forcing operators to migrate all instances to convert their allocations to nested is reasonable | |
| 16:11:34 | mnaser | because i don't particularly understand it that much :) | |
| 16:11:48 | mnaser | but maybe i should chime in, because i don't know all that much about the whole story | |
| 16:11:51 | SpamapS | We have 20gbit networking for migrations, and we're not ceph based. We aren't going to do a full migration for kernel/kvm updates, we'll schedule downtime. Reboot time is about 8 minutes. | |
| 16:12:12 | SpamapS | We do AZ's, and make sure to complete one AZ before doing the next. | |
| 16:12:14 | superdan | SpamapS: ah sweet, that's a data point I was looking for | |
| 16:12:21 | superdan | my cloud provider does the same | |
| 16:12:33 | superdan | basically tells me that my instances will reboot on X at Y hour, downtime is minutes | |
| 16:12:48 | mnaser | i think in subjects like this, it would be helpful if operators can get a much more simple question like the one superdan asked rather than having to understand whats going on with nested resource providers | |
| 16:12:50 | superdan | and lets me do it early if I want | |
| 16:12:50 | jgwentworth | my cloud provider does that too | |
| 16:13:15 | hansmoleman | can't we have an allocation transformer tool or something for NRP? | |
| 16:13:22 | superdan | mnaser: right, knowing if you're affected by the NRP change, or how, is also something I don't want to mix into the upgrade decision for them either | |
| 16:13:22 | SpamapS | And when people are sad because their one pet VM was down for 10 minutes causing their service to be down, we definitely pat them on the back and say "there there". https://vignette.wikia.nocookie.net/glee/images/7/7a/Sheldon-leonard-there-there.gif/revision/latest?cb=20140908045609 | |
| 16:13:34 | superdan | hansmoleman: that's what I'm saying we owe to the users | |
| 16:13:43 | superdan | SpamapS: ++ | |
| 16:13:51 | superdan | oh shit | |
| 16:13:54 | hansmoleman | nova-manage placement heal_allocations --fix-nrp | |
| 16:13:55 | superdan | you woke the bear | |
| 16:14:12 | hansmoleman | --easy-button | |
| 16:14:21 | superdan | hansmoleman: unfortunately, we can't really do it completely outside with a tool I think, because we need info from the compute node | |
| 16:14:34 | SpamapS | superdan: the "lets me do it early" is interesting. So you're saying that if you hard reboot your instance, you don't get a downtime during the window? | |
| 16:14:53 | hansmoleman | superdan: maybe on restart of the compute service then? like we did for ironic instance flavors? | |
| 16:15:03 | superdan | SpamapS: I can opt to take the downtime early, which is just them (cold) migrating me to another node that is already fixed, but on my schedule instead of theirs, yeah | |
| 16:15:03 | hansmoleman | and we'll be doing for legacy bdm attachments | |
| 16:15:09 | SpamapS | AH | |
| 16:15:11 | superdan | hansmoleman: right, that's what I think we need to do | |
| 16:15:12 | SpamapS | that's neat | |
| 16:15:19 | SpamapS | I think I'm going to put that on our todo list. | |
| 16:15:33 | superdan | SpamapS: sometimes I opt fo that so I can check the health ofmy pet immediately, and sometimes I don't care, depending on which instance it is | |
| 16:15:41 | jgwentworth | superdan, hansmoleman: ++ cause that also works with FFU | |
| 16:15:57 | hansmoleman | well, | |
| 16:15:58 | superdan | jgwentworth: well, actually it doesn't | |
| 16:16:01 | hansmoleman | right | |