| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-01 | |||
| 15:35:20 | superdan | yup | |
| 15:35:29 | hansmoleman | so i'd nack until they can give specifics | |
| 15:36:24 | fried_rice | Roger wilco. | |
| 15:36:27 | fried_rice | thanks guys | |
| 15:37:03 | hansmoleman | i'd be willing to bet Kevin_Zheng knows if we (huawei) have a need for this | |
| 15:37:37 | hansmoleman | like, maybe i need to get my chinese unicode name passed through user metadata to config drive for something running in the image | |
| 15:37:47 | hansmoleman | to register with some internal system | |
| 15:37:48 | hansmoleman | idk | |
| 15:39:36 | PapaOurs | silly question but... does cloud-init support unicode ? | |
| 15:42:38 | hansmoleman | https://cloudinit.readthedocs.io/en/latest/search.html?q=unicode&check_keywords=yes&area=default | |
| 15:42:43 | hansmoleman | the answer is, shrug | |
| 15:43:25 | fried_rice | Marked Incomplete asking for details: https://bugs.launchpad.net/nova/+bug/1737711 | |
| 15:43:27 | SpamapS | PapaOurs: it should, if it doesn't that's a bug. | |
| 15:43:27 | openstack | Launchpad bug 1737711 in OpenStack Compute (nova) "nova boot failed when use the chinese metadata key and value" [Undecided,Incomplete] - Assigned to wanghongtao (hongtao.wang) | |
| 15:44:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Restrict CONF.quota.driver to DB and noop quota drivers https://review.openstack.org/410996 | |
| 15:45:10 | hansmoleman | finucannot: i made that release note update you wanted ^ if you want to just fast approve | |
| 15:46:20 | finucannot | hansmoleman: Done and done | |
| 15:46:23 | hansmoleman | thanks | |
| 15:46:44 | mnaser | so wild Friday cells v2 question: how much does latency affect operations overall? | |
| 15:47:09 | superdan | mnaser: latency to the cell db? no more or less than with one cell | |
| 15:47:40 | mnaser | superdan: well, say you have your api in NA and a cell in .. say.. thailand. | |
| 15:48:02 | superdan | mnaser: that's probably not the best plan | |
| 15:48:17 | mnaser | (nfv use case, and company has world wide presence, so i suggested 4 regions with a single cell split geographically) | |
| 15:48:26 | mnaser | but i was thrown "cells v2" at me | |
| 15:48:40 | SpamapS | mnaser: you'll always get the latency of the slowest round trip from api->cellbits .. so it will suffer as much as your latency varies | |
| 15:48:47 | jgwentworth | mnaser: the api connects directly to cell databases, so that scenario would be slow | |
| 15:48:51 | superdan | mnaser: if unified control plane is what you care about the most, then... | |
| 15:48:54 | superdan | SpamapS: that's not true | |
| 15:49:20 | mnaser | so the thing is, api operations isn't a big deal for them, but i don't know how much slower they'd be | |
| 15:49:32 | jgwentworth | superdan: I think he means for a scatter-gather, yeah? | |
| 15:49:38 | mnaser | aka if the value of having a unified control plane is enough to take that performance hit | |
| 15:49:39 | SpamapS | Well, worst latency that has instances in the project you're listing I suppose | |
| 15:49:40 | superdan | SpamapS: if you spread things evenly across cells then it will, but if you have a tenant per cell (as one example) then you'll get normal performance if you're not in one of theother cells | |
| 15:50:00 | superdan | jgwentworth: yes | |
| 15:50:40 | superdan | mnaser: right, so relatively static workload, they just want a unified view of it, even if operations take a while, because they're infrequentish? | |
| 15:50:48 | SpamapS | I didn't mean to say all operations will suffer that much | |
| 15:50:55 | mnaser | superdan: according to them, correct | |
| 15:51:27 | superdan | mnaser: yeah, so if that's the preference, that's cool. we're not as good as we should be on tolerating faults in the remote links right now, | |
| 15:51:41 | superdan | mnaser: which means you're increasing your chances for things not working if one link is down, but that's on the books to improve | |
| 15:52:01 | mnaser | i guess that's for them to understand as a limitation if we do take this path | |
| 15:52:04 | mnaser | maybe good feedback to bring back :p | |
| 15:52:21 | superdan | at least the story is on the path to improve, but yeah | |
| 15:52:55 | mnaser | i'm suggesting 4 regions instead (na-east, na-west, europe, asia).. sure its not a unified control plane but still a lot better in terms of surprises we'll run into | |
| 15:53:11 | superdan | yep, that's a better approach if they can tolerate it | |
| 15:54:18 | hansmoleman | there are projects meant to abstract the multi-region thing https://wiki.openstack.org/wiki/Kingbird | |
| 15:54:22 | hansmoleman | but no idea how good those are | |
| 15:54:29 | SpamapS | mnaser: those are the 4 regions we have, and we do not have a unified control plane. It's nice that they're isolated from eachother. | |
| 15:54:32 | mnaser | yeah i've heard of kingbird before but it's a bit scary as a concept :p | |
| 15:54:40 | SpamapS | The only thing they share is LDAP. | |
| 15:55:00 | mnaser | yeah, i was thinking share a single keystone, multiple regions in the catalog | |
| 15:55:31 | mnaser | if this works out, hopefully we can share the story, what they're doing with nova is really, really cool. | |
| 15:56:11 | hansmoleman | that would be great | |
| 15:56:12 | jgwentworth | bpoulos: just fyi cert validation is back in a review runway, so be on the lookout for reviews https://etherpad.openstack.org/p/nova-runways-rocky | |
| 15:58:01 | mnaser | jgwentworth: superdan SpamapS thanks for the input | |
| 16:02:21 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ocata: libvirt: handle DiskNotFound during update_available_resource https://review.openstack.org/571432 | |
| 16:02:22 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ocata: libvirt: Skip fetching the virtual size of block devices https://review.openstack.org/571433 | |
| 16:03:44 | superdan | mnaser: since you're around, and since you do upgrades a lot, let me ask... | |
| 16:04:03 | mnaser | i'm always around :p what's up? | |
| 16:04:24 | superdan | mnaser: when you do an upgrade from say pike to queens, do you move everything off of every compute node before you do anything to that node, or do you just stop compute services, upgrade packages, and then re-start services? | |
| 16:04:58 | mnaser | we have the ability to live migrate everything because of ceph, but i've never needed to do that, my thought process is -- services are control plane and they don't affect vms | |
| 16:05:04 | mnaser | so it's always been in place upgrades | |
| 16:05:13 | superdan | mnaser: right okay, that's what I'd expect | |
| 16:05:23 | jgwentworth | it was in place upgrades when I was at yahoo too | |
| 16:05:35 | mnaser | superdan: and that's also a lot of what 99% of deployment tools do as well, afaik | |
| 16:06:11 | superdan | mnaser: if we forced you to migrate every instance in your cloud to roll to rocky, that would be a big deal right? | |
| 16:06:33 | superdan | jgwentworth: yeah, I think it's the only case for very large clouds because it's not feasible to move everything, so that's good data | |
| 16:06:44 | superdan | pretty sure I can say RAX didn't either | |
| 16:06:48 | jgwentworth | yep, that | |
| 16:07:13 | mnaser | superdan: uh, very big, and id hope its <blink> bright red in the release notes | |
| 16:07:34 | mnaser | and also the problem is that it's not possible for some other users which don't have the infrastructure to do migrations | |
| 16:07:37 | hansmoleman | superdan: i want to say huawei public cloud does a lot of live migrations, but Kevin_Zheng would know better than me on their upgrade strategy | |
| 16:07:38 | jgwentworth | yeah, I think penick would have a problem with that too | |
| 16:07:41 | superdan | mnaser: what about when you need to patch hypervisors or kernels? you do full slide puzzle to prevent any instance downtime? | |
| 16:08:07 | mnaser | yes, hypervisor and kernel patching (as well as ceph for customers who we do hyperconverged deployments) all get emptied out first | |
| 16:08:22 | mnaser | reasoning is: hypervisor and kernel touches something that affects the actual running vm, nova-compute does not | |
| 16:08:28 | superdan | mnaser: okay, you never schedule downtime per instance? my cloud provider does it that way | |
| 16:08:54 | mnaser | nope, since we have ceph everywhere, live migrations are easy and function extremely well | |
| 16:09:06 | mnaser | 10g network + ceph = no problems | |
| 16:09:19 | superdan | mnaser: yeah, so what if you didn't have that? I know it's a bit theoretical but, | |
| 16:09:35 | superdan | what if you had to transfer the entire disk and memory across the network for every migration? | |
| 16:09:48 | hansmoleman | we know that would kill NTT.. | |
| 16:09:49 | hansmoleman | at least | |
| 16:09:50 | superdan | would that change the calculus? | |
| 16:10:02 | mnaser | yeah, big time, it would make the upgrade a huge pain | |
| 16:10:09 | superdan | ack, okay thanks | |
| 16:10:10 | mnaser | especially in a public cloud, you have to coordinate across so many customers.. | |
| 16:10:37 | hansmoleman | is there something specifically up for review or being debated that would require us to move all instances to migrate to rocky? | |
| 16:10:57 | superdan | yeah, the NRP allocation conversion thread | |
| 16:11:10 | hansmoleman | ok i figured | |
| 16:11:18 | hansmoleman | i've been blissfully ignorant on that so far | |
| 16:11:25 | mnaser | ^^ me too | |
| 16:11:31 | superdan | I don't think forcing operators to migrate all instances to convert their allocations to nested is reasonable | |
| 16:11:34 | mnaser | because i don't particularly understand it that much :) | |
| 16:11:48 | mnaser | but maybe i should chime in, because i don't know all that much about the whole story | |
| 16:11:51 | SpamapS | We have 20gbit networking for migrations, and we're not ceph based. We aren't going to do a full migration for kernel/kvm updates, we'll schedule downtime. Reboot time is about 8 minutes. | |
| 16:12:12 | SpamapS | We do AZ's, and make sure to complete one AZ before doing the next. | |
| 16:12:14 | superdan | SpamapS: ah sweet, that's a data point I was looking for | |
| 16:12:21 | superdan | my cloud provider does the same | |
| 16:12:33 | superdan | basically tells me that my instances will reboot on X at Y hour, downtime is minutes | |
| 16:12:48 | mnaser | i think in subjects like this, it would be helpful if operators can get a much more simple question like the one superdan asked rather than having to understand whats going on with nested resource providers | |