| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-29 | |||
| 14:08:53 | artom | belmoreira, yep, that's very helpful, thanks! | |
| 14:09:38 | belmoreira | artom stephenfin thanks for ping me on this | |
| 14:09:42 | sean-k-mooney | well there are two parts first the query to get all the instance that need to be migrated then we need to lock the db and update all recoreds | |
| 14:10:01 | sean-k-mooney | you can actully do that in one update query | |
| 14:10:18 | sean-k-mooney | in general bug not in this case | |
| 14:10:24 | artom | belmoreira, just to make sure I understand, you're saying that your numa_topology-having instances are only about 4 to 8 instances per compute host, right? | |
| 14:10:25 | gibi | gmann: ack, thanks | |
| 14:10:25 | dansmith | belmoreira: I'm not sure I got what you said.. you don't have a db with tons of instances with a numa topology to test this on right? | |
| 14:11:49 | sean-k-mooney | if i parsed it correctly the cells with numa instance only run large vms so they have fewer isntances then the other cells | |
| 14:12:01 | dansmith | right | |
| 14:12:01 | belmoreira | artom yes | |
| 14:12:46 | artom | belmoreira, aha, ok - well, it tells us that the sql engine (or sqlalchemy itself?) is smart and checks the more restrictive stuff first - iow having a numa_topology at all | |
| 14:13:03 | artom | But it doesn't tell us what happens if there are thousands of numa_topology-having instances... :( | |
| 14:13:06 | sean-k-mooney | to be fair i would suspect that is likly common of many deployments. e.g. most nuam instance will be large flavors and therefore there will be less of them then standard instances | |
| 14:13:27 | artom | sean-k-mooney, that does seem very likely... | |
| 14:13:47 | dansmith | stephenfin: without the migrate on load, any idea what I can do to tickle those instances to rewrite them? | |
| 14:13:47 | belmoreira | dansmith you're right. I don't have a cell with a lot of instances. <2000 instances only | |
| 14:13:54 | dansmith | belmoreira: gotcha, thanks | |
| 14:14:12 | sean-k-mooney | belmoreira: could you do a count(*) on instance that have a numa topology so we can get a feel for the amount you have | |
| 14:14:32 | belmoreira | sean-k-mooney yes, give a sec | |
| 14:14:53 | sean-k-mooney | just wondering if its in the 100s or roughly what you were messuring | |
| 14:15:53 | sean-k-mooney | artom: i mean we can always fake it if we need too | |
| 14:16:08 | artom | sean-k-mooney, fake what? | |
| 14:16:24 | sean-k-mooney | a db with 1000s of numa instances | |
| 14:16:39 | artom | True | |
| 14:16:49 | belmoreira | our typical use case: | |
| 14:16:52 | belmoreira | select count(*) from instance_extra where deleted = 0; #711 | |
| 14:16:52 | belmoreira | select count(*) from compute_nodes nodes; #176 | |
| 14:16:53 | belmoreira | select * from instance_extra where deleted = 0 and numa_topology is not null and numa_topology like "%nova_object.name%"; #took 38.2 ms | |
| 14:17:10 | sean-k-mooney | ok so 700 ish is not bad | |
| 14:17:27 | sean-k-mooney | its not 2000 but its not 10 | |
| 14:18:07 | artom | So based on those number it might actually be OK to go ahead as is... | |
| 14:18:11 | sean-k-mooney | if we were to assume this was liniar then i think it would be accpetable | |
| 14:18:13 | artom | *numbers | |
| 14:19:09 | dansmith | so, if we just convert to migrate-on-load, we don't take any overhead for string searching at all, migrate only the instances that need it, and are assured that they have migrated all instances by the end of the cycle, right? | |
| 14:19:22 | dansmith | even with nova-manage, we can't make them run it without a blocker migration (which is also expensive) | |
| 14:19:48 | belmoreira | in this case nova_topology is not defined, but may help in the analysis: | |
| 14:19:55 | belmoreira | select count(*) from instance_extra where deleted = 0; #1573 | |
| 14:19:55 | belmoreira | select count(*) from compute_nodes nodes; #88 | |
| 14:19:57 | belmoreira | select * from instance_extra where deleted = 0 and numa_topology is not null and numa_topology like "%nova_object.name%"; #took 106 ms | |
| 14:19:58 | dansmith | if we did that, stephenfin could just remove his nova-manage bits entirely, no impact for anyone that doesn't have old instances | |
| 14:20:35 | stephenfin | Yeah, I can do that too. Let me see if I can wrangle something up | |
| 14:20:39 | artom | dansmith, the end of cycle stuff is complicated by FFUs and operators skipping releases, no? | |
| 14:20:47 | sean-k-mooney | ya if we migrate on load that also works but the only realy implciation is keeping that for a relase or two for FFU | |
| 14:21:00 | dansmith | artom: ah, yeah, good point... | |
| 14:21:34 | artom | So 2 or 3 cycles, I guess | |
| 14:21:39 | dansmith | artom: if we had migrate-on-load for a while, then it'd be fine, but that's the problem trying to do the switch quickly | |
| 14:21:41 | sean-k-mooney | we coudl do both e.g. do the migrate on load and then wen we drop that provide a nova manange command with a blocker migration | |
| 14:22:15 | dansmith | sean-k-mooney: stephenfin has been trying to get this done for a long time so we're trying not to make this a career-long arc | |
| 14:22:46 | artom | dansmith, it's a thing he'll bequeath to his grandchildren | |
| 14:23:08 | sean-k-mooney | yep i know if this was a few weeks ato i woudl have suggeted doing the migrate on load in ussuri and then the blocker/migrate command in victoria | |
| 14:23:16 | stephenfin | 826 days | |
| 14:23:55 | stephenfin | oh my, I appear to have broken zuul https://review.opendev.org/#/c/724332/ | |
| 14:24:24 | artom | Dayyyyuuuuum | |
| 14:24:30 | sean-k-mooney | hehe let me check with infra | |
| 14:27:43 | sean-k-mooney | zuul is being restart | |
| 14:28:00 | sean-k-mooney | it hit an out of memory issue and they are currently trying to fix it | |
| 14:29:17 | sean-k-mooney | so hold rechecks for a few minutes while they sort this out | |
| 14:29:50 | sean-k-mooney | well clearly his patch ate all the memory | |
| 14:35:00 | sean-k-mooney | ok zuul is back up. changes running before 14:00 UTC have been requed anything uploaded or approved bettween 14:00 and 14:30 needs to be rechecked | |
| 14:35:19 | sean-k-mooney | infra are going to send a staus update for the same shortly | |
| 14:52:11 | kashyap | sean-k-mooney: Since you've reviewed an older version (PS-5) and a newer one (PS-7), I'll just address the PS-7 bits here: https://review.opendev.org/#/c/631154/ | |
| 14:52:15 | kashyap | sean-k-mooney: That okay? | |
| 14:55:54 | sean-k-mooney | kashyap: am sure | |
| 14:56:25 | sean-k-mooney | i think i coppied most of the relevent bits although if you read both and just resond on 7 that is fine with me | |
| 14:56:42 | sean-k-mooney | or well update it in version 8 | |
| 14:56:54 | kashyap | sean-k-mooney: While I comment in the spec, on the "increased memory usage" bit -- I knew that thing, but haven't explicitly mentioned it because it requires precise tests, in what scenarios, etc | |
| 14:57:06 | kashyap | You can't just put a generic: "in all cases memory is increased" | |
| 14:57:32 | sean-k-mooney | kashyap: not really even with 1 pci root port dan found it used more memeory the pc | |
| 14:57:34 | kashyap | It requires more testing; so somebody ought to do the "performance testing guy's job"... | |
| 14:57:45 | sean-k-mooney | so i think in all configurtion it has more memory overhead | |
| 14:58:06 | kashyap | sean-k-mooney: Right; I'll mention that, but need to carefully write it in context and with a config example | |
| 14:58:42 | sean-k-mooney | or we can jsut say we expect that q35 will use more memroy as we have never seen a case where it uses less | |
| 14:58:44 | kashyap | sean-k-mooney: Also I don't want us to get sucked into that black-hole and get derailed... | |
| 14:58:56 | kashyap | But that begs the question: "how much more memory than before" | |
| 14:59:13 | sean-k-mooney | sure an that we can leave to the operator | |
| 14:59:14 | kashyap | Which requires clear a example benchmark | |
| 14:59:18 | sean-k-mooney | or perfomance guy | |
| 14:59:42 | kashyap | Okido; I actually first mentioned it locally and then removed it, as I was still thinking of it | |
| 15:00:37 | sean-k-mooney | i mainly just want them to be aware that they should consider it when upgrading so they can factor it in to there host memory reservation and capastity planning | |
| 15:01:55 | kashyap | Yeah, definitely. Thx for the taking time respond. | |
| 15:35:28 | spatel | sean-k-mooney: do you know how much CPU would be enough to reserve for hypervisor? using isolcpus option? | |
| 15:45:11 | sean-k-mooney | i dont adviase using isolcpus | |
| 15:46:11 | sean-k-mooney | generally 1 phsycial core is more then enough on a compute node or 1 per numa nodes if you want to do affintiy of interupts | |
| 15:47:59 | spatel | sean-k-mooney: but mostly for NFV they suggest using isolcpus for isolation | |
| 15:48:15 | sean-k-mooney | spatel: i generally recommend you use the vcpu_pin_set or in train+ cpu_dedicate_set and cpu_shared_set to do the reservation | |
| 15:48:34 | spatel | you are saying 2 cpu core would be more than enough for hypervisor (per NUMA)? | |
| 15:48:40 | sean-k-mooney | spatel: you should only use isolcpus on realtime hosts and only on the cpus used for pinned vms | |
| 15:49:32 | spatel | I am doing all cpu pinning (dedicated) option for all my workload | |
| 15:49:55 | spatel | we need performance not quantity. | |
| 15:51:14 | spatel | I am planning to use isolcpus + vcpu_pin_set (both option to allocate dedicated CPU) | |
| 15:55:21 | openstackgerrit | Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301 | |
| 15:55:24 | spatel | sean-k-mooney: ^^ | |
| 15:56:13 | sean-k-mooney | 1 phsyical core(2 hyperthreads) is normally enouch for a compute node if you are not using heavy telemetry | |
| 15:56:40 | sean-k-mooney | spatel: you can use isolcpus + vcpu_pin_set but only if the vm is pinned | |
| 15:57:52 | sean-k-mooney | when you use isolcpus it disables the linux kernel shcduler for those cores | |
| 15:58:05 | sean-k-mooney | so if you have floating vms then they wont float | |
| 15:58:30 | spatel | sean-k-mooney: sounds good, yes we do pinned VM (currently i have assigned 8 cores but wanted to see what people mostly recommend ) | |
| 15:58:32 | sean-k-mooney | in generally isolcpus is only a good idea if you are running realtime wrokloads | |
| 15:58:59 | sean-k-mooney | spatel: you might want to look into tuned by the way | |