| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-17 | |||
| 16:22:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208 | |
| 16:36:44 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201 | |
| 16:41:33 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201 | |
| 16:43:41 | johnthetubaguy | melwitt alex_xu I have attempted to capture the unified limits discussion here in here, still not complete, but getting closer: https://review.openstack.org/602201 | |
| 16:50:34 | bauzas | mriedem: thanks for the last update on libvirt reshape, I now see your TODO(sbauza) and the direction it goes | |
| 16:50:56 | bauzas | mriedem: starting from now, I'll test this by changing to create a child per pGPU | |
| 17:46:19 | cfriesen | sean-k-mooney: bauzas: curious what you think of https://bugs.launchpad.net/nova/+bug/1792985. Huge bug write-up for what would be a small code change. | |
| 17:46:19 | openstack | Launchpad bug 1792985 in OpenStack Compute (nova) "strict NUMA memory allocation for 4K pages leads to OOM-killer" [Undecided,New] | |
| 17:53:34 | sean-k-mooney | cfriesen: we have spoken about that in the past | |
| 17:53:50 | sean-k-mooney | cfriesen: its because the reserved memory is global not per numa node | |
| 17:54:28 | cfriesen | sean-k-mooney: I'm talking about actual instance memory, so reserved shouldn't be involved | |
| 17:54:53 | cfriesen | sean-k-mooney: for a floating instance, it can allocate memory from any numa node | |
| 17:54:55 | sean-k-mooney | cfriesen: reserved should because it will prevent schduing to that node so we dont exaust the memory on one numa node | |
| 17:55:15 | sean-k-mooney | cfriesen: when you set a mem page size it created a numa node of 1 | |
| 17:55:30 | cfriesen | sean-k-mooney: if you don't specify a page size, there is no instance numa_topology | |
| 17:55:32 | sean-k-mooney | cfriesen: so its now confied to float over one numa node | |
| 17:56:11 | sean-k-mooney | yes but if you dont specify a page size of 4 k we dont set the numa tune element at all unless you specify some else that created the numa topologyy | |
| 17:57:20 | cfriesen | sean-k-mooney: the scenario is this: one floating instance that can allocate from any numa node. one instance (with pci device maybe) that is pinned to a single numa node and can't allocate because the other instance consumed all the memory from that node. | |
| 17:57:38 | sean-k-mooney | cfriesen: this xml | |
| 17:57:40 | sean-k-mooney | <numatune> | |
| 17:57:43 | sean-k-mooney | <memory mode='strict' nodeset='0'/> | |
| 17:57:45 | sean-k-mooney | <memnode cellid='0' mode='strict' nodeset='0'/> | |
| 17:57:47 | sean-k-mooney | </numatune> | |
| 17:58:01 | sean-k-mooney | will not be genergated unless the guest has a numa toployt of 1 either implictly or implcitly | |
| 17:58:10 | sean-k-mooney | *explictly | |
| 17:58:14 | cfriesen | sean-k-mooney: yes, that's the one with the pci device | |
| 17:58:21 | cfriesen | which is pinned to a numa node | |
| 17:58:40 | sean-k-mooney | right so the behavor is correct | |
| 17:59:01 | cfriesen | no, because the "floating" instance may have consumed all the memory from that node | |
| 17:59:11 | sean-k-mooney | the issue is we do not have a per numa reserved memory so we can exaust one numa node and trigger oom | |
| 17:59:28 | cfriesen | so either we restrict floating instances to a single numa node, or else we can't use "strict" for the "memory mode" line | |
| 18:00:07 | sean-k-mooney | cfriesen: we already no you cannot mix numa affined instances with non numa instances on a singel host for this exact reason | |
| 18:00:30 | cfriesen | sean-k-mooney: but you can, because you can specify a pci device which implicitly makes it numa-affined | |
| 18:00:46 | sean-k-mooney | cfriesen: we could fix this but we tell operators that this is not ok and use host aggreate to seperate these flavors today | |
| 18:01:03 | sean-k-mooney | cfriesen: you mean by neutron sriov ports | |
| 18:01:09 | sean-k-mooney | cfriesen: that is true | |
| 18:02:31 | sean-k-mooney | cfriesen: i would be ok with introduceing the hw:numa_mem_policy extra spec that was originally proposed for hyper v to allowchanging the mode but it messes up our accounting in placement if we do | |
| 18:04:23 | sean-k-mooney | that said another way to resolve this would be to allways specify a page size so your floating instances would be hw:mem_page_size=small or 4k | |
| 18:04:35 | cfriesen | sean-k-mooney: I was thinking we could literally just remove the "memory mode=strict" line. That would make us use the linux default, which is local allocation. | |
| 18:05:09 | sean-k-mooney | cfriesen: if we remove it for 4k why would we not allow it for any page size | |
| 18:05:09 | cfriesen | sean-k-mooney: explicitly specifying page size would limit them to a single numa node, which we had before but removed. | |
| 18:05:45 | cfriesen | sean-k-mooney: adding hugepages explicitly results in a numa-topology and being restricted to a single node. | |
| 18:06:04 | sean-k-mooney | cfriesen: in nova yes but qemu does not require that | |
| 18:06:42 | cfriesen | sean-k-mooney: I suspect that people using hugepages care about performance, while people using the default pagesize and floating instances don't. :) | |
| 18:07:08 | sean-k-mooney | cfriesen: so the defautl is stict and you opt out | |
| 18:07:40 | sean-k-mooney | cfriesen: that argument will never win with me because i argued that none of the implit behavior should have existied for the start | |
| 18:09:09 | cfriesen | sean-k-mooney: I'd rather have the default be something that doesn't lead to the OOM-killer. Then if you've engineered the cluster to avoid the problem you can opt-in to "strict" | |
| 18:10:01 | sean-k-mooney | cfriesen: prefer will break us one memory is reported in placement as we will nologer no where the memory was consumed form | |
| 18:10:23 | cfriesen | sean-k-mooney: we already don't know where it's consumed from. we just think we do. :) | |
| 18:10:36 | cfriesen | for the floating instances, at least | |
| 18:10:42 | sean-k-mooney | cfriesen: we know when we explcitly state a page size | |
| 18:10:47 | cfriesen | sean-k-mooney: yes | |
| 18:11:13 | sean-k-mooney | for floating instance we dont touch the numa toplogy blob. | |
| 18:11:17 | sean-k-mooney | so its fine | |
| 18:11:41 | sean-k-mooney | the numa toplogy blob heals based on the periodic task | |
| 18:11:57 | cfriesen | sean-k-mooney: sure, each is fine in isolation. but as soon as you have numa-affined and non-numa-affined instances on the same numa node all bets are off. | |
| 18:12:45 | cfriesen | (with "strict" at least) | |
| 18:12:49 | sean-k-mooney | cfriesen: yes so perhaps we should use a filter to prevent that | |
| 18:13:35 | cfriesen | worth considering. anyways, gotta run. thanks for the feedback | |
| 18:13:54 | sean-k-mooney | :) sorry it was not more positve | |
| 18:27:58 | openstackgerrit | Merged openstack/nova stable/rocky: Fix soft deleting vm fails after "nova resize" vm https://review.openstack.org/603140 | |
| 18:53:36 | mriedem | johnthetubaguy: you asked why remove the old warts from the compute API in a microversion? i have one answer: because i have to always lookup those stupid "OS-EXT-SRV-ATTR" prefixes when writing client side code for the server response | |
| 18:53:54 | mriedem | i know the host is in the response when showing server details as admin, but i can never remember the stupid prefix on the response field | |
| 18:54:30 | mriedem | mordred: can i get an amen?! ^ | |
| 19:19:45 | mriedem | gmann: alex_xu: i've left some comments on https://review.openstack.org/#/c/599276/ but we should probably discuss when we can change additionalProperties=False in the GET /servers(/detail) APIs | |
| 19:19:59 | mriedem | i'll send something to the ML | |
| 19:32:49 | mriedem | http://lists.openstack.org/pipermail/openstack-dev/2018-September/134748.html | |
| 20:13:54 | mordred | mriedem: amen | |
| 21:18:15 | jaypipes | mriedem: do you want code reviews on https://review.openstack.org/#/c/599208/ or not? | |
| 21:18:53 | mriedem | jaypipes: no | |
| 21:19:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Merge security groups extension response into server view builder https://review.openstack.org/585475 | |
| 21:19:03 | mriedem | sounds like it's going to need more work from bauzas | |
| 21:27:48 | jaypipes | mriedem: ack | |
| 21:47:16 | alex_xu | johnthetubaguy: thanks a lot, will review it | |
| 22:00:01 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104 | |
| 22:00:12 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106 | |
| 22:00:27 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110 | |
| 22:23:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Merge extended_status extension response into server view builder https://review.openstack.org/592092 | |
| 22:42:28 | mriedem | when your house is full of the sounds of screaming 7 year old girls and "metal militia" by metallica comes on, | |
| 22:42:34 | mriedem | one must drown out the screaming | |
| 22:46:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947 | |
| 22:46:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Merge extended_volumes extension response into server view builder https://review.openstack.org/596285 | |
| 22:51:25 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (6) https://review.openstack.org/574113 | |
| 22:51:35 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (7) https://review.openstack.org/574974 | |
| 22:51:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311 | |
| 22:54:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Making instance/migration listing skipping down cells configurable https://review.openstack.org/592428 | |
| 22:57:43 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.openstack.org/575581 | |
| 22:57:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.openstack.org/576017 | |
| 22:58:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.openstack.org/576018 | |
| 23:09:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add get_by_cell_and_project() method to InstanceMappingList https://review.openstack.org/591656 | |
| 23:14:27 | mriedem | gmann: there is one open change left in the extension merge series https://review.openstack.org/#/q/topic:bp/api-extensions-merge-stein+(status:open+OR+status:merged) - if you're feeling up to it the extended_volumes one needs to be fixed for the same kind of performance issue as the security groups one | |
| 23:14:42 | mriedem | i.e. call it with the list of servers to batch the results rather than one per server in the detail() Case | |
| 23:44:41 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.openstack.org/576019 | |
| 23:44:52 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.openstack.org/576020 | |
| 23:45:18 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.openstack.org/576027 | |
| #openstack-nova - 2018-09-18 | |||
| 00:34:16 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.openstack.org/576031 | |
| 00:34:40 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299 | |
| 00:34:59 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.openstack.org/576344 | |
| 01:20:36 | naichuans_ | bauzas: got it, thanks. Please ping me when you online. | |
| 01:25:48 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.openstack.org/576673 | |