Posted	Nick	Remark
#openstack-nova - 2018-09-17
17:53:50	sean-k-mooney	cfriesen: its because the reserved memory is global not per numa node
17:54:28	cfriesen	sean-k-mooney: I'm talking about actual instance memory, so reserved shouldn't be involved
17:54:53	cfriesen	sean-k-mooney: for a floating instance, it can allocate memory from any numa node
17:54:55	sean-k-mooney	cfriesen: reserved should because it will prevent schduing to that node so we dont exaust the memory on one numa node
17:55:15	sean-k-mooney	cfriesen: when you set a mem page size it created a numa node of 1
17:55:30	cfriesen	sean-k-mooney: if you don't specify a page size, there is no instance numa_topology
17:55:32	sean-k-mooney	cfriesen: so its now confied to float over one numa node
17:56:11	sean-k-mooney	yes but if you dont specify a page size of 4 k we dont set the numa tune element at all unless you specify some else that created the numa topologyy
17:57:20	cfriesen	sean-k-mooney: the scenario is this: one floating instance that can allocate from any numa node. one instance (with pci device maybe) that is pinned to a single numa node and can't allocate because the other instance consumed all the memory from that node.
17:57:38	sean-k-mooney	cfriesen: this xml
17:57:40	sean-k-mooney	<numatune>
17:57:43	sean-k-mooney	<memory mode='strict' nodeset='0'/>
17:57:45	sean-k-mooney	<memnode cellid='0' mode='strict' nodeset='0'/>
17:57:47	sean-k-mooney	</numatune>
17:58:01	sean-k-mooney	will not be genergated unless the guest has a numa toployt of 1 either implictly or implcitly
17:58:10	sean-k-mooney	*explictly
17:58:14	cfriesen	sean-k-mooney: yes, that's the one with the pci device
17:58:21	cfriesen	which is pinned to a numa node
17:58:40	sean-k-mooney	right so the behavor is correct
17:59:01	cfriesen	no, because the "floating" instance may have consumed all the memory from that node
17:59:11	sean-k-mooney	the issue is we do not have a per numa reserved memory so we can exaust one numa node and trigger oom
17:59:28	cfriesen	so either we restrict floating instances to a single numa node, or else we can't use "strict" for the "memory mode" line
18:00:07	sean-k-mooney	cfriesen: we already no you cannot mix numa affined instances with non numa instances on a singel host for this exact reason
18:00:30	cfriesen	sean-k-mooney: but you can, because you can specify a pci device which implicitly makes it numa-affined
18:00:46	sean-k-mooney	cfriesen: we could fix this but we tell operators that this is not ok and use host aggreate to seperate these flavors today
18:01:03	sean-k-mooney	cfriesen: you mean by neutron sriov ports
18:01:09	sean-k-mooney	cfriesen: that is true
18:02:31	sean-k-mooney	cfriesen: i would be ok with introduceing the hw:numa_mem_policy extra spec that was originally proposed for hyper v to allowchanging the mode but it messes up our accounting in placement if we do
18:04:23	sean-k-mooney	that said another way to resolve this would be to allways specify a page size so your floating instances would be hw:mem_page_size=small or 4k
18:04:35	cfriesen	sean-k-mooney: I was thinking we could literally just remove the "memory mode=strict" line. That would make us use the linux default, which is local allocation.
18:05:09	cfriesen	sean-k-mooney: explicitly specifying page size would limit them to a single numa node, which we had before but removed.
18:05:09	sean-k-mooney	cfriesen: if we remove it for 4k why would we not allow it for any page size
18:05:45	cfriesen	sean-k-mooney: adding hugepages explicitly results in a numa-topology and being restricted to a single node.
18:06:04	sean-k-mooney	cfriesen: in nova yes but qemu does not require that
18:06:42	cfriesen	sean-k-mooney: I suspect that people using hugepages care about performance, while people using the default pagesize and floating instances don't. :)
18:07:08	sean-k-mooney	cfriesen: so the defautl is stict and you opt out
18:07:40	sean-k-mooney	cfriesen: that argument will never win with me because i argued that none of the implit behavior should have existied for the start
18:09:09	cfriesen	sean-k-mooney: I'd rather have the default be something that doesn't lead to the OOM-killer. Then if you've engineered the cluster to avoid the problem you can opt-in to "strict"
18:10:01	sean-k-mooney	cfriesen: prefer will break us one memory is reported in placement as we will nologer no where the memory was consumed form
18:10:23	cfriesen	sean-k-mooney: we already don't know where it's consumed from. we just think we do. :)
18:10:36	cfriesen	for the floating instances, at least
18:10:42	sean-k-mooney	cfriesen: we know when we explcitly state a page size
18:10:47	cfriesen	sean-k-mooney: yes
18:11:13	sean-k-mooney	for floating instance we dont touch the numa toplogy blob.
18:11:17	sean-k-mooney	so its fine
18:11:41	sean-k-mooney	the numa toplogy blob heals based on the periodic task
18:11:57	cfriesen	sean-k-mooney: sure, each is fine in isolation. but as soon as you have numa-affined and non-numa-affined instances on the same numa node all bets are off.
18:12:45	cfriesen	(with "strict" at least)
18:12:49	sean-k-mooney	cfriesen: yes so perhaps we should use a filter to prevent that
18:13:35	cfriesen	worth considering. anyways, gotta run. thanks for the feedback
18:13:54	sean-k-mooney	:) sorry it was not more positve
18:27:58	openstackgerrit	Merged openstack/nova stable/rocky: Fix soft deleting vm fails after "nova resize" vm https://review.openstack.org/603140
18:53:36	mriedem	johnthetubaguy: you asked why remove the old warts from the compute API in a microversion? i have one answer: because i have to always lookup those stupid "OS-EXT-SRV-ATTR" prefixes when writing client side code for the server response
18:53:54	mriedem	i know the host is in the response when showing server details as admin, but i can never remember the stupid prefix on the response field
18:54:30	mriedem	mordred: can i get an amen?! ^
19:19:45	mriedem	gmann: alex_xu: i've left some comments on https://review.openstack.org/#/c/599276/ but we should probably discuss when we can change additionalProperties=False in the GET /servers(/detail) APIs
19:19:59	mriedem	i'll send something to the ML
19:32:49	mriedem	http://lists.openstack.org/pipermail/openstack-dev/2018-September/134748.html
20:13:54	mordred	mriedem: amen
21:18:15	jaypipes	mriedem: do you want code reviews on https://review.openstack.org/#/c/599208/ or not?
21:18:53	mriedem	jaypipes: no
21:19:00	openstackgerrit	Matt Riedemann proposed openstack/nova master: Merge security groups extension response into server view builder https://review.openstack.org/585475
21:19:03	mriedem	sounds like it's going to need more work from bauzas
21:27:48	jaypipes	mriedem: ack
21:47:16	alex_xu	johnthetubaguy: thanks a lot, will review it
22:00:01	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104
22:00:12	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106
22:00:27	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110
22:23:40	openstackgerrit	Matt Riedemann proposed openstack/nova master: Merge extended_status extension response into server view builder https://review.openstack.org/592092
22:42:28	mriedem	when your house is full of the sounds of screaming 7 year old girls and "metal militia" by metallica comes on,
22:42:34	mriedem	one must drown out the screaming
22:46:21	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947
22:46:22	openstackgerrit	Matt Riedemann proposed openstack/nova master: Merge extended_volumes extension response into server view builder https://review.openstack.org/596285
22:51:25	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (6) https://review.openstack.org/574113
22:51:35	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (7) https://review.openstack.org/574974
22:51:45	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311
22:54:07	openstackgerrit	Matt Riedemann proposed openstack/nova master: Making instance/migration listing skipping down cells configurable https://review.openstack.org/592428
22:57:43	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.openstack.org/575581
22:57:57	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.openstack.org/576017
22:58:09	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.openstack.org/576018
23:09:03	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add get_by_cell_and_project() method to InstanceMappingList https://review.openstack.org/591656
23:14:27	mriedem	gmann: there is one open change left in the extension merge series https://review.openstack.org/#/q/topic:bp/api-extensions-merge-stein+(status:open+OR+status:merged) - if you're feeling up to it the extended_volumes one needs to be fixed for the same kind of performance issue as the security groups one
23:14:42	mriedem	i.e. call it with the list of servers to batch the results rather than one per server in the detail() Case
23:44:41	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.openstack.org/576019
23:44:52	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.openstack.org/576020
23:45:18	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.openstack.org/576027
#openstack-nova - 2018-09-18
00:34:16	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.openstack.org/576031
00:34:40	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299
00:34:59	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.openstack.org/576344
01:20:36	naichuans_	bauzas: got it, thanks. Please ping me when you online.
01:25:48	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.openstack.org/576673
01:26:13	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.openstack.org/576676
01:26:32	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.openstack.org/576689
01:32:01	openstackgerrit	fupingxie proposed openstack/nova master: Delete allocations for instances that is not in this node https://review.openstack.org/582899
01:37:47	openstackgerrit	fupingxie proposed openstack/nova master: Delete allocations for instances that is not in this node https://review.openstack.org/582899
02:08:53	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.openstack.org/576709
02:09:10	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.openstack.org/576712
02:09:42	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (6) https://review.openstack.org/571330
02:10:00	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992
02:56:16	cburgess	mriedem Oh I'm still lurking about.

Earlier   Later