Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-05
17:58:40 openstackgerrit Corey Bryant proposed openstack/nova master: Add Python 3 Train unit tests https://review.opendev.org/669437
18:00:20 openstackgerrit Corey Bryant proposed openstack/os-vif master: Add Python 3 Train unit tests https://review.opendev.org/669438
18:01:06 openstackgerrit Corey Bryant proposed openstack/python-novaclient master: Add Python 3 Train unit tests https://review.opendev.org/669439
18:08:50 mriedem redkrieg: which release? placement could be filtering it out - enable debug logging in the placement logs and you should see the allocation candidate filtering messages (on newer releases), there is the reserved_host_ram_mb config option on the compute that will be reporting reserved MEMORY_MB to placement
18:08:57 mriedem which is part of the overall capacity calculation
18:12:20 Nick_A mriedem - rocky, phys_ram=32141MB used_ram=0MB, reserved_host_memory_mb = 0 in nova.conf
18:12:44 Nick_A we found https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html which explains why 32768 doesn't work, but we're not seeing why 32000 is also failing
18:17:27 mriedem are you using the RamFilter or AggregateRamFilter?
18:21:55 redkrieg yes, we do have the AggregateRamFilter active. I don't see this host being filtered by it though.
18:25:07 openstackgerrit Merged openstack/nova master: libvirt: Remove MIN_{QEMU,LIBVIRT}_LUKS_VERSION https://review.opendev.org/668924
18:31:18 openstackgerrit Merged openstack/nova-specs master: Fix the cross_az_attach value in AZ-unshelved spec https://review.opendev.org/669261
18:34:48 mriedem the reason i ask is because those filters return a "limit" from the scheduler which gets passed down to the compute which is part of the resource claim,
18:35:00 mriedem and if the scheduler is passing down a limit for something that you don't expect, that might be the cause
18:35:39 mriedem https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios
18:37:17 mriedem iow, what you think and what placement is using might be different allocation ratios
18:37:28 mriedem hopefully that doc gives some things to look at
18:48:34 Nick_A It actually looks like we're dealing with some sort of qemu/kernel bug... It successfully attempts a claim on compute if we set the flavor RAM less than the physical RAM, but libvirt errors at certain amounts with "qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory."
19:15:56 dansmith Nick_A: using anything that requires pinned memory? cpu pinning, huge pages, etc?
19:16:36 dansmith also, even if you have 32g of memory, you can't necessarily expect to land a 30g instance there if you have no swap and have a large enough footprint of other stuff
19:17:04 dansmith but, if you're getting to libvirt, then you're out of the realm of it being a nova thing
19:17:21 Nick_A we are doing cpu pinning on here - yeah, I think the only nova issue was the hard limit based on physical ram
19:17:46 Nick_A I am curious why that's the case if you happen to know
19:18:17 Nick_A we have non-openstack centos libvirt nodes that are happy to create KVMs with over 32GB on the same hardware
19:18:19 dansmith do you also have a numa topology specified?
19:18:34 dansmith there's no 32g limit specifically of course
19:19:16 Nick_A "Regardless of the overcommit ratio, an instance can not be placed on any physical node with fewer raw (pre-overcommit) resources than the instance flavor requires." Am I misunderstanding that?
19:19:40 Nick_A re: numa - we're using openstack-ansible defaults, not sure
19:21:14 Nick_A I got that from https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html
19:21:29 dansmith no, I'm just saying there's no specific 32g limit. nova's overcommit is for landing multiple instances on a node, however ill-advised
19:21:47 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old TODO about forced_host policy check https://review.opendev.org/669474
19:22:08 dansmith a single instance larger than physical memory is silly, so yeah, no support for it
19:22:41 Nick_A right, but there is a limit based on the physical RAM preventing a single instance from being allocated above that amount. I'm curious why it allows multiple instances to do so but not a single one.
19:23:50 dansmith it allows it because it was designed that way for earlier cloud providers that did it for cost scale, but it's never a good idea for performance, obviously, as you're just using host disk for the overage
19:24:13 dansmith a single instance that is larger than the host doesn't make sense, you might as well run a smaller guest and let it do its own swapping
19:25:10 dansmith anyway, if you're doing cpu pinning you might be running up against something that requires page pinning which is getting you to the "cannot allocate memory" thing
19:25:18 dansmith I doubt it's a kernel/qemu bug
19:25:36 dansmith best bet would be to ask around monday morning when more of the folks that are familiar with that stuff are around
19:25:42 Nick_A thank you
19:25:43 dansmith sean, sfinucan, etc
19:43:47 openstackgerrit Corey Bryant proposed openstack/os-resource-classes master: Add Python 3 Train unit tests https://review.opendev.org/669479
19:44:20 openstackgerrit Corey Bryant proposed openstack/os-traits master: Add Python 3 Train unit tests https://review.opendev.org/669480
20:15:16 mriedem bauzas: how is your audit command coming? it's been a couple of weeks and this came up in the ML today http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007534.html
20:35:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove Newton-era min compute checks for server create with device tags https://review.opendev.org/669523
20:37:36 mriedem dansmith: is https://review.opendev.org/#/c/667177/ just waiting for my eye-ball diff and if we're good, fix the thing you noted and then we can move forward?
20:37:48 dansmith mriedem: yeah
21:16:46 mriedem ok comments in there now
21:17:24 mriedem also - https://review.opendev.org/#/c/668631/ - it's a good thing we didn't do the interface change in the same patch...
21:31:24 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Init HostState.failed_builds https://review.opendev.org/669538
21:37:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove Rocky-era min compute trusted certs compat check https://review.opendev.org/669539
22:02:18 sean-k-mooney Nick_A: im not sure if this is the issue but i do not belive we allow vms to oversubsribe resouces against itself.
22:02:41 sean-k-mooney in placment the max_unit for MEMORY_MB should be set to the total capasity of the host
22:03:10 sean-k-mooney so we will not allow vms that request more ram then the host has to be schduled to a node
22:03:19 sean-k-mooney the same is true from ram and disk
22:03:54 sean-k-mooney i havent read the full scrollback but that is the first limitation that came to mind
22:05:09 sean-k-mooney reading back
22:05:22 sean-k-mooney we reserve 512MB of ram for the host by default
22:08:04 openstackgerrit Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451
22:08:04 openstackgerrit Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012
22:08:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013
22:08:05 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source https://review.opendev.org/637058
22:08:06 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.opendev.org/637070
22:08:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075
22:08:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.opendev.org/637316
22:08:07 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add revert_snapshot_based_resize_at_dest compute method https://review.opendev.org/637630
22:08:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations https://review.opendev.org/639453
22:08:08 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add finish_revert_snapshot_based_resize_at_source compute method https://review.opendev.org/637647
22:08:09 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.opendev.org/638046
22:08:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.opendev.org/638047
22:08:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.opendev.org/638048
22:08:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.opendev.org/638268
22:08:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.opendev.org/651650
22:08:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.opendev.org/614353
22:08:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional test for anti-affinity cross-cell migration https://review.opendev.org/661859
22:08:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478
22:10:32 Nick_A sean-k-mooney thanks
22:11:53 sean-k-mooney i would have to think about it some more but im guessing you are hiting one of the limit we have to make sure a single vm does not exceed the total amount of one of the resouce we have on the host
22:30:06 Nick_A yeah that's clearly why 32768 wasn't working. We were just surprised it had a limit of that sort when the RAM allocated to multiple smaller instances could still cross that threshold.
22:36:17 sean-k-mooney i think the qemu error is proably a similar similar safte check
22:37:25 sean-k-mooney the os and qemu both need some ram to work properly. so there is some overhad per host that should not be allocated to a guest unless you have enough swap space
22:37:42 sean-k-mooney but even then i generally do not recommend ram over commit at all
22:38:40 sean-k-mooney it causes more issues the in sovles as the kernel OOM killer will reap the vms first if it need to allocate memory and it can swap the vm memory to make room
22:54:08 openstackgerrit Merged openstack/nova master: libvirt: Remove MIN_LIBVIRT_PERF_VERSION https://review.opendev.org/669128
23:00:14 openstackgerrit Merged openstack/nova master: libvirt: vif: Remove MIN_LIBVIRT_MACVTAP_PASSTHROUGH_VLAN https://review.opendev.org/669176
#openstack-nova - 2019-07-06
07:33:33 openstackgerrit garyk proposed openstack/nova master: Nova compute: add in log eception to help debug failures https://review.opendev.org/669545
#openstack-nova - 2019-07-07
00:05:16 spatel sean-k-mooney: can you take a look here https://bugs.launchpad.net/nova/+bug/1835637
00:05:16 openstack Launchpad bug 1835637 in OpenStack Compute (nova) "(404) NOT_FOUND - failed to perform operation on queue 'notifications.info' in vhost '/nova' due to timeout" [Undecided,New]
00:57:46 openstackgerrit Merged openstack/nova master: Set COMPUTE_STATUS_DISABLED trait from update_provider_tree flow https://review.opendev.org/668565
16:31:53 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing os-agents policies https://review.opendev.org/669578
22:05:59 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.opendev.org/576020
22:06:11 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027
22:06:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031
22:06:43 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299
22:06:59 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344
22:07:17 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673
22:07:34 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.opendev.org/576676
22:07:52 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.opendev.org/576689
22:08:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.opendev.org/576709
22:08:30 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.opendev.org/576712
22:08:51 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens https://review.opendev.org/637716
#openstack-nova - 2019-07-08
00:39:20 openstackgerrit Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.opendev.org/631242

Earlier   Later