Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-12
20:41:18 openstack Launchpad bug 1716593 in OpenStack Compute (nova) "reservations table in nova mysql db grows constantly" [Undecided,New]
20:41:35 mriedem nova-manage db archive_deleted_rows
20:41:39 bauzas I can surely ask to run nova-manage, but a link could be awesome
20:41:43 bauzas mriedem: yup I know
20:41:47 bauzas mriedem: I just wanted some docs :)
20:42:10 mriedem http://paste.openstack.org/show/620973/
20:42:28 mriedem bauzas: https://docs.openstack.org/nova/latest/cli/nova-manage.html#nova-database
20:43:27 bauzas mriedem: ta, adding it to the bug report
20:44:40 bauzas mriedem: interesting, trying locally
20:45:22 mwynne Hi guys. I'm trying to allocate an instance that would overcommit RAM but remain within my ratio and it fails. This document (https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html) states that you can overcommit RAM, but the "Note" is basically saying you can't overcommit RAM. Am I missing something here? Something doesn't add up to me.
20:45:40 mriedem dansmith: hey, we're friends right?
20:45:53 dansmith mriedem: um, yes?
20:45:55 mriedem https://review.openstack.org/#/c/501025/ <3
20:46:41 cdent mwynne: you’re not trying to allocate more virtual RAM than physical RAM are you?
20:47:31 mwynne cdent: I'm trying to allocate a VM that would push me over the limit of how much total physical RAM I have on my compute nodes.
20:47:43 mwynne But isn't that what overcommit is enabling me to do?
20:47:49 mwynne within the specified ratio?
20:47:56 cdent mwynne: for multiple vms, yes, but not for a single vm
20:48:21 mwynne ?
20:48:26 bauzas mriedem: oh snap, you just need to pass things after your venv call
20:48:31 bauzas hence the indexerror
20:48:43 bauzas something like tox -e venv -- do_something
20:49:08 mwynne cdent: I have something like 4GB free and I'm trying to allocate a VM with 8GB. Is that not allowed?
20:50:45 mriedem stephenfin: do you know of a way with nova-manage to say if an arg is an option or an argument (required)?
20:50:46 cdent mwynne: correct that is not allowed, since ocata
20:50:48 bauzas mriedem: I do confirm running something like tox -evenv -- python -i myscript.py works like a charm
20:50:58 cdent mwynne: and maybe before, but the mechanism for why has changed
20:51:01 mwynne cdent: I'm running newton..
20:51:17 mriedem bauzas: never had to do that before
20:51:32 bauzas ?
20:51:52 bauzas I never saw tox -evenv without a following string
20:51:55 bauzas :)
20:51:58 cdent mriedem, bauzas, can you comment on the state of things in newton for mwynne ?
20:52:11 cdent mwynne: in any case: whatever the limitations, you don’t want to do that
20:52:14 bauzas about overcommitting ?
20:52:26 mwynne cdent: So if I have VMs powered down they're still "taking up memory" from nova's perspective?
20:52:26 cdent bauzas: the mechanism thereof prior to ocata
20:52:35 bauzas well in Newton, we did changed a bit where allocation ratios were defined
20:53:10 bauzas nova.conf ratio opts were defaulted to be read on compute nodes, else in the scheduler
20:53:36 bauzas apart from that, I don't see a problem with defining a ratio of 1.5 for the RAM and trying to allocate more than what you have
20:54:12 bauzas but asking for 8GB when you have a ratio of 1.5 and a physical host of 4GB isn't mathematically acceptable :)
20:55:21 bauzas I remind the default ratios : 16.0 for CPU, 1.5 for RAM and 1.0 for disk (well, the latter isn't really fully workable)
20:55:56 mwynne bauzas: The default for newton seems to be 0.0 for RAM
20:56:16 openstackgerrit Dan Smith proposed openstack/nova master: Add nova-manage db command for ironic flavor migrations https://review.openstack.org/501025
20:56:44 dansmith mriedem: ^
20:56:47 bauzas mwynne: that's a flag for expressing that you need to define your ratios in the compute nova.conf
20:56:56 bauzas mwynne: but programmatically, it'll respect 1.5
20:57:03 bauzas (if 0.0 is set by defaultà)
20:57:56 mwynne bauzas: Ok. I've been looking at the number of free huge pages on my compute nodes, and comparing that to what "nova hypervisor-stats" says, and the math doesn't check out.
20:58:18 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/502700
20:58:18 mwynne I have tons of free pages on my nodes, but that cmd says I have 4 free gb of RAM. Something doesn't seem right there.
20:58:59 mwynne (And yes I'm using a flavor with HP enabled)
20:59:13 openstackgerrit Merged openstack/nova master: Handle keypair not found from metadata server using cells https://review.openstack.org/476122
20:59:35 bauzas mwynne: sec, you want to use huge pages ?
20:59:42 mwynne yes
20:59:50 bauzas okay, that's a separate thing
21:00:07 mwynne Ah, ok. Sorry..
21:00:36 bauzas no worries, huge pages in Nova isn't tracked the same way that a regular RAM consumption
21:01:24 bauzas mwynne: https://docs.openstack.org/nova/pike/admin/huge-pages.html is the one I have in mind
21:02:03 bauzas mwynne: have you followed that and modified your flavors by mentioning the mem_page_size for the instance you want to create ?
21:02:21 mwynne yes
21:02:25 bauzas even if that's a Pike doc, the flavor thing is identical in Newton
21:02:42 mwynne Yeah, I've done that.
21:02:48 bauzas mwynne: ok, so you did that, you wanna boot an instance and you end up having NoValidHosts ?
21:03:09 openstackgerrit Merged openstack/nova master: Create allocations against forced dest host during evacuate https://review.openstack.org/499399
21:03:19 mwynne Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology
21:03:36 openstackgerrit Merged openstack/nova master: Add recreate test for evacuate claim failure https://review.openstack.org/499874
21:03:42 bauzas mwynne: I actually wonder if huge pages supports and ratios can be mixed up
21:03:53 mwynne bauzas: I deleted some unused vms and then I could spawn it.
21:04:01 bauzas in general, people using huge pages are not wanting to have overallocated instances
21:04:14 bauzas so they default ratios to 1.0
21:04:44 bauzas from what I remember about the huge pages implementation, I know we don't really care of any overallocation
21:04:57 bauzas so I wouldn't be surprised if that wouldn't fit
21:05:05 openstackgerrit Merged openstack/nova master: Add a test to make sure failed evacuate cleans up dest allocation https://review.openstack.org/499877
21:05:14 mwynne Yeah. I assumed that.. What I want to be able to do is basically this: Use heat to spin up a really big stack, consuming almost all my RAM. When I'm finished with it, just power them all off so I don't have to re-deploy each time. I was hoping that having them powered down meant I could deploy and use other vms until I want to spin them all back up again.
21:05:37 openstackgerrit Merged openstack/nova master: Remove dest node allocation if evacuate MoveClaim fails https://review.openstack.org/499878
21:06:10 melwitt mwynne: there was a bug around that but the fix was backported to newton https://bugs.launchpad.net/nova/+bug/1635367
21:06:11 openstack Launchpad bug 1635367 in OpenStack Compute (nova) ocata "Ram filter is broken since Mitaka" [High,In progress] - Assigned to Matt Riedemann (mriedem)
21:06:48 melwitt mwynne: are you using at least version 14.0.8?
21:06:59 mwynne melwitt: how can I check that?
21:07:57 mwynne melwitt: 14.0.7... Nooooooo! haha
21:08:34 melwitt mwynne: okay, hopefully if you pull down 14.0.8 it will fix your problem but I'm not 100% sure that's what's causing what you're seeing with NUMA etc
21:09:05 bauzas melwitt: oh good catch, I forgot it
21:09:22 mwynne What's the best way to update? I'm running RDO packages.
21:09:42 bauzas mwynne: just upgrade your packages if you're on a 14.0.z version already
21:09:47 bauzas and restart your services
21:10:01 mwynne bauzas: Ok. I figured.. Just wasn't sure if that would break stuff :S
21:10:04 bauzas but yeah, I just feel we don't mix up overcommits and huge pages
21:11:08 mwynne bauzas: Ok. I'
21:11:23 mwynne I'll see if updating helps. But I think you might be right about huge pages...
21:11:34 mwynne Thanks for all the help guys. I appreciate it :)
21:26:40 efried mikal FYI, https://review.openstack.org/#/c/503078/ totally doesn't work. I had a feeling... it's the difference, when we hijack the namespace, of putting our code in a new module within an existing package, versus making a new package.
21:27:14 efried mikal I suppose I could try structuring it as nova/virt/powervm/__init__.py <== with the code in (or at least exported from) here.
21:35:14 openstackgerrit Michael Still proposed openstack/nova master: Fix missed chown call https://review.openstack.org/503079
21:35:21 efried mikal (and esberglu edmondsw) we'll see if this flies: https://review.openstack.org/#/c/503078/
21:37:44 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/502700
21:38:44 med_ did you just find that by grep'ing for dangling chowns?
21:38:51 med_ smells that way
21:40:29 efried mikal BTW, why do we have to be in the nova.privsep namespace to make this work?
21:40:41 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708

Earlier   Later