| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-14 | |||
| 15:30:00 | hamzy | hey mriedem I think that I am running into https://bugs.launchpad.net/nova/+bug/1670627 is there a way to clean up the instances from the CLI? I guess that I need to do some DB cleanup? | |
| 15:30:56 | dansmith | bauzas: okay, I need to see how much change that is to the other stuff, but probably okay | |
| 15:31:16 | bauzas | dansmith: it's just a nit tho | |
| 15:31:31 | bauzas | dansmith: but I'd prefer both of them rather than a filter | |
| 15:31:59 | mriedem | hamzy: the instances are deleted right? | |
| 15:32:13 | mriedem | hamzy: the problem is the quota usage isn't decremented | |
| 15:32:17 | hamzy | mriedem, yes "nova list --all-tenants" shows nothing | |
| 15:32:32 | hamzy | My compute overview page shows "RAM Used 48GB of 50GB" and 6 used instances... | |
| 15:32:57 | dansmith | bauzas: yeah, I don't know why I didn't think to do this when johnthetubaguy and I were talking about it.. probably just misunderstanding the cachingscheduler at the time | |
| 15:33:06 | mriedem | hamzy: yeah, and the "nova_cell0.usages" table should have that information in it, can you confirm | |
| 15:33:07 | mriedem | ? | |
| 15:33:14 | mriedem | oh wait, sorry, | |
| 15:33:16 | mriedem | not nova_cell0, | |
| 15:33:19 | mriedem | just nova.usages | |
| 15:33:34 | bauzas | dansmith: the problem with a filter is that operators would need to modify their config for an upgrade | |
| 15:33:38 | hamzy | mriedem, I have an AIO installation with a bunch of containers... do I need to attach to one of them> | |
| 15:33:45 | bauzas | dansmith: or they wouldn't check the cells | |
| 15:34:03 | bauzas | if they forget to modify their conf opt | |
| 15:34:09 | dansmith | bauzas: they wouldn't until they add another cell | |
| 15:34:10 | mriedem | hamzy: let me check something, we might have a CLI to run | |
| 15:34:19 | dansmith | bauzas: but I know, this is better, I'm not arguing | |
| 15:34:30 | bauzas | dansmith: sure, but they could also opt-out verifying cells, which I also dislike :) | |
| 15:34:40 | bauzas | anyway | |
| 15:34:42 | mriedem | cdent: replied to the ML thread on the placement client topic for completeness | |
| 15:34:51 | bauzas | mriedem: thanks for that, appreciatedf | |
| 15:34:52 | cdent | mriedem: yeah, saw that, thanks | |
| 15:34:52 | mriedem | and so i don't forget what we agreed on in 4 months time | |
| 15:34:53 | dansmith | bauzas: I'm not arguing for a filter | |
| 15:35:05 | bauzas | yeah no worries :) | |
| 15:35:20 | mriedem | hamzy: https://docs.openstack.org/developer/nova/man/nova-manage.html#nova-project | |
| 15:35:29 | mriedem | hamzy: i'm wondering if "nova-manage project quota_usage_refresh <project_id> [--user <user_id>] [--key <key>]" will help | |
| 15:35:47 | mriedem | hamzy: can you try that for the project in question | |
| 15:36:39 | hamzy | yeah let me figure out the project_id | |
| 15:38:25 | hamzy | mriedem, is there an openstack CLI equivalent? | |
| 15:38:32 | mriedem | hamzy: nope | |
| 15:38:35 | mriedem | it's direct to the nova db | |
| 15:38:53 | mriedem | and something in osc cli would require a rest api for this in the compute endpoint, which we don't have | |
| 15:41:06 | hamzy | mriedem, trying to figure out how to install it in my AIO environment | |
| 15:41:35 | mriedem | hamzy: nova-manage is available on the controller | |
| 15:41:42 | mriedem | it's a CLI | |
| 15:41:46 | mriedem | so "which nova-manage" | |
| 15:42:15 | mriedem | AIO == all in one? | |
| 15:42:20 | hamzy | yes | |
| 15:42:27 | hamzy | installed with openstack-ansible | |
| 15:44:06 | hamzy | mriedem, http://paste.openstack.org/show/602707/ that CLI is not in the utility-container | |
| 15:44:44 | mriedem | hamzy: i've never used openstack-ansible, so not sure how things are deployed, | |
| 15:45:09 | mriedem | hamzy: how about aio1_nova_conductor_container-41603c3d ? | |
| 15:45:55 | johnthetubaguy | not sure which container they run the nova-manage stuff in, worth asking andymccr in #openstack-ansible | |
| 15:45:57 | hamzy | nova command is there but not nova-manage | |
| 15:46:13 | hamzy | same with aio1_nova_console_container-fb9219e4... sigh | |
| 15:46:25 | andymccr | hamzy: it will be in the nova containers, but inside the venv for nova | |
| 15:46:34 | hamzy | ah | |
| 15:47:57 | hamzy | \o/ | |
| 15:48:54 | mriedem | now to see if that command fixes the usage | |
| 15:51:09 | raj_singh | johnthetubaguy: regarding https://review.openstack.org/#/c/438467/, I am wondering if new "force_complete" option could have choices of "abort, pause, post-copy", then we can deprecate live_migration_permit_post_copy? | |
| 15:51:44 | johnthetubaguy | raj_singh: but what about the force complete API that still uses that config? | |
| 15:51:47 | hamzy | mriedem, I don't think it did, but the overview page no longer shows the pie charts anymore | |
| 15:52:24 | johnthetubaguy | raj_singh: I think folks want to opt into post copy, most want it turned off, so a single config to do that makes sense I think | |
| 15:53:07 | raj_singh | johnthetubaguy: ok, let me rework the spec | |
| 15:53:29 | mriedem | hamzy: ok, we can check via nova cli, sec | |
| 15:54:27 | hamzy | mriedem, FYI I ran quota_usage_refresh on all four projects that I had | |
| 15:54:36 | mriedem | hamzy: https://docs.openstack.org/cli-reference/nova.html#nova-quota-show | |
| 15:54:37 | johnthetubaguy | raj_singh: we should add that context in the spec | |
| 15:54:51 | mriedem | nova quota-show --tenant <tenant id> | |
| 15:54:59 | mriedem | plus maybe --detail | |
| 15:55:05 | mriedem | nova quota-show --tenant <tenant id> --detail | |
| 15:55:16 | mriedem | see if the instances are back to 10 for limit and 0 for usage | |
| 15:55:37 | raj_singh | johnthetubaguy: yes, will update it soon. | |
| 15:57:31 | hamzy | mriedem, http://paste.openstack.org/show/602711/ | |
| 15:58:22 | johnthetubaguy | raj_singh: got a few nits on the follow on spec too | |
| 15:58:32 | mriedem | | instances | {u'limit': 10, u'reserved': 0, u'in_use': 6} | | |
| 15:58:40 | mriedem | yeah usage is still wrong | |
| 15:58:51 | raj_singh | johnthetubaguy: thanks for the review | |
| 16:00:09 | mriedem | hamzy: is the [database]/connection option in the nova.conf that the aio1-nova-conductor-container-41603c3d container is using pointing at the 'nova' database? | |
| 16:01:34 | openstackgerrit | sahid proposed openstack/nova master: numa: fit instance NUMA node with cpus reserved onto host NUMA node https://review.openstack.org/400875 | |
| 16:01:34 | openstackgerrit | sahid proposed openstack/nova master: libvirt: return a CPU overhead if isolate emulator threads requested https://review.openstack.org/385364 | |
| 16:01:36 | openstackgerrit | sahid proposed openstack/nova master: virt: update overhead to take into account vCPUs https://review.openstack.org/385024 | |
| 16:01:39 | hamzy | mriedem, there is a line "connection = mysql+pymysql://nova:random_password@172.29.236.100/nova?charset=utf8" | |
| 16:01:42 | openstackgerrit | sahid proposed openstack/nova master: libvirt: make emulator threads to run on the reserved pCPU https://review.openstack.org/400878 | |
| 16:01:44 | openstackgerrit | sahid proposed openstack/nova master: numa: take into account cpus reserved https://review.openstack.org/400876 | |
| 16:01:46 | openstackgerrit | sahid proposed openstack/nova master: numa: update numa usage incording the cpu reserved https://review.openstack.org/400877 | |
| 16:02:00 | mriedem | hamzy: yeah that's the nova db | |
| 16:02:06 | mriedem | /nova is the nova db | |
| 16:04:19 | mriedem | so i'm thinking the refresh CLI doesn't fix things because the refresh operation thinks that usage is correct, because of the bug, we never decremented usage | |
| 16:04:37 | hamzy | so no --force option? :) | |
| 16:05:21 | mriedem | nova quota-update will just update the limits, not the usages, | |
| 16:06:00 | mriedem | i'm not sure if "nova quota-delete" will do what you want, i suspect that will reset quota limits to the default, but not touch usages, | |
| 16:06:02 | mriedem | i can look quick | |
| 16:06:13 | hamzy | thanks | |
| 16:07:15 | mriedem | actually it looks like it will, | |
| 16:07:28 | mriedem | it should delete the quota limit, usage and any pending reservations | |
| 16:08:01 | mriedem | it sucks, but if you can reset the quota for that project, then that's what you want "nova quota-delete --tenant <tenant id>" | |
| 16:08:33 | hamzy | mriedem, and then how do you recreate the quota? I don't see a quota-create | |
| 16:08:53 | mriedem | there are global defaults for quota in nova.conf | |
| 16:09:12 | mriedem | now might be a good time for me to cleanup https://review.openstack.org/#/c/417563/ | |
| 16:09:21 | mriedem | but ^ is an attempt and documenting some of how quotas work in nova | |
| 16:11:07 | openstackgerrit | Tadas Ustinavičius proposed openstack/nova master: Fixing flake8 errors https://review.openstack.org/445558 | |
| 16:12:10 | melwitt | mriedem: no, didn't do anything with a functional test yet. so feel free :) | |
| 16:12:59 | mriedem | melwitt: i think i'm going to just get ocata 13.0.1 out the door for this quotas issue, and add a release note with a known issue for the retry bug, | |
| 16:13:09 | mriedem | and then once that's fixed (with a functional recreate test) we can do a 13.0.2 shortly after | |