| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-19 | |||
| 21:22:35 | mriedem | :( | |
| 21:22:54 | lbragstad | but i'm trying to update that with all context of what i've found across projects (since operators are on that note, too) | |
| 21:24:44 | mriedem | so you'd like to see compute:server:cross_cell_resize | |
| 21:24:56 | mriedem | note this is an action, | |
| 21:25:01 | mriedem | so it won't cleanly map to a method | |
| 21:25:15 | lbragstad | right | |
| 21:25:29 | lbragstad | which i guess is another reason to not put http methods in policy names? | |
| 21:25:32 | mriedem | i mean, we could do compute:servers:resize:cross_cell | |
| 21:26:29 | lbragstad | yeah - i don't think nova would be super special in that case... there are other projects have more than just <service>:<resource>:<action> | |
| 21:26:54 | lbragstad | ironic and heat for example | |
| 21:27:09 | mriedem | alright, this is months/releases away from landing so i'll just do that and move on | |
| 21:27:20 | mriedem | this is the amount of progress i've made on this since last week | |
| 21:27:47 | lbragstad | what's months away from landing? | |
| 21:27:57 | mriedem | cross-cell resize | |
| 21:28:01 | lbragstad | oh - sure | |
| 21:28:04 | mriedem | not your thing | |
| 21:28:26 | mriedem | being cross project, your thing is *years* away from landing :) | |
| 21:28:28 | lbragstad | i'd like to get the convention "established" as soon as possible, because I think it's going to affect gmann's work too | |
| 21:28:35 | lbragstad | right - let's be real here | |
| 21:41:03 | mnaser | is there a documented process for when you end up with: "Instance 13a09ff5-7124-4b3c-8229-a8691a2f43b4 has allocations against this compute host but is not found in the database." | |
| 21:41:35 | mnaser | context: dns was borked on a compute, live migration would complete, but the placement migration allocation thing fails because compute cant talk to placement | |
| 21:43:34 | mriedem | that doesn't sound like a scenario in which you'd see that message | |
| 21:44:01 | mnaser | well, it happened after fixing dns | |
| 21:44:16 | mnaser | the compute node finally started talking to placement properly and now complains forever | |
| 21:44:30 | mriedem | is that instance in the db? | |
| 21:44:52 | mnaser | hmm | |
| 21:45:02 | mriedem | the code calls out 2 cases that could happen: | |
| 21:45:03 | mnaser | no, i think the exception i was seeing was unrelated | |
| 21:45:08 | mnaser | i think it was no placement + vm deleted | |
| 21:45:23 | mnaser | i think i jumped to a conclusion there | |
| 21:45:25 | mriedem | 1. compute RT is racing with the scheduler where the scheduler created allocations in placement but didn't yet create the instance in the cell db, | |
| 21:45:45 | mriedem | 2. the instance was deleted and archived/purged from the db, but the alloctions are still in placement for that node | |
| 21:45:52 | mriedem | yeah, | |
| 21:46:03 | mriedem | so if the compute couldn't talk to placement when the vm was deleted, we'd fail to cleanup the allocations | |
| 21:46:32 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete | |
| 21:46:44 | mnaser | thanks for making life easy and writing those clie stuf | |
| 21:46:46 | mriedem | openstack resource provider allocation delete 13a09ff5-7124-4b3c-8229-a8691a2f43b4 | |
| 21:46:57 | mriedem | if the instance is truly deleted | |
| 21:47:13 | mriedem | if it were moved and allocations were just messed up, doing ^ would be bad | |
| 21:47:42 | mnaser | ill double check they're all deleted | |
| 21:47:45 | mriedem | might be good to update that log message with the same info | |
| 21:47:55 | mnaser | tail -1000 /var/log/nova/nova-compute.log | grep 'has allocations against this compute host but is not found in the database.' | cut -d' ' -f13 | sort | uniq | xargs -n1 echo nova show | |
| 21:47:56 | mnaser | magic | |
| 21:48:30 | mriedem | "If you continue to see this message and have confirmed the instance is truly gone from the database, you can run 'openstack resource provider allocation delete <uuid>' to remove the allocation from the placement service." | |
| 21:48:46 | mnaser | yeah just even document it somewhere i guess | |
| 21:48:54 | mnaser | i mean | |
| 21:48:56 | mriedem | well, | |
| 21:48:58 | mnaser | couldnt nova confirm that it doesnt exist in the db | |
| 21:49:05 | mriedem | not if it's a race | |
| 21:49:06 | mnaser | and technically just delete it when it polls | |
| 21:49:09 | mnaser | ah | |
| 21:49:21 | mnaser | i guess we'd have to check N times that it's not there but that starts becoming silly i guess | |
| 21:49:26 | mriedem | as noted, we create the allocations in scheduling before the instance is created in the cell db | |
| 21:49:33 | mriedem | right | |
| 21:49:47 | mnaser | or if its been running for a while | |
| 21:49:53 | mriedem | putting it into some kind of faq could work, but people might not see that and come here asking the same thing | |
| 21:49:59 | mnaser | like 300s age | |
| 21:50:08 | mnaser | 300s isn't a race condition at that point i guess | |
| 21:50:24 | mnaser | but i guess log is the easiest path | |
| 21:50:31 | mriedem | rather than nova auto-delete your stuff, i'd prefer to just document weird edge cases for manual intervention | |
| 21:51:07 | mriedem | we've also talked about (last week it came up too) a command to compare placement allocations against nova instances, | |
| 21:51:13 | mriedem | the opposite of heal_allocations, | |
| 21:51:24 | mriedem | more of a "tell me what garbage is in the placement db b/c nova didn't clean up after itself" | |
| 21:51:54 | mriedem | but we can't reliably do that until we have consumer types in placement... | |
| 21:54:47 | mriedem | takashin: is this todo still valid https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3509 given we check the specified host is in the same cell above? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3405 | |
| 21:55:55 | mriedem | i think we can remove that todo; when we call objects.ComputeNode.get_first_node_by_host_for_old_compat the context is targeted to the cell in which the instance currently exists | |
| 21:56:08 | mriedem | so if the specified host is in another cell, we'll get ComputeHostNotFound there | |
| 21:58:57 | mriedem | johnthetubaguy: we should remove this rebuild parameter from the conductor migrate_server task API :) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L263 | |
| 21:59:12 | mriedem | that must have been something someone talked about way back in 2013 to unify the various move APIs | |
| 22:10:08 | takashin | mriedem: you are right. We can remove the 'todo'. | |
| 22:10:42 | mriedem | takashin: please push a change to remove it and i'll +2 | |
| 22:11:11 | takashin | mriedem: okay. I will submit the patch. | |
| 22:13:52 | vishwanathj | hi looking for guidance on how to limit the number of instances on certain set of hosts... | |
| 22:15:06 | vishwanathj | not clear on sequence of steps to be executed on controller and compute nodes to get this working | |
| 22:31:47 | mriedem | there is a NumInstancesFilter | |
| 22:32:11 | mriedem | vishwanathj: https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#numinstancesfilter | |
| 22:32:25 | mriedem | and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregatenuminstancesfilter for aggregates | |
| 22:33:16 | openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | |
| 22:33:53 | mriedem | there is a known bug with those filters https://bugs.launchpad.net/nova/+bug/1740320 | |
| 22:33:53 | openstack | Launchpad bug 1740320 in OpenStack Compute (nova) "nova-scheduler does not honor max_instances_per_host set to a host aggregate" [Undecided,Confirmed] | |
| 22:34:01 | mriedem | race bug | |
| 22:47:40 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove an unnecessary comment https://review.openstack.org/603924 | |
| 22:49:34 | takashin | mriedem: I submitted https://review.openstack.org/#/c/603924 . | |
| 22:57:15 | sorrison | Anyone know how to re kick a gate job that failed due to a timeout. https://review.openstack.org/#/c/526558/ I tried a recheck but that aint it | |
| 22:57:42 | sorrison | mriedem: ^ | |
| 23:02:34 | mriedem | sorrison: the gate has been eating a ton of dairy | |
| 23:02:49 | mriedem | plug 526558 into http://zuul.openstack.org/ | |
| 23:03:06 | mriedem | it's queued up | |
| 23:03:54 | mriedem | http://lists.openstack.org/pipermail/openstack-dev/2018-September/134867.html | |
| 23:04:48 | sorrison | ok thanks | |
| 23:19:24 | mriedem | takashin: got it, thanks | |
| 23:20:11 | takashin | mriedem: Thank you for your review. | |
| 23:40:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930 | |
| 23:40:39 | mriedem | ship it! | |
| 23:52:34 | gmann | lbragstad: i will check the ML today and get back to you. | |
| #openstack-nova - 2018-09-20 | |||
| 00:57:24 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549 | |
| 01:07:32 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova-specs master: Spec: allocation candidates in tree https://review.openstack.org/603585 | |
| 01:12:57 | openstackgerrit | Tao Li proposed openstack/python-novaclient master: Remove the unused instance-name https://review.openstack.org/602520 | |
| 01:35:04 | lbragstad | gmann sounds good | |
| 02:33:47 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955 | |