| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-12 | |||
| 00:17:07 | mriedem | but we can also be smarter and check quota before trying to create volumes | |
| 00:17:09 | mriedem | like we do for ports | |
| 00:17:27 | mnaser | mriedem: yeah, i think that's better, do we check quota at api layer for ports? | |
| 00:17:29 | mriedem | could do that in conductor so we don't block the API response | |
| 00:17:33 | mriedem | mnaser: yes | |
| 00:17:40 | mriedem | it's the validate_networks call | |
| 00:17:50 | mnaser | that sounds reasonable and backport-able too i think | |
| 00:17:52 | mriedem | based on the number of requested networks and instances, we check quota | |
| 00:18:22 | mnaser | the thing is if i have a quota of 10 volumes, and i launch a 100 instances (one by one), i might still run into that issue i guess | |
| 00:18:23 | mriedem | yeah - either way you'd fail after the 202 you get from the API | |
| 00:18:31 | mriedem | you either fail in conductor or you fail in compute | |
| 00:18:45 | mriedem | sure but we don't disable your compute :) | |
| 00:18:54 | mnaser | because of the race as conductor still sees you using 0 volumes but as computes start creating volumes | |
| 00:19:05 | mriedem | oh, sure | |
| 00:19:23 | mriedem | we could also just handle volume quota issues in the compute as a whitelist of things to not disable the compute | |
| 00:19:29 | mriedem | multi-part fixes | |
| 00:19:49 | mnaser | i think both out-of-ports and out-of-volumes both seem like reasonable 'skip' failures | |
| 00:20:01 | mriedem | yeah | |
| 00:20:11 | mriedem | the bug above started as a port / fixed ips quota issue | |
| 00:20:15 | mriedem | but also extends to volumes | |
| 00:20:21 | mnaser | i think any api exception should be skipped tbh | |
| 00:20:29 | mnaser | because that means the compute node is fine | |
| 00:20:51 | mnaser | sure it sounds vague but if your cinder is not having a good time then you're slowly disabling all compute nodes | |
| 00:20:52 | mriedem | well, depends on what it is, | |
| 00:21:05 | mriedem | if nova.conf is misconfigured on the compute to talk to neutron, that's a different problem | |
| 00:21:08 | mriedem | likely a 403 | |
| 00:21:21 | mnaser | ah yes | |
| 00:21:41 | mriedem | if we get NeutronClientException, then we made a request and it failed but the endpoint, config and token should be ok | |
| 00:21:57 | mriedem | same with CinderClientException | |
| 00:22:19 | mnaser | so we can skip those 2 exceptions because if we get them, it doesn't mean that the compute node has any problems | |
| 00:22:55 | mriedem | well, if it did turn out that excluding those were too broad, we could narrow it down over time | |
| 00:23:06 | mriedem | vif plug is something that could fail... | |
| 00:23:10 | mriedem | based on bad config of the host | |
| 00:23:17 | mriedem | that's a hard one though | |
| 00:23:46 | mriedem | because we don't get vif plug failures directly | |
| 00:23:59 | mriedem | we get a callback event from neutron that just says plug failed but not why | |
| 00:24:19 | mriedem | but yeah, that's not a NeutronClientException, so ignore me | |
| 00:24:21 | mnaser | i can't imagine a single time that i remember seeing a failed to plug callback | |
| 00:24:27 | mnaser | it either never comes or comes ok :P | |
| 00:24:29 | mnaser | haha | |
| 00:27:06 | mnaser | https://github.com/openstack/python-cinderclient/blob/master/cinderclient/exceptions.py (ClientException) and https://github.com/openstack/python-neutronclient/blob/master/neutronclient/common/exceptions.py (NeutronClientException) | |
| 00:27:13 | mnaser | both seem to be fairly reasonable in terms of things they handle | |
| 00:28:15 | mnaser | thing is NoAuthURLProvided/EndpointNotFound/EndpointTypeNotFound/AmbiguousEndpoints/ConnectionFailed and a bunch of other stuff | |
| 00:28:19 | mnaser | are also under neutronclientexcpetion | |
| 00:32:37 | mriedem | we won't hit those anymore, | |
| 00:32:44 | mriedem | because we constructor neutronclient with a ksa session | |
| 00:32:53 | mriedem | so if we fail auth stuff, it will be early with ksa | |
| 00:32:56 | mriedem | *construct | |
| 00:33:16 | mriedem | that's true in at least queens, i'm not sure about pike... | |
| 00:35:07 | mnaser | so that affects the backportable-itiy | |
| 00:35:26 | mnaser | not that i want to stay on pike for much longer | |
| 00:36:12 | mriedem | maybe, would have to dig into it | |
| 00:36:22 | mriedem | trying to fix bug 1679750 atm | |
| 00:36:22 | openstack | bug 1679750 in OpenStack Compute (nova) queens "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,Confirmed] https://launchpad.net/bugs/1679750 | |
| 00:37:32 | mnaser | o | |
| 00:37:35 | mnaser | that's a bad time | |
| 00:48:52 | arvindn05 | mriedem: quick question, i think you answered it once but wanted to reconfirm. "A rebuild is staying on the same host, with optionally a new image, but the flavor stays the same." | |
| 00:49:47 | arvindn05 | https://developer.openstack.org/api-guide/compute/server_concepts.html the documentation does not specify that it stays on the same host...just wanted to confirm if rebuild ALWAYS means staying on same host | |
| 00:53:41 | mriedem | yes it's always the same host | |
| 00:54:46 | mriedem | https://developer.openstack.org/api-guide/compute/server_concepts.html#recover-from-a-failed-compute-host could probably be tightened up | |
| 00:55:20 | mriedem | where it says, "Evacuate does the same operation as a rebuild. " - that could say, "Evacuate does the same operation as a rebuild except evacuate is on a new host while rebuild is on the same host." | |
| 00:55:22 | mriedem | something like that | |
| 00:55:41 | mriedem | and you can't evacuate with a new image | |
| 00:56:00 | mriedem | arvindn05: when you start talking about evacuate and rebuild, you must read http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 00:56:43 | arvindn05 | thx for the hint :) | |
| 00:57:58 | arvindn05 | where is the same host enforced? | |
| 00:58:46 | arvindn05 | in code i mean...all that the manager checks is if the hints have the check_type set to rebuild | |
| 00:59:00 | arvindn05 | as per scheduler.utils.request_is_rebuild | |
| 01:02:36 | arvindn05 | thikning out loud, why would select destination be called in the first place if we dont need to select a host? is it just for some validation? | |
| 01:09:05 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Use XAPI pool instead of aggregate pool for shared SR migration https://review.openstack.org/554154 | |
| 01:13:52 | arvindn05 | nvm...we want to run image related filters so thats why we want to call select destination... | |
| 01:26:59 | Kevin_Zheng | mriedem as for your comments in https://review.openstack.org/#/c/560296/ | |
| 01:27:43 | Kevin_Zheng | I thought that making the order constant in all files make it more easier to read and compare.... | |
| 01:30:16 | mriedem | Kevin_Zheng: i wouldn't mix that into the deduplicate payload change | |
| 01:30:35 | mriedem | if you want to sort the keys globally for those samples, i'd do that once all of the dedup stuff is done | |
| 01:30:52 | Kevin_Zheng | OK, then I will undo them | |
| 01:31:31 | mriedem | arvindn05: for rebuild we set a destination on the request_spec which forces the scheduler to only consider the host that the instance is already one | |
| 01:32:10 | mriedem | arvindn05: here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3158 | |
| 01:34:17 | arvindn05 | mriedem: thank you. starting to understand more of the workflow | |
| 01:35:10 | mriedem | arvindn05: np, it only took me about 5 years to understand how rebuild works | |
| 01:37:59 | arvindn05 | mriedem: lol...i am shifting languages as well so its harder to find workflows | |
| 01:38:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanup RP and HM records while deleting a compute service. https://review.openstack.org/554920 | |
| 01:38:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete allocations from API if nova-compute is down https://review.openstack.org/560706 | |
| 01:38:59 | arvindn05 | mriedem: i saw def _reset_image_metadata and did not read any code after since in java world seeing a function definition meant i was done with the original function | |
| 01:39:11 | mriedem | inner method | |
| 01:39:17 | mriedem | you can have inline functions in java | |
| 01:40:11 | arvindn05 | inline functions? you mean lambda's? | |
| 01:40:29 | mriedem | no, | |
| 01:40:37 | mriedem | maybe i'm thinking of inline class impls, like inline interfaces | |
| 01:40:48 | mriedem | idk, it's been 7 years since i've written java | |
| 01:41:06 | arvindn05 | yea..anonymous classes you can define them | |
| 01:43:38 | arvindn05 | but cannot define method withhin - quora speaks https://www.quora.com/Can-we-write-a-method-inside-a-method-in-Java | |
| 01:45:07 | arvindn05 | anyway...i learn something new everyday...weird pattern to have methods within methods....havent see it a lot in openstack code either | |
| 01:45:16 | arvindn05 | will keep my eye out from now on | |
| 02:16:33 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 1 https://review.openstack.org/560296 | |
| 02:22:27 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 1 https://review.openstack.org/560296 | |
| 02:31:22 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Deduplicate notification samples Rocky - 2 https://review.openstack.org/560299 | |
| 02:32:37 | takashin | ||
| 02:55:32 | Spaz-Work | Morning Nova | |
| 02:56:42 | openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Handle rebuild of instance with new image https://review.openstack.org/560718 | |