| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-11 | |||
| 22:26:11 | mriedem | cfriesen: i leave the conflicts in the commit message and make a note about what they are to help reviewers | |
| 22:26:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for deleting a compute service https://review.openstack.org/560626 | |
| 22:26:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Cleanup RP and HM records while deleting a compute service. https://review.openstack.org/554920 | |
| 22:26:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Block deleting compute services which are hosting instances https://review.openstack.org/560674 | |
| 22:26:31 | mriedem | efried: ^ should make it more clear | |
| 22:26:42 | efried | mriedem: ack, will look mañana | |
| 22:26:59 | mriedem | manana iguana? https://www.amazon.com/Manana-Iguana-Ann-Whitford-Paul/dp/0823419800 | |
| 22:27:08 | mriedem | 2 childrens books references in one day | |
| 22:27:36 | mriedem | those are for dansmith and jaypipes btw | |
| 22:35:23 | jaypipes | ewww | |
| 22:36:00 | mriedem | there goes a gate reset so that functional test fix is going to be awhile yet | |
| 23:01:42 | openstackgerrit | Eric Fried proposed openstack/nova master: test.stub_object https://review.openstack.org/560683 | |
| 23:30:46 | openstackgerrit | Chris Friesen proposed openstack/nova stable/pike: libvirt: disconnect volume from host during detach https://review.openstack.org/560690 | |
| 23:42:32 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Example verification for servers.inc https://review.openstack.org/529520 | |
| 23:50:17 | gmann | sdague: can you feedback on this comment (one of TODO from you) - https://review.openstack.org/#/c/529520/18/api-ref/source/servers.inc@765 | |
| 23:51:18 | gmann | sdague: i feel single latest sample response file is good enough instead of separate sample files per request attribute for update | |
| 23:51:36 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_neutron_security_groups.py https://review.openstack.org/560267 | |
| #openstack-nova - 2018-04-12 | |||
| 00:04:01 | mnaser | hey | |
| 00:04:03 | mnaser | fun bug time | |
| 00:05:04 | mnaser | if a user builds an instance that if boot from volume (where nova creates the volume) and nova-compute fails to create the volume, the vm build fails, and it actually counts as a 'consecutive build failures' | |
| 00:05:29 | mnaser | so if i'm $bad_user and i try to launch 40 instances but keep hitting quota limits but keep trying again, i will effectively slowly but surely disable every compute node in the cloud | |
| 00:06:14 | mnaser | so while that feature is nice, i think we have to find a way to identify the type of failure it is | |
| 00:08:10 | mnaser | i mean we could disable that feature but that feels dirty | |
| 00:14:36 | mriedem | mnaser: yeah https://bugs.launchpad.net/nova/+bug/1742102 | |
| 00:14:37 | openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "Simple user can disable compute" [Undecided,Confirmed] - Assigned to jichenjc (jichenjc) | |
| 00:14:40 | mriedem | there is an ops ML thread about similar issues | |
| 00:14:59 | mriedem | i think we likely need to consider a whitelist of acceptable, not threshold inducing exceptions | |
| 00:15:30 | mriedem | the latest comments in there from jichen are actually the exact issue you're describing | |
| 00:17:07 | mriedem | but we can also be smarter and check quota before trying to create volumes | |
| 00:17:09 | mriedem | like we do for ports | |
| 00:17:27 | mnaser | mriedem: yeah, i think that's better, do we check quota at api layer for ports? | |
| 00:17:29 | mriedem | could do that in conductor so we don't block the API response | |
| 00:17:33 | mriedem | mnaser: yes | |
| 00:17:40 | mriedem | it's the validate_networks call | |
| 00:17:50 | mnaser | that sounds reasonable and backport-able too i think | |
| 00:17:52 | mriedem | based on the number of requested networks and instances, we check quota | |
| 00:18:22 | mnaser | the thing is if i have a quota of 10 volumes, and i launch a 100 instances (one by one), i might still run into that issue i guess | |
| 00:18:23 | mriedem | yeah - either way you'd fail after the 202 you get from the API | |
| 00:18:31 | mriedem | you either fail in conductor or you fail in compute | |
| 00:18:45 | mriedem | sure but we don't disable your compute :) | |
| 00:18:54 | mnaser | because of the race as conductor still sees you using 0 volumes but as computes start creating volumes | |
| 00:19:05 | mriedem | oh, sure | |
| 00:19:23 | mriedem | we could also just handle volume quota issues in the compute as a whitelist of things to not disable the compute | |
| 00:19:29 | mriedem | multi-part fixes | |
| 00:19:49 | mnaser | i think both out-of-ports and out-of-volumes both seem like reasonable 'skip' failures | |
| 00:20:01 | mriedem | yeah | |
| 00:20:11 | mriedem | the bug above started as a port / fixed ips quota issue | |
| 00:20:15 | mriedem | but also extends to volumes | |
| 00:20:21 | mnaser | i think any api exception should be skipped tbh | |
| 00:20:29 | mnaser | because that means the compute node is fine | |
| 00:20:51 | mnaser | sure it sounds vague but if your cinder is not having a good time then you're slowly disabling all compute nodes | |
| 00:20:52 | mriedem | well, depends on what it is, | |
| 00:21:05 | mriedem | if nova.conf is misconfigured on the compute to talk to neutron, that's a different problem | |
| 00:21:08 | mriedem | likely a 403 | |
| 00:21:21 | mnaser | ah yes | |
| 00:21:41 | mriedem | if we get NeutronClientException, then we made a request and it failed but the endpoint, config and token should be ok | |
| 00:21:57 | mriedem | same with CinderClientException | |
| 00:22:19 | mnaser | so we can skip those 2 exceptions because if we get them, it doesn't mean that the compute node has any problems | |
| 00:22:55 | mriedem | well, if it did turn out that excluding those were too broad, we could narrow it down over time | |
| 00:23:06 | mriedem | vif plug is something that could fail... | |
| 00:23:10 | mriedem | based on bad config of the host | |
| 00:23:17 | mriedem | that's a hard one though | |
| 00:23:46 | mriedem | because we don't get vif plug failures directly | |
| 00:23:59 | mriedem | we get a callback event from neutron that just says plug failed but not why | |
| 00:24:19 | mriedem | but yeah, that's not a NeutronClientException, so ignore me | |
| 00:24:21 | mnaser | i can't imagine a single time that i remember seeing a failed to plug callback | |
| 00:24:27 | mnaser | it either never comes or comes ok :P | |
| 00:24:29 | mnaser | haha | |
| 00:27:06 | mnaser | https://github.com/openstack/python-cinderclient/blob/master/cinderclient/exceptions.py (ClientException) and https://github.com/openstack/python-neutronclient/blob/master/neutronclient/common/exceptions.py (NeutronClientException) | |
| 00:27:13 | mnaser | both seem to be fairly reasonable in terms of things they handle | |
| 00:28:15 | mnaser | thing is NoAuthURLProvided/EndpointNotFound/EndpointTypeNotFound/AmbiguousEndpoints/ConnectionFailed and a bunch of other stuff | |
| 00:28:19 | mnaser | are also under neutronclientexcpetion | |
| 00:32:37 | mriedem | we won't hit those anymore, | |
| 00:32:44 | mriedem | because we constructor neutronclient with a ksa session | |
| 00:32:53 | mriedem | so if we fail auth stuff, it will be early with ksa | |
| 00:32:56 | mriedem | *construct | |
| 00:33:16 | mriedem | that's true in at least queens, i'm not sure about pike... | |
| 00:35:07 | mnaser | so that affects the backportable-itiy | |
| 00:35:26 | mnaser | not that i want to stay on pike for much longer | |
| 00:36:12 | mriedem | maybe, would have to dig into it | |
| 00:36:22 | openstack | bug 1679750 in OpenStack Compute (nova) queens "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,Confirmed] https://launchpad.net/bugs/1679750 | |
| 00:36:22 | mriedem | trying to fix bug 1679750 atm | |
| 00:37:32 | mnaser | o | |
| 00:37:35 | mnaser | that's a bad time | |
| 00:48:52 | arvindn05 | mriedem: quick question, i think you answered it once but wanted to reconfirm. "A rebuild is staying on the same host, with optionally a new image, but the flavor stays the same." | |
| 00:49:47 | arvindn05 | https://developer.openstack.org/api-guide/compute/server_concepts.html the documentation does not specify that it stays on the same host...just wanted to confirm if rebuild ALWAYS means staying on same host | |
| 00:53:41 | mriedem | yes it's always the same host | |
| 00:54:46 | mriedem | https://developer.openstack.org/api-guide/compute/server_concepts.html#recover-from-a-failed-compute-host could probably be tightened up | |
| 00:55:20 | mriedem | where it says, "Evacuate does the same operation as a rebuild. " - that could say, "Evacuate does the same operation as a rebuild except evacuate is on a new host while rebuild is on the same host." | |
| 00:55:22 | mriedem | something like that | |
| 00:55:41 | mriedem | and you can't evacuate with a new image | |
| 00:56:00 | mriedem | arvindn05: when you start talking about evacuate and rebuild, you must read http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 00:56:43 | arvindn05 | thx for the hint :) | |
| 00:57:58 | arvindn05 | where is the same host enforced? | |
| 00:58:46 | arvindn05 | in code i mean...all that the manager checks is if the hints have the check_type set to rebuild | |
| 00:59:00 | arvindn05 | as per scheduler.utils.request_is_rebuild | |
| 01:02:36 | arvindn05 | thikning out loud, why would select destination be called in the first place if we dont need to select a host? is it just for some validation? | |
| 01:09:05 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Use XAPI pool instead of aggregate pool for shared SR migration https://review.openstack.org/554154 | |
| 01:13:52 | arvindn05 | nvm...we want to run image related filters so thats why we want to call select destination... | |
| 01:26:59 | Kevin_Zheng | mriedem as for your comments in https://review.openstack.org/#/c/560296/ | |