| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-03 | |||
| 20:32:16 | mriedem | cool | |
| 20:54:20 | superdan | melwitt: I think one of these fails is due to quota: http://logs.openstack.org/94/436094/11/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f2d1af0/logs/testr_results.html.gz | |
| 20:54:29 | superdan | melwitt: down near the bottom, test_update_instance_port_admin_state | |
| 20:54:53 | superdan | runs in isolation, but not as part of the larger thing | |
| 20:55:22 | superdan | I'm wondering if we're reserving in cell0 and claiming in cell1 or something like that | |
| 20:56:20 | melwitt | superdan: ah, okay. yeah, I think it would work that way. the quota commit (done by compute) would probably end up in the compute cell's db, so that would end up not freeing the reservation at the top | |
| 20:56:30 | superdan | right | |
| 20:57:02 | superdan | is reservations where that gets made in the api? | |
| 20:57:14 | superdan | I thought it was something else | |
| 20:57:23 | superdan | but it seems that I don't know at all where these things go | |
| 20:57:27 | melwitt | what gets made? | |
| 20:57:53 | melwitt | the quota reserve happens in the api, which will create the reservation record | |
| 20:58:25 | superdan | what I mean is.. we create reservation in ... the reservations table? | |
| 20:58:34 | melwitt | oh, yes | |
| 20:58:37 | superdan | where does that get converted to a claim or whatever? | |
| 20:59:04 | melwitt | it will create a record in the reservations table and then also update records in the quota_usages table to mark resources as 'reserved' | |
| 20:59:28 | melwitt | then the quota commit will delete the reservation record from the reservations table and update the quota_usages records to unmark them as 'reserved' and mark them as 'in use' | |
| 20:59:29 | superdan | okay I have things in nova_cell0.reservations | |
| 20:59:54 | superdan | 21 in reservations and 15 in quota_usages, in cell0 interestingly | |
| 21:00:00 | melwitt | quota_usages records have a in_use field and reserved field which represent int counts | |
| 21:00:17 | superdan | ugh | |
| 21:00:25 | melwitt | the commit will subtract from reserve and increment in_use, if it's an alloc | |
| 21:00:33 | melwitt | yeah. | |
| 21:00:51 | superdan | I heard "monday, monday. then monday and monday, followed by monday" | |
| 21:04:19 | melwitt | lol | |
| 21:04:22 | melwitt | that's fair | |
| 21:05:31 | openstack | siva_krishnan: Error: Could not gather data from Launchpad for bug #1315201 (https://launchpad.net/bugs/1315201). The error has been logged | |
| 21:05:31 | siva_krishnan | mriedem: Looked at your bug report https://bugs.launchpad.net/nova/+bug/1315201. I have been hitting this issue for one of my patches. Any suggestion/help on how to debug these tempest test failures? | |
| 21:06:41 | mriedem | siva_krishnan: that's super old | |
| 21:07:11 | mriedem | siva_krishnan: see my last comment in there? need to debug what's going on in the service logs | |
| 21:09:14 | superdan | melwitt: actually, maybe it should be working because we hard target the context, which seems to be how the quota stuff works | |
| 21:10:30 | siva_krishnan | mriedem: I am getting NoValidHost: No valid host was found. There are not enough hosts available in n-cpu | |
| 21:10:48 | superdan | oh, but. | |
| 21:10:53 | superdan | we don't have an instance at that point | |
| 21:10:55 | mriedem | siva_krishnan: that could be caused by many different things | |
| 21:11:07 | mriedem | siva_krishnan: usually check n-sch, n-cond and/or n-cpu logs | |
| 21:11:20 | superdan | melwitt: so we kinda have to have your counting quotas thing before we can do this... | |
| 21:12:22 | melwitt | superdan: this is where the reserve happens https://github.com/openstack/nova/blob/master/nova/compute/api.py#L312 | |
| 21:12:27 | sneti | Hi alex_xu, can you please look at this when you have time: https://review.openstack.org/#/c/392305/. Current fix doesn't work with query param validation . Do you suggest any other way to fix this bug. | |
| 21:13:59 | siva_krishnan | mriedem: apart from the above error other thing which I could find in logs is this one "SSLError: SSL exception connecting to https://10.28.135.25/placement/resource_providers/dba7216f-e694-49ee-821b-098e692a64c2/allocations: ("bad handshake: SysCallError(104, 'ECONNRESET')",)" | |
| 21:14:02 | superdan | melwitt: which is before we know what cell | |
| 21:14:31 | mriedem | siva_krishnan: do you have a link to the patch or job that failed? | |
| 21:14:46 | siva_krishnan | mriedem: https://review.openstack.org/#/c/434870/3 | |
| 21:16:17 | melwitt | superdan: yeah. I think what I have up right now for quotas works, so it would be interesting to rebase on that and see what happens | |
| 21:16:39 | superdan | melwitt: like passing tempest? | |
| 21:16:57 | melwitt | the todos I have for it have to do with changing how to deal with quota count races and not counting flavor blob in instance_extra | |
| 21:16:58 | melwitt | yeah | |
| 21:17:18 | superdan | okay | |
| 21:17:26 | melwitt | well, I guess it has a merge conflict now. | |
| 21:17:32 | melwitt | sorry, I forgot that | |
| 21:17:52 | superdan | melwitt: so this? | |
| 21:17:53 | superdan | https://review.openstack.org/#/c/416521/ | |
| 21:17:54 | melwitt | it was passing tempest | |
| 21:18:14 | melwitt | yeah. I think I have to rebase it | |
| 21:18:33 | superdan | okay, well, that's encouraging either way | |
| 21:19:17 | melwitt | yeah, definitely will be easy to get a run and see how many failures go away | |
| 21:23:14 | mriedem | siva_krishnan: your patch kills libvirt http://logs.openstack.org/70/434870/3/check/gate-tempest-dsvm-py35-ubuntu-xenial/a2c2fdd/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-03-01_01_51_37_675 | |
| 21:23:17 | mriedem | which takes down the compute service | |
| 21:23:24 | mriedem | and in a single node job, there is nothing to schedule to | |
| 21:23:28 | mriedem | so you get NoValidHost | |
| 21:25:37 | mriedem | i left a comment | |
| 21:25:43 | mriedem | post copy is not available until libvirt 1.3.3 | |
| 21:25:45 | mriedem | we test against 1.3. | |
| 21:25:49 | mriedem | *1.3.1 | |
| 21:27:06 | siva_krishnan | mriedem: ah ok . didn't check this job. thanks a lot. will makes changes accordingly | |
| 21:27:59 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Get instance availability_zone without hitting the api db https://review.openstack.org/439754 | |
| 21:28:00 | openstackgerrit | Dan Smith proposed openstack/nova master: Change MQ targeting to honor only what is in the context https://review.openstack.org/438023 | |
| 21:28:00 | openstackgerrit | Dan Smith proposed openstack/nova master: Target cell in super conductor operations https://review.openstack.org/438022 | |
| 21:28:01 | openstackgerrit | Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891 | |
| 21:28:02 | openstackgerrit | Dan Smith proposed openstack/nova master: Make consoleauth target the proper cell https://review.openstack.org/441461 | |
| 21:28:03 | openstackgerrit | Dan Smith proposed openstack/nova master: Add cell field to Destination object https://review.openstack.org/438024 | |
| 21:28:07 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Make conductor ask scheduler to limit migrates to same cell https://review.openstack.org/438025 | |
| 21:31:17 | openstackgerrit | Maciej Szankin proposed openstack/nova master: conf: remove console_driver opt https://review.openstack.org/441485 | |
| 21:33:50 | openstackgerrit | Dan Smith proposed openstack/nova master: Count resources to check quota for cells https://review.openstack.org/416521 | |
| 21:33:52 | openstackgerrit | Dan Smith proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946 | |
| 21:33:52 | superdan | melwitt: rebasificated ^ | |
| 21:33:52 | openstackgerrit | Dan Smith proposed openstack/nova master: Make Quotas object favor the API database https://review.openstack.org/410945 | |
| 21:34:45 | melwitt | superdan: I was just in the middle of doing that. thanks | |
| 21:34:51 | superdan | melwitt: oh sorry | |
| 21:36:07 | melwitt | superdan: np. I didn't mention I was doing it so :P | |
| 21:37:53 | melwitt | I was being paranoid running all the unit and functional tests on it | |
| 21:38:19 | superdan | meh, jenkins'll do that :P | |
| 21:39:09 | superdan | stacking with your stuff under mine now | |
| 21:39:11 | melwitt | it's gonna fail pep8 because a new hacking change went in since last I updated it, I think | |
| 21:39:23 | melwitt | everything else should be okay. *should* be okay | |
| 21:39:44 | superdan | meh, pep8 is for losers | |
| 21:39:53 | melwitt | that's true | |
| 21:41:09 | cburgess | lol | |
| 22:00:50 | efried_ | If an op like pause or suspend isn't implemented in my driver, and I do that op from the UI, shouldn't it fail? | |
| 22:01:19 | efried_ | n-cpu shows the NotImplementedError, but the UI gives me the green box claiming it succeeded. Though the state doesn't change (so I can't e.g. resume after that) | |
| 22:03:55 | sdague | efried_: the commands are async | |
| 22:04:14 | sdague | it succeeds with having submitted the command | |
| 22:04:43 | efried_ | Whoah. Okay. That's news to me. sdague Not all of the commands are async, are they? | |
| 22:04:54 | sdague | efried_: most of them | |
| 22:05:26 | efried_ | Guess I'm not used to using anything other than spawn, destroy, and migrate. Thanks for the info sdague | |
| 22:05:40 | sdague | all of those are async | |
| 22:06:29 | efried_ | sdague Is that new, then? Cause I swear when something goes wrong in the middle of e.g. spawn, the UI shows me a red failure box. | |
| 22:06:39 | efried_ | I haven't done that from the UI in a while... | |
| 22:07:12 | superdan | melwitt: well it stacked | |
| 22:07:15 | sdague | efried_: it might be polling, but the API call does some basic sanity checking, then returns success well before a vm is ever booting | |