| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-28 | |||
| 12:33:08 | ebbex | there is no binary that has this name | |
| 12:33:30 | ebbex | also, nice to use a function name as a column name :) | |
| 12:34:05 | jaypipes | ebbex: yeah... :( | |
| 12:34:35 | jaypipes | ebbex: so I'm pretty sure this is the source of the bug: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2422 | |
| 12:34:53 | jaypipes | ebbex: or rather, that is the source of your woes... :) | |
| 12:35:14 | jaypipes | ebbex: either the service version being returned by the call here: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2417 | |
| 12:35:24 | jaypipes | is not >= 15 | |
| 12:35:34 | jaypipes | or you have cellsv1 set up, which I doubt :) | |
| 12:35:42 | ebbex | no cellsv1 | |
| 12:36:09 | ebbex | so where do I set the service_version if its not in the services table, and what can cause it to be there | |
| 12:36:21 | jaypipes | ebbex: can you try removing lines 2422-2424 from the code and retrying the scenario? | |
| 12:37:10 | jaypipes | ebbex: I'm thinking that "nova-osapi_compute" is wrong for existing installations maybe. Perhaps that should be "nova-api"? | |
| 12:37:23 | jaypipes | ebbex: what does this give you if you run it on the cell DB: | |
| 12:37:31 | ebbex | jaypipes: could I have bootstrapped something wrong with cellsv2? should there be an entry for nova-os-compute-api? | |
| 12:37:38 | jaypipes | SELECT binary, count(*) as c from services group by binary; | |
| 12:37:52 | jaypipes | ebbex: not sure, trying to find out :) | |
| 12:38:27 | ebbex | remember `binary` ;) | |
| 12:38:34 | jaypipes | heh, yes | |
| 12:38:46 | jaypipes | was just about to say that :) | |
| 12:38:53 | ebbex | | nova-conductor | 3 | | |
| 12:38:53 | ebbex | | nova-compute | 2 | | |
| 12:38:54 | ebbex | | nova-scheduler | 3 | | |
| 12:38:54 | ebbex | | nova-consoleauth | 3 | | |
| 12:38:54 | ebbex | | nova-console | 3 | | |
| 12:39:31 | jaypipes | is there a services table in the nova_api db? | |
| 12:39:39 | ebbex | nope. | |
| 12:39:49 | jaypipes | hmm.. | |
| 12:39:57 | jaypipes | I wonder why there's no entry for the api services... | |
| 12:41:52 | jaypipes | ebbex: I need to run out for a bit. I'm hoping dansmith will have some ideas when he gets online about this. | |
| 12:42:39 | ebbex | jaypipes: cool, thanks :) | |
| 12:45:07 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for images instead of stevedore https://review.openstack.org/478498 | |
| 12:45:08 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for image-metadata instead of stevedore https://review.openstack.org/478499 | |
| 12:45:49 | alex_xu | nova api meeting is in 15 mins at #openstack-meeting-4 | |
| 12:59:50 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for volumes instead of stevedore https://review.openstack.org/478502 | |
| 13:01:20 | alex_xu | nova api meeting is running | |
| 13:04:48 | openstackgerrit | Eric Fried proposed openstack/nova master: Monkey patch the blockdiag extension https://review.openstack.org/476159 | |
| 13:04:49 | openstackgerrit | Eric Fried proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810 | |
| 13:05:08 | efried | sfinucan ^^ resolve merge conflict (test-requirements.txt) | |
| 13:05:33 | openstackgerrit | Eric Fried proposed openstack/nova master: Add formatting to scheduling activity diagram https://review.openstack.org/476204 | |
| 13:05:40 | jaypipes | artom: any ideas on that drive-scsi0-0-0-0 issue from a couple days ago? is lyarwood around by any chance? | |
| 13:06:19 | artom | jaypipes, I haven't looked into it further since then | |
| 13:10:53 | dimak | Hey, I'm trying to bring up a large VM on a recent devstack, and I'm getting 'No valid host was found' | |
| 13:11:43 | dimak | And the scheduler logs say 'Starting with 0 host(s)' while I have 2 hypervisors and a bit smaller VM is able to get scheduled | |
| 13:12:09 | kashyap | dimak: I recall fixing that, but now I forget how or what I did | |
| 13:12:30 | kashyap | There are other people who meditate on scheduler stuff here, though. Let's wait for them to chime in | |
| 13:12:46 | dimak | Sure, thanks :) | |
| 13:13:26 | ebbex | jaypipes: dansmith : I think you've helped track it down, any idea which nova-manage command puts the 'nova-osapi_compute' binary into the services table? | |
| 13:14:57 | klipka | dimak: Could you provide log from nova-scheduler with debug set to true? | |
| 13:16:05 | dimak | klipka, sure, a moment | |
| 13:17:48 | klipka | first wild guess is that Placement API is not configured | |
| 13:18:12 | klipka | refer to second bullet here: https://docs.openstack.org/developer/nova/placement.html#ocata-15-0-0 | |
| 13:20:33 | dimak | klipka, https://paste.fedoraproject.org/paste/E6FuxcJaKPJ5qcKUauoG9g | |
| 13:21:25 | dimak | klipka, I have placement API configured on my other node, should I have some placement service running there as well? | |
| 13:23:55 | klipka | dimak: if it's configured on controller it should be ok | |
| 13:24:36 | klipka | in log which you provided there is 'Starting with 1 host(s)' and it seems to choose host: 'Selected host: WeighedHost [host: (bishop, bishop)' | |
| 13:24:58 | dimak | klipka, yes, thats the smaller VM that starts | |
| 13:25:26 | dimak | But I also have another host, 'rogue', and it is not being considered | |
| 13:27:00 | klipka | and you want to know why this 'rogue' is not taken into consideration? | |
| 13:27:06 | dimak | yes | |
| 13:27:44 | dimak | It's in the hypervisor list and up | |
| 13:32:01 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for images instead of stevedore https://review.openstack.org/478498 | |
| 13:32:01 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for image-metadata instead of stevedore https://review.openstack.org/478499 | |
| 13:32:02 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for volumes instead of stevedore https://review.openstack.org/478502 | |
| 13:35:32 | openstackgerrit | Steve Noyes proposed openstack/nova master: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 13:36:23 | klipka | dimak: seems that before creating instance you didn't received info about 'rogue', sync comes after VM scheduling | |
| 13:38:14 | dimak | I've tried again and it is still starting with 0 hosts | |
| 13:43:42 | klipka | I'm little confused, because you are talking about two cases: 1. creating small VM, where you are starting with only one host and want to see two of them | |
| 13:44:19 | klipka | 2. creating large VM, where you are starting with 0 hosts and want to see possibly two of them | |
| 13:44:26 | dansmith | ebbex: this sounds like a support question and this channel is for development only | |
| 13:44:46 | dansmith | ebbex: I may be able to help you, but we should talk in #openstack about it | |
| 13:45:41 | dimak | klipka, I just think that the one host I do see is not large enough for the big vm so it gets filtered out before the log | |
| 13:46:31 | ebbex | dansmith: I think we've tracked it down. the nova/wsgi/nova-api.py doesn't register in the services table, but the "nova-api-os-compute" does. | |
| 13:46:56 | dansmith | ebbex: it's supposed to, cdent did that work | |
| 13:49:54 | ebbex | dansmith: well, we've been running the nova-api.py been running under uwsgi, and the table is empty. but once I fired up the nova-api-os-compute, the services table was populated. | |
| 13:50:45 | dansmith | ebbex: are your api services configured to point at the nova (cell) database or the cell0 one? | |
| 13:51:13 | ebbex | the nova (cell1) database. | |
| 13:52:14 | klipka | dimak: I think scheduler should start with both hosts and filter them out in such case | |
| 13:52:20 | dansmith | ebbex: and what is the behavior you think is being caused by those records not being created? | |
| 13:52:29 | klipka | if I am wrong, then somebody, please correct me | |
| 13:53:35 | ebbex | dansmith: Not sure yet, I'll have to look a bit deeper into the code, but something has to be called in nova-api-os-compute, that's not being done in the wsgi app. | |
| 13:53:55 | dansmith | ebbex: no, | |
| 13:54:18 | dansmith | I mean why are you concerning yourself with those records? they should be created, agreed, but why are you even chasing this? | |
| 13:56:03 | ebbex | dansmith: because we have a race-condition where using the command "server create" in openstack-cli returns a 202 with a location to servers/<uuid>, but a query there returns a 404. | |
| 13:56:58 | dansmith | ebbex: and you think this service record thing is related? | |
| 13:57:07 | ebbex | dansmith: but the server is created and starts up, and issuing the "server show" command, or a simple query to that location a bit later returns a 200. | |
| 13:57:37 | ebbex | Yeah, in the code there is a check to see if sevice_version > 15 look in build_instance database. | |
| 13:58:35 | ebbex | otherwise it doesn't look there, only in "instances", and it's not there quite yet. still building i assume. | |
| 13:59:35 | ebbex | hence the 404. | |
| 13:59:46 | dansmith | https://review.openstack.org/#/c/457283/4/nova/api/openstack/compute/wsgi.py | |
| 13:59:46 | dansmith | so this is the app setup that does the service registration: | |
| 14:00:00 | dansmith | but I don't see how that's called from nova/wsgi/nova-api.py | |
| 14:00:46 | dansmith | the commit message there implies that you should be loading that as your wsgi script and that nova/wsgi is old | |
| 14:00:52 | dansmith | but it appears we're loading the old one in devstack, so I dunno | |
| 14:01:52 | ebbex | dansmith: well, ok. so this could all be the result of a mix-up. | |
| 14:02:11 | dansmith | I'm not sure why that was landed in a different place without removing the old one, | |
| 14:02:19 | dansmith | but also not sure why devstack isn't loading the new one | |
| 14:02:21 | dansmith | maybe sdague knows | |
| 14:02:48 | dansmith | ebbex: you might notice that the nova/wsgi one says "don't use this" :) | |
| 14:03:31 | sdague | dnawhat's the question? | |
| 14:03:36 | sdague | dansmith: ^^^ | |