| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-28 | |||
| 13:13:26 | ebbex | jaypipes: dansmith : I think you've helped track it down, any idea which nova-manage command puts the 'nova-osapi_compute' binary into the services table? | |
| 13:14:57 | klipka | dimak: Could you provide log from nova-scheduler with debug set to true? | |
| 13:16:05 | dimak | klipka, sure, a moment | |
| 13:17:48 | klipka | first wild guess is that Placement API is not configured | |
| 13:18:12 | klipka | refer to second bullet here: https://docs.openstack.org/developer/nova/placement.html#ocata-15-0-0 | |
| 13:20:33 | dimak | klipka, https://paste.fedoraproject.org/paste/E6FuxcJaKPJ5qcKUauoG9g | |
| 13:21:25 | dimak | klipka, I have placement API configured on my other node, should I have some placement service running there as well? | |
| 13:23:55 | klipka | dimak: if it's configured on controller it should be ok | |
| 13:24:36 | klipka | in log which you provided there is 'Starting with 1 host(s)' and it seems to choose host: 'Selected host: WeighedHost [host: (bishop, bishop)' | |
| 13:24:58 | dimak | klipka, yes, thats the smaller VM that starts | |
| 13:25:26 | dimak | But I also have another host, 'rogue', and it is not being considered | |
| 13:27:00 | klipka | and you want to know why this 'rogue' is not taken into consideration? | |
| 13:27:06 | dimak | yes | |
| 13:27:44 | dimak | It's in the hypervisor list and up | |
| 13:32:01 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for image-metadata instead of stevedore https://review.openstack.org/478499 | |
| 13:32:01 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for images instead of stevedore https://review.openstack.org/478498 | |
| 13:32:02 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for volumes instead of stevedore https://review.openstack.org/478502 | |
| 13:35:32 | openstackgerrit | Steve Noyes proposed openstack/nova master: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 13:36:23 | klipka | dimak: seems that before creating instance you didn't received info about 'rogue', sync comes after VM scheduling | |
| 13:38:14 | dimak | I've tried again and it is still starting with 0 hosts | |
| 13:43:42 | klipka | I'm little confused, because you are talking about two cases: 1. creating small VM, where you are starting with only one host and want to see two of them | |
| 13:44:19 | klipka | 2. creating large VM, where you are starting with 0 hosts and want to see possibly two of them | |
| 13:44:26 | dansmith | ebbex: this sounds like a support question and this channel is for development only | |
| 13:44:46 | dansmith | ebbex: I may be able to help you, but we should talk in #openstack about it | |
| 13:45:41 | dimak | klipka, I just think that the one host I do see is not large enough for the big vm so it gets filtered out before the log | |
| 13:46:31 | ebbex | dansmith: I think we've tracked it down. the nova/wsgi/nova-api.py doesn't register in the services table, but the "nova-api-os-compute" does. | |
| 13:46:56 | dansmith | ebbex: it's supposed to, cdent did that work | |
| 13:49:54 | ebbex | dansmith: well, we've been running the nova-api.py been running under uwsgi, and the table is empty. but once I fired up the nova-api-os-compute, the services table was populated. | |
| 13:50:45 | dansmith | ebbex: are your api services configured to point at the nova (cell) database or the cell0 one? | |
| 13:51:13 | ebbex | the nova (cell1) database. | |
| 13:52:14 | klipka | dimak: I think scheduler should start with both hosts and filter them out in such case | |
| 13:52:20 | dansmith | ebbex: and what is the behavior you think is being caused by those records not being created? | |
| 13:52:29 | klipka | if I am wrong, then somebody, please correct me | |
| 13:53:35 | ebbex | dansmith: Not sure yet, I'll have to look a bit deeper into the code, but something has to be called in nova-api-os-compute, that's not being done in the wsgi app. | |
| 13:53:55 | dansmith | ebbex: no, | |
| 13:54:18 | dansmith | I mean why are you concerning yourself with those records? they should be created, agreed, but why are you even chasing this? | |
| 13:56:03 | ebbex | dansmith: because we have a race-condition where using the command "server create" in openstack-cli returns a 202 with a location to servers/<uuid>, but a query there returns a 404. | |
| 13:56:58 | dansmith | ebbex: and you think this service record thing is related? | |
| 13:57:07 | ebbex | dansmith: but the server is created and starts up, and issuing the "server show" command, or a simple query to that location a bit later returns a 200. | |
| 13:57:37 | ebbex | Yeah, in the code there is a check to see if sevice_version > 15 look in build_instance database. | |
| 13:58:35 | ebbex | otherwise it doesn't look there, only in "instances", and it's not there quite yet. still building i assume. | |
| 13:59:35 | ebbex | hence the 404. | |
| 13:59:46 | dansmith | so this is the app setup that does the service registration: | |
| 13:59:46 | dansmith | https://review.openstack.org/#/c/457283/4/nova/api/openstack/compute/wsgi.py | |
| 14:00:00 | dansmith | but I don't see how that's called from nova/wsgi/nova-api.py | |
| 14:00:46 | dansmith | the commit message there implies that you should be loading that as your wsgi script and that nova/wsgi is old | |
| 14:00:52 | dansmith | but it appears we're loading the old one in devstack, so I dunno | |
| 14:01:52 | ebbex | dansmith: well, ok. so this could all be the result of a mix-up. | |
| 14:02:11 | dansmith | I'm not sure why that was landed in a different place without removing the old one, | |
| 14:02:19 | dansmith | but also not sure why devstack isn't loading the new one | |
| 14:02:21 | dansmith | maybe sdague knows | |
| 14:02:48 | dansmith | ebbex: you might notice that the nova/wsgi one says "don't use this" :) | |
| 14:03:31 | sdague | dnawhat's the question? | |
| 14:03:36 | sdague | dansmith: ^^^ | |
| 14:04:02 | ebbex | dansmith: it says EXPERIMENTAL ;) | |
| 14:04:03 | dansmith | sdague: apparently cdent's updated wsgi app for nova-api was landed in a different place in the tree, and the old broken one is still there | |
| 14:04:13 | dansmith | sdague: and devstack seems to be using the old one | |
| 14:04:46 | dansmith | sdague: I thought we enabled apache wsgi by default in the gate for nova now, and would expect things to be broken if it's not using the new one | |
| 14:04:47 | sdague | dansmith: nova using the new one change hasn't landed yet, iirc, I'll go review | |
| 14:04:52 | dansmith | and also not sure why it was landed in tree | |
| 14:04:56 | sdague | no, it's an outstanding change | |
| 14:04:57 | dansmith | sdague: ah, okay well that explains that | |
| 14:05:00 | dansmith | gotcha | |
| 14:05:28 | dansmith | sdague: is that why we haven't removed the old one? because that'd be super confusing to our users, hence the above couple hours of debug :) | |
| 14:06:18 | sdague | yes, that's why the current stuff wasn't removed | |
| 14:06:41 | sdague | https://review.openstack.org/#/c/457715/ | |
| 14:06:45 | dansmith | okay well, the current one probably needs an even bigger, louder comment about the "the other one" | |
| 14:07:01 | dansmith | ack | |
| 14:07:16 | dansmith | ebbex: I assume you're swapping out to the other one to report back here right? | |
| 14:07:54 | ebbex | dansmith: yessir ;) | |
| 14:08:32 | dansmith | sdague: so I thought that we told tripleo to go ahead and swap back to apache mode because we had this running in our gate now.. I guess I missed something | |
| 14:09:02 | dansmith | I'll ask mriedem when/if he pops up | |
| 14:11:35 | sdague | dansmith: I don't know if anyone told tripleo that | |
| 14:11:47 | dansmith | sdague: mriedem definitely did | |
| 14:12:04 | sdague | so, if they use the new scripts, they should be fine | |
| 14:12:18 | sdague | I guess that was miscommunicated | |
| 14:12:49 | dansmith | they should, and we'd have heard about problems I'm sure, but I think it was under the assumption that we were testing it ourselves too | |
| 14:14:02 | ebbex | dansmith: that did the trick in registering the services. | |
| 14:14:45 | ebbex | dansmith: and the 404 race-condition thing :) | |
| 14:15:12 | dansmith | sdague: I guess he qualified it enough: [13:07:25] <mriedem> EmilienM: so kick the tires, but don't come screaming if there are issues :) | |
| 14:15:44 | dansmith | sdague: that was in may and the conversation definitely made it sound like we'd be running it before July or whenever this is likely to merge at this point :) | |
| 14:15:56 | dansmith | ebbex: good | |
| 14:16:17 | EmilienM | what's up? | |
| 14:16:19 | dansmith | sdague: anyway, I'll put another comment in the old file to try to reduce further confusion on the old one | |
| 14:16:24 | sdague | dansmith: sure | |
| 14:16:30 | dansmith | EmilienM: nothing, just quoting ancient history, carry on :) | |
| 14:16:36 | EmilienM | what did I do again? :P | |
| 14:16:43 | sdague | cdent did ask me to take a look while he was out, I'm rebasing now, we'll see what the tests look like | |
| 14:16:51 | dansmith | sdague: k | |
| 14:17:09 | ebbex | jaypipes, dansmith, sdague : thank you for all your assistance! :D | |
| 14:18:24 | jaypipes | ebbex: solved the problem? | |
| 14:20:30 | ebbex | jaypipes: Yes indeedy :) | |
| 14:20:42 | jaypipes | w00t. hats off to dansmith | |
| 14:21:45 | openstackgerrit | Dan Smith proposed openstack/nova master: Strengthen the warning on the old broken WSGI script https://review.openstack.org/478521 | |
| 14:21:46 | dansmith | jaypipes: sdague ebbex ^ | |
| 14:22:29 | sdague | dansmith: actually, you don't run the other thing instead | |
| 14:22:46 | dansmith | sdague: eh? | |
| 14:22:49 | sdague | you need to run the command that's built and put in /usr/local/bin with the pbr scripts | |
| 14:23:09 | sdague | nova-api-wsgi | |
| 14:23:30 | sdague | https://github.com/openstack/nova/blob/d5634769b47f7879892e34db0ad05483bf4b0652/setup.cfg#L69-L72 | |