| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-28 | |||
| 12:06:48 | jaypipes | ebbex: let me explain. | |
| 12:07:32 | ebbex | jaypipes: ok... atleast that is what we see here, a select ... where ...uuid= ... from instances .. returning no entries and nova-api returning us 404 | |
| 12:08:01 | jaypipes | ebbex: the information returned from the GET /servers/<UUID> will be information about the build request (flavor, image, the instance UUID, etc) *until the instances record in the cell DB has been written, in which case the build_requests table entry will be deleted* | |
| 12:08:35 | jaypipes | ebbex: where is your cell DB vs. your API db? | |
| 12:08:36 | ebbex | jaypipes: so what then causes the 404 from the GET servers/UUID? | |
| 12:08:51 | Dinesh_Bhor | artom: ok, ill try, thanks for the help | |
| 12:10:05 | jaypipes | ebbex: have you looked in your nova-conductor log for errors? | |
| 12:10:06 | ebbex | cell0 is nova_cell0, cell1 is nova, nova-api has nova_api and placement-api uses nova_api database. all on the same server. | |
| 12:10:22 | jaypipes | ebbex: I'm wondering perhaps if you're hitting this: https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L936 | |
| 12:11:00 | jaypipes | ebbex: have you checked all the logs for errors? anything in there? | |
| 12:11:50 | ebbex | We dont get any errors and the instance is started just fine | |
| 12:12:26 | ebbex | and if we do a server show after the server create has returned the 404 error, it is there | |
| 12:12:53 | jaypipes | ebbex: may I ask how you are creating the server? | |
| 12:12:57 | ebbex | so the problem only exists for small amount of time, but still causes throuble in openstackclient and what-not | |
| 12:13:22 | ebbex | server create --wait --image cirros.raw --flavor m1.small --nic net-id=network1 server1 | |
| 12:14:03 | jaypipes | ebbex: k, thank you. sounds like a bug indeed. would you mind creating a bug for this? | |
| 12:15:04 | sfinucan | sdague: comments left on the controversial ones | |
| 12:18:25 | ebbex | jaypipes: I do not see nova-api doing any other sql queries than the one against instances that returns empty | |
| 12:18:59 | jaypipes | ebbex: would you mind putting all of this into a bug report? | |
| 12:19:24 | ebbex | jaypipes: so if that you said earlier is correct, that is should also look in build_requests or whatever, it is not happening | |
| 12:19:38 | jaypipes | efried: lol. Deputy Dan :) | |
| 12:19:46 | efried | :) | |
| 12:21:38 | sdague | sfinucan: where is the blueprint for this, the blueprint link is broken | |
| 12:21:41 | jaypipes | ebbex: nova-api definitely is looking in build_requests. https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2564 | |
| 12:22:06 | ebbex | jaypipes: how come I do not see the sql query in the tcpdump? | |
| 12:22:57 | openstackgerrit | Alex Xu proposed openstack/nova master: DNM: debug traits sync https://review.openstack.org/478495 | |
| 12:23:01 | alex_xu | jaypipes: reproduce same fail in the local, after adding some logs...the test successful now...even remove those logs | |
| 12:23:22 | alex_xu | jaypipes: ^ adding a log, then let see if they still reproducable in the gate | |
| 12:23:36 | jaypipes | alex_xu: k, ty sir | |
| 12:23:41 | jaypipes | ebbex: not sure :( | |
| 12:23:42 | alex_xu | np | |
| 12:24:00 | jaypipes | ebbex: what version of Nova are you on? | |
| 12:24:24 | ebbex | jaypipes: we are talking about "server show" here right, not "server list"? | |
| 12:24:28 | ebbex | ocata branch | |
| 12:25:01 | ebbex | jaypipes: because the line you refer to seems to be related to "server list" | |
| 12:26:22 | artom | ebbex, you said this is load balanced, right? What's the db setup like? | |
| 12:26:23 | jaypipes | ebbex: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2435 | |
| 12:26:31 | jaypipes | ebbex: same thing for show. | |
| 12:26:55 | ebbex | artom: it is not load-balanced any more, disabled to eliminate errors caused by this. | |
| 12:26:59 | jaypipes | ebbex: ooooh | |
| 12:27:20 | artom | ebbex, so everything on a single machine? | |
| 12:27:21 | jaypipes | ebbex: try setting the service version of nova-osapi_compute to 16. | |
| 12:27:58 | ebbex | ok.. lets try that | |
| 12:28:17 | jaypipes | ebbex: go into the API database's services table and update the version column to 16 for all services where the binary column is "nova-osapi_compute" | |
| 12:29:09 | jaypipes | ebbex: sorry, meant the cell1 db | |
| 12:29:12 | jaypipes | not the api db | |
| 12:32:09 | ebbex | hmm.. | |
| 12:33:08 | ebbex | there is no binary that has this name | |
| 12:33:30 | ebbex | also, nice to use a function name as a column name :) | |
| 12:34:05 | jaypipes | ebbex: yeah... :( | |
| 12:34:35 | jaypipes | ebbex: so I'm pretty sure this is the source of the bug: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2422 | |
| 12:34:53 | jaypipes | ebbex: or rather, that is the source of your woes... :) | |
| 12:35:14 | jaypipes | ebbex: either the service version being returned by the call here: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2417 | |
| 12:35:24 | jaypipes | is not >= 15 | |
| 12:35:34 | jaypipes | or you have cellsv1 set up, which I doubt :) | |
| 12:35:42 | ebbex | no cellsv1 | |
| 12:36:09 | ebbex | so where do I set the service_version if its not in the services table, and what can cause it to be there | |
| 12:36:21 | jaypipes | ebbex: can you try removing lines 2422-2424 from the code and retrying the scenario? | |
| 12:37:10 | jaypipes | ebbex: I'm thinking that "nova-osapi_compute" is wrong for existing installations maybe. Perhaps that should be "nova-api"? | |
| 12:37:23 | jaypipes | ebbex: what does this give you if you run it on the cell DB: | |
| 12:37:31 | ebbex | jaypipes: could I have bootstrapped something wrong with cellsv2? should there be an entry for nova-os-compute-api? | |
| 12:37:38 | jaypipes | SELECT binary, count(*) as c from services group by binary; | |
| 12:37:52 | jaypipes | ebbex: not sure, trying to find out :) | |
| 12:38:27 | ebbex | remember `binary` ;) | |
| 12:38:34 | jaypipes | heh, yes | |
| 12:38:46 | jaypipes | was just about to say that :) | |
| 12:38:53 | ebbex | | nova-compute | 2 | | |
| 12:38:53 | ebbex | | nova-conductor | 3 | | |
| 12:38:54 | ebbex | | nova-console | 3 | | |
| 12:38:54 | ebbex | | nova-consoleauth | 3 | | |
| 12:38:54 | ebbex | | nova-scheduler | 3 | | |
| 12:39:31 | jaypipes | is there a services table in the nova_api db? | |
| 12:39:39 | ebbex | nope. | |
| 12:39:49 | jaypipes | hmm.. | |
| 12:39:57 | jaypipes | I wonder why there's no entry for the api services... | |
| 12:41:52 | jaypipes | ebbex: I need to run out for a bit. I'm hoping dansmith will have some ideas when he gets online about this. | |
| 12:42:39 | ebbex | jaypipes: cool, thanks :) | |
| 12:45:07 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for images instead of stevedore https://review.openstack.org/478498 | |
| 12:45:08 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for image-metadata instead of stevedore https://review.openstack.org/478499 | |
| 12:45:49 | alex_xu | nova api meeting is in 15 mins at #openstack-meeting-4 | |
| 12:59:50 | openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for volumes instead of stevedore https://review.openstack.org/478502 | |
| 13:01:20 | alex_xu | nova api meeting is running | |
| 13:04:48 | openstackgerrit | Eric Fried proposed openstack/nova master: Monkey patch the blockdiag extension https://review.openstack.org/476159 | |
| 13:04:49 | openstackgerrit | Eric Fried proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810 | |
| 13:05:08 | efried | sfinucan ^^ resolve merge conflict (test-requirements.txt) | |
| 13:05:33 | openstackgerrit | Eric Fried proposed openstack/nova master: Add formatting to scheduling activity diagram https://review.openstack.org/476204 | |
| 13:05:40 | jaypipes | artom: any ideas on that drive-scsi0-0-0-0 issue from a couple days ago? is lyarwood around by any chance? | |
| 13:06:19 | artom | jaypipes, I haven't looked into it further since then | |
| 13:10:53 | dimak | Hey, I'm trying to bring up a large VM on a recent devstack, and I'm getting 'No valid host was found' | |
| 13:11:43 | dimak | And the scheduler logs say 'Starting with 0 host(s)' while I have 2 hypervisors and a bit smaller VM is able to get scheduled | |
| 13:12:09 | kashyap | dimak: I recall fixing that, but now I forget how or what I did | |
| 13:12:30 | kashyap | There are other people who meditate on scheduler stuff here, though. Let's wait for them to chime in | |
| 13:12:46 | dimak | Sure, thanks :) | |
| 13:13:26 | ebbex | jaypipes: dansmith : I think you've helped track it down, any idea which nova-manage command puts the 'nova-osapi_compute' binary into the services table? | |
| 13:14:57 | klipka | dimak: Could you provide log from nova-scheduler with debug set to true? | |
| 13:16:05 | dimak | klipka, sure, a moment | |
| 13:17:48 | klipka | first wild guess is that Placement API is not configured | |
| 13:18:12 | klipka | refer to second bullet here: https://docs.openstack.org/developer/nova/placement.html#ocata-15-0-0 | |
| 13:20:33 | dimak | klipka, https://paste.fedoraproject.org/paste/E6FuxcJaKPJ5qcKUauoG9g | |
| 13:21:25 | dimak | klipka, I have placement API configured on my other node, should I have some placement service running there as well? | |