| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-28 | |||
| 10:57:41 | artom | If it's isn't there's not much you can unless you have access to the db and want to run SQL queries | |
| 10:58:17 | artom | If what you're looking for is in there you can bypass the python client and hit the API directly with curl, for example | |
| 10:59:12 | openstackgerrit | Chason Chan proposed openstack/nova master: import installation guide pages from openstack-manuals https://review.openstack.org/477488 | |
| 11:01:32 | Dinesh_Bhor | artom: no I don't have access to db. Do you think that we can returning system_metadata in show call ? | |
| 11:01:49 | Dinesh_Bhor | s/ returning/return | |
| 11:02:52 | artom | Dinesh_Bhor, there would be a lot of overhead (spec, api microversion) if we were to implement that, and I suspect it'll get shot down | |
| 11:03:30 | artom | Dinesh_Bhor, why are you looking for specific system metadata though? Maybe there's another way to accomplish what you're trying to do | |
| 11:09:44 | sdague | sfinucan: because you were handling a bunch of these deprecations, can you look at https://bugs.launchpad.net/nova/+bug/1698010 | |
| 11:09:44 | openstack | Launchpad bug 1698010 in OpenStack Compute (nova) "dhcp-domain is deprecated, but required for correct FQDN behavior" [Undecided,New] | |
| 11:12:23 | openstackgerrit | Chason Chan proposed openstack/nova master: import the admin guide content from openstack-manuals https://review.openstack.org/477497 | |
| 11:13:06 | Dinesh_Bhor | artom: Instance can be resized in two vm_states "active" and "stopped". I want to find out what is the previous vm_state of an Instance before it was resized through API. | |
| 11:13:18 | Dinesh_Bhor | artom: I can see that while reverting resized instance it gets the instance "old_vm_state" from instance system_metadata [1] and depending on it nova puts that instance back to that old_vm_state. | |
| 11:13:29 | Dinesh_Bhor | artom: [1] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4022 | |
| 11:14:29 | sfinucan | sdague: will do | |
| 11:18:42 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Format NIC features using os-traits definitions https://review.openstack.org/466051 | |
| 11:20:56 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Exception Error logs shown in Citrix XenServer CI https://review.openstack.org/478437 | |
| 11:35:59 | artom | Dinesh_Bhor, instance_actions might help you | |
| 11:36:42 | artom | (Using "might" in the literal sense - ie, look into it, but I'm not sure) | |
| 11:43:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Populate the 'user' section https://review.openstack.org/478475 | |
| 11:43:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Switch to openstackdocstheme https://review.openstack.org/477751 | |
| 11:43:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Populate the 'reference' section https://review.openstack.org/478484 | |
| 11:43:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Rework index page per new sections https://review.openstack.org/478485 | |
| 11:43:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: fixup! doc: Populate the 'reference' section https://review.openstack.org/478486 | |
| 11:43:24 | sfinucan | jaypipes, sdague: Those docs patches should be easy reviews and have to get in this cycle. Would appreciate it if either of ye could take a look sometime this week | |
| 11:43:34 | sfinucan | Will ping mriedem too once he's about | |
| 11:43:52 | jaypipes | will do sfinucan | |
| 11:44:12 | sfinucan | ta | |
| 11:44:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Populate the 'user' section https://review.openstack.org/478475 | |
| 11:44:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Switch to openstackdocstheme https://review.openstack.org/477751 | |
| 11:44:29 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Populate the 'reference' section https://review.openstack.org/478484 | |
| 11:44:29 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Rework index page per new sections https://review.openstack.org/478485 | |
| 11:44:33 | alex_xu | jaypipes: hope this is helpful https://review.openstack.org/478464 | |
| 11:47:48 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | |
| 11:48:56 | jaypipes | alex_xu: thx! reviewing now. BTW, I can't figure out why https://review.openstack.org/#/c/475448/ is failing that one functional test (about the traits sync...) | |
| 11:49:19 | jaypipes | alex_xu: it's passing locally and the version of os-traits in requirements.txt is the same. | |
| 11:49:29 | jaypipes | alex_xu: perhaps you might take a look? | |
| 11:50:04 | ebbex | artom: so we've possibly traced it down a bit further, on "server create" we see inserts into request_spec, build_requests and instance_mappings. But the GET we see right after has a SELECT from instances. But we don't see what is inserting the uuid into this table. Any idea how something ends up in the "instances" table? | |
| 11:50:12 | alex_xu | jaypipes: yea, let me try | |
| 11:51:03 | jaypipes | alex_xu: if I can't figure it out, I can revert to trying to create MISC_SHARES_VIA_AGGREGATE manually, like we do here currently: https://github.com/openstack/nova/blob/master/nova/tests/functional/db/test_resource_provider.py#L2533-L2546 | |
| 11:51:36 | jaypipes | alex_xu: edleafe had asked me to use Trait.get_by_name() in the new patch now that we have os-traits 0.3.1 in the requirements. | |
| 11:51:42 | ebbex | Is that table populated by something else via rpc? nova-scheduler maybe? | |
| 11:51:59 | alex_xu | jaypipes: ok, got it, will let you know whehter I can found something | |
| 11:52:04 | jaypipes | cheers | |
| 11:52:13 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add BDM to InstancePayload https://review.openstack.org/448779 | |
| 11:52:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add key_name field to InstancePayload https://review.openstack.org/463002 | |
| 11:52:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add keypairs field to InstanceCreatePayload https://review.openstack.org/419730 | |
| 11:53:17 | jaypipes | ebbex: the instance UUID is generated in the nova-api server and placed in the instance_mappings table before scheduling. i.e. we pre-create the instance's UUID when we create the request spec. | |
| 11:56:16 | jaypipes | ebbex: the first time the cell DB's instances table is written to (and populates the instances.uuid field) is here: https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L964 | |
| 11:56:58 | jaypipes | ebbex: you'll note that is after the scheduler has selected the destination host to attempt the build on and before the compute service is notified to build the instance (line 999) | |
| 11:57:17 | jaypipes | ebbex: hope this helps. | |
| 11:57:53 | ebbex | jaypipes: so it is nova conductor that places the instance in the instance table, right? But.. as this is done via rpc, am I correct in assuming nova-api is not waiting for a reply from conductor (instance inserted in instances table) before returning 202? | |
| 11:58:23 | sdague | sfinucan: I just went through most of that stack, I've got some concerns by some of the changes. | |
| 11:58:56 | sfinucan | sdague: Anything in particular, or is it all in Gerrit? | |
| 11:59:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send soft_delete from context manager https://review.openstack.org/476459 | |
| 11:59:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297 | |
| 11:59:22 | jaypipes | ebbex: no, the (super)conductor writes the instances record to the cell DB on line 964 *before* RPC'ing the compute service. | |
| 11:59:33 | jaypipes | ebbex: and that write is synchronous. | |
| 12:00:09 | sdague | sfinucan: it's all in gerrir | |
| 12:00:12 | sdague | gerrit | |
| 12:00:20 | ebbex | jaypipes: but then how come the select on the instances table _after_ nova-api has returned 202 on to the create instance call returns no match? | |
| 12:01:22 | jaypipes | ebbex: nova-api returns a 202 when it creates the *build request*. The build request is in the API database (top-level DB). The instances table is in the cell DB and is written to after scheduling has selected a target host (and therefore a target cell). | |
| 12:01:36 | jaypipes | ebbex: nova-api calls nova-conductor after passing a 202 back to the user. | |
| 12:01:42 | ebbex | jaypipes: there must be a race condition here, and even tho we are using galera, we (currently) send all of nova*'s sql connections to a single galera instance (trying to eliminate "slave-lag" problems) | |
| 12:02:23 | jaypipes | ebbex: nova-conductor then calls nova-scheduler, which selects the host and returns that selection back to the conductor, which then writes the instances record to the cell DB and sends the launch instance RPC call to the target compute host. | |
| 12:02:50 | jaypipes | ebbex: it's not a race condition. it's working as intended :) | |
| 12:03:15 | jaypipes | ebbex: the instances table of the cell DB will not have a record for the instance with that UUID until the scheduler selects a host. | |
| 12:03:31 | jaypipes | ebbex: the UUID you get back is the one from the build_requests table in the top-level API database. | |
| 12:04:26 | ebbex | jaypipes: but nova-api then serves a location with the 202 response that returns 404... hmm, that cant be intentional? | |
| 12:04:30 | jaypipes | ebbex: when the compute host is selected by the scheduler, the instances table in the cell database is written and the build_requests table entry is deleted (here: https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L986) | |
| 12:04:55 | artom | ebbex, what database are you looking in? | |
| 12:04:57 | jaypipes | ebbex: no. the location returned by nova-api *does* exist. | |
| 12:05:07 | ebbex | jaypipes: nope | |
| 12:05:39 | jaypipes | ebbex: execute list instances with an admin user. | |
| 12:05:56 | ebbex | jaypipes: so, after a post to /v2.1/servers, nova-api replies with 202 and location: http://nova-api/v2.1/servers/899d5996-a030-44ac-ad34-28d261255d2d | |
| 12:06:08 | jaypipes | ebbex: something is causing the build to go into an error state somewhere I suspect. | |
| 12:06:20 | jaypipes | ebbex: yes, that URI is valid. | |
| 12:06:36 | ebbex | jaypipes: that url is not valid until the instance is inserted into the instances table | |
| 12:06:43 | jaypipes | ebbex: no, that's not true. | |
| 12:06:46 | ebbex | and there you have your race condition | |
| 12:06:48 | jaypipes | ebbex: let me explain. | |
| 12:07:32 | ebbex | jaypipes: ok... atleast that is what we see here, a select ... where ...uuid= ... from instances .. returning no entries and nova-api returning us 404 | |
| 12:08:01 | jaypipes | ebbex: the information returned from the GET /servers/<UUID> will be information about the build request (flavor, image, the instance UUID, etc) *until the instances record in the cell DB has been written, in which case the build_requests table entry will be deleted* | |
| 12:08:35 | jaypipes | ebbex: where is your cell DB vs. your API db? | |
| 12:08:36 | ebbex | jaypipes: so what then causes the 404 from the GET servers/UUID? | |
| 12:08:51 | Dinesh_Bhor | artom: ok, ill try, thanks for the help | |
| 12:10:05 | jaypipes | ebbex: have you looked in your nova-conductor log for errors? | |
| 12:10:06 | ebbex | cell0 is nova_cell0, cell1 is nova, nova-api has nova_api and placement-api uses nova_api database. all on the same server. | |
| 12:10:22 | jaypipes | ebbex: I'm wondering perhaps if you're hitting this: https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L936 | |
| 12:11:00 | jaypipes | ebbex: have you checked all the logs for errors? anything in there? | |
| 12:11:50 | ebbex | We dont get any errors and the instance is started just fine | |
| 12:12:26 | ebbex | and if we do a server show after the server create has returned the 404 error, it is there | |
| 12:12:53 | jaypipes | ebbex: may I ask how you are creating the server? | |
| 12:12:57 | ebbex | so the problem only exists for small amount of time, but still causes throuble in openstackclient and what-not | |
| 12:13:22 | ebbex | server create --wait --image cirros.raw --flavor m1.small --nic net-id=network1 server1 | |
| 12:14:03 | jaypipes | ebbex: k, thank you. sounds like a bug indeed. would you mind creating a bug for this? | |
| 12:15:04 | sfinucan | sdague: comments left on the controversial ones | |
| 12:18:25 | ebbex | jaypipes: I do not see nova-api doing any other sql queries than the one against instances that returns empty | |
| 12:18:59 | jaypipes | ebbex: would you mind putting all of this into a bug report? | |
| 12:19:24 | ebbex | jaypipes: so if that you said earlier is correct, that is should also look in build_requests or whatever, it is not happening | |
| 12:19:38 | jaypipes | efried: lol. Deputy Dan :) | |