| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-31 | |||
| 19:08:35 | mriedem | because all that api does is literally change the vm_state in the db https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/admin_actions.py#L82 | |
| 19:10:41 | mriedem | the upcall to the api cell is an rpc cast, | |
| 19:10:47 | mriedem | i wonder if that's why it's a 202, for cells v1 | |
| 19:13:40 | artom | Where does the cells call happen in that? | |
| 19:13:51 | mriedem | in nova.objects.Instance.save | |
| 19:14:35 | melwitt | I would think that would save it at the top and should be returned as 'error' as expected by the test. the save() would sync the vm_state change to the child. the get_all in compute/api is reading from child cells, so that's probably how it's happening | |
| 19:15:13 | melwitt | we've put in some conditionals "if cells v1, read from the top" but it looks like this was missed, for the get_all | |
| 19:15:20 | artom | Bah, vim ctags has failed me | |
| 19:15:34 | artom | It took me to agents.py save method for some reason | |
| 19:16:04 | openstack | Launchpad bug 1660747 in OpenStack Compute (nova) "test_list_servers_filter_by_error_status intermittently fails with MismatchError on no servers in response" [Medium,Confirmed] | |
| 19:16:04 | mriedem | here is the bug btw https://bugs.launchpad.net/nova/+bug/1660747 | |
| 19:16:25 | melwitt | thanks | |
| 19:16:38 | openstackgerrit | Lee Yarwood proposed openstack/nova: WIP - libvirt: Remove redundant bdm serial mangling during swap_volume https://review.openstack.org/427364 | |
| 19:17:15 | mriedem | sfinucan: do you know the ReST magic to find the root of a doc tree? | |
| 19:17:19 | mriedem | for a link | |
| 19:18:25 | artom | melwitt, you wanna propose a fix to Tempest? Shall I? | |
| 19:18:53 | dansmith | melwitt: I definitely had that in one of my patches at some point | |
| 19:19:03 | melwitt | artom: I'll propose a fix to nova, it's the get_all that needs a tweak I think | |
| 19:19:15 | dansmith | melwitt: for get_all, but I don't see it right now, so might've been the victim of a rebase, although I was failing the cells job regularly without it | |
| 19:19:34 | artom | melwitt, aha! | |
| 19:19:35 | melwitt | dansmith: yeah, I think it must have gotten lost in the mix | |
| 19:19:56 | artom | I still think Tempest should wait after a request that returns a 202, not just assume it's been completed... | |
| 19:20:04 | openstackgerrit | Eric Fried proposed openstack/nova: Introduce base PowerVM driver https://review.openstack.org/391288 | |
| 19:20:22 | openstackgerrit | Sarafraj Singh proposed openstack/nova-specs: Send service token along with user tokens https://review.openstack.org/426933 | |
| 19:20:26 | mriedem | melwitt: technically, we don't have to care about fixing latent bugs in cells v1 | |
| 19:21:08 | dansmith | mriedem: this isn't latent | |
| 19:21:10 | melwitt | mriedem: it's not a latent bug in cells v1. it should have been working fine until we added cells v2 instance listing from all cells | |
| 19:22:59 | openstackgerrit | Matt Riedemann proposed openstack/nova: doc: add upgrade notes to the placement devref https://review.openstack.org/427367 | |
| 19:23:01 | mriedem | cdent: bauzas: ^ | |
| 19:23:11 | mriedem | dansmith: melwitt: ok | |
| 19:24:02 | browne | mriedem: do the RPC versions need to be updated for Ocata? seemed so, so i put up this patch: https://review.openstack.org/#/c/425958/ | |
| 19:24:36 | lyarwood | dansmith: https://review.openstack.org/#/c/427267 should be ready for a review if you have time later today | |
| 19:25:06 | melwitt | artom: I think we've left the test alone since it helps catch our bugs, but technically you're right, a 202 means wait | |
| 19:26:03 | artom | melwitt, well, is the problem that the server state update takes too long, or doesn't happen at all? Because the latter isn't affected by waiting | |
| 19:26:22 | mriedem | it would have to be a race, | |
| 19:26:26 | mriedem | else this would be 100% failure | |
| 19:26:35 | melwitt | artom: the problem is our code is reading instances from the child cells instead of the parent cell, for cells v1 | |
| 19:26:40 | artom | It'll still catch the bugs, except it'll do it by timing out on the wait_for_server_status call | |
| 19:27:10 | melwitt | most of the time, the sync down gets to the child before our read does, and in that case, it will pass the test | |
| 19:27:23 | melwitt | no, it wouldn't catch this bug I don't think | |
| 19:27:45 | melwitt | this is like a split second difference | |
| 19:28:29 | artom | Ah, so the status ends up being synced in all cases | |
| 19:28:36 | melwitt | yes | |
| 19:28:39 | mriedem | lyarwood: i threw the rc-potential tag on that one since it's a regression | |
| 19:29:03 | lyarwood | mriedem: ack thanks | |
| 19:29:23 | artom | melwitt, wait, just to make this clear in my head: the write happens on the parent cell, the read from the child cells, so if you time your read really quickly after the write, you won't get the new state? | |
| 19:29:42 | artom | And the fix you have in mind is to read from the parent cell? | |
| 19:30:05 | melwitt | yes. cells v1 normally reads from the parent cell. it has instances table in the parent and the childs and it syncs them | |
| 19:31:32 | artom | Hrmpf, this is indeed a case of a test being technically wrong but actually useful | |
| 19:31:47 | dansmith | artom: no | |
| 19:31:54 | dansmith | artom: the test case can't tell where it's being read from | |
| 19:32:23 | melwitt | he means the 202 return code implies async so it shouldn't just assume the state change is available after the call returns | |
| 19:32:35 | artom | melwitt, words out of my mouth :) | |
| 19:32:44 | melwitt | we know that it should be, from the code. but the API doesn't tell that | |
| 19:32:51 | dansmith | is it not polling? | |
| 19:33:00 | melwitt | no, it's just immediately reading after | |
| 19:33:00 | artom | dansmith, nope | |
| 19:33:03 | dansmith | ah, okay | |
| 19:33:07 | dansmith | then yeah | |
| 19:33:30 | jaypipes | mriedem, cdent: yes, I'm still working on functional tests for the Ironic patch. Didn't want you to think I forgot or am slacking. | |
| 19:34:08 | mriedem | i'm pretty far removed from that patch by now, | |
| 19:34:12 | mriedem | once john manhandled it | |
| 19:34:44 | mriedem | like, before i start reviewing it again, someone is probably going to have to give me a recap of what happened | |
| 19:35:57 | mriedem | browne: probably... | |
| 19:36:26 | mriedem | i feel like russellb used to have a cheat sheet for this | |
| 19:42:08 | openstackgerrit | Matt Riedemann proposed openstack/nova: doc: add upgrade notes to the placement devref https://review.openstack.org/427367 | |
| 19:45:37 | mordred | mriedem: I've got this error coming up on nodepool patches: http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/screen-nodepool.txt.gz#_2017-01-31_19_02_00_177 | |
| 19:45:57 | mordred | mriedem: which is reporting a 500 error from nova and <class 'sqlalchemy.exc.ProgrammingError'> as the error | |
| 19:46:33 | mriedem | nice http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/screen-n-api.txt.gz?level=TRACE#_2017-01-31_19_02_00_169 | |
| 19:46:41 | mordred | yah | |
| 19:46:51 | mordred | isn't that pretty? (I was just about to paste that link next :) ) | |
| 19:46:53 | mriedem | Table 'nova_cell0.instance_extra' doesn't exist | |
| 19:47:00 | mriedem | mordred: you don't have your cells v2 stuff setup | |
| 19:47:02 | mriedem | in whatever that thing is | |
| 19:47:18 | mordred | well, it's nodepool's functional tests that run on top of a devstack node | |
| 19:47:36 | mordred | does devstack-gate need cellsv2 now but not set it up by default? :( | |
| 19:47:47 | mriedem | devstack sets up cells v2 | |
| 19:48:05 | mriedem | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L684 | |
| 19:48:09 | mordred | neat. well, apparently something I'm doing there is not letting it I guess - Ill look fruther | |
| 19:48:10 | mriedem | ^ creates the cell0 db | |
| 19:48:28 | mordred | wow. so it should just do it without any flags it looks like | |
| 19:48:43 | mriedem | mordred: and this stuff at the end https://github.com/openstack-dev/devstack/blob/master/lib/nova#L947 | |
| 19:48:53 | mriedem | mordred: yes, cells v2 is required now in ocata | |
| 19:49:12 | mriedem | all ocata nova deployments will be at least a cell of one | |
| 19:49:24 | mordred | mriedem: is there a cell service I need to enable? | |
| 19:49:41 | mriedem | no | |
| 19:49:49 | mriedem | that's cells v1 | |
| 19:49:55 | mriedem | cells v2 is baked in | |
| 19:50:05 | mriedem | here is the nova_cell0 db create in your run http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_18_58_29_045 | |
| 19:50:20 | mriedem | map_cell0 http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_19_01_45_588 | |
| 19:50:31 | mriedem | and simple_cell_setup http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_19_01_48_335 | |
| 19:50:48 | mordred | WEIRD | |
| 19:51:03 | mriedem | that's an appropriate reaction | |
| 19:51:25 | mriedem | http://docs.openstack.org/developer/nova/cells.html | |
| 19:52:37 | mriedem | so when your thing blew up in n-api, cell0 would be mapped | |
| 19:52:42 | mriedem | so i'm not sure why it's not finding the table | |
| 19:52:43 | mriedem | *db | |
| 19:52:53 | mriedem | or, maybe it found the db but the schema wasn't migrated? | |
| 19:53:05 | mriedem | simple_cell_setup will migrate the db schema on the cell0 db | |
| 19:53:16 | mordred | does it do the migration asynchronously? | |