| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-08 | |||
| 17:38:34 | fried_rice | cdent You going to be around in a couple hours? For a "warm handoff"? | |
| 17:38:41 | mriedem | gross | |
| 17:39:25 | cdent | fried_rice: yessir, I should be around, or at least nearby, make a ping and I’ll (eventually) come running | |
| 17:40:35 | fried_rice | cdent ack | |
| 17:40:59 | fried_rice | cdent Leaving a bit more of a mess than anticipated, I'm afraid. But nothing insurmountable. | |
| 17:49:26 | mnaser | is there anyone here who worked on glance image signature verification? | |
| 17:50:03 | mnaser | we have a change in puppet-nova to add the ability to configure it, but i'd like to know if its a global nova config (aka, api/etc need it) or nova-compute only | |
| 17:50:20 | mnaser | from my understanding of code, the option seems to be used in the glance image download code which probably only happens on computes? | |
| 17:52:09 | mriedem | peter-hamilton: ^ | |
| 17:52:24 | mriedem | mnaser: dane-fichter is who i'd usually go to but haven't seem him in channel in a long time | |
| 17:52:28 | alaski | mriedem: in case this didn't already get answered, instance actions are not synced between a cell and the api level in v1. There are spurious records at the API level that never get used, but it would have taken a lot of effort to not write them there. Requests for instance actions are proxied to the cell and results are returned from there. | |
| 17:52:54 | mnaser | mriedem: ok cool, thanks for the info.. i'll research the code a tad bit more | |
| 17:53:03 | mriedem | alaski: hmm, i'm not sure how @wrap_instance_event in the compute service would ever work then | |
| 17:53:08 | mriedem | alaski: because the api creates the action record | |
| 17:53:15 | mriedem | and the compute tries to look it up to record events against it | |
| 17:53:24 | mriedem | but if the action isn't in the cell db, how are they ever found? | |
| 17:53:34 | superdan | mriedem: they get replayed in the cell I imagine | |
| 17:53:42 | mriedem | superdan: where/how? | |
| 17:53:51 | superdan | so they're down there, but the api never looks at them, only at the stale ones that are left by the start | |
| 17:54:04 | superdan | mriedem: because we replay calls into compute/api down in the cell | |
| 17:54:11 | superdan | which would create them there too | |
| 17:54:37 | mriedem | so if a compute service fails to find an action created in the api, it could just be because of a timing issue? | |
| 17:54:55 | superdan | no, I don't think so, | |
| 17:55:02 | mriedem | ok, was going to say - i find that hard to believe | |
| 17:55:07 | mriedem | because we'd be randomly failing all the time | |
| 17:55:09 | superdan | because it should still get created in the cell before you kick off anything | |
| 17:56:08 | mriedem | ok that doesn't seem to be happening | |
| 17:56:23 | mriedem | i dug through the cellsv1 code for awhile yesterday and don't see anything special about instance actions in there thoguh | |
| 17:56:27 | mriedem | *though, like how they are handled | |
| 17:56:28 | superdan | right | |
| 17:56:31 | superdan | because they're not | |
| 17:57:21 | superdan | mriedem: for example, this one: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3098-L3099 | |
| 17:57:33 | superdan | mriedem: we run that in the api node as part of the cells_api indirection | |
| 17:57:46 | superdan | mriedem: then we pass it down to the cell and run that same code again in the cell, | |
| 17:57:46 | openstackgerrit | Merged openstack/nova-specs master: Document Queens review priorities https://review.openstack.org/526573 | |
| 17:57:57 | superdan | then we call to compute, which would finish it, but only in the cell | |
| 17:58:01 | superdan | and then you're done | |
| 17:58:07 | mriedem | and that's all synchronous? | |
| 17:58:24 | superdan | if you go look at it from the api, you'd only see the start, because you never finished it in the api cell, and nothing ever sync'd the finish | |
| 17:58:52 | superdan | mriedem: is what synchronous? the api cell before the child cell? sure, but they don't know about each other really anyway, so it doesn't matter | |
| 17:59:58 | mriedem | is the replay of the instance action create from the top level to the child cell synchronous | |
| 18:00:00 | mriedem | was my question | |
| 18:00:10 | mriedem | trying to figure out what is special that makes this fail the cellsv1 job https://review.openstack.org/#/c/523676 | |
| 18:00:15 | mriedem | http://logs.openstack.org/76/523676/11/check/legacy-tempest-dsvm-cells/1d155c6/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_07_09_02_09_579991 | |
| 18:00:33 | superdan | the thing that is replayed is the call to compute/api::confirm_resize() | |
| 18:00:34 | superdan | the action_create is | |
| 18:00:35 | superdan | not replayed itself, | |
| 18:00:36 | superdan | the whole call to that method is | |
| 18:01:27 | mriedem | ok, so backup_instance would all be replayed in the child cell api too right? | |
| 18:01:44 | superdan | yeah | |
| 18:01:49 | superdan | I don't see where the cells job ran on that | |
| 18:02:00 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2594 | |
| 18:02:09 | mriedem | the latest patch timed out in on the cells job | |
| 18:02:23 | superdan | okay | |
| 18:02:31 | superdan | would have thought that would be reported | |
| 18:02:40 | mriedem | zuul in the last 48 hours... | |
| 18:02:55 | superdan | oh I see, it is i the comment | |
| 18:02:57 | superdan | but not the summary | |
| 18:03:45 | alaski | stepped away, but it's exactly as superdan describes | |
| 18:03:54 | superdan | \o/ | |
| 18:04:23 | mriedem | so i wonder if it has something to do with this comment from comstud https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2594 | |
| 18:04:55 | mriedem | but i think that is just about setting the task_state | |
| 18:04:59 | superdan | mriedem: that's a remote call, so maybe it's calling to conductor which is looking in the wrong db? | |
| 18:05:15 | superdan | er, a remote exception I mean | |
| 18:06:02 | mriedem | well, | |
| 18:06:13 | mriedem | i don't know why it would just fail on that one lookup and not all of the others | |
| 18:06:13 | superdan | not sure that makes sense actually | |
| 18:06:19 | mriedem | that's what i don't understand about this | |
| 18:06:57 | mriedem | fried_rolls: have you ever mocked something that can't be imported? | |
| 18:07:01 | superdan | oh Iknow why | |
| 18:07:39 | superdan | mriedem: backup_instance doesn't get replayed in the cell for some reason, unlike everything else, which means it calls straight to the compute I think: https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L915-L925 | |
| 18:07:45 | superdan | mriedem: compare that to rebuild right below | |
| 18:08:01 | mriedem | ok yeah that's what i was looking at | |
| 18:08:27 | mriedem | https://github.com/openstack/nova/commit/2be50f0f3f02830798c25442dc54e92faa18106b | |
| 18:09:43 | superdan | I'd be extremely concerned about changing _any_ of that stuff at this point | |
| 18:09:50 | mriedem | weird, ok, so i guess the note about "any changes made here have to be made there" is what that's about | |
| 18:10:06 | mriedem | including creating the action record | |
| 18:10:14 | mriedem | well i see 2 options | |
| 18:10:29 | mriedem | 1. do the action create for backup and snapshot in the cells messaging code like it says, | |
| 18:10:46 | mriedem | 2. do something in the EventReporter where if cells is enabled, we blacklist some actions | |
| 18:10:55 | mriedem | we could remove #2 when we remove cellsv1 | |
| 18:11:29 | superdan | I don't love either of those | |
| 18:11:40 | peter-hamilton | mriedem: thanks! | |
| 18:11:49 | mriedem | superdan: me neither | |
| 18:11:57 | peter-hamilton | mnaser: bpoulos is who you want to talk to; I'll ping her for you | |
| 18:12:09 | mriedem | superdan: but i'm not sure what else to do at this point | |
| 18:12:15 | mnaser | peter-hamilton: cool, thanks! | |
| 18:12:16 | mriedem | so, i'll go to lunch and not think about this | |
| 18:17:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't launch guestfs in a thread pool if guestfs.debug is enabled https://review.openstack.org/526748 | |
| 18:17:58 | mriedem | dmsimard: clarkb: ^ | |
| 18:21:28 | dmsimard | mriedem: so we toggled the bug by setting it to debug ? :/ | |
| 18:21:40 | dmsimard | mriedem: the force_tcg patch would work without debug ? | |
| 18:25:55 | bpoulos | mnaser: only nova-compute needs the verify_glance_signatures option set in order to enable glance image signature verification | |
| 18:26:27 | mnaser | bpoulos: awesome, thank you so much (fyi, asking in context of https://review.openstack.org/#/c/526514/ ) | |
| 18:27:38 | bpoulos | mnaser: no problem, happy to help | |
| 18:56:42 | openstackgerrit | Jay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg https://review.openstack.org/526753 | |
| 19:10:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795 | |
| 19:16:35 | openstackgerrit | Merged openstack/nova master: Updated from global requirements https://review.openstack.org/526386 | |
| 19:27:26 | openstackgerrit | Merged openstack/nova master: Fix 'force' parameter in os-quota-sets PUT schema https://review.openstack.org/523194 | |
| 19:47:12 | mriedem | dmsimard: that's what i hope to find out | |