| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-08 | |||
| 17:55:09 | superdan | because it should still get created in the cell before you kick off anything | |
| 17:56:08 | mriedem | ok that doesn't seem to be happening | |
| 17:56:23 | mriedem | i dug through the cellsv1 code for awhile yesterday and don't see anything special about instance actions in there thoguh | |
| 17:56:27 | mriedem | *though, like how they are handled | |
| 17:56:28 | superdan | right | |
| 17:56:31 | superdan | because they're not | |
| 17:57:21 | superdan | mriedem: for example, this one: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3098-L3099 | |
| 17:57:33 | superdan | mriedem: we run that in the api node as part of the cells_api indirection | |
| 17:57:46 | superdan | mriedem: then we pass it down to the cell and run that same code again in the cell, | |
| 17:57:46 | openstackgerrit | Merged openstack/nova-specs master: Document Queens review priorities https://review.openstack.org/526573 | |
| 17:57:57 | superdan | then we call to compute, which would finish it, but only in the cell | |
| 17:58:01 | superdan | and then you're done | |
| 17:58:07 | mriedem | and that's all synchronous? | |
| 17:58:24 | superdan | if you go look at it from the api, you'd only see the start, because you never finished it in the api cell, and nothing ever sync'd the finish | |
| 17:58:52 | superdan | mriedem: is what synchronous? the api cell before the child cell? sure, but they don't know about each other really anyway, so it doesn't matter | |
| 17:59:58 | mriedem | is the replay of the instance action create from the top level to the child cell synchronous | |
| 18:00:00 | mriedem | was my question | |
| 18:00:10 | mriedem | trying to figure out what is special that makes this fail the cellsv1 job https://review.openstack.org/#/c/523676 | |
| 18:00:15 | mriedem | http://logs.openstack.org/76/523676/11/check/legacy-tempest-dsvm-cells/1d155c6/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_07_09_02_09_579991 | |
| 18:00:33 | superdan | the thing that is replayed is the call to compute/api::confirm_resize() | |
| 18:00:34 | superdan | the action_create is | |
| 18:00:35 | superdan | not replayed itself, | |
| 18:00:36 | superdan | the whole call to that method is | |
| 18:01:27 | mriedem | ok, so backup_instance would all be replayed in the child cell api too right? | |
| 18:01:44 | superdan | yeah | |
| 18:01:49 | superdan | I don't see where the cells job ran on that | |
| 18:02:00 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2594 | |
| 18:02:09 | mriedem | the latest patch timed out in on the cells job | |
| 18:02:23 | superdan | okay | |
| 18:02:31 | superdan | would have thought that would be reported | |
| 18:02:40 | mriedem | zuul in the last 48 hours... | |
| 18:02:55 | superdan | oh I see, it is i the comment | |
| 18:02:57 | superdan | but not the summary | |
| 18:03:45 | alaski | stepped away, but it's exactly as superdan describes | |
| 18:03:54 | superdan | \o/ | |
| 18:04:23 | mriedem | so i wonder if it has something to do with this comment from comstud https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2594 | |
| 18:04:55 | mriedem | but i think that is just about setting the task_state | |
| 18:04:59 | superdan | mriedem: that's a remote call, so maybe it's calling to conductor which is looking in the wrong db? | |
| 18:05:15 | superdan | er, a remote exception I mean | |
| 18:06:02 | mriedem | well, | |
| 18:06:13 | mriedem | i don't know why it would just fail on that one lookup and not all of the others | |
| 18:06:13 | superdan | not sure that makes sense actually | |
| 18:06:19 | mriedem | that's what i don't understand about this | |
| 18:06:57 | mriedem | fried_rolls: have you ever mocked something that can't be imported? | |
| 18:07:01 | superdan | oh Iknow why | |
| 18:07:39 | superdan | mriedem: backup_instance doesn't get replayed in the cell for some reason, unlike everything else, which means it calls straight to the compute I think: https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L915-L925 | |
| 18:07:45 | superdan | mriedem: compare that to rebuild right below | |
| 18:08:01 | mriedem | ok yeah that's what i was looking at | |
| 18:08:27 | mriedem | https://github.com/openstack/nova/commit/2be50f0f3f02830798c25442dc54e92faa18106b | |
| 18:09:43 | superdan | I'd be extremely concerned about changing _any_ of that stuff at this point | |
| 18:09:50 | mriedem | weird, ok, so i guess the note about "any changes made here have to be made there" is what that's about | |
| 18:10:06 | mriedem | including creating the action record | |
| 18:10:14 | mriedem | well i see 2 options | |
| 18:10:29 | mriedem | 1. do the action create for backup and snapshot in the cells messaging code like it says, | |
| 18:10:46 | mriedem | 2. do something in the EventReporter where if cells is enabled, we blacklist some actions | |
| 18:10:55 | mriedem | we could remove #2 when we remove cellsv1 | |
| 18:11:29 | superdan | I don't love either of those | |
| 18:11:40 | peter-hamilton | mriedem: thanks! | |
| 18:11:49 | mriedem | superdan: me neither | |
| 18:11:57 | peter-hamilton | mnaser: bpoulos is who you want to talk to; I'll ping her for you | |
| 18:12:09 | mriedem | superdan: but i'm not sure what else to do at this point | |
| 18:12:15 | mnaser | peter-hamilton: cool, thanks! | |
| 18:12:16 | mriedem | so, i'll go to lunch and not think about this | |
| 18:17:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't launch guestfs in a thread pool if guestfs.debug is enabled https://review.openstack.org/526748 | |
| 18:17:58 | mriedem | dmsimard: clarkb: ^ | |
| 18:21:28 | dmsimard | mriedem: so we toggled the bug by setting it to debug ? :/ | |
| 18:21:40 | dmsimard | mriedem: the force_tcg patch would work without debug ? | |
| 18:25:55 | bpoulos | mnaser: only nova-compute needs the verify_glance_signatures option set in order to enable glance image signature verification | |
| 18:26:27 | mnaser | bpoulos: awesome, thank you so much (fyi, asking in context of https://review.openstack.org/#/c/526514/ ) | |
| 18:27:38 | bpoulos | mnaser: no problem, happy to help | |
| 18:56:42 | openstackgerrit | Jay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg https://review.openstack.org/526753 | |
| 19:10:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795 | |
| 19:16:35 | openstackgerrit | Merged openstack/nova master: Updated from global requirements https://review.openstack.org/526386 | |
| 19:27:26 | openstackgerrit | Merged openstack/nova master: Fix 'force' parameter in os-quota-sets PUT schema https://review.openstack.org/523194 | |
| 19:47:12 | mriedem | dmsimard: that's what i hope to find out | |
| 19:50:24 | fried_rolls | mriedem No, I don't think I have ever (successfully) mocked something that can't be imported. It will fail at "compile" time. | |
| 19:50:44 | fried_rolls | mriedem You can cheat by mocking it after it's imported, though. | |
| 19:52:07 | fried_rolls | I assume this is in a situation where the thing imports correctly on the prod side but for some reason not on the test side? Which... why? | |
| 19:53:39 | fried_rice | cdent How's now? | |
| 19:53:48 | cdent | one moment caller | |
| 19:54:34 | cdent | blargh, timeouts in migration tests: http://logs.openstack.org/39/521639/8/gate/openstack-tox-functional/f6880ac/testr_results.html.gz | |
| 20:02:55 | cdent | fried_rice: okay, sorry, had to feed that cat and put dinner dishes aweay | |
| 20:03:09 | fried_rice | How domestic. | |
| 20:03:40 | fried_rice | Should we do this here, or maybe snag an empty meeting channel so we don't consume this one? | |
| 20:04:04 | figleaf | stay here - I'm eavesdropping | |
| 20:04:09 | fried_rice | ight | |
| 20:04:19 | cdent | agree with figleaf, probably best to spread the word as much as possible | |
| 20:05:14 | fried_rice | I'll lead by saying I've got several balls in the air, including the 'granular' effort and a non-placement-related bp for ksa adapter stuff; but I'll be happy if just the nrp series gets shepherded while I'm away. | |
| 20:05:40 | fried_rice | That's the series starting at the hopefully-just-about-to-merge https://review.openstack.org/#/c/385693/91 | |
| 20:06:18 | fried_rice | The culmination of this series will be the ability for the compute driver to tell the scheduler (almost) everything about how it wants to perceive its world, placement-wise. | |
| 20:07:21 | cdent | “the scheduler”? or “placment” or both? | |
| 20:07:33 | fried_rice | As currently conceived in the top patch in the series (https://review.openstack.org/#/c/520246/12/nova/compute/resource_tracker.py) the resource tracker will hand off the ProviderTree to the compute driver; the compute driver will mangle it as it sees fit; and the resource tracker will flush any changes back to placement. | |
| 20:08:12 | fried_rice | cdent I should have said "resource tracker" instead of "scheduler". The main point there is that the compute driver does not directly tell placement anything; that is brokered by resource tracker via ProviderTree. | |
| 20:08:20 | fried_rice | ...via report client. | |
| 20:08:53 | cdent | okay, so the virt driver tells the compute manager and its resource tracker about what it knows about its world | |
| 20:09:07 | fried_rice | Yup. | |
| 20:09:15 | cdent | ✔ | |
| 20:09:58 | fried_rice | I think there are reasons we don't want virt talking to placement directly. leakypipes and superdan have those in their heads. But I think it has to do with narrowing the points of control. | |
| 20:10:25 | leakypipes | fried_rice: ya | |
| 20:10:50 | leakypipes | fried_rice: the scheduler reporting client is a good bright-line interface for nova-compute with the placement API. | |