Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-08
17:58:24 superdan if you go look at it from the api, you'd only see the start, because you never finished it in the api cell, and nothing ever sync'd the finish
17:58:52 superdan mriedem: is what synchronous? the api cell before the child cell? sure, but they don't know about each other really anyway, so it doesn't matter
17:59:58 mriedem is the replay of the instance action create from the top level to the child cell synchronous
18:00:00 mriedem was my question
18:00:10 mriedem trying to figure out what is special that makes this fail the cellsv1 job https://review.openstack.org/#/c/523676
18:00:15 mriedem http://logs.openstack.org/76/523676/11/check/legacy-tempest-dsvm-cells/1d155c6/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_07_09_02_09_579991
18:00:33 superdan the thing that is replayed is the call to compute/api::confirm_resize()
18:00:34 superdan the action_create is
18:00:35 superdan not replayed itself,
18:00:36 superdan the whole call to that method is
18:01:27 mriedem ok, so backup_instance would all be replayed in the child cell api too right?
18:01:44 superdan yeah
18:01:49 superdan I don't see where the cells job ran on that
18:02:00 mriedem https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2594
18:02:09 mriedem the latest patch timed out in on the cells job
18:02:23 superdan okay
18:02:31 superdan would have thought that would be reported
18:02:40 mriedem zuul in the last 48 hours...
18:02:55 superdan oh I see, it is i the comment
18:02:57 superdan but not the summary
18:03:45 alaski stepped away, but it's exactly as superdan describes
18:03:54 superdan \o/
18:04:23 mriedem so i wonder if it has something to do with this comment from comstud https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2594
18:04:55 mriedem but i think that is just about setting the task_state
18:04:59 superdan mriedem: that's a remote call, so maybe it's calling to conductor which is looking in the wrong db?
18:05:15 superdan er, a remote exception I mean
18:06:02 mriedem well,
18:06:13 mriedem i don't know why it would just fail on that one lookup and not all of the others
18:06:13 superdan not sure that makes sense actually
18:06:19 mriedem that's what i don't understand about this
18:06:57 mriedem fried_rolls: have you ever mocked something that can't be imported?
18:07:01 superdan oh Iknow why
18:07:39 superdan mriedem: backup_instance doesn't get replayed in the cell for some reason, unlike everything else, which means it calls straight to the compute I think: https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L915-L925
18:07:45 superdan mriedem: compare that to rebuild right below
18:08:01 mriedem ok yeah that's what i was looking at
18:08:27 mriedem https://github.com/openstack/nova/commit/2be50f0f3f02830798c25442dc54e92faa18106b
18:09:43 superdan I'd be extremely concerned about changing _any_ of that stuff at this point
18:09:50 mriedem weird, ok, so i guess the note about "any changes made here have to be made there" is what that's about
18:10:06 mriedem including creating the action record
18:10:14 mriedem well i see 2 options
18:10:29 mriedem 1. do the action create for backup and snapshot in the cells messaging code like it says,
18:10:46 mriedem 2. do something in the EventReporter where if cells is enabled, we blacklist some actions
18:10:55 mriedem we could remove #2 when we remove cellsv1
18:11:29 superdan I don't love either of those
18:11:40 peter-hamilton mriedem: thanks!
18:11:49 mriedem superdan: me neither
18:11:57 peter-hamilton mnaser: bpoulos is who you want to talk to; I'll ping her for you
18:12:09 mriedem superdan: but i'm not sure what else to do at this point
18:12:15 mnaser peter-hamilton: cool, thanks!
18:12:16 mriedem so, i'll go to lunch and not think about this
18:17:54 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't launch guestfs in a thread pool if guestfs.debug is enabled https://review.openstack.org/526748
18:17:58 mriedem dmsimard: clarkb: ^
18:21:28 dmsimard mriedem: so we toggled the bug by setting it to debug ? :/
18:21:40 dmsimard mriedem: the force_tcg patch would work without debug ?
18:25:55 bpoulos mnaser: only nova-compute needs the verify_glance_signatures option set in order to enable glance image signature verification
18:26:27 mnaser bpoulos: awesome, thank you so much (fyi, asking in context of https://review.openstack.org/#/c/526514/ )
18:27:38 bpoulos mnaser: no problem, happy to help
18:56:42 openstackgerrit Jay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg https://review.openstack.org/526753
19:10:05 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795
19:16:35 openstackgerrit Merged openstack/nova master: Updated from global requirements https://review.openstack.org/526386
19:27:26 openstackgerrit Merged openstack/nova master: Fix 'force' parameter in os-quota-sets PUT schema https://review.openstack.org/523194
19:47:12 mriedem dmsimard: that's what i hope to find out
19:50:24 fried_rolls mriedem No, I don't think I have ever (successfully) mocked something that can't be imported. It will fail at "compile" time.
19:50:44 fried_rolls mriedem You can cheat by mocking it after it's imported, though.
19:52:07 fried_rolls I assume this is in a situation where the thing imports correctly on the prod side but for some reason not on the test side? Which... why?
19:53:39 fried_rice cdent How's now?
19:53:48 cdent one moment caller
19:54:34 cdent blargh, timeouts in migration tests: http://logs.openstack.org/39/521639/8/gate/openstack-tox-functional/f6880ac/testr_results.html.gz
20:02:55 cdent fried_rice: okay, sorry, had to feed that cat and put dinner dishes aweay
20:03:09 fried_rice How domestic.
20:03:40 fried_rice Should we do this here, or maybe snag an empty meeting channel so we don't consume this one?
20:04:04 figleaf stay here - I'm eavesdropping
20:04:09 fried_rice ight
20:04:19 cdent agree with figleaf, probably best to spread the word as much as possible
20:05:14 fried_rice I'll lead by saying I've got several balls in the air, including the 'granular' effort and a non-placement-related bp for ksa adapter stuff; but I'll be happy if just the nrp series gets shepherded while I'm away.
20:05:40 fried_rice That's the series starting at the hopefully-just-about-to-merge https://review.openstack.org/#/c/385693/91
20:06:18 fried_rice The culmination of this series will be the ability for the compute driver to tell the scheduler (almost) everything about how it wants to perceive its world, placement-wise.
20:07:21 cdent “the scheduler”? or “placment” or both?
20:07:33 fried_rice As currently conceived in the top patch in the series (https://review.openstack.org/#/c/520246/12/nova/compute/resource_tracker.py) the resource tracker will hand off the ProviderTree to the compute driver; the compute driver will mangle it as it sees fit; and the resource tracker will flush any changes back to placement.
20:08:12 fried_rice cdent I should have said "resource tracker" instead of "scheduler". The main point there is that the compute driver does not directly tell placement anything; that is brokered by resource tracker via ProviderTree.
20:08:20 fried_rice ...via report client.
20:08:53 cdent okay, so the virt driver tells the compute manager and its resource tracker about what it knows about its world
20:09:07 fried_rice Yup.
20:09:15 cdent ✔
20:09:58 fried_rice I think there are reasons we don't want virt talking to placement directly. leakypipes and superdan have those in their heads. But I think it has to do with narrowing the points of control.
20:10:25 leakypipes fried_rice: ya
20:10:50 leakypipes fried_rice: the scheduler reporting client is a good bright-line interface for nova-compute with the placement API.
20:11:02 fried_rice Nod.
20:11:41 cdent yeah, my concern was not so much about violating that line, rather trying to clarify the use of the term scheduler, which is ambiguous, but it’s clear now
20:11:47 fried_rice As currently written in the series, the first time around, resource tracker => report client will do get_provider_tree to bootstrap the ProviderTree, which will _ensure_resource_provider with the compute nodename & uuid. That will create the entry in placement for the compute host RP and populate the local ProviderTree with that.
20:12:10 fried_rice And then pass that ProviderTree to virt, via ComputeDriver.update_provider_tree.
20:12:35 fried_rice So it'll have just the compute RP as a root (no children), no inventory, no traits, no aggs, no sharing RPs.
20:13:26 cdent ✔
20:14:06 fried_rice virt does whatever it wants, as long as it's possible via the public ProviderTree interfaces. This includes creating new root RPs (which will probably typically be sharing RPs, though we're not enforcing that) or child RPs (usually descendants of the compute RP, though we're not enforcing that).
20:14:19 fried_rice It also includes setting traits and inventories on any of those guys.
20:14:30 fried_rice And assigning aggregate UUIDs.
20:15:16 fried_rice virt is responsible for returning True or False from that method to let us know whether it changed anything. (TBH, I'm not sure that's necessary, cause it's pretty easy for us to figure it out, and that might be better than trusting virt to get it right.)
20:16:27 fried_rice Anyway, resource tracker gets that provider tree back and diffs it with what it passed in originally (which, I should have mentioned, is a copy of what it had built up); and then makes report client calls to push those changes back to the placement db.
20:16:33 cdent can I pause you for a moment?
20:16:36 fried_rice sho

Earlier   Later