| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-03 | |||
| 13:31:16 | pooja_jadhav | efried: I am eager to learn what DB query is generated when we call method https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L2691. | |
| 13:31:46 | pooja_jadhav | efied: so want to know under which scenarios this method will get called? | |
| 13:37:51 | openstackgerrit | Margarita Shakhova proposed openstack/nova master: Do not skip migrations in _destroy_evacuated_instances() https://review.openstack.org/563623 | |
| 13:42:02 | efried | pooja_jadhav: Okay. If you create a query with a granular request group, you should hit it. | |
| 13:42:33 | efried | pooja_jadhav: Take a look at nova/tests/functional/api/openstack/placement/gabbits/granular.yaml | |
| 13:43:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send soft_delete from context manager https://review.openstack.org/476459 | |
| 13:43:18 | efried | pooja_jadhav: Anywhere you see `resources<Number>` is a query that should hit that code path. | |
| 13:44:57 | pooja_jadhav | efried: ok, will check and if any doubts will let u know. thanks for your time :) | |
| 13:45:21 | efried | Good luck pooja_jadhav | |
| 13:45:35 | jroll | efried: does this ring a bell? sambetts is hitting it in third party CI: http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jul_03_12_53_48_638704 | |
| 13:45:49 | jroll | and/or do you have tips on debugging that? or do we need to add more logging | |
| 13:46:01 | efried | jroll: Is it causing a problem? | |
| 13:46:05 | pooja_jadhav | efried: :) | |
| 13:46:15 | efried | jroll: It's actually a semi-expected thing, supposed to self-heal on the next periodic hit. | |
| 13:46:28 | jroll | efried: yes, causes a reschedule (and maybe a failure due to lack of hardware?) | |
| 13:46:47 | jroll | yes, failure | |
| 13:46:49 | jroll | http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cond-cell1.txt.gz#_Jul_03_12_53_47_517588 | |
| 13:47:06 | sambetts | yeah, we only have one node, so the retry filter kills the instance | |
| 13:47:13 | jroll | not sure if it's a constant failure or occassional, sam can clarify | |
| 13:47:37 | sambetts | all my CI failures seem to be this at the moment | |
| 13:47:57 | efried | That's weird that it would be affecting a sched operation. At least the one you're showing me ought to be part of periodic, not part of spawn. | |
| 13:48:28 | efried | but in any case, yeah, we have to close some of these gaps in rocky | |
| 13:48:41 | efried | Is this rocky? | |
| 13:48:57 | jroll | oh, that is a periodic, odd | |
| 13:49:50 | efried | What we need to do is put a retry loop around this chunk in the resource tracker: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L883-L914 | |
| 13:50:30 | efried | The devil is in the details, of course. | |
| 13:51:49 | sambetts | looking at the n-cond log that placement sync error is bubbling up in the _build_and_run_instance function | |
| 13:52:37 | sambetts | although its been wrapped in a ReschduledException :/ | |
| 13:52:43 | sambetts | somwhere along the way | |
| 13:52:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Wait for vif plugging during live migration job https://review.openstack.org/578551 | |
| 13:56:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop setting glance_api_version in cinder.conf in nova-live-migration https://review.openstack.org/579871 | |
| 13:57:05 | efried | sambetts: Are you sure it's the same ResourceProviderSyncFailed exception? (Though that's the only one I'm seeing in your compute log.) | |
| 13:59:17 | sambetts | efried: what interesting is that the node UUID in the exeception in n-cpu isn't the node UUID in the n-cond logs | |
| 13:59:48 | sambetts | so I don't know if this is being caused by other nodes async in the background being created and deleted by the API tests | |
| 14:00:16 | sambetts | jroll: ^ | |
| 14:00:37 | efried | sambetts: It isn't? It looks the same to me | |
| 14:00:52 | sambetts | oh acually may bad I had two different cases of this happening up *face plam* | |
| 14:01:14 | jroll | yeah, it is the same :) | |
| 14:01:15 | efried | n-cpu: Error updating resources for node b5bb2c51-3960-4d9c-b47e-223d83cb6139 | |
| 14:01:15 | efried | n-cond: node b5bb2c51-3960-4d9c-b47e-223d83cb6139 | |
| 14:01:23 | efried | okay phew | |
| 14:01:37 | sambetts | I had two jobs for two different ironic drivers and I'm seeing the same error in both so I have like 10 tabs open with all the different logs XD | |
| 14:01:40 | efried | Still mysterious why a failed periodic in n-cpu would cause a spawn to fail. Unless it's not periodic. | |
| 14:01:56 | efried | or unless sambetts is looking at two different sets of logs :P | |
| 14:02:12 | efried | But yeah, I see the resched failure and I'm sure I'm only looking at one :) | |
| 14:02:34 | sambetts | yeah, its definately the same, double checked | |
| 14:04:03 | jroll | hrm, it's the same request id in the reschedule log as the request id in the periodics logs | |
| 14:05:59 | sambetts | the trace isn't quite deep enough in the n-cpu logs to know what called _update_available_resource_for_node | |
| 14:06:51 | sambetts | only place I can see it being called is here: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7479 | |
| 14:06:54 | jroll | sambetts: it's only from the periodic task: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7500 | |
| 14:07:16 | jroll | odd | |
| 14:08:34 | efried | I'm pretty sure there's a code path that hits that guy during spawn. | |
| 14:09:04 | efried | I don't remember where in the stack it happens. We're, what, five layers deep in methods called update-something? | |
| 14:09:35 | efried | but I would have expected to see it in this stack trace. | |
| 14:09:49 | jroll | efried: calls which guy, specifically? | |
| 14:10:00 | efried | that's what I don't remember. | |
| 14:10:27 | efried | jroll: I'm saying I think there's a code path during spawn that winds up here in _update, but I don't remember where in the call stack it's initiated. | |
| 14:10:37 | efried | that didn't make sense, gimme a sec to find what I'm talking about. | |
| 14:10:48 | jroll | hm, ok | |
| 14:11:06 | jroll | the compute traceback is pretty clearly from the periodic task | |
| 14:11:15 | jroll | per http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jul_03_12_53_48_624944 | |
| 14:11:35 | jroll | er wait, that's later | |
| 14:11:49 | jroll | ignore me | |
| 14:12:01 | jroll | (also, I see what you're saying) | |
| 14:13:26 | sambetts | its a same that because a new exepcetion is raised in the n-cond code the line that through the original exeception is lost | |
| 14:14:27 | sambetts | s/thought/threw | |
| 14:18:30 | efried | I am now unable to find the code path that calls this from not-a-periodic. | |
| 14:18:46 | jroll | instance_claim() calls into _update() | |
| 14:19:17 | jroll | which is in _build_and_run_instance | |
| 14:19:29 | jroll | so that's probably where the build is bombing | |
| 14:19:42 | jroll | and the n-cpu traceback is an unrelated red herring | |
| 14:20:18 | sambetts | same exception on both sides though :/ | |
| 14:20:33 | efried | jroll: Then why is that exception message in the resched... yeah | |
| 14:20:34 | jroll | yes | |
| 14:20:45 | jroll | same exception, different call stack | |
| 14:21:00 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Transform instance-live_migration_post notification https://review.openstack.org/480119 | |
| 14:21:00 | sambetts | maybe the reason they are both raising that execetion is because of a collision? | |
| 14:21:13 | jroll | note that the conductor exception has _do_build_and_run_instance as the source | |
| 14:21:51 | jroll | _do_build_and_run_instance calls _build_and_run_instance calls rt.instance_claim() calls rt._update() calls.... down to where that exception happens | |
| 14:25:21 | dansmith | mriedem: does this mean you don't need to have the hangout this morning? https://review.openstack.org/#/c/564445/11/nova/network/neutronv2/api.py | |
| 14:26:14 | sambetts | jroll, efried: I can't see anything in the placement logs to indicate why it could sync :/ | |
| 14:26:37 | sambetts | and I would have thought if this was racey it would be so consistently failing :/ | |
| 14:26:45 | efried | sambetts: Which call is 409ing? | |
| 14:26:48 | dansmith | I'm still not clear on why we just need to know if any of the networks have the attributes specified, because that doesn't seem like nearly enough information for us to do something differently that we couldn't do by default, but.. | |
| 14:27:19 | sambetts | efried: I don't see any | |
| 14:28:33 | efried | well wtf | |
| 14:28:42 | mriedem | dansmith: if it's satisfactory for you then maybe not, but as you said, i'm still unclear on how this is going to be used and i haven't gotten that far in the series yet to find out - at least i'd like to hold off on approving the object changes at the bottom of the series until i'm more comfortable with how they are being used in the rest of the series | |
| 14:29:06 | dansmith | okay I'm not sure what revelation you've had that makes you okay with it now, if not "why" | |
| 14:29:17 | mriedem | but i think it answers my question about physnet and tunneled being mutually exclusive for a given network | |
| 14:29:40 | dansmith | okay, yeah I get that although it's on faith about the larger thing | |
| 14:33:17 | tssurya | dansmith: do you have some time now to discuss about aggregates behaviour ? | |
| 14:33:42 | dansmith | tssurya: yeah, I was going to hassle mriedem about one other thing, but it can wait | |
| 14:34:07 | tssurya | so, since we have aggregate "hosts" concept, the thing doesn't work well with ironic | |
| 14:34:21 | tssurya | since in ironic 1 host = many compute nodes | |
| 14:34:38 | tssurya | and we would need something like aggregate "compute bodes" | |
| 14:34:42 | tssurya | nodes* | |
| 14:34:58 | tssurya | because now we have the pre-placement filters which allow scheduling | |
| 14:35:02 | tssurya | based on aggregates | |
| 14:35:02 | efried | sambetts, jroll: Let me spin up a quick and dirty fix and we can see if it clears this up. Need a couple hours tho. | |
| 14:35:24 | sambetts | efried: thanks that would be really useful :) | |