| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-03 | |||
| 13:59:17 | sambetts | efried: what interesting is that the node UUID in the exeception in n-cpu isn't the node UUID in the n-cond logs | |
| 13:59:48 | sambetts | so I don't know if this is being caused by other nodes async in the background being created and deleted by the API tests | |
| 14:00:16 | sambetts | jroll: ^ | |
| 14:00:37 | efried | sambetts: It isn't? It looks the same to me | |
| 14:00:52 | sambetts | oh acually may bad I had two different cases of this happening up *face plam* | |
| 14:01:14 | jroll | yeah, it is the same :) | |
| 14:01:15 | efried | n-cond: node b5bb2c51-3960-4d9c-b47e-223d83cb6139 | |
| 14:01:15 | efried | n-cpu: Error updating resources for node b5bb2c51-3960-4d9c-b47e-223d83cb6139 | |
| 14:01:23 | efried | okay phew | |
| 14:01:37 | sambetts | I had two jobs for two different ironic drivers and I'm seeing the same error in both so I have like 10 tabs open with all the different logs XD | |
| 14:01:40 | efried | Still mysterious why a failed periodic in n-cpu would cause a spawn to fail. Unless it's not periodic. | |
| 14:01:56 | efried | or unless sambetts is looking at two different sets of logs :P | |
| 14:02:12 | efried | But yeah, I see the resched failure and I'm sure I'm only looking at one :) | |
| 14:02:34 | sambetts | yeah, its definately the same, double checked | |
| 14:04:03 | jroll | hrm, it's the same request id in the reschedule log as the request id in the periodics logs | |
| 14:05:59 | sambetts | the trace isn't quite deep enough in the n-cpu logs to know what called _update_available_resource_for_node | |
| 14:06:51 | sambetts | only place I can see it being called is here: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7479 | |
| 14:06:54 | jroll | sambetts: it's only from the periodic task: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7500 | |
| 14:07:16 | jroll | odd | |
| 14:08:34 | efried | I'm pretty sure there's a code path that hits that guy during spawn. | |
| 14:09:04 | efried | I don't remember where in the stack it happens. We're, what, five layers deep in methods called update-something? | |
| 14:09:35 | efried | but I would have expected to see it in this stack trace. | |
| 14:09:49 | jroll | efried: calls which guy, specifically? | |
| 14:10:00 | efried | that's what I don't remember. | |
| 14:10:27 | efried | jroll: I'm saying I think there's a code path during spawn that winds up here in _update, but I don't remember where in the call stack it's initiated. | |
| 14:10:37 | efried | that didn't make sense, gimme a sec to find what I'm talking about. | |
| 14:10:48 | jroll | hm, ok | |
| 14:11:06 | jroll | the compute traceback is pretty clearly from the periodic task | |
| 14:11:15 | jroll | per http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jul_03_12_53_48_624944 | |
| 14:11:35 | jroll | er wait, that's later | |
| 14:11:49 | jroll | ignore me | |
| 14:12:01 | jroll | (also, I see what you're saying) | |
| 14:13:26 | sambetts | its a same that because a new exepcetion is raised in the n-cond code the line that through the original exeception is lost | |
| 14:14:27 | sambetts | s/thought/threw | |
| 14:18:30 | efried | I am now unable to find the code path that calls this from not-a-periodic. | |
| 14:18:46 | jroll | instance_claim() calls into _update() | |
| 14:19:17 | jroll | which is in _build_and_run_instance | |
| 14:19:29 | jroll | so that's probably where the build is bombing | |
| 14:19:42 | jroll | and the n-cpu traceback is an unrelated red herring | |
| 14:20:18 | sambetts | same exception on both sides though :/ | |
| 14:20:33 | efried | jroll: Then why is that exception message in the resched... yeah | |
| 14:20:34 | jroll | yes | |
| 14:20:45 | jroll | same exception, different call stack | |
| 14:21:00 | sambetts | maybe the reason they are both raising that execetion is because of a collision? | |
| 14:21:00 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Transform instance-live_migration_post notification https://review.openstack.org/480119 | |
| 14:21:13 | jroll | note that the conductor exception has _do_build_and_run_instance as the source | |
| 14:21:51 | jroll | _do_build_and_run_instance calls _build_and_run_instance calls rt.instance_claim() calls rt._update() calls.... down to where that exception happens | |
| 14:25:21 | dansmith | mriedem: does this mean you don't need to have the hangout this morning? https://review.openstack.org/#/c/564445/11/nova/network/neutronv2/api.py | |
| 14:26:14 | sambetts | jroll, efried: I can't see anything in the placement logs to indicate why it could sync :/ | |
| 14:26:37 | sambetts | and I would have thought if this was racey it would be so consistently failing :/ | |
| 14:26:45 | efried | sambetts: Which call is 409ing? | |
| 14:26:48 | dansmith | I'm still not clear on why we just need to know if any of the networks have the attributes specified, because that doesn't seem like nearly enough information for us to do something differently that we couldn't do by default, but.. | |
| 14:27:19 | sambetts | efried: I don't see any | |
| 14:28:33 | efried | well wtf | |
| 14:28:42 | mriedem | dansmith: if it's satisfactory for you then maybe not, but as you said, i'm still unclear on how this is going to be used and i haven't gotten that far in the series yet to find out - at least i'd like to hold off on approving the object changes at the bottom of the series until i'm more comfortable with how they are being used in the rest of the series | |
| 14:29:06 | dansmith | okay I'm not sure what revelation you've had that makes you okay with it now, if not "why" | |
| 14:29:17 | mriedem | but i think it answers my question about physnet and tunneled being mutually exclusive for a given network | |
| 14:29:40 | dansmith | okay, yeah I get that although it's on faith about the larger thing | |
| 14:33:17 | tssurya | dansmith: do you have some time now to discuss about aggregates behaviour ? | |
| 14:33:42 | dansmith | tssurya: yeah, I was going to hassle mriedem about one other thing, but it can wait | |
| 14:34:07 | tssurya | so, since we have aggregate "hosts" concept, the thing doesn't work well with ironic | |
| 14:34:21 | tssurya | since in ironic 1 host = many compute nodes | |
| 14:34:38 | tssurya | and we would need something like aggregate "compute bodes" | |
| 14:34:42 | tssurya | nodes* | |
| 14:34:58 | tssurya | because now we have the pre-placement filters which allow scheduling | |
| 14:35:02 | efried | sambetts, jroll: Let me spin up a quick and dirty fix and we can see if it clears this up. Need a couple hours tho. | |
| 14:35:02 | tssurya | based on aggregates | |
| 14:35:24 | sambetts | efried: thanks that would be really useful :) | |
| 14:35:46 | tssurya | I guess what I am trying to say is, does it make sense to expand aggregate add hosts to actually be aggregate add compute nodes ? | |
| 14:37:02 | dansmith | tssurya: well, that's a sticky situation of course | |
| 14:37:12 | mriedem | tssurya: there is a whole series i just abandoned for that last week | |
| 14:37:13 | mriedem | and a spec | |
| 14:37:24 | dansmith | tssurya: you can create aggregates in placement and add compute nodes to them because the resource providers are actually the ironic nodes | |
| 14:37:24 | tssurya | mriedem: oh really ? | |
| 14:37:28 | jroll | efried: <3 | |
| 14:37:29 | mriedem | and we said at the ptg you can do aggregates with pre-placement request filters and placement aggregates | |
| 14:37:44 | dansmith | tssurya: but in terms of making it automatic we'd need to change nova's api | |
| 14:38:00 | tssurya | dansmith: right, I mean nova api change | |
| 14:38:12 | tssurya | because it is a pain to add this manually on the nova side | |
| 14:38:15 | dansmith | yeah, that's a problem | |
| 14:38:16 | tssurya | for the ironic nodes | |
| 14:38:41 | mriedem | tssurya: https://review.openstack.org/#/q/topic:bp/aggregate-affinity | |
| 14:39:02 | tssurya | mriedem: thanks looking | |
| 14:41:17 | dansmith | mriedem: https://review.openstack.org/#/c/559032/ | |
| 14:44:24 | tssurya | mriedem, dansmith: oh right so the work around was to just have them in the placement aggregate! | |
| 14:44:27 | tssurya | thanks | |
| 14:44:34 | dansmith | yeah | |
| 14:44:47 | tssurya | I guess I totally missed/forgot | |
| 14:44:49 | tssurya | that | |
| 15:00:38 | sahid | stephenfin: do you want reconsider your vote for https://review.openstack.org/#/c/579611/ ? | |
| 15:00:51 | sahid | mriedem: ^ | |
| 15:01:03 | openstackgerrit | jiang wei proposed openstack/nova master: Add action initiator attribute to the instance payload https://review.openstack.org/536243 | |
| 15:01:15 | stephenfin | sahid: In a meeting but I can take a look once I'm done | |
| 15:02:24 | dansmith | mriedem: coming? it's awkward with just stephenfin and myself | |
| 15:02:29 | mriedem | 4ever in bluejeans | |
| 15:02:32 | mriedem | this isn't hangouts at all | |
| 15:02:33 | openstackgerrit | Konstantinos Samaras-Tsakiris proposed openstack/nova master: Hide hypervisor id on windows guests https://review.openstack.org/579897 | |
| 15:03:43 | sahid | stephenfin: ok np, nothing hurry actually | |
| 15:28:40 | efried | sambetts, jroll: Can you identify whether there are/were/should be nodes with the following UUIDs in the environment for | |
| 15:28:40 | efried | http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz | |