Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-03
14:19:17 jroll which is in _build_and_run_instance
14:19:29 jroll so that's probably where the build is bombing
14:19:42 jroll and the n-cpu traceback is an unrelated red herring
14:20:18 sambetts same exception on both sides though :/
14:20:33 efried jroll: Then why is that exception message in the resched... yeah
14:20:34 jroll yes
14:20:45 jroll same exception, different call stack
14:21:00 sambetts maybe the reason they are both raising that execetion is because of a collision?
14:21:00 openstackgerrit Takashi NATSUME proposed openstack/nova master: Transform instance-live_migration_post notification https://review.openstack.org/480119
14:21:13 jroll note that the conductor exception has _do_build_and_run_instance as the source
14:21:51 jroll _do_build_and_run_instance calls _build_and_run_instance calls rt.instance_claim() calls rt._update() calls.... down to where that exception happens
14:25:21 dansmith mriedem: does this mean you don't need to have the hangout this morning? https://review.openstack.org/#/c/564445/11/nova/network/neutronv2/api.py
14:26:14 sambetts jroll, efried: I can't see anything in the placement logs to indicate why it could sync :/
14:26:37 sambetts and I would have thought if this was racey it would be so consistently failing :/
14:26:45 efried sambetts: Which call is 409ing?
14:26:48 dansmith I'm still not clear on why we just need to know if any of the networks have the attributes specified, because that doesn't seem like nearly enough information for us to do something differently that we couldn't do by default, but..
14:27:19 sambetts efried: I don't see any
14:28:33 efried well wtf
14:28:42 mriedem dansmith: if it's satisfactory for you then maybe not, but as you said, i'm still unclear on how this is going to be used and i haven't gotten that far in the series yet to find out - at least i'd like to hold off on approving the object changes at the bottom of the series until i'm more comfortable with how they are being used in the rest of the series
14:29:06 dansmith okay I'm not sure what revelation you've had that makes you okay with it now, if not "why"
14:29:17 mriedem but i think it answers my question about physnet and tunneled being mutually exclusive for a given network
14:29:40 dansmith okay, yeah I get that although it's on faith about the larger thing
14:33:17 tssurya dansmith: do you have some time now to discuss about aggregates behaviour ?
14:33:42 dansmith tssurya: yeah, I was going to hassle mriedem about one other thing, but it can wait
14:34:07 tssurya so, since we have aggregate "hosts" concept, the thing doesn't work well with ironic
14:34:21 tssurya since in ironic 1 host = many compute nodes
14:34:38 tssurya and we would need something like aggregate "compute bodes"
14:34:42 tssurya nodes*
14:34:58 tssurya because now we have the pre-placement filters which allow scheduling
14:35:02 efried sambetts, jroll: Let me spin up a quick and dirty fix and we can see if it clears this up. Need a couple hours tho.
14:35:02 tssurya based on aggregates
14:35:24 sambetts efried: thanks that would be really useful :)
14:35:46 tssurya I guess what I am trying to say is, does it make sense to expand aggregate add hosts to actually be aggregate add compute nodes ?
14:37:02 dansmith tssurya: well, that's a sticky situation of course
14:37:12 mriedem tssurya: there is a whole series i just abandoned for that last week
14:37:13 mriedem and a spec
14:37:24 dansmith tssurya: you can create aggregates in placement and add compute nodes to them because the resource providers are actually the ironic nodes
14:37:24 tssurya mriedem: oh really ?
14:37:28 jroll efried: <3
14:37:29 mriedem and we said at the ptg you can do aggregates with pre-placement request filters and placement aggregates
14:37:44 dansmith tssurya: but in terms of making it automatic we'd need to change nova's api
14:38:00 tssurya dansmith: right, I mean nova api change
14:38:12 tssurya because it is a pain to add this manually on the nova side
14:38:15 dansmith yeah, that's a problem
14:38:16 tssurya for the ironic nodes
14:38:41 mriedem tssurya: https://review.openstack.org/#/q/topic:bp/aggregate-affinity
14:39:02 tssurya mriedem: thanks looking
14:41:17 dansmith mriedem: https://review.openstack.org/#/c/559032/
14:44:24 tssurya mriedem, dansmith: oh right so the work around was to just have them in the placement aggregate!
14:44:27 tssurya thanks
14:44:34 dansmith yeah
14:44:47 tssurya I guess I totally missed/forgot
14:44:49 tssurya that
15:00:38 sahid stephenfin: do you want reconsider your vote for https://review.openstack.org/#/c/579611/ ?
15:00:51 sahid mriedem: ^
15:01:03 openstackgerrit jiang wei proposed openstack/nova master: Add action initiator attribute to the instance payload https://review.openstack.org/536243
15:01:15 stephenfin sahid: In a meeting but I can take a look once I'm done
15:02:24 dansmith mriedem: coming? it's awkward with just stephenfin and myself
15:02:29 mriedem 4ever in bluejeans
15:02:32 mriedem this isn't hangouts at all
15:02:33 openstackgerrit Konstantinos Samaras-Tsakiris proposed openstack/nova master: Hide hypervisor id on windows guests https://review.openstack.org/579897
15:03:43 sahid stephenfin: ok np, nothing hurry actually
15:28:40 efried sambetts, jroll: Can you identify whether there are/were/should be nodes with the following UUIDs in the environment for
15:28:40 efried http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz
15:28:58 efried => fbd2e932-ea59-4b57-a206-a839f40fa182
15:28:58 efried => 7e1567e7-4d7c-4ad1-9563-3d277f4fb04d
15:29:07 efried => 72b0a2e0-277b-4698-bfd6-aa15623e44bd
15:29:51 jroll efried: those are likely fake nodes created by other tempest tests
15:30:04 jroll that said, this is a third party CI environment that isn't mine
15:30:10 jroll so I'll let sam speak from here on :)
15:31:05 efried ahhh. So, that appears to be what's actually causing the failure here. Those guys show up in the provider tree, so we try to update them, but by the time we get to hitting placement for them, they're gone.
15:31:19 sambetts efried: those nodes were created by the ironic tempest api tests and then deleted after, the only real node in the test environment is b5bb2c51-3960-4d9c-b47e-223d83cb6139
15:31:40 jroll oh, we update everything at build time?
15:31:50 jroll or everything for that compute service
15:32:13 efried jroll: Yes, update_from_provider_tree is supposed to make the host's placement presence right.
15:32:36 jroll mmmm
15:32:48 stephenfin sahid: Yup, makes sense. +2 now
15:36:54 openstackgerrit Merged openstack/nova master: Update xenapi_disable_agent config option usage in docs https://review.openstack.org/578848
15:43:03 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: use dest host vif migrate details for live migration https://review.openstack.org/551370
15:43:04 openstackgerrit Matt Riedemann proposed openstack/nova master: Annotate flows and handle PortBindingDeletionFailed in ComputeManager https://review.openstack.org/551371
15:43:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Port binding based on events during live migration https://review.openstack.org/434870
15:43:06 openstackgerrit Matt Riedemann proposed openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537
15:45:09 efried sambetts: jroll: So I think this is probably the problem here, which a retry isn't going to fix: ironic's update_provider_tree isn't getting rid of nodes when they disappear.
15:45:56 efried sambetts, jroll: How (via what code path) are these fake nodes "created by the ironic tempest api tests and then deleted after" ?
15:46:04 jroll efried: right, and it races with the thing that cleans up orphan nodes
15:46:29 jroll efried: via api tests in our tempest plugin being run (in parallel?) with the scenario tests
15:46:32 efried jroll: Oh, we have a thing that cleans up orphan nodes? Does that thing actually remove them from placement? Where does that code live?
15:47:15 jroll efried: not sure where the code is offhand, but see e.g. http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz#_Jul_03_12_53_45_414036
15:47:23 efried jroll: Okay, but the tempest tests presumably use some kind of API to create/delete the nodes. Because they're ending up registered in placement somehow.
15:47:28 jroll I think it's handled in update_available_resource or something
15:47:49 sambetts they are just created in ironic, then nova syncs with ironic and picks up the new nodes
15:47:55 jroll efried: yes, POST ironic-host/v1/nodes, then the ..... yes that
15:48:31 efried oho
15:48:42 efried it's happening *after* _update_available_resource_for_node
15:49:32 efried sambetts: How reproducible is this thang?
15:51:00 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Always retrieve network information if available https://review.openstack.org/564444
15:51:01 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Retrieve tunneled status in '_get_physnet_info' https://review.openstack.org/564445
15:51:02 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add NUMACell.network_metadata https://review.openstack.org/564439
15:51:03 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add NUMATopologyLimits.networks https://review.openstack.org/575486
15:51:04 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Start accounting for networks in NUMA placement https://review.openstack.org/564448

Earlier   Later