Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-03
14:35:02 tssurya based on aggregates
14:35:02 efried sambetts, jroll: Let me spin up a quick and dirty fix and we can see if it clears this up. Need a couple hours tho.
14:35:24 sambetts efried: thanks that would be really useful :)
14:35:46 tssurya I guess what I am trying to say is, does it make sense to expand aggregate add hosts to actually be aggregate add compute nodes ?
14:37:02 dansmith tssurya: well, that's a sticky situation of course
14:37:12 mriedem tssurya: there is a whole series i just abandoned for that last week
14:37:13 mriedem and a spec
14:37:24 tssurya mriedem: oh really ?
14:37:24 dansmith tssurya: you can create aggregates in placement and add compute nodes to them because the resource providers are actually the ironic nodes
14:37:28 jroll efried: <3
14:37:29 mriedem and we said at the ptg you can do aggregates with pre-placement request filters and placement aggregates
14:37:44 dansmith tssurya: but in terms of making it automatic we'd need to change nova's api
14:38:00 tssurya dansmith: right, I mean nova api change
14:38:12 tssurya because it is a pain to add this manually on the nova side
14:38:15 dansmith yeah, that's a problem
14:38:16 tssurya for the ironic nodes
14:38:41 mriedem tssurya: https://review.openstack.org/#/q/topic:bp/aggregate-affinity
14:39:02 tssurya mriedem: thanks looking
14:41:17 dansmith mriedem: https://review.openstack.org/#/c/559032/
14:44:24 tssurya mriedem, dansmith: oh right so the work around was to just have them in the placement aggregate!
14:44:27 tssurya thanks
14:44:34 dansmith yeah
14:44:47 tssurya I guess I totally missed/forgot
14:44:49 tssurya that
15:00:38 sahid stephenfin: do you want reconsider your vote for https://review.openstack.org/#/c/579611/ ?
15:00:51 sahid mriedem: ^
15:01:03 openstackgerrit jiang wei proposed openstack/nova master: Add action initiator attribute to the instance payload https://review.openstack.org/536243
15:01:15 stephenfin sahid: In a meeting but I can take a look once I'm done
15:02:24 dansmith mriedem: coming? it's awkward with just stephenfin and myself
15:02:29 mriedem 4ever in bluejeans
15:02:32 mriedem this isn't hangouts at all
15:02:33 openstackgerrit Konstantinos Samaras-Tsakiris proposed openstack/nova master: Hide hypervisor id on windows guests https://review.openstack.org/579897
15:03:43 sahid stephenfin: ok np, nothing hurry actually
15:28:40 efried http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz
15:28:40 efried sambetts, jroll: Can you identify whether there are/were/should be nodes with the following UUIDs in the environment for
15:28:58 efried => 7e1567e7-4d7c-4ad1-9563-3d277f4fb04d
15:28:58 efried => fbd2e932-ea59-4b57-a206-a839f40fa182
15:29:07 efried => 72b0a2e0-277b-4698-bfd6-aa15623e44bd
15:29:51 jroll efried: those are likely fake nodes created by other tempest tests
15:30:04 jroll that said, this is a third party CI environment that isn't mine
15:30:10 jroll so I'll let sam speak from here on :)
15:31:05 efried ahhh. So, that appears to be what's actually causing the failure here. Those guys show up in the provider tree, so we try to update them, but by the time we get to hitting placement for them, they're gone.
15:31:19 sambetts efried: those nodes were created by the ironic tempest api tests and then deleted after, the only real node in the test environment is b5bb2c51-3960-4d9c-b47e-223d83cb6139
15:31:40 jroll oh, we update everything at build time?
15:31:50 jroll or everything for that compute service
15:32:13 efried jroll: Yes, update_from_provider_tree is supposed to make the host's placement presence right.
15:32:36 jroll mmmm
15:32:48 stephenfin sahid: Yup, makes sense. +2 now
15:36:54 openstackgerrit Merged openstack/nova master: Update xenapi_disable_agent config option usage in docs https://review.openstack.org/578848
15:43:03 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: use dest host vif migrate details for live migration https://review.openstack.org/551370
15:43:04 openstackgerrit Matt Riedemann proposed openstack/nova master: Annotate flows and handle PortBindingDeletionFailed in ComputeManager https://review.openstack.org/551371
15:43:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Port binding based on events during live migration https://review.openstack.org/434870
15:43:06 openstackgerrit Matt Riedemann proposed openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537
15:45:09 efried sambetts: jroll: So I think this is probably the problem here, which a retry isn't going to fix: ironic's update_provider_tree isn't getting rid of nodes when they disappear.
15:45:56 efried sambetts, jroll: How (via what code path) are these fake nodes "created by the ironic tempest api tests and then deleted after" ?
15:46:04 jroll efried: right, and it races with the thing that cleans up orphan nodes
15:46:29 jroll efried: via api tests in our tempest plugin being run (in parallel?) with the scenario tests
15:46:32 efried jroll: Oh, we have a thing that cleans up orphan nodes? Does that thing actually remove them from placement? Where does that code live?
15:47:15 jroll efried: not sure where the code is offhand, but see e.g. http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz#_Jul_03_12_53_45_414036
15:47:23 efried jroll: Okay, but the tempest tests presumably use some kind of API to create/delete the nodes. Because they're ending up registered in placement somehow.
15:47:28 jroll I think it's handled in update_available_resource or something
15:47:49 sambetts they are just created in ironic, then nova syncs with ironic and picks up the new nodes
15:47:55 jroll efried: yes, POST ironic-host/v1/nodes, then the ..... yes that
15:48:31 efried oho
15:48:42 efried it's happening *after* _update_available_resource_for_node
15:49:32 efried sambetts: How reproducible is this thang?
15:51:00 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Always retrieve network information if available https://review.openstack.org/564444
15:51:01 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Retrieve tunneled status in '_get_physnet_info' https://review.openstack.org/564445
15:51:02 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add NUMACell.network_metadata https://review.openstack.org/564439
15:51:03 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add NUMATopologyLimits.networks https://review.openstack.org/575486
15:51:04 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Start accounting for networks in NUMA placement https://review.openstack.org/564448
15:51:05 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add RequestSpec.network_metadata https://review.openstack.org/564442
15:51:06 openstackgerrit Stephen Finucane proposed openstack/nova master: scheduler: Start utilizing RequestSpec.network_metadata https://review.openstack.org/564452
15:51:07 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Add '[neutron] physnets' and related options https://review.openstack.org/564440
15:51:08 sambetts every single one of my CI runs hits it at the moment, I expect the reason the upstream ironic CI isn't hitting it is because setting tempest concurrency=1
15:51:08 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start populating NUMACell.network_metadata field https://review.openstack.org/564441
16:08:57 openstack Launchpad bug 1739325 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields" [Medium,Confirmed] - Assigned to Balazs Gibizer (balazs-gibizer)
16:08:57 mriedem gibi: so i'm looking at https://bugs.launchpad.net/nova/+bug/1739325 again and Flavor.disabled isn't something you can set from the API as far as i can tell
16:11:21 gibi mriedem: but then why we even have a disabled field?
16:11:43 mriedem idk, i'm not seeing in git history where it was added, or if there is a way to ever set it via cli or something
16:11:47 mriedem dansmith: do you remember?
16:13:15 openstack Launchpad bug 1259262 in python-novaclient "there is no api or cli to enable/disable a flavor" [Wishlist,Opinion]
16:13:15 gibi mriedem: https://bugs.launchpad.net/nova/+bug/1259262
16:13:48 gibi pretty old whishlist bug to make the disabled flag changeable
16:14:06 mriedem https://github.com/openstack/nova/commit/f371198
16:16:20 gibi mriedem: nice finding. Even this patch uses direct db access to created a disabled instance_type https://github.com/openstack/nova/commit/f371198#diff-59404890aec6423ece734730c153addbR752
16:17:08 gibi I think we can conclude that there is no way to set it outside of the API
16:17:21 mriedem so i think the issue might be related to https://github.com/openstack/nova/blob/master/nova/compute/flavors.py#L52
16:17:35 mriedem and how we used to store the embedded flavor in the instance based on the instance system_metadata
16:17:41 mriedem which didn't include the 'disabled' key
16:18:27 mriedem so likely any really old instance didn't have the instance_type_disabled key in it's system_metadata
16:18:58 mriedem which might explain why we're hitting "Field `disabled' cannot be None" when extracting these things and loading them into Flavor objects
16:19:37 gibi this very well be the explanation
16:20:27 gibi dansmith: would this ^^ be enough explanation to revive https://review.openstack.org/#/c/529194/ ?
16:21:09 gibi dansmith: you were -1 on that patch as we did not know why can be the instance.flavor.disabled undefined
16:21:46 gibi mriedem: thanks for the investigation I was not able to find this by myself
16:22:40 dansmith gibi: I'd really rather not embed that behavior in the flavor object.. either we should set those properties when we load an older flavor from the instance_extra, or (better) just be graceful about it when we're emitting the notification
16:26:11 mriedem left a comment here about how i think it all tied together https://bugs.launchpad.net/nova/+bug/1739325/comments/9
16:26:12 openstack Launchpad bug 1739325 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields" [Medium,Confirmed] - Assigned to Balazs Gibizer (balazs-gibizer)
16:26:29 gibi mriedem: thanks

Earlier   Later