Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-03
15:29:07 efried => 72b0a2e0-277b-4698-bfd6-aa15623e44bd
15:29:51 jroll efried: those are likely fake nodes created by other tempest tests
15:30:04 jroll that said, this is a third party CI environment that isn't mine
15:30:10 jroll so I'll let sam speak from here on :)
15:31:05 efried ahhh. So, that appears to be what's actually causing the failure here. Those guys show up in the provider tree, so we try to update them, but by the time we get to hitting placement for them, they're gone.
15:31:19 sambetts efried: those nodes were created by the ironic tempest api tests and then deleted after, the only real node in the test environment is b5bb2c51-3960-4d9c-b47e-223d83cb6139
15:31:40 jroll oh, we update everything at build time?
15:31:50 jroll or everything for that compute service
15:32:13 efried jroll: Yes, update_from_provider_tree is supposed to make the host's placement presence right.
15:32:36 jroll mmmm
15:32:48 stephenfin sahid: Yup, makes sense. +2 now
15:36:54 openstackgerrit Merged openstack/nova master: Update xenapi_disable_agent config option usage in docs https://review.openstack.org/578848
15:43:03 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: use dest host vif migrate details for live migration https://review.openstack.org/551370
15:43:04 openstackgerrit Matt Riedemann proposed openstack/nova master: Annotate flows and handle PortBindingDeletionFailed in ComputeManager https://review.openstack.org/551371
15:43:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Port binding based on events during live migration https://review.openstack.org/434870
15:43:06 openstackgerrit Matt Riedemann proposed openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537
15:45:09 efried sambetts: jroll: So I think this is probably the problem here, which a retry isn't going to fix: ironic's update_provider_tree isn't getting rid of nodes when they disappear.
15:45:56 efried sambetts, jroll: How (via what code path) are these fake nodes "created by the ironic tempest api tests and then deleted after" ?
15:46:04 jroll efried: right, and it races with the thing that cleans up orphan nodes
15:46:29 jroll efried: via api tests in our tempest plugin being run (in parallel?) with the scenario tests
15:46:32 efried jroll: Oh, we have a thing that cleans up orphan nodes? Does that thing actually remove them from placement? Where does that code live?
15:47:15 jroll efried: not sure where the code is offhand, but see e.g. http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz#_Jul_03_12_53_45_414036
15:47:23 efried jroll: Okay, but the tempest tests presumably use some kind of API to create/delete the nodes. Because they're ending up registered in placement somehow.
15:47:28 jroll I think it's handled in update_available_resource or something
15:47:49 sambetts they are just created in ironic, then nova syncs with ironic and picks up the new nodes
15:47:55 jroll efried: yes, POST ironic-host/v1/nodes, then the ..... yes that
15:48:31 efried oho
15:48:42 efried it's happening *after* _update_available_resource_for_node
15:49:32 efried sambetts: How reproducible is this thang?
15:51:00 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Always retrieve network information if available https://review.openstack.org/564444
15:51:01 openstackgerrit Stephen Finucane proposed openstack/nova master: network: Retrieve tunneled status in '_get_physnet_info' https://review.openstack.org/564445
15:51:02 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add NUMACell.network_metadata https://review.openstack.org/564439
15:51:03 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add NUMATopologyLimits.networks https://review.openstack.org/575486
15:51:04 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Start accounting for networks in NUMA placement https://review.openstack.org/564448
15:51:05 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add RequestSpec.network_metadata https://review.openstack.org/564442
15:51:06 openstackgerrit Stephen Finucane proposed openstack/nova master: scheduler: Start utilizing RequestSpec.network_metadata https://review.openstack.org/564452
15:51:07 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Add '[neutron] physnets' and related options https://review.openstack.org/564440
15:51:08 sambetts every single one of my CI runs hits it at the moment, I expect the reason the upstream ironic CI isn't hitting it is because setting tempest concurrency=1
15:51:08 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start populating NUMACell.network_metadata field https://review.openstack.org/564441
16:08:57 openstack Launchpad bug 1739325 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields" [Medium,Confirmed] - Assigned to Balazs Gibizer (balazs-gibizer)
16:08:57 mriedem gibi: so i'm looking at https://bugs.launchpad.net/nova/+bug/1739325 again and Flavor.disabled isn't something you can set from the API as far as i can tell
16:11:21 gibi mriedem: but then why we even have a disabled field?
16:11:43 mriedem idk, i'm not seeing in git history where it was added, or if there is a way to ever set it via cli or something
16:11:47 mriedem dansmith: do you remember?
16:13:15 openstack Launchpad bug 1259262 in python-novaclient "there is no api or cli to enable/disable a flavor" [Wishlist,Opinion]
16:13:15 gibi mriedem: https://bugs.launchpad.net/nova/+bug/1259262
16:13:48 gibi pretty old whishlist bug to make the disabled flag changeable
16:14:06 mriedem https://github.com/openstack/nova/commit/f371198
16:16:20 gibi mriedem: nice finding. Even this patch uses direct db access to created a disabled instance_type https://github.com/openstack/nova/commit/f371198#diff-59404890aec6423ece734730c153addbR752
16:17:08 gibi I think we can conclude that there is no way to set it outside of the API
16:17:21 mriedem so i think the issue might be related to https://github.com/openstack/nova/blob/master/nova/compute/flavors.py#L52
16:17:35 mriedem and how we used to store the embedded flavor in the instance based on the instance system_metadata
16:17:41 mriedem which didn't include the 'disabled' key
16:18:27 mriedem so likely any really old instance didn't have the instance_type_disabled key in it's system_metadata
16:18:58 mriedem which might explain why we're hitting "Field `disabled' cannot be None" when extracting these things and loading them into Flavor objects
16:19:37 gibi this very well be the explanation
16:20:27 gibi dansmith: would this ^^ be enough explanation to revive https://review.openstack.org/#/c/529194/ ?
16:21:09 gibi dansmith: you were -1 on that patch as we did not know why can be the instance.flavor.disabled undefined
16:21:46 gibi mriedem: thanks for the investigation I was not able to find this by myself
16:22:40 dansmith gibi: I'd really rather not embed that behavior in the flavor object.. either we should set those properties when we load an older flavor from the instance_extra, or (better) just be graceful about it when we're emitting the notification
16:26:11 mriedem left a comment here about how i think it all tied together https://bugs.launchpad.net/nova/+bug/1739325/comments/9
16:26:12 openstack Launchpad bug 1739325 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields" [Medium,Confirmed] - Assigned to Balazs Gibizer (balazs-gibizer)
16:26:29 gibi mriedem: thanks
16:27:58 gibi dansmith: do we still load those old flavors from the system_metadata or we loaded once and copied into the instance without the disabled field?
16:28:17 dansmith no, we don't, they've all been migrated to extra now
16:29:02 gibi dansmith: then the first option is not a real solution
16:29:40 dansmith gibi: no, the first option I meant was when we're in the context of the instance, loading a flavor from instance_extra
16:29:43 mriedem wouldn't we just implement obj_load_attr (lazy-load) for Flavor.disabled?
16:30:14 dansmith we could, but that also seems obscure to me,
16:30:28 dansmith because the flavor could be coming from the instance, or from the actual db
16:30:40 dansmith and we wouldn't want the disabled default to affect the latter
16:30:54 dansmith it seems like we should patch it up if/when we're loading from extra and notice that field is gone
16:30:54 mriedem so then add default=False to 'disabled': fields.BooleanField(), ?
16:31:29 dansmith that's basically the same as the lazy-load, although slightly less icky, and one of the things I suggested in the original patch
16:32:22 mriedem ok so maybe handle in _flavor_from_db
16:32:25 mriedem Instance._flavor_from_db
16:32:31 mriedem if 'disabled' not in flavor_info
16:32:39 mriedem flavor_info['disabled'] = False
16:32:41 dansmith exactly
16:32:45 mriedem ok i'll push that up
16:32:46 gibi mriedem, dansmith I got it
16:32:48 dansmith because there we know what we're dealing with
16:33:00 dansmith and a comment about old instance flavors will be quite relevant
16:33:47 dansmith _load_flavor will get it implicitly if we're actually lazy-loading instance.flavor too, so that is good
16:34:35 zzzeek nova peeps, dansmith / mriedem , what happens when someone in Newton sets "workers=0" in nova.conf, does that mean default to number of CPUs ?
16:35:00 gibi mriedem, dansmith: thanks for the help
16:36:49 mriedem dansmith: gibi: pre-tests https://gist.github.com/mriedem/3f1560a7e9697ade633ab11b6aab8c62
16:37:01 mriedem zzzeek: yes
16:37:05 zzzeek mriedem: thanks!
16:37:10 dansmith mriedem: sure
16:37:54 gibi mriedem: LGTM
16:49:39 efried sambetts: Sorry, it's been a day. Can you tell me where/how to run this thing?
16:50:16 efried If I push a nova patch, do I have to push something in ironic with a Depends-On? Looks like an experimental job, needs special incantation?
16:51:30 efried sambetts: Can I mod the commit message on https://review.openstack.org/#/c/514312/ to add the Depends-On?
16:52:26 openstackgerrit Jay Pipes proposed openstack/nova master: move lookup of provider from _new_allocations() https://review.openstack.org/579920
16:52:27 openstackgerrit Jay Pipes proposed openstack/nova master: placement: delete auto-created consumers on fail https://review.openstack.org/579921
16:55:05 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Delete orphan nodes before updating resources https://review.openstack.org/579922
16:57:14 sambetts efried: you can mod the commit message to add the Depends-On, then you need to leave a "cisco-experimental" comment to trigger our experimental job queue which has some other fixes for our ironic job
16:57:30 efried sambetts: Roger wilco, it's on the way...
16:57:39 sambetts efried: thanks so much for looking into this!

Earlier   Later