Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-08-11
11:10:40 bauzas mriedem: snap, my bad
11:10:42 mriedem now you have to run https://review.openstack.org/#/c/487954/ back through twice
11:10:47 mriedem i'm already tagging rc1
11:10:48 bauzas mriedem: yeah, good point
11:11:06 bauzas mriedem: I just updated the RC1 patch
11:11:17 bauzas mriedem: with the latest merge sha1
11:11:41 bauzas mriedem: I can revert back to PS14 so we won't need to run yet again jenkins
11:11:49 bauzas and I'll do what you say
11:11:53 vdrok dtantsur: I'll just test this locally now I think. and then we'll try to make the grenade do this resource class setting to see the whole process
11:11:55 bauzas mriedem: ack ?
11:12:39 mriedem bauzas: i think https://review.openstack.org/#/c/492788/ is ready to go
11:12:40 dtantsur vdrok: cool, thanks!
11:12:44 mriedem the ironic stuff is rc2
11:13:19 dtantsur that will require backporting to stable/pike, right?
11:13:23 dtantsur also morning mriedem
11:13:35 mriedem dtantsur: yes
11:13:42 bauzas mriedem: okay, I'm fine then
11:13:43 dtantsur ack
11:13:46 bauzas mriedem: removing my -1
11:13:47 bauzas smcginnis: ^
11:13:48 mriedem but that's just because we don't have milestone-proposed anymore
11:16:03 openstackgerrit Sylvain Bauza proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954
11:16:17 mriedem dtantsur: i was not sure what to make of http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621
11:16:27 mriedem dtantsur: when i dug through those logs,
11:16:37 mriedem the ironic driver wasn't populating inventory in placement,
11:16:44 mriedem which would have auto-created the custom resource class
11:17:01 mriedem i'm not sure if we have some chicken and egg issue
11:17:15 dtantsur mriedem: it seems to be that it tries to proceed, and actually fails on http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_588272
11:17:21 dtantsur I'm not sure it's expected or not
11:17:37 mriedem dtantsur: i think that's a side effect
11:17:37 dtantsur if it is, then I'm pretty sure we have a chicked and egg situation
11:18:04 mriedem there is a periodic task in the compute service that pulls inventory from ironic https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L775
11:18:13 mriedem ^ includes any custom resource class on the node
11:18:42 mriedem the resource tracker in the compute service calls that from here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L834
11:19:11 mriedem and if there is a custom resource class in that inventory, it would auto-create it in placement here https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L775
11:19:29 mriedem when i was looking at the logs on that failed job, i never saw _update_inventory get called
11:20:16 mriedem so with my limited understanding of how the ironic driver works, when does the node cache in the driver actually have something show up here? https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L735
11:20:27 vdrok 57-44dc-8b96-4c17182a9a64 have been updated for custom resource class 'baremetal'."
11:20:27 vdrok dtantsur: locally, I see "The flavor extra_specs for Ironic instance 0b8460b1-22
11:20:36 cdent mriedem, bauzas any pending crises that need attention?
11:20:55 mriedem cdent: not really, just trying to sort out https://review.openstack.org/#/c/487954/1
11:20:59 mriedem https://review.openstack.org/#/c/487954/ i mean
11:21:05 vdrok dtantsur: mriedem http://paste.openstack.org/show/618160/
11:21:06 mriedem and this http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621
11:21:39 vdrok so same thing as you see I guess
11:21:59 openstackgerrit Sylvain Bauza proposed openstack/nova master: DNM: Add more logging + hook devstack https://review.openstack.org/492957
11:22:00 bauzas cdent: nothing really critical
11:22:12 bauzas cdent: ironic flavor migration is hold for RC2
11:22:17 bauzas cdent: and RC1 is on its way
11:22:17 mriedem "Unable to allocate inventory for resource provider 68d57495-2daa-46b3-8e2c-f2f0a19dbaf8: No such resource class CUSTOM_BAREMETAL."
11:22:32 bauzas mriedem: dtantsur: https://review.openstack.org/492957 HTH
11:22:55 dtantsur mriedem: I guess it does it once in 2 minutes. this may be the cause of the problem: we may need to wait for Placement to get it. Do we have any CLI for placement so far?
11:23:18 mriedem dtantsur: there is a series of unmerged osc changes
11:23:26 bauzas (lunch)
11:23:30 mriedem https://blueprints.launchpad.net/nova/+spec/placement-osc-plugin
11:23:50 vdrok dtantsur: don't think there is a problem with waiting, I still see the same error after 4 minutes
11:24:18 dtantsur oh
11:24:35 dtantsur thanks vdrok. this does look like a problem to me now
11:25:44 mriedem this is really weird
11:25:45 mriedem Aug 11 11:18:56 ubuntu nova-compute[7647]: INFO nova.scheduler.client.report [None req-267b8c81-5ea5-4e13-9ea8-02354628c37f None None] Compute node 68d57495-2daa-46b3-8e2c-f2f0a19dbaf8 reported no inventory but previous inventory was detected. Deleting existing inventory records.
11:26:00 mriedem ^ is if the get_inventory method in the driver reported some inventory at one point, but now it doesn't
11:26:58 mriedem was the node disabled?
11:27:10 dtantsur vdrok: ^^^ what was your testing scenario?
11:27:12 mriedem https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L741
11:28:38 vdrok dtantsur: mriedem I had an instance booted yesterday, devstack setup also from yesterday, with no custom non-merged patches. then I updated nova code with this flavor migration change, restarted n-cpu, updated the resource class of the node the instance was on from None to baremetal
11:29:09 dtantsur mriedem: oh. it seems that we don't report inventory for ACTIVE nodes
11:29:19 dtantsur https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L324-L329
11:29:27 mriedem Aug 11 11:18:56 ubuntu nova-compute[7647]: DEBUG nova.virt.ironic.driver [None req-267b8c81-5ea5-4e13-9ea8-02354628c37f None None] The flavor extra_specs for Ironic instance 0b8460b1-22 57-44dc-8b96-4c17182a9a64 have been updated for custom resource class 'baremetal'. {{(pid=7647) _pike_flavor_migration /opt/stack/nova/nova/virt/ironic/driver.py:561}}
11:29:27 mriedem vdrok: ok and we see that get picked up here
11:29:39 vdrok yup
11:29:40 dtantsur which is... wrong?
11:30:19 mriedem dtantsur: ACTIVE?
11:30:22 mriedem https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L176
11:30:26 mriedem that state checking seems fine
11:30:33 mriedem if it's not available, don't say it is
11:30:48 dtantsur states.ACTIVE is not in good state
11:30:58 dtantsur which is the state we have when we have an instance provisioned
11:31:26 dtantsur yeah, but should we return an empty inventory for nodes with an instance? actually, I guess, we should, right
11:31:46 dtantsur but maybe instead of an empty dict we should return all values with zeroes?
11:31:46 mriedem no idea
11:31:49 dtantsur mriedem: ^^^
11:31:59 dtantsur like CUSTOM_FOOBAR=0, VCPU=0, etc?
11:32:29 mriedem we can't report 0 inventory to placement
11:32:30 mriedem https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L752
11:32:38 mriedem min_unit is minimum of 1
11:32:52 dtantsur hmmm
11:33:04 dtantsur then the placement indeed has no idea about the new resource class
11:33:13 dtantsur we never return it from virt/ironic
11:33:17 mriedem https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/handlers/inventory.py#L39
11:33:23 mriedem yeah inventory total has to be at least 1
11:33:30 dtantsur but wait, cannot a hypervisor have 0 free memory, for example?
11:33:33 mriedem dtantsur: is that the chicken and egg?
11:33:41 dtantsur (please pardon my ignorance)
11:34:39 dtantsur but yes, I suspect we cannot create resource classes for such nodes, because we don't report them back ever
11:35:19 mriedem ok so the node has the resource_class set after there is an instance associated and that's how we migrate the flavor extra spec,
11:35:35 mriedem but once the instance is associated, the node is in an ACTIVE state which means we don't report inventory for it?
11:35:40 dtantsur correct
11:35:43 mriedem and thus can't auto-create the newly added resource class
11:36:13 mriedem and that instance <> node is what's already consuming the existing vcpu/memory_mb/disk_gb inventory that we can't delete now
11:36:28 mriedem Aug 11 11:18:56 ubuntu nova-compute[7647]: WARNING nova.scheduler.client.report [None req-267b8c81-5ea5-4e13-9ea8-02354628c37f None None] [req-27dbace7-c527-4f8f-add8-d732047d7382] We c annot delete inventory 'VCPU, MEMORY_MB, DISK_GB' for resource provider 68d57495-2daa-46b3-8e2c-f2f0a19dbaf8 because the inventory is in use.

Earlier   Later