Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-08-11
11:04:12 dtantsur bauzas: done
11:04:20 bauzas dtantsur: I have to apologize, I wasn't having a full view of the situation
11:04:26 vdrok dtantsur: so, for that code to be triggered, we have to have an active instance booted with old flavor, and afterwards being updated with resource class right?
11:04:35 bauzas dtantsur: okay, I'm on https://review.openstack.org/#/c/487954/
11:04:36 vdrok code in https://review.openstack.org/#/c/487954/14 I mean
11:04:36 dtantsur bauzas: no problem, thanks for helping us with this stuff anyway
11:05:08 bauzas vdrok: for the nova code to be triggered, you have to set resource classes for ironic nodes firsrt
11:05:29 bauzas vdrok: that would be done by devstack in the job we discuss
11:06:15 dtantsur vdrok: I think vdrok's point is that we still won't see the log message, because it needs the resource_class to not be present initially..
11:06:30 vdrok dtantsur: bauzas exactly
11:06:50 openstackgerrit Sylvain Bauza proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954
11:06:57 dtantsur so I wonder if the only option to test it is to actually get a devstack locally, and try it
11:07:02 bauzas dtantsur: done ^
11:07:31 bauzas dtantsur: just to make it clear, I'm just updating it but just for testing purposes
11:07:52 dtantsur vdrok: in any case, could you please review https://review.openstack.org/#/c/491777/ ? this is something we must get in today to not block nova further
11:08:01 bauzas dtantsur: once we're sure the flavor is correctly updated, I feel fine to just revert it to PS14 and +W it since melwitt already gave her +2
11:08:17 vdrok dtantsur: yeah that one looks fine to me
11:10:22 mriedem bauzas: you know you could have pushed a change on top that depended on https://review.openstack.org/#/c/491777/
11:10:40 bauzas mriedem: snap, my bad
11:10:42 mriedem now you have to run https://review.openstack.org/#/c/487954/ back through twice
11:10:47 mriedem i'm already tagging rc1
11:10:48 bauzas mriedem: yeah, good point
11:11:06 bauzas mriedem: I just updated the RC1 patch
11:11:17 bauzas mriedem: with the latest merge sha1
11:11:41 bauzas mriedem: I can revert back to PS14 so we won't need to run yet again jenkins
11:11:49 bauzas and I'll do what you say
11:11:53 vdrok dtantsur: I'll just test this locally now I think. and then we'll try to make the grenade do this resource class setting to see the whole process
11:11:55 bauzas mriedem: ack ?
11:12:39 mriedem bauzas: i think https://review.openstack.org/#/c/492788/ is ready to go
11:12:40 dtantsur vdrok: cool, thanks!
11:12:44 mriedem the ironic stuff is rc2
11:13:19 dtantsur that will require backporting to stable/pike, right?
11:13:23 dtantsur also morning mriedem
11:13:35 mriedem dtantsur: yes
11:13:42 bauzas mriedem: okay, I'm fine then
11:13:43 dtantsur ack
11:13:46 bauzas mriedem: removing my -1
11:13:47 bauzas smcginnis: ^
11:13:48 mriedem but that's just because we don't have milestone-proposed anymore
11:16:03 openstackgerrit Sylvain Bauza proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954
11:16:17 mriedem dtantsur: i was not sure what to make of http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621
11:16:27 mriedem dtantsur: when i dug through those logs,
11:16:37 mriedem the ironic driver wasn't populating inventory in placement,
11:16:44 mriedem which would have auto-created the custom resource class
11:17:01 mriedem i'm not sure if we have some chicken and egg issue
11:17:15 dtantsur mriedem: it seems to be that it tries to proceed, and actually fails on http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_588272
11:17:21 dtantsur I'm not sure it's expected or not
11:17:37 dtantsur if it is, then I'm pretty sure we have a chicked and egg situation
11:17:37 mriedem dtantsur: i think that's a side effect
11:18:04 mriedem there is a periodic task in the compute service that pulls inventory from ironic https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L775
11:18:13 mriedem ^ includes any custom resource class on the node
11:18:42 mriedem the resource tracker in the compute service calls that from here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L834
11:19:11 mriedem and if there is a custom resource class in that inventory, it would auto-create it in placement here https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L775
11:19:29 mriedem when i was looking at the logs on that failed job, i never saw _update_inventory get called
11:20:16 mriedem so with my limited understanding of how the ironic driver works, when does the node cache in the driver actually have something show up here? https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L735
11:20:27 vdrok dtantsur: locally, I see "The flavor extra_specs for Ironic instance 0b8460b1-22
11:20:27 vdrok 57-44dc-8b96-4c17182a9a64 have been updated for custom resource class 'baremetal'."
11:20:36 cdent mriedem, bauzas any pending crises that need attention?
11:20:55 mriedem cdent: not really, just trying to sort out https://review.openstack.org/#/c/487954/1
11:20:59 mriedem https://review.openstack.org/#/c/487954/ i mean
11:21:05 vdrok dtantsur: mriedem http://paste.openstack.org/show/618160/
11:21:06 mriedem and this http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621
11:21:39 vdrok so same thing as you see I guess
11:21:59 openstackgerrit Sylvain Bauza proposed openstack/nova master: DNM: Add more logging + hook devstack https://review.openstack.org/492957
11:22:00 bauzas cdent: nothing really critical
11:22:12 bauzas cdent: ironic flavor migration is hold for RC2
11:22:17 mriedem "Unable to allocate inventory for resource provider 68d57495-2daa-46b3-8e2c-f2f0a19dbaf8: No such resource class CUSTOM_BAREMETAL."
11:22:17 bauzas cdent: and RC1 is on its way
11:22:32 bauzas mriedem: dtantsur: https://review.openstack.org/492957 HTH
11:22:55 dtantsur mriedem: I guess it does it once in 2 minutes. this may be the cause of the problem: we may need to wait for Placement to get it. Do we have any CLI for placement so far?
11:23:18 mriedem dtantsur: there is a series of unmerged osc changes
11:23:26 bauzas (lunch)
11:23:30 mriedem https://blueprints.launchpad.net/nova/+spec/placement-osc-plugin
11:23:50 vdrok dtantsur: don't think there is a problem with waiting, I still see the same error after 4 minutes
11:24:18 dtantsur oh
11:24:35 dtantsur thanks vdrok. this does look like a problem to me now
11:25:44 mriedem this is really weird
11:25:45 mriedem Aug 11 11:18:56 ubuntu nova-compute[7647]: INFO nova.scheduler.client.report [None req-267b8c81-5ea5-4e13-9ea8-02354628c37f None None] Compute node 68d57495-2daa-46b3-8e2c-f2f0a19dbaf8 reported no inventory but previous inventory was detected. Deleting existing inventory records.
11:26:00 mriedem ^ is if the get_inventory method in the driver reported some inventory at one point, but now it doesn't
11:26:58 mriedem was the node disabled?
11:27:10 dtantsur vdrok: ^^^ what was your testing scenario?
11:27:12 mriedem https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L741
11:28:38 vdrok dtantsur: mriedem I had an instance booted yesterday, devstack setup also from yesterday, with no custom non-merged patches. then I updated nova code with this flavor migration change, restarted n-cpu, updated the resource class of the node the instance was on from None to baremetal
11:29:09 dtantsur mriedem: oh. it seems that we don't report inventory for ACTIVE nodes
11:29:19 dtantsur https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L324-L329
11:29:27 mriedem vdrok: ok and we see that get picked up here
11:29:27 mriedem Aug 11 11:18:56 ubuntu nova-compute[7647]: DEBUG nova.virt.ironic.driver [None req-267b8c81-5ea5-4e13-9ea8-02354628c37f None None] The flavor extra_specs for Ironic instance 0b8460b1-22 57-44dc-8b96-4c17182a9a64 have been updated for custom resource class 'baremetal'. {{(pid=7647) _pike_flavor_migration /opt/stack/nova/nova/virt/ironic/driver.py:561}}
11:29:39 vdrok yup
11:29:40 dtantsur which is... wrong?
11:30:19 mriedem dtantsur: ACTIVE?
11:30:22 mriedem https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L176
11:30:26 mriedem that state checking seems fine
11:30:33 mriedem if it's not available, don't say it is
11:30:48 dtantsur states.ACTIVE is not in good state
11:30:58 dtantsur which is the state we have when we have an instance provisioned
11:31:26 dtantsur yeah, but should we return an empty inventory for nodes with an instance? actually, I guess, we should, right
11:31:46 mriedem no idea
11:31:46 dtantsur but maybe instead of an empty dict we should return all values with zeroes?
11:31:49 dtantsur mriedem: ^^^
11:31:59 dtantsur like CUSTOM_FOOBAR=0, VCPU=0, etc?

Earlier   Later