Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-08-11
10:58:17 dtantsur bauzas: why should we see any updates, given that the nodes don't have resource classes yet?
10:58:38 bauzas oh f****
10:59:01 bauzas dtantsur: you killed me :p
10:59:22 dtantsur bear metal powerzzz!
10:59:25 bauzas dtantsur: those ironic nodes aren't having resource classes
10:59:26 bauzas ?
10:59:31 bauzas yet, I mean ?
11:00:04 dtantsur bauzas: yep. your logging line should show up in https://review.openstack.org/491777 instead - hence I made it depends-on the nova patch
11:01:20 bauzas dtantsur: oh snap https://review.openstack.org/#/c/491777/9/devstack/lib/ironic@1821 right?
11:01:48 bauzas until that devstack change, the gate nodes aren't yet correctly having resource classes
11:01:55 bauzas I thought it was already the case
11:02:08 bauzas dtantsur: IMHO, we should invert the depends-on
11:02:10 dtantsur hah, sorry for not figuring out the confusion earlier
11:02:48 bauzas dtantsur: why would you make the devstack change dependent on the nova change ?
11:02:52 dtantsur bauzas: yeah, good call probably. wanna me drop the depends-on from my patch?
11:03:10 dtantsur I wanted one of them to depend on the other, I don't care which exactly :)
11:03:10 bauzas if the nova change uses what's provided by the devstack one ?
11:03:29 bauzas dtantsur: yeah, please remove the depends-on on the devstack one
11:03:51 bauzas dtantsur: and then I'll update https://review.openstack.org/#/c/487954/ to include devstack
11:04:12 dtantsur bauzas: done
11:04:20 bauzas dtantsur: I have to apologize, I wasn't having a full view of the situation
11:04:26 vdrok dtantsur: so, for that code to be triggered, we have to have an active instance booted with old flavor, and afterwards being updated with resource class right?
11:04:35 bauzas dtantsur: okay, I'm on https://review.openstack.org/#/c/487954/
11:04:36 dtantsur bauzas: no problem, thanks for helping us with this stuff anyway
11:04:36 vdrok code in https://review.openstack.org/#/c/487954/14 I mean
11:05:08 bauzas vdrok: for the nova code to be triggered, you have to set resource classes for ironic nodes firsrt
11:05:29 bauzas vdrok: that would be done by devstack in the job we discuss
11:06:15 dtantsur vdrok: I think vdrok's point is that we still won't see the log message, because it needs the resource_class to not be present initially..
11:06:30 vdrok dtantsur: bauzas exactly
11:06:50 openstackgerrit Sylvain Bauza proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954
11:06:57 dtantsur so I wonder if the only option to test it is to actually get a devstack locally, and try it
11:07:02 bauzas dtantsur: done ^
11:07:31 bauzas dtantsur: just to make it clear, I'm just updating it but just for testing purposes
11:07:52 dtantsur vdrok: in any case, could you please review https://review.openstack.org/#/c/491777/ ? this is something we must get in today to not block nova further
11:08:01 bauzas dtantsur: once we're sure the flavor is correctly updated, I feel fine to just revert it to PS14 and +W it since melwitt already gave her +2
11:08:17 vdrok dtantsur: yeah that one looks fine to me
11:10:22 mriedem bauzas: you know you could have pushed a change on top that depended on https://review.openstack.org/#/c/491777/
11:10:40 bauzas mriedem: snap, my bad
11:10:42 mriedem now you have to run https://review.openstack.org/#/c/487954/ back through twice
11:10:47 mriedem i'm already tagging rc1
11:10:48 bauzas mriedem: yeah, good point
11:11:06 bauzas mriedem: I just updated the RC1 patch
11:11:17 bauzas mriedem: with the latest merge sha1
11:11:41 bauzas mriedem: I can revert back to PS14 so we won't need to run yet again jenkins
11:11:49 bauzas and I'll do what you say
11:11:53 vdrok dtantsur: I'll just test this locally now I think. and then we'll try to make the grenade do this resource class setting to see the whole process
11:11:55 bauzas mriedem: ack ?
11:12:39 mriedem bauzas: i think https://review.openstack.org/#/c/492788/ is ready to go
11:12:40 dtantsur vdrok: cool, thanks!
11:12:44 mriedem the ironic stuff is rc2
11:13:19 dtantsur that will require backporting to stable/pike, right?
11:13:23 dtantsur also morning mriedem
11:13:35 mriedem dtantsur: yes
11:13:42 bauzas mriedem: okay, I'm fine then
11:13:43 dtantsur ack
11:13:46 bauzas mriedem: removing my -1
11:13:47 bauzas smcginnis: ^
11:13:48 mriedem but that's just because we don't have milestone-proposed anymore
11:16:03 openstackgerrit Sylvain Bauza proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954
11:16:17 mriedem dtantsur: i was not sure what to make of http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621
11:16:27 mriedem dtantsur: when i dug through those logs,
11:16:37 mriedem the ironic driver wasn't populating inventory in placement,
11:16:44 mriedem which would have auto-created the custom resource class
11:17:01 mriedem i'm not sure if we have some chicken and egg issue
11:17:15 dtantsur mriedem: it seems to be that it tries to proceed, and actually fails on http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_588272
11:17:21 dtantsur I'm not sure it's expected or not
11:17:37 mriedem dtantsur: i think that's a side effect
11:17:37 dtantsur if it is, then I'm pretty sure we have a chicked and egg situation
11:18:04 mriedem there is a periodic task in the compute service that pulls inventory from ironic https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L775
11:18:13 mriedem ^ includes any custom resource class on the node
11:18:42 mriedem the resource tracker in the compute service calls that from here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L834
11:19:11 mriedem and if there is a custom resource class in that inventory, it would auto-create it in placement here https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L775
11:19:29 mriedem when i was looking at the logs on that failed job, i never saw _update_inventory get called
11:20:16 mriedem so with my limited understanding of how the ironic driver works, when does the node cache in the driver actually have something show up here? https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L735
11:20:27 vdrok 57-44dc-8b96-4c17182a9a64 have been updated for custom resource class 'baremetal'."
11:20:27 vdrok dtantsur: locally, I see "The flavor extra_specs for Ironic instance 0b8460b1-22
11:20:36 cdent mriedem, bauzas any pending crises that need attention?
11:20:55 mriedem cdent: not really, just trying to sort out https://review.openstack.org/#/c/487954/1
11:20:59 mriedem https://review.openstack.org/#/c/487954/ i mean
11:21:05 vdrok dtantsur: mriedem http://paste.openstack.org/show/618160/
11:21:06 mriedem and this http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621
11:21:39 vdrok so same thing as you see I guess
11:21:59 openstackgerrit Sylvain Bauza proposed openstack/nova master: DNM: Add more logging + hook devstack https://review.openstack.org/492957
11:22:00 bauzas cdent: nothing really critical
11:22:12 bauzas cdent: ironic flavor migration is hold for RC2
11:22:17 bauzas cdent: and RC1 is on its way
11:22:17 mriedem "Unable to allocate inventory for resource provider 68d57495-2daa-46b3-8e2c-f2f0a19dbaf8: No such resource class CUSTOM_BAREMETAL."
11:22:32 bauzas mriedem: dtantsur: https://review.openstack.org/492957 HTH
11:22:55 dtantsur mriedem: I guess it does it once in 2 minutes. this may be the cause of the problem: we may need to wait for Placement to get it. Do we have any CLI for placement so far?
11:23:18 mriedem dtantsur: there is a series of unmerged osc changes
11:23:26 bauzas (lunch)
11:23:30 mriedem https://blueprints.launchpad.net/nova/+spec/placement-osc-plugin
11:23:50 vdrok dtantsur: don't think there is a problem with waiting, I still see the same error after 4 minutes
11:24:18 dtantsur oh
11:24:35 dtantsur thanks vdrok. this does look like a problem to me now
11:25:44 mriedem this is really weird
11:25:45 mriedem Aug 11 11:18:56 ubuntu nova-compute[7647]: INFO nova.scheduler.client.report [None req-267b8c81-5ea5-4e13-9ea8-02354628c37f None None] Compute node 68d57495-2daa-46b3-8e2c-f2f0a19dbaf8 reported no inventory but previous inventory was detected. Deleting existing inventory records.
11:26:00 mriedem ^ is if the get_inventory method in the driver reported some inventory at one point, but now it doesn't
11:26:58 mriedem was the node disabled?
11:27:10 dtantsur vdrok: ^^^ what was your testing scenario?

Earlier   Later