Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-08-11
10:21:25 dtantsur bauzas: I can make https://review.openstack.org/#/c/491777/ depends-on this patch, to see how it behaves with resource_classes. wdyt?
10:22:14 bauzas dtantsur: sure
10:22:43 bauzas dtantsur: if that doesn't trample you waiting for devstack patch being merged
10:23:00 bauzas dtantsur: I mean, if you can wait for that devstack patch to be merged, that's fine to me
10:23:01 dtantsur we need to make sure it won't blow up after merging the both changes :)
10:23:06 bauzas yeah
10:23:32 bauzas anyway, just doing an urgent internal bug thingy and then I'm back to the DNM patch
10:23:43 dtantsur ack
10:27:44 dtantsur ok, both patches updated, waiting for the CI
10:41:06 openstackgerrit Dmitry Tantsur proposed openstack/nova master: Deprecate bare metal filters https://review.openstack.org/492563
10:42:04 dtantsur bauzas: meanwhile, do you think we can also get ^^ in?
10:53:11 bauzas dtantsur: looks to me hard for RC1
10:53:21 bauzas dtantsur: even if we haven't yet merged it
10:53:26 bauzas tagged it, sorry
10:53:50 dtantsur ok, that's fine. I just have an ironic docs patch depending on it, I may need to split it
10:53:53 bauzas dtantsur: we're already past the deadline but I leave matt make the hard call :)
10:54:29 dtantsur vdrok: first of all, please review https://review.openstack.org/#/c/487954/
10:54:52 vdrok dtantsur: looking
10:56:09 dtantsur vdrok: it fails the ironic CI for some reason. I see network connection problems between various services, so it is not necessary related to the patch itself
10:56:36 vdrok dtantsur: yup, there is some socket error in the vbmc log as well
10:56:51 dtantsur I've rechecked it, let's see
10:57:16 dtantsur vdrok: our next step would be to make https://review.openstack.org/491777 and https://review.openstack.org/476968 pass the CI - reviews welcome there too
10:57:26 bauzas dtantsur: I'm more concerned by the fact I don't see the logs mentioning the flavor update rather than the Ironic job giving us -1 :)
10:57:57 bauzas dtantsur: in other words, I feel brave enough to +2 some ironic change if I'm sure the job issues are unrelated
10:58:17 dtantsur bauzas: why should we see any updates, given that the nodes don't have resource classes yet?
10:58:38 bauzas oh f****
10:59:01 bauzas dtantsur: you killed me :p
10:59:22 dtantsur bear metal powerzzz!
10:59:25 bauzas dtantsur: those ironic nodes aren't having resource classes
10:59:26 bauzas ?
10:59:31 bauzas yet, I mean ?
11:00:04 dtantsur bauzas: yep. your logging line should show up in https://review.openstack.org/491777 instead - hence I made it depends-on the nova patch
11:01:20 bauzas dtantsur: oh snap https://review.openstack.org/#/c/491777/9/devstack/lib/ironic@1821 right?
11:01:48 bauzas until that devstack change, the gate nodes aren't yet correctly having resource classes
11:01:55 bauzas I thought it was already the case
11:02:08 bauzas dtantsur: IMHO, we should invert the depends-on
11:02:10 dtantsur hah, sorry for not figuring out the confusion earlier
11:02:48 bauzas dtantsur: why would you make the devstack change dependent on the nova change ?
11:02:52 dtantsur bauzas: yeah, good call probably. wanna me drop the depends-on from my patch?
11:03:10 bauzas if the nova change uses what's provided by the devstack one ?
11:03:10 dtantsur I wanted one of them to depend on the other, I don't care which exactly :)
11:03:29 bauzas dtantsur: yeah, please remove the depends-on on the devstack one
11:03:51 bauzas dtantsur: and then I'll update https://review.openstack.org/#/c/487954/ to include devstack
11:04:12 dtantsur bauzas: done
11:04:20 bauzas dtantsur: I have to apologize, I wasn't having a full view of the situation
11:04:26 vdrok dtantsur: so, for that code to be triggered, we have to have an active instance booted with old flavor, and afterwards being updated with resource class right?
11:04:35 bauzas dtantsur: okay, I'm on https://review.openstack.org/#/c/487954/
11:04:36 vdrok code in https://review.openstack.org/#/c/487954/14 I mean
11:04:36 dtantsur bauzas: no problem, thanks for helping us with this stuff anyway
11:05:08 bauzas vdrok: for the nova code to be triggered, you have to set resource classes for ironic nodes firsrt
11:05:29 bauzas vdrok: that would be done by devstack in the job we discuss
11:06:15 dtantsur vdrok: I think vdrok's point is that we still won't see the log message, because it needs the resource_class to not be present initially..
11:06:30 vdrok dtantsur: bauzas exactly
11:06:50 openstackgerrit Sylvain Bauza proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954
11:06:57 dtantsur so I wonder if the only option to test it is to actually get a devstack locally, and try it
11:07:02 bauzas dtantsur: done ^
11:07:31 bauzas dtantsur: just to make it clear, I'm just updating it but just for testing purposes
11:07:52 dtantsur vdrok: in any case, could you please review https://review.openstack.org/#/c/491777/ ? this is something we must get in today to not block nova further
11:08:01 bauzas dtantsur: once we're sure the flavor is correctly updated, I feel fine to just revert it to PS14 and +W it since melwitt already gave her +2
11:08:17 vdrok dtantsur: yeah that one looks fine to me
11:10:22 mriedem bauzas: you know you could have pushed a change on top that depended on https://review.openstack.org/#/c/491777/
11:10:40 bauzas mriedem: snap, my bad
11:10:42 mriedem now you have to run https://review.openstack.org/#/c/487954/ back through twice
11:10:47 mriedem i'm already tagging rc1
11:10:48 bauzas mriedem: yeah, good point
11:11:06 bauzas mriedem: I just updated the RC1 patch
11:11:17 bauzas mriedem: with the latest merge sha1
11:11:41 bauzas mriedem: I can revert back to PS14 so we won't need to run yet again jenkins
11:11:49 bauzas and I'll do what you say
11:11:53 vdrok dtantsur: I'll just test this locally now I think. and then we'll try to make the grenade do this resource class setting to see the whole process
11:11:55 bauzas mriedem: ack ?
11:12:39 mriedem bauzas: i think https://review.openstack.org/#/c/492788/ is ready to go
11:12:40 dtantsur vdrok: cool, thanks!
11:12:44 mriedem the ironic stuff is rc2
11:13:19 dtantsur that will require backporting to stable/pike, right?
11:13:23 dtantsur also morning mriedem
11:13:35 mriedem dtantsur: yes
11:13:42 bauzas mriedem: okay, I'm fine then
11:13:43 dtantsur ack
11:13:46 bauzas mriedem: removing my -1
11:13:47 bauzas smcginnis: ^
11:13:48 mriedem but that's just because we don't have milestone-proposed anymore
11:16:03 openstackgerrit Sylvain Bauza proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954
11:16:17 mriedem dtantsur: i was not sure what to make of http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621
11:16:27 mriedem dtantsur: when i dug through those logs,
11:16:37 mriedem the ironic driver wasn't populating inventory in placement,
11:16:44 mriedem which would have auto-created the custom resource class
11:17:01 mriedem i'm not sure if we have some chicken and egg issue
11:17:15 dtantsur mriedem: it seems to be that it tries to proceed, and actually fails on http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_588272
11:17:21 dtantsur I'm not sure it's expected or not
11:17:37 dtantsur if it is, then I'm pretty sure we have a chicked and egg situation
11:17:37 mriedem dtantsur: i think that's a side effect
11:18:04 mriedem there is a periodic task in the compute service that pulls inventory from ironic https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L775
11:18:13 mriedem ^ includes any custom resource class on the node
11:18:42 mriedem the resource tracker in the compute service calls that from here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L834
11:19:11 mriedem and if there is a custom resource class in that inventory, it would auto-create it in placement here https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L775
11:19:29 mriedem when i was looking at the logs on that failed job, i never saw _update_inventory get called
11:20:16 mriedem so with my limited understanding of how the ironic driver works, when does the node cache in the driver actually have something show up here? https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L735
11:20:27 vdrok dtantsur: locally, I see "The flavor extra_specs for Ironic instance 0b8460b1-22
11:20:27 vdrok 57-44dc-8b96-4c17182a9a64 have been updated for custom resource class 'baremetal'."

Earlier   Later