Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-18
20:19:02 melwitt (the multiple compute host part I mean)
20:20:42 openstackgerrit Jay Pipes proposed openstack/nova master: WIP - Support aggregate affinity filters https://review.openstack.org/528804
20:20:49 jaypipes jroll: they are stowed away in padded rooms. not to worry.
20:20:57 jroll excellent.
20:21:52 jaypipes melwitt: see the patch above... it will fail with this for no discernable reason whatsoever: http://paste.openstack.org/show/629242/
20:22:01 jaypipes edleafe: very good question...
20:22:13 jroll ahem, projects
20:22:17 jaypipes lol
20:23:19 jaypipes jroll: well, disregarding my current functional test woes, your eyeballs on https://review.openstack.org/#/c/528804/1/nova/scheduler/filters/affinity_filter.py would be useful.
20:23:41 jroll jaypipes: sure thing, on my list
20:23:49 jaypipes ya, no rush..
20:24:44 windsn mriedem: I ran the command ""nova-manage cell_v2 discover_hosts", but i still dont see the hypervisor-list getting populated with the ironic node
20:26:31 windsn mriedem: is the "nova-manage cell v2 discover_host" supposed to create a "host_mapping" entry in the nova db ? I don't see that happening
20:28:01 mriedem edleafe: ah i think i know why the ironic multinode job is failing to reschedule
20:28:02 mriedem "Re-scheduling is disabled"
20:28:20 mriedem if the retryfilter isn't running, then max_attempts = 0 and we wouldn't get alternates yeah?
20:29:20 mriedem jaypipes: likely using the latest microversion
20:29:26 mriedem so we don't proxy for images
20:29:36 mriedem you have to pass a specific image to _build_minimal_create_server_request i think
20:29:43 mriedem something like that
20:30:12 mriedem windsn: supposed to create a host_mapping entry in the nova_api db
20:30:42 mriedem jaypipes: yeah "GET /v2.1/6f70656e737461636b20342065766572/images/detail" status: 404 len: 78 microversion: 2.59
20:31:07 mriedem that's capped at 2.35
20:31:25 jaypipes mriedem: sorry, I'm not following you...
20:31:31 melwitt aha, yup. good thinkin mriedem
20:31:41 mriedem jaypipes: your test is using the latest microversion by default
20:31:51 mriedem or you've set self.microversion = 'latest'
20:31:59 mriedem the /images proxy API is capped at 2.35
20:32:00 jaypipes mriedem: it's not *my* test :) it's a copy of one of the tests that is already in there...
20:32:04 melwitt as of that microversion, we no longer proxy image API requests to glance, so you're getting not found
20:32:12 mriedem anyway, you'll need to hard-code the image_uuid http://git.openstack.org/cgit/openstack/nova/tree/nova/tests/functional/integrated_helpers.py#n140
20:32:19 mriedem to one of the images in our fake image service
20:32:30 jaypipes mriedem: ah... since the test I copied from sets microversion to 2.15, it didn't bomb it.
20:32:44 edleafe mriedem: also, alternates are hosts, not nodes, so unless there are multiple computes per cell for ironic, there won't be alternates
20:32:48 mriedem jaypipes: https://review.openstack.org/#/c/528804/1/nova/tests/functional/test_server_group.py@840
20:32:54 mriedem you are setting the microversion to 2.59
20:32:54 jaypipes mriedem: that is SO clear :/
20:32:58 melwitt right, it's setting microversion = '2.59' that does it
20:33:15 mriedem edleafe: yeah i know, but in this case it's saying it's finding 12 hosts to start from
20:33:23 mriedem edleafe: http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-sch.txt.gz#_Dec_12_22_23_47_783670
20:33:36 jroll "alternates are hosts, not nodes" O_O
20:33:41 jaypipes mriedem: right, what I was saying was the test case I copied that from sets microversion to 2.15 and it just so happened that 2.15 < 2.35
20:34:21 jroll edleafe: that feels very wrong, is that code merged or is there still time to review it? :)
20:34:22 mriedem jroll: yeah it's not completely baked for ironic yet obviously
20:34:28 jroll right
20:34:30 mriedem jroll: the code to use it isn't merged yet
20:34:35 mriedem https://review.openstack.org/#/c/511358/40
20:34:39 jroll awesome, ty
20:34:46 mriedem i have https://review.openstack.org/#/c/527289/ and am debugging it
20:34:47 jaypipes mriedem: so I have to figure out how to adapt the functional test base so that only the _create_server_group() uses the microversion at 2.59 and everything else uses 2.1...
20:34:50 jaypipes fun.
20:34:52 edleafe jroll: yeah, we discussed updating that for unique host/node, but the first round it's just host
20:35:20 jroll edleafe: gotcha, will take a look around
20:35:27 edleafe jroll: the code to create the alternates has merged.
20:40:17 melwitt jaypipes: it's ghetto, but you could just do what mriedem suggested and add something like this to _boot_a_server_to_group https://github.com/openstack/nova/blob/master/nova/tests/functional/test_servers.py#L1449
20:40:54 melwitt I don't know who figured out that's always the image uuid of the first image but that's what it appears to me
20:41:20 mriedem jaypipes: you can use the self.api client directly
20:41:30 mriedem if you want to avoid the base class method to create the server group
20:44:01 jaypipes mriedem: I am looking at nova/tests/functional/api/client.py to try and understand how to pass a specific microversion... I don't see it.
20:44:11 mriedem oh right...
20:44:22 mriedem you can't, and the client specifically blows up if you try passing your own header
20:44:39 mriedem jaypipes: please place your nuts here < > so i can kick you square in them
20:44:40 jaypipes mriedem: looks like I will need to do somethin glike this:
20:44:59 mriedem self.api.microversion = '2.59'
20:45:05 mriedem self.api.post_server_group()
20:45:07 jaypipes self.api.microversion = '2.59'; do something...; self.api.microversion = orig
20:45:11 mriedem self.api.microversion = old_value
20:45:12 mriedem yeah
20:45:13 jaypipes heh, jinux
20:45:20 jaypipes or even jinx.
20:45:33 mriedem jinux is your personal linux distro?
20:45:35 jaypipes I guess jinux is linux jinx.
20:45:36 mriedem jay's linux
20:45:40 jaypipes heh
20:45:57 mriedem this is the thing that blows up if you try to pass a header override https://github.com/openstack/nova/blob/master/nova/tests/functional/api/client.py#L179
20:46:23 mriedem however, i think that was added when this was all written by sdague and was very specific to how the tests worked with scenarios
20:46:50 jaypipes I'll just add the microversion override as above with the reset right after
20:46:54 mriedem i'd be cool with removing that so you can run tests at a default minimum microversion and then make certain calls with a higher microversion, or vice-versa
20:48:37 mriedem btw, this is a useful code comment http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/filters/retry_filter.py#n37
20:49:09 jaypipes mriedem: about equally as useful as the majority of code comments in the scheduler filters.
20:49:43 melwitt lol
20:50:05 jaypipes also, god help whomever needs to change "tenant" -> "project" when we no longer support keystone 2.0.'
20:50:57 mriedem edleafe: ok this is really confusing http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-sch.txt.gz#_Dec_12_22_23_47_783964
20:51:04 mriedem we have 12 candidate hosts,
20:51:15 mriedem initial select_destinations call for this create request
20:51:25 mriedem and we log "Re-scheduling is disabled" 12 times, once per host, because of the RetryFilter,
20:51:43 mriedem and i think it's all because we don't populate the 'retry' field in the RequestSpec before calling select_destinations the first time
20:52:04 mriedem during create we call the scheduler here http://git.openstack.org/cgit/openstack/nova/tree/nova/conductor/manager.py#n1016
20:52:15 mriedem and we don't populate retry until http://git.openstack.org/cgit/openstack/nova/tree/nova/conductor/manager.py#n1108
20:52:23 mriedem i'm not sure if that was ever by design
20:52:49 mriedem looking at the old build_instances method, we'd populate retry before calling the scheduler http://git.openstack.org/cgit/openstack/nova/tree/nova/conductor/manager.py#n543
20:53:19 mriedem again, probably not our issue here since max_attempts=3 by default, and i verified that in this job, but it sure is confusing in the logs
20:57:52 openstack Launchpad bug 1738876 in OpenStack Compute (nova) "Scheduler logs "Re-scheduling is disabled" on initial server create scheduling even though max_attempts>0" [Medium,Triaged]
20:57:52 mriedem https://bugs.launchpad.net/nova/+bug/1738876
21:09:32 edleafe mriedem: with alternates, isn't the RetryFilter supposed to go away?
21:09:58 mriedem i suppose it would over time
21:10:07 edleafe since you won't try the same host more than once?
21:10:32 edleafe IOW, it'll always be a noop
21:17:18 mriedem edleafe: this shows that we're getting duplicate alternates http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/22826a2/logs/screen-n-super-cond.txt.gz#_Dec_15_23_17_04_949754
21:17:29 mriedem Selected host: ubuntu-xenial-inap-mtl01-0001472200; Alternates: [u'ubuntu-xenial-inap-mtl01-0001472200', u'ubuntu-xenial-inap-mtl01-0001472200']

Earlier   Later