| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-18 | |||
| 20:00:01 | jaypipes | mriedem: done | |
| 20:02:27 | mriedem | thanks | |
| 20:05:03 | mriedem | edleafe: melwitt: i was thinking about something with the alternate hosts change during create server - https://review.openstack.org/#/c/511358/ - there is no functional test in that change that shows we're actually using the alternate hosts. we already have at least a couple functional tests which use 2 hosts and make the first fail to trigger a reschedule. but that worked before this because the compute and cell conductor c | |
| 20:05:03 | mriedem | it the scheduler. i wonder if there is an easy way to make the cell conductor fail to hit the scheduler | |
| 20:05:12 | mriedem | in our functional tests we're just running one conductor service | |
| 20:06:18 | windsn | Hi, Im having issues with trying to use nova with ironic.... I see this error in the nova-compute logs, https://thepasteb.in/p/pghQo0OnW5vcR, and when i create an ironic node, i dont see the nova-hypervisor-list getting populated with an entry for the ironic node... I'm not too sure what seems to be the issue | |
| 20:06:55 | mriedem | windsn: ocata, pike, master? | |
| 20:07:01 | windsn | Pike | |
| 20:07:36 | mriedem | windsn: have you run "nova-manage cell_v2 discover_hosts"? | |
| 20:08:09 | mriedem | edleafe: melwitt: an easy think we could do is just make a select_destinations fixture which blows up if it's called more than once... | |
| 20:08:50 | windsn | mriedem: No I haven't run that... I'm not sure what that does | |
| 20:08:51 | mriedem | otherwise i think we'd need to do something where we start a 2nd conductor service with modified transport_url/db connection, and modify the compute service to point at the cell transport_url also | |
| 20:09:17 | mriedem | windsn: https://docs.openstack.org/nova/latest/cli/nova-manage.html#nova-cells-v2 | |
| 20:09:45 | edleafe | mriedem: I did think about doing something like that, but got tangled up in the multiple cell/conductor creation | |
| 20:10:14 | mriedem | edleafe: by default the tests are running with 2 cells, cell0 and cell1 (unless the SimpleCellFixture or whatever it's called is used) | |
| 20:10:36 | mriedem | so the trick would be running the cell conductor service and the compute service configured for the cell1 transport_url i think | |
| 20:10:49 | mriedem | but we use a fake mq so i'm not sure it would actually help anything | |
| 20:10:59 | edleafe | mriedem: ah, then I was trying for a third cell to show that it didn't touch the other cell | |
| 20:11:15 | edleafe | I guess I don't think of cell0 as a "real" cell | |
| 20:11:42 | mriedem | RPCFixture i think | |
| 20:12:23 | edleafe | mriedem: ok, I'll start playing around with that in a bit | |
| 20:13:13 | mriedem | looks like dan left a breadcrumb https://review.openstack.org/#/c/396417/23/nova/tests/fixtures.py | |
| 20:13:36 | cdent | dansel | |
| 20:14:10 | jaypipes | gdi, these server group functional tests are inscrutable. :( | |
| 20:14:17 | jaypipes | make me want to stab something. | |
| 20:14:23 | mriedem | anyway, just thought about this before digging into the ironic multinode reschedule failures | |
| 20:16:54 | jaypipes | AAAARRRRGGGGGGGGGGGGGGHHHHHHHHHHH I frigggin hate the integrate test helpers. | |
| 20:17:19 | melwitt | jaypipes: one thing I learned about those tests is if you want to have scheduling consider > 1 compute host, you need a compute service and set_nodes per host | |
| 20:18:05 | jaypipes | melwitt: just doing a simple copy/paste of one of the "negative" server group func tests results in completely random NotFound errors from the faked-out image service... | |
| 20:18:13 | jaypipes | melwitt: makes no sense whatsoever. | |
| 20:18:25 | jaypipes | melwitt: and the tests are completely unintelligible. | |
| 20:18:28 | melwitt | hm, yeah I would not have expected that | |
| 20:18:45 | melwitt | that, I do know from having used them as examples for other tests | |
| 20:19:02 | melwitt | (the multiple compute host part I mean) | |
| 20:20:42 | openstackgerrit | Jay Pipes proposed openstack/nova master: WIP - Support aggregate affinity filters https://review.openstack.org/528804 | |
| 20:20:49 | jaypipes | jroll: they are stowed away in padded rooms. not to worry. | |
| 20:20:57 | jroll | excellent. | |
| 20:21:52 | jaypipes | melwitt: see the patch above... it will fail with this for no discernable reason whatsoever: http://paste.openstack.org/show/629242/ | |
| 20:22:01 | jaypipes | edleafe: very good question... | |
| 20:22:13 | jroll | ahem, projects | |
| 20:22:17 | jaypipes | lol | |
| 20:23:19 | jaypipes | jroll: well, disregarding my current functional test woes, your eyeballs on https://review.openstack.org/#/c/528804/1/nova/scheduler/filters/affinity_filter.py would be useful. | |
| 20:23:41 | jroll | jaypipes: sure thing, on my list | |
| 20:23:49 | jaypipes | ya, no rush.. | |
| 20:24:44 | windsn | mriedem: I ran the command ""nova-manage cell_v2 discover_hosts", but i still dont see the hypervisor-list getting populated with the ironic node | |
| 20:26:31 | windsn | mriedem: is the "nova-manage cell v2 discover_host" supposed to create a "host_mapping" entry in the nova db ? I don't see that happening | |
| 20:28:01 | mriedem | edleafe: ah i think i know why the ironic multinode job is failing to reschedule | |
| 20:28:02 | mriedem | "Re-scheduling is disabled" | |
| 20:28:20 | mriedem | if the retryfilter isn't running, then max_attempts = 0 and we wouldn't get alternates yeah? | |
| 20:29:20 | mriedem | jaypipes: likely using the latest microversion | |
| 20:29:26 | mriedem | so we don't proxy for images | |
| 20:29:36 | mriedem | you have to pass a specific image to _build_minimal_create_server_request i think | |
| 20:29:43 | mriedem | something like that | |
| 20:30:12 | mriedem | windsn: supposed to create a host_mapping entry in the nova_api db | |
| 20:30:42 | mriedem | jaypipes: yeah "GET /v2.1/6f70656e737461636b20342065766572/images/detail" status: 404 len: 78 microversion: 2.59 | |
| 20:31:07 | mriedem | that's capped at 2.35 | |
| 20:31:25 | jaypipes | mriedem: sorry, I'm not following you... | |
| 20:31:31 | melwitt | aha, yup. good thinkin mriedem | |
| 20:31:41 | mriedem | jaypipes: your test is using the latest microversion by default | |
| 20:31:51 | mriedem | or you've set self.microversion = 'latest' | |
| 20:31:59 | mriedem | the /images proxy API is capped at 2.35 | |
| 20:32:00 | jaypipes | mriedem: it's not *my* test :) it's a copy of one of the tests that is already in there... | |
| 20:32:04 | melwitt | as of that microversion, we no longer proxy image API requests to glance, so you're getting not found | |
| 20:32:12 | mriedem | anyway, you'll need to hard-code the image_uuid http://git.openstack.org/cgit/openstack/nova/tree/nova/tests/functional/integrated_helpers.py#n140 | |
| 20:32:19 | mriedem | to one of the images in our fake image service | |
| 20:32:30 | jaypipes | mriedem: ah... since the test I copied from sets microversion to 2.15, it didn't bomb it. | |
| 20:32:44 | edleafe | mriedem: also, alternates are hosts, not nodes, so unless there are multiple computes per cell for ironic, there won't be alternates | |
| 20:32:48 | mriedem | jaypipes: https://review.openstack.org/#/c/528804/1/nova/tests/functional/test_server_group.py@840 | |
| 20:32:54 | jaypipes | mriedem: that is SO clear :/ | |
| 20:32:54 | mriedem | you are setting the microversion to 2.59 | |
| 20:32:58 | melwitt | right, it's setting microversion = '2.59' that does it | |
| 20:33:15 | mriedem | edleafe: yeah i know, but in this case it's saying it's finding 12 hosts to start from | |
| 20:33:23 | mriedem | edleafe: http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-sch.txt.gz#_Dec_12_22_23_47_783670 | |
| 20:33:36 | jroll | "alternates are hosts, not nodes" O_O | |
| 20:33:41 | jaypipes | mriedem: right, what I was saying was the test case I copied that from sets microversion to 2.15 and it just so happened that 2.15 < 2.35 | |
| 20:34:21 | jroll | edleafe: that feels very wrong, is that code merged or is there still time to review it? :) | |
| 20:34:22 | mriedem | jroll: yeah it's not completely baked for ironic yet obviously | |
| 20:34:28 | jroll | right | |
| 20:34:30 | mriedem | jroll: the code to use it isn't merged yet | |
| 20:34:35 | mriedem | https://review.openstack.org/#/c/511358/40 | |
| 20:34:39 | jroll | awesome, ty | |
| 20:34:46 | mriedem | i have https://review.openstack.org/#/c/527289/ and am debugging it | |
| 20:34:47 | jaypipes | mriedem: so I have to figure out how to adapt the functional test base so that only the _create_server_group() uses the microversion at 2.59 and everything else uses 2.1... | |
| 20:34:50 | jaypipes | fun. | |
| 20:34:52 | edleafe | jroll: yeah, we discussed updating that for unique host/node, but the first round it's just host | |
| 20:35:20 | jroll | edleafe: gotcha, will take a look around | |
| 20:35:27 | edleafe | jroll: the code to create the alternates has merged. | |
| 20:40:17 | melwitt | jaypipes: it's ghetto, but you could just do what mriedem suggested and add something like this to _boot_a_server_to_group https://github.com/openstack/nova/blob/master/nova/tests/functional/test_servers.py#L1449 | |
| 20:40:54 | melwitt | I don't know who figured out that's always the image uuid of the first image but that's what it appears to me | |
| 20:41:20 | mriedem | jaypipes: you can use the self.api client directly | |
| 20:41:30 | mriedem | if you want to avoid the base class method to create the server group | |
| 20:44:01 | jaypipes | mriedem: I am looking at nova/tests/functional/api/client.py to try and understand how to pass a specific microversion... I don't see it. | |
| 20:44:11 | mriedem | oh right... | |
| 20:44:22 | mriedem | you can't, and the client specifically blows up if you try passing your own header | |
| 20:44:39 | mriedem | jaypipes: please place your nuts here < > so i can kick you square in them | |
| 20:44:40 | jaypipes | mriedem: looks like I will need to do somethin glike this: | |
| 20:44:59 | mriedem | self.api.microversion = '2.59' | |
| 20:45:05 | mriedem | self.api.post_server_group() | |
| 20:45:07 | jaypipes | self.api.microversion = '2.59'; do something...; self.api.microversion = orig | |
| 20:45:11 | mriedem | self.api.microversion = old_value | |