| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-11 | |||
| 21:53:25 | edmondsw | so I'm not sure what it's doing without digging into the automation | |
| 21:53:37 | edmondsw | this is the undercloud | |
| 21:54:01 | mriedem | ok i guess check to see if it's forcing a host/node | |
| 21:54:07 | edmondsw | will do | |
| 21:54:21 | mriedem | or if CONF.scheduler.max_attempts = 1 | |
| 21:55:47 | edmondsw | max_attempts is the default (3) | |
| 21:55:56 | mriedem | ok | |
| 21:59:12 | edmondsw | mriedem I think these instances are being created by nodepool, if that tells you anything | |
| 22:00:31 | mriedem | i'm going to see if i can recreate with a functional test quick | |
| 22:00:35 | edmondsw | I doubt we're specifying a specific host/node but I'm looking | |
| 22:00:36 | mriedem | but no nodepool doesn't tell me anything | |
| 22:00:50 | openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Create specs directory for Stein https://review.openstack.org/573602 | |
| 22:01:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (5) https://review.openstack.org/570842 | |
| 22:01:10 | edmondsw | and whether we do or not, I don't know whether nodepool would | |
| 22:01:36 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (6) https://review.openstack.org/571330 | |
| 22:02:06 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992 | |
| 22:02:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993 | |
| 22:06:44 | edmondsw | PowerVM_Ubuntu_Base\"}", "groups": "[\"PowerVM_DevStacked\", \"PowerVM_Ubuntu_Base\"]"}}} {{(pid=31995) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:604}} | |
| 22:06:44 | edmondsw | mriedem I think this is an example request body: body: {"server": {"name": "PowerVM_CI-PowerVM_DevStacked-440", "imageRef": "0dd943fd-ae60-4cdf-85fb-709d00f87db1", "flavorRef": "55b0569d-4f51-47fd-a326-8b333fedcce3", "max_count": 1, "min_count": 1, "networks": [{"uuid": "a4a8157c-da2f-462f-aa73-5c97fc7e17f1"}], "metadata": {"nodepool": "{\"provider_name\": \"PowerVM_DevStacked\", \"node_id\": 440, \"image_name\": \" | |
| 22:07:32 | edmondsw | no availability zone specified | |
| 22:11:28 | mriedem | i don't know what \"node_id\": 440 is | |
| 22:13:54 | edmondsw | mriedem I think that's just something nodepool wants to remember, not important to nova | |
| 22:14:15 | edmondsw | mriedem the Exception block you pasted in comment 1 is not entered | |
| 22:14:26 | edmondsw | the error isn't until after that | |
| 22:18:40 | mriedem | ok log local_filter_props before the other instance of calling _destroy_build_request | |
| 22:21:48 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104 | |
| 22:22:05 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106 | |
| 22:22:20 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110 | |
| 22:23:31 | mriedem | edmondsw: what is the devstack env like? a single host but with multiple nova-compute services running the powervm driver? | |
| 22:23:37 | mriedem | so a 1:M host:node setup? | |
| 22:23:53 | edmondsw | local_filter_props isn't set until a few lines after the error | |
| 22:24:24 | edmondsw | the environment is a controller that runs most things (api, single conductor, etc.) and then multiple compute nodes | |
| 22:24:58 | edmondsw | the multiple compute nodes are all on actual physical nodes | |
| 22:25:08 | edmondsw | since this is the undercloud | |
| 22:25:34 | mriedem | get_host_availability_zone | |
| 22:25:34 | mriedem | oh shite i see, | |
| 22:25:41 | mriedem | that's an up-call to get aggregates for the host | |
| 22:25:43 | mriedem | to find the az | |
| 22:27:45 | mriedem | so we reschedule, pop another host off the selection list, | |
| 22:28:02 | mriedem | and then call host | |
| 22:28:04 | mriedem | oops | |
| 22:28:08 | mriedem | host.service_host)) | |
| 22:28:08 | mriedem | availability_zones.get_host_availability_zone(context, | |
| 22:28:08 | mriedem | instance.availability_zone = ( | |
| 22:28:17 | mriedem | which is an up-call to get aggregates in the api db | |
| 22:28:23 | mriedem | dansmith: ^ add it to the list :) | |
| 22:28:59 | mriedem | hmm, i wouldn't think it's possible to reschedule out of an az once you're initially scheduled into one | |
| 22:29:49 | melwitt | you can if you never requested a specific AZ and went to the default AZ. you won't necessarily stay in the default AZ during a move | |
| 22:30:12 | mriedem | i have a patch to test that.. | |
| 22:30:22 | mriedem | https://review.openstack.org/#/c/567701/ | |
| 22:30:26 | melwitt | so while a 'nova show' will show you're in the default AZ, your requested AZ is actually "None" aka I don't care what AZ I'm in | |
| 22:30:33 | melwitt | (this came up recently on our bug call) | |
| 22:31:51 | mriedem | https://bugs.launchpad.net/nova/+bug/1771860 | |
| 22:31:52 | openstack | Launchpad bug 1771860 in OpenStack Compute (nova) "instance.availability_zone reports wrong az after live migration" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 22:32:01 | mriedem | so if you don't specify an az during server create, you can freely move across azs, | |
| 22:32:12 | mriedem | but if you do specify an az during server create, you must be restricted to that az during moves | |
| 22:32:26 | melwitt | right | |
| 22:32:33 | mriedem | for live migration if you don't specify an az on create and are live migrated to another one, we aren't updating the instance az | |
| 22:32:33 | edmondsw | that's what I'd expect | |
| 22:32:36 | mriedem | which is that bug fix | |
| 22:33:08 | melwitt | k, that makes sense | |
| 22:36:45 | mriedem | edmondsw: ok updated comments in your bug | |
| 22:36:50 | edmondsw | mriedem tx | |
| 22:37:15 | mriedem | hmm, | |
| 22:37:34 | mriedem | so we could put the az on the Selection object for a given host while we're in the controller and pass that down to the cell, | |
| 22:37:53 | mriedem | but there would be a window of time where the host could be moved into / out of an aggregate / AZ during the reschedule loop | |
| 22:37:59 | mriedem | not likely, and kind of tough luck if that happens... | |
| 22:38:29 | mriedem | this might also help us resolve up-call #5 at https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls | |
| 22:38:37 | mriedem | when [cinder]/cross_az_attach=False | |
| 22:38:45 | mriedem | because the compute could get the host az off the Selection object | |
| 22:41:15 | mriedem | we really need a way to do split mq fixturization in our functional testes | |
| 22:41:17 | mriedem | *tests | |
| 22:41:42 | edmondsw | why does the AZ have to be set if the user didn't specify one? I assume it's to fill that on a later GET, but couldn't it just be populated dynamically there? | |
| 22:41:59 | mriedem | https://review.openstack.org/#/c/446053/ | |
| 22:43:07 | edmondsw | so that's what caused it | |
| 22:43:11 | mriedem | yes, | |
| 22:43:13 | mriedem | i believe the reason is, | |
| 22:43:29 | mriedem | set the az on the instance at the top where we can reach aggregates in the api db, | |
| 22:43:32 | mriedem | so we don't need to do that in the computes | |
| 22:43:36 | mriedem | where we can't reach the api db | |
| 22:43:45 | mriedem | but dansmith would have to confirm that | |
| 22:44:08 | mriedem | "which isn't as useful to the things that need to consider what AZ the instance is actually in (without doing all the aggregate math to determine it on the fly)." | |
| 22:44:13 | mriedem | that reads "up call" to me ^ | |
| 22:44:17 | edmondsw | but what needs to consider the AZ if the user didn't specify? | |
| 22:44:48 | edmondsw | e.g. we just said migrate wouldn't care about the AZ in that case | |
| 22:45:13 | mriedem | not sure off the top of my head honestly | |
| 22:46:10 | mriedem | maybe [cinder]/cross_az_attach=False ? | |
| 22:47:13 | mriedem | but as far as i know that is still broken | |
| 22:49:05 | mriedem | ugh, so we have a cache in nova.availability_zones, | |
| 22:49:15 | mriedem | but we don't populate it when calling get_host_availability_zone at the top | |
| 22:49:46 | mriedem | so when we get to the compute and check the instance.az (set at the top) against the host az from the cache, the latter is None and we say, 'the cache is invalid, so fetch the host az again' | |
| 22:49:47 | edmondsw | mriedem, actually git blame shows this was introduced with https://github.com/openstack/nova/commit/03b4c67b22f49d325386bc3ebd2ade79b44fa699 | |
| 22:49:50 | mriedem | which is the up-call that fails | |
| 22:50:10 | mriedem | edmondsw: that's the same change | |
| 22:50:52 | mriedem | oh but the cache on the compute won't have the az set in it anyway | |
| 22:50:55 | edmondsw | oh, dang it it is... I saw "Updated 9 weeks ago" in that review and thought it was new, whereas the commit was old, and assumed they were different | |
| 22:51:00 | mriedem | b/c we get the host az at in a different process | |
| 22:51:14 | mriedem | unless you have a distributed cache... | |
| 23:14:36 | mriedem | edmondsw: fyi, your bug is also a problem for resize reschedules | |
| 23:23:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add another up-call to the cells v2 caveats list https://review.openstack.org/581910 | |