| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-11 | |||
| 21:52:14 | edmondsw | I think I can get the data you asked for | |
| 21:52:15 | mriedem | http://www.wikicu.com/ROLM_Phone | |
| 21:52:24 | mriedem | edmondsw: are you by chance forcing the host/node during server create | |
| 21:52:33 | mriedem | using --availability-zone nova:host:node? | |
| 21:52:36 | mriedem | something like that? | |
| 21:52:51 | edmondsw | not that I know of, but I'm not sure | |
| 21:53:02 | edmondsw | this is in the PowerVM CI environment | |
| 21:53:05 | mriedem | well, are you issuing a nova boot? | |
| 21:53:10 | mriedem | oh... | |
| 21:53:12 | edmondsw | so it's all automatated (hit this while redeploying it) | |
| 21:53:25 | edmondsw | so I'm not sure what it's doing without digging into the automation | |
| 21:53:37 | edmondsw | this is the undercloud | |
| 21:54:01 | mriedem | ok i guess check to see if it's forcing a host/node | |
| 21:54:07 | edmondsw | will do | |
| 21:54:21 | mriedem | or if CONF.scheduler.max_attempts = 1 | |
| 21:55:47 | edmondsw | max_attempts is the default (3) | |
| 21:55:56 | mriedem | ok | |
| 21:59:12 | edmondsw | mriedem I think these instances are being created by nodepool, if that tells you anything | |
| 22:00:31 | mriedem | i'm going to see if i can recreate with a functional test quick | |
| 22:00:35 | edmondsw | I doubt we're specifying a specific host/node but I'm looking | |
| 22:00:36 | mriedem | but no nodepool doesn't tell me anything | |
| 22:00:50 | openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Create specs directory for Stein https://review.openstack.org/573602 | |
| 22:01:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (5) https://review.openstack.org/570842 | |
| 22:01:10 | edmondsw | and whether we do or not, I don't know whether nodepool would | |
| 22:01:36 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (6) https://review.openstack.org/571330 | |
| 22:02:06 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992 | |
| 22:02:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993 | |
| 22:06:44 | edmondsw | mriedem I think this is an example request body: body: {"server": {"name": "PowerVM_CI-PowerVM_DevStacked-440", "imageRef": "0dd943fd-ae60-4cdf-85fb-709d00f87db1", "flavorRef": "55b0569d-4f51-47fd-a326-8b333fedcce3", "max_count": 1, "min_count": 1, "networks": [{"uuid": "a4a8157c-da2f-462f-aa73-5c97fc7e17f1"}], "metadata": {"nodepool": "{\"provider_name\": \"PowerVM_DevStacked\", \"node_id\": 440, \"image_name\": \" | |
| 22:06:44 | edmondsw | PowerVM_Ubuntu_Base\"}", "groups": "[\"PowerVM_DevStacked\", \"PowerVM_Ubuntu_Base\"]"}}} {{(pid=31995) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:604}} | |
| 22:07:32 | edmondsw | no availability zone specified | |
| 22:11:28 | mriedem | i don't know what \"node_id\": 440 is | |
| 22:13:54 | edmondsw | mriedem I think that's just something nodepool wants to remember, not important to nova | |
| 22:14:15 | edmondsw | mriedem the Exception block you pasted in comment 1 is not entered | |
| 22:14:26 | edmondsw | the error isn't until after that | |
| 22:18:40 | mriedem | ok log local_filter_props before the other instance of calling _destroy_build_request | |
| 22:21:48 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104 | |
| 22:22:05 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106 | |
| 22:22:20 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110 | |
| 22:23:31 | mriedem | edmondsw: what is the devstack env like? a single host but with multiple nova-compute services running the powervm driver? | |
| 22:23:37 | mriedem | so a 1:M host:node setup? | |
| 22:23:53 | edmondsw | local_filter_props isn't set until a few lines after the error | |
| 22:24:24 | edmondsw | the environment is a controller that runs most things (api, single conductor, etc.) and then multiple compute nodes | |
| 22:24:58 | edmondsw | the multiple compute nodes are all on actual physical nodes | |
| 22:25:08 | edmondsw | since this is the undercloud | |
| 22:25:34 | mriedem | oh shite i see, | |
| 22:25:34 | mriedem | get_host_availability_zone | |
| 22:25:41 | mriedem | that's an up-call to get aggregates for the host | |
| 22:25:43 | mriedem | to find the az | |
| 22:27:45 | mriedem | so we reschedule, pop another host off the selection list, | |
| 22:28:02 | mriedem | and then call host | |
| 22:28:04 | mriedem | oops | |
| 22:28:08 | mriedem | instance.availability_zone = ( | |
| 22:28:08 | mriedem | availability_zones.get_host_availability_zone(context, | |
| 22:28:08 | mriedem | host.service_host)) | |
| 22:28:17 | mriedem | which is an up-call to get aggregates in the api db | |
| 22:28:23 | mriedem | dansmith: ^ add it to the list :) | |
| 22:28:59 | mriedem | hmm, i wouldn't think it's possible to reschedule out of an az once you're initially scheduled into one | |
| 22:29:49 | melwitt | you can if you never requested a specific AZ and went to the default AZ. you won't necessarily stay in the default AZ during a move | |
| 22:30:12 | mriedem | i have a patch to test that.. | |
| 22:30:22 | mriedem | https://review.openstack.org/#/c/567701/ | |
| 22:30:26 | melwitt | so while a 'nova show' will show you're in the default AZ, your requested AZ is actually "None" aka I don't care what AZ I'm in | |
| 22:30:33 | melwitt | (this came up recently on our bug call) | |
| 22:31:51 | mriedem | https://bugs.launchpad.net/nova/+bug/1771860 | |
| 22:31:52 | openstack | Launchpad bug 1771860 in OpenStack Compute (nova) "instance.availability_zone reports wrong az after live migration" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 22:32:01 | mriedem | so if you don't specify an az during server create, you can freely move across azs, | |
| 22:32:12 | mriedem | but if you do specify an az during server create, you must be restricted to that az during moves | |
| 22:32:26 | melwitt | right | |
| 22:32:33 | edmondsw | that's what I'd expect | |
| 22:32:33 | mriedem | for live migration if you don't specify an az on create and are live migrated to another one, we aren't updating the instance az | |
| 22:32:36 | mriedem | which is that bug fix | |
| 22:33:08 | melwitt | k, that makes sense | |
| 22:36:45 | mriedem | edmondsw: ok updated comments in your bug | |
| 22:36:50 | edmondsw | mriedem tx | |
| 22:37:15 | mriedem | hmm, | |
| 22:37:34 | mriedem | so we could put the az on the Selection object for a given host while we're in the controller and pass that down to the cell, | |
| 22:37:53 | mriedem | but there would be a window of time where the host could be moved into / out of an aggregate / AZ during the reschedule loop | |
| 22:37:59 | mriedem | not likely, and kind of tough luck if that happens... | |
| 22:38:29 | mriedem | this might also help us resolve up-call #5 at https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls | |
| 22:38:37 | mriedem | when [cinder]/cross_az_attach=False | |
| 22:38:45 | mriedem | because the compute could get the host az off the Selection object | |
| 22:41:15 | mriedem | we really need a way to do split mq fixturization in our functional testes | |
| 22:41:17 | mriedem | *tests | |
| 22:41:42 | edmondsw | why does the AZ have to be set if the user didn't specify one? I assume it's to fill that on a later GET, but couldn't it just be populated dynamically there? | |
| 22:41:59 | mriedem | https://review.openstack.org/#/c/446053/ | |
| 22:43:07 | edmondsw | so that's what caused it | |
| 22:43:11 | mriedem | yes, | |
| 22:43:13 | mriedem | i believe the reason is, | |
| 22:43:29 | mriedem | set the az on the instance at the top where we can reach aggregates in the api db, | |
| 22:43:32 | mriedem | so we don't need to do that in the computes | |
| 22:43:36 | mriedem | where we can't reach the api db | |
| 22:43:45 | mriedem | but dansmith would have to confirm that | |
| 22:44:08 | mriedem | "which isn't as useful to the things that need to consider what AZ the instance is actually in (without doing all the aggregate math to determine it on the fly)." | |
| 22:44:13 | mriedem | that reads "up call" to me ^ | |
| 22:44:17 | edmondsw | but what needs to consider the AZ if the user didn't specify? | |
| 22:44:48 | edmondsw | e.g. we just said migrate wouldn't care about the AZ in that case | |
| 22:45:13 | mriedem | not sure off the top of my head honestly | |
| 22:46:10 | mriedem | maybe [cinder]/cross_az_attach=False ? | |
| 22:47:13 | mriedem | but as far as i know that is still broken | |
| 22:49:05 | mriedem | ugh, so we have a cache in nova.availability_zones, | |
| 22:49:15 | mriedem | but we don't populate it when calling get_host_availability_zone at the top | |