Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-27
22:17:52 superdan however, if you run that while things are going on, you could get retval=2 due to a transient luck-based event
22:18:12 superdan but doing the online migrations should fix both cases up for any non-booting instances
22:18:29 superdan so the answer to "what to do" is run the map command
22:19:13 superdan map_instances for 1 and map_cell_and_hosts for 2 (modulo transientness)
22:19:17 superdan clear as mud?
22:21:13 diana_clarke superdan: I going to re-read that 10 times in a row, and then hopefully it will be ;) much appreciated!
22:21:45 superdan let me say it again linearly now that I have it in my head...
22:22:06 superdan If you get retval=1, that means you need to run map_instances as some instances (which were created with a previous version) don't have instance mappings yet
22:22:48 superdan if you get retval=2, that may be because you're racing with instances that are booting, or it may mean that you are missing cell mappings, which means you need to run map_cell_and_hosts
22:23:24 superdan someone could modify that code to not return 2 if the instance has a building-related task state such that we know it's the former case for retval=2
22:24:49 diana_clarke nice, thx!
22:43:57 openstackgerrit Ed Leafe proposed openstack/nova: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472
22:44:29 figleaf johnthetubaguy: cdent: ^^ adds the workarounds for the problem we identified earlier.
22:48:26 mriedem_beach hola
22:48:55 mriedem donde esta el progresso?
22:49:12 superdan where is the progress?
22:49:21 mriedem si
22:49:25 mriedem grenade patch is +W
22:49:32 superdan is it?
22:49:38 mriedem i thought you were going to +W it
22:49:51 superdan I will when I see the nova one yeah
22:49:57 mriedem oh
22:50:04 superdan I think the nova patch was running against a stale version of the grenade one,
22:50:08 superdan so it's going again now
22:50:16 superdan john fixed up a bunch of the unit and pep8 issues
22:50:34 mriedem yeah i see the series must have been rebased
22:50:38 mriedem i'll review that all again
22:51:34 superdan still getting NoValidHost on 11 tests on regular grenade ...
22:51:56 superdan jeez
22:52:05 superdan compute is still not starting because of detecting the placement config
22:52:11 mriedem i thought things were peachy this morning?
22:52:18 mriedem or was that not valid b/c of bad deps?
22:52:48 superdan yeah, I dunno now, I thought we were past that
22:53:36 mriedem maybe changing the grenade patch messed something up?
22:53:55 superdan PS3 worked. but that was the last time
22:54:11 openstackgerrit Pushkar Umaranikar proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634
22:54:21 superdan but I see no reason why, there's been almost no change since
22:56:53 superdan yeah no placement config in nova.conf
22:56:55 superdan grr
22:58:00 mriedem hmm,
22:58:13 mriedem we're now deploying placement before nova in grenade right?
22:58:18 superdan yeah
22:58:24 mriedem which should configure nova.conf on the primary node,
22:58:33 mriedem which would actually be newton at that point
22:58:36 mriedem since it's pre-upgrade
22:58:44 mriedem 55 is updating a newton compute i mean
22:58:57 superdan but we still call the configure bit
22:59:03 mriedem sure,
22:59:13 mriedem and that shouldn't get wiped out or anything by the 60_nova part
22:59:19 openstackgerrit Pushkar Umaranikar proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634
23:00:46 mriedem huh, so now both of the nova changes depend on the grenade change
23:00:50 mriedem doesn't hurt anything
23:01:02 superdan I did that because it ran against a stale grenade patch for some reason
23:01:06 superdan so when I rebased I just added it on
23:01:11 superdan didn't know why
23:01:32 superdan ahhh
23:01:47 superdan I wonder if we still need to do configure_placement_nova_compute in the 60_nova bit
23:02:02 superdan not sure why it wouldn't survive, but that's the part that gets us placemetn config
23:02:35 superdan also, what about this? http://logs.openstack.org/06/425806/6/check/gate-grenade-dsvm-neutron-ubuntu-xenial/fb281f1/logs/grenade.sh.txt.gz#_2017-01-27_20_02_44_142
23:02:38 mriedem https://review.openstack.org/#/c/424730/15/projects/55_placement/from-newton/upgrade-placement@17
23:02:47 mriedem ^ should configure nova-compute on the primary node for placement
23:02:57 superdan I know
23:03:02 superdan but
23:03:09 superdan after the run it's not set
23:03:18 superdan oh wait
23:03:26 superdan it doesn't have $NOVA_CONF in the run
23:04:16 mriedem because we didn't source the nova stuff yet?
23:04:22 superdan yeah
23:04:25 mriedem teehee
23:04:30 mriedem i believe i called this ~6 hours ago
23:04:35 superdan yes you did
23:05:18 superdan it's definitely weird doing this out of body but still part of nova
23:05:53 mriedem superdan: so do we no longer need the thing you added where it saves off the number of computes before and checks for that after?
23:06:09 superdan mriedem: that is being challenged, so I split it out
23:06:49 superdan with this set we're introducing a very specific race because the scheduler checks the min vers at startup
23:06:50 superdan which is why I think we need it
23:06:51 superdan but we haven't hit it yet, so ...
23:07:49 superdan mriedem: this is the standalone patch: https://review.openstack.org/#/c/426310/
23:08:09 superdan which I see I have tabs in.. gdi
23:17:56 mriedem ok the 55_placement thing looks good now
23:19:18 mriedem now we play the waiting game
23:21:29 superdan cool
23:21:34 superdan personally,
23:21:38 superdan I hate the effing waiting game
23:21:45 superdan I've been playing it all week and I'm done with it
23:40:59 openstackgerrit Pushkar Umaranikar proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634
23:42:19 openstackgerrit Pushkar Umaranikar proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634
23:48:16 mriedem diana_clarke: just went through https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:man - some comments inline,
23:48:21 mriedem i think there are only a few that need some tweaks
23:48:44 mriedem like discover_hosts should come after create_cell because you can't map the hosts to a cell w/o a cell
23:50:45 openstackgerrit Matt Riedemann proposed openstack/nova: DNM: hack ironic with resource providers https://review.openstack.org/426296
23:52:53 diana_clarke mriedem: Yes, thanks a ton! I'll circle back and add more detail etc monday at the latest.
23:53:32 mriedem superdan: passed http://logs.openstack.org/61/417961/40/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/5fb6f74/
23:53:58 superdan like, yay
23:54:21 mriedem http://logs.openstack.org/61/417961/40/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/5fb6f74/logs/new/screen-n-sch.txt.gz#_2017-01-27_23_44_07_736
23:54:30 mriedem 2017-01-27 23:44:07.736 12381 DEBUG nova.scheduler.filter_scheduler [req-f76d4b01-c248-4de4-8675-a4cd3cb6d4cb tempest-AttachInterfacesTestJSON-881960388 tempest-AttachInterfacesTestJSON-881960388] Skipping call to placement, as upgrade in progress. _get_all_host_states /opt/stack/new/nova/nova/scheduler/filter_scheduler.py:179
23:54:37 superdan woot
23:55:45 mriedem wonder what the whiny ass grenade + live migration job is complaining about
23:56:44 mriedem heh wtf http://logs.openstack.org/61/417961/40/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/8c363cd/logs/devstack-gate-post_test_hook.txt.gz
23:57:02 mriedem tdurakov: ^

Earlier   Later