Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-18
21:47:54 mriedem http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cpu.txt.gz#_Dec_12_22_23_48_399346
21:47:58 mriedem ComputeHostNotFound: Compute host 5 could not be found.
21:49:05 edleafe any idea why that host couldn't be found?
21:49:22 mriedem nope
21:49:34 mriedem we might not be using the proper cell context
21:52:02 mriedem so we hit the first selected node, which fails because of compute node not found,
21:52:09 mriedem and we cast up to the cell conductor to reschedule and hit this http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cond-cell1.txt.gz#_Dec_12_22_23_48_751837
21:55:30 mriedem AHA
21:55:38 mriedem i see the problemo
21:55:39 mriedem oh ed
21:55:43 mriedem ed ed ed
21:56:24 edleafe you were getting me confused. How could the compute call back when the compute wasn't found?
21:58:52 mriedem comments line
21:58:53 mriedem *inline
21:59:01 mriedem you were passing host_list using the wrong arg
21:59:04 edleafe looking
21:59:10 mriedem that's why it was None when it came back up and we tried to hit the scheduler again
21:59:27 mriedem use kwargs ftw
22:00:10 mriedem so i think ironic reschedules should be totally fine
22:00:11 mriedem jroll: ^
22:00:36 mriedem code as is is busted so it wasn't trying alternates
22:00:42 mriedem once we fix that, things should be good
22:01:33 jroll nice
22:04:25 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
22:06:40 edleafe mriedem: ^^ updated
22:07:28 mriedem edleafe: ok, that should fix the actual reschedule issue, but still missing comments from PS41 so re-applied my -1
22:07:31 mriedem if you're about end of day,
22:07:38 mriedem then ill kick off the ironic job again
22:07:44 mriedem and we can check results in the morn
22:08:04 edleafe I just wanted to get you that fix first
22:08:09 edleafe working on the other comments
22:08:11 mriedem ok
22:11:59 edleafe mriedem: re: your question on https://review.openstack.org/#/c/511358/41/nova/conductor/manager.py@578
22:12:18 edleafe Should that failure be handled at the compute level, or up here in the conductor?
22:13:57 mriedem edleafe: conductor
22:14:07 mriedem scheduler_utils.populate_retry is what raises MaxRetriesExceeded today
22:14:39 mriedem so, maybe populate_retry just handles my issue on L578
22:14:43 mriedem since it comes before we'd get that far
22:16:16 edleafe The question is at this point, if we have an empty host_list, should I raise a MaxRetriesExceeded exception?
22:16:53 mriedem i don't think we'll get that far
22:16:58 mriedem populate_retry should fail before that
22:17:09 mriedem if you're asking about just being explicit...
22:17:31 mriedem it's definitely confusing that populate_retry is also the "check if you've gone over the retry limit thing"
22:17:46 edleafe that confused me too
22:17:54 mriedem i'd be fine with an explicit check
22:18:13 edleafe so if we reach 578 and have an empty host_list, something is seriously wrong
22:18:27 mriedem yes
22:19:13 edleafe what would be the best exception to raise there? Besides a YoureHosedException?
22:20:01 openstackgerrit Sylvain Bauza proposed openstack/nova master: libvirt: Provide VGPU inventory for a single GPU type https://review.openstack.org/514763
22:20:02 openstackgerrit Sylvain Bauza proposed openstack/nova master: WIP: libvirt: create vGPU for instance https://review.openstack.org/528832
22:20:16 mriedem edleafe: MaxRetriesExceedd?
22:20:21 mriedem *exceeded
22:20:24 mriedem that, or NoValidHost
22:20:57 mriedem honestly L578 shouldn't be there
22:20:58 edleafe Well, we found valid hosts, but just couldn't build on 'em
22:21:09 mriedem the loop should assume we have a host per instance
22:21:18 mriedem if we don't, something higher up in the method should have caught it
22:21:25 edleafe ok, I'll remove it
22:21:48 mriedem this is what should catch that case https://review.openstack.org/#/c/511358/42/nova/conductor/manager.py@545
22:22:17 mriedem it's definitely confusing to have the retry information spread and maintained across filter properties, request spec and the host list stuff, but that's what we have
22:22:48 mriedem if for some reason populate_retry didn't catch it and host_lists was empty, then we have to handle it explicitly
22:23:01 mriedem and, host_lists wouldn't be empty, because it'd be [[]] right?
22:23:11 mriedem a list with a single entry, which is an empty list
22:23:14 edleafe yeah, that's the plan
22:23:33 mriedem empty lists are fun
22:23:33 mriedem ...
22:23:33 mriedem ... print 'empty lists are fun'
22:23:33 mriedem >>> if [[]]:
22:23:42 edleafe ok, what about this scenario:
22:23:49 edleafe max_attempts=5
22:23:58 edleafe only 2 alternates were found
22:24:19 edleafe the populate_retry thing won't catch that, will it?
22:24:30 mriedem no it wouldn't,
22:24:41 mriedem in that case, with the old flow, we'd call the scheduler and get NoValidHost
22:25:01 mriedem ok so given that, we can't rely on populate_retry
22:25:14 edleafe So I'll leave the check for an empty list, and raise MaxRetriesExceeded
22:25:35 mriedem well, L578 isn't the place to do that
22:25:41 mriedem we need it to fail in the try/except above
22:25:51 mriedem so it sets the vm_state to ERROR
22:26:09 mriedem so how about an else to this? "if not is_retry:"
22:26:48 mriedem if not is_retry: select_dests elif not host_list[0]: raise MaxRetriesExceeded
22:26:49 edleafe ok, I'll move the check there
22:27:49 mriedem note that in that case, we shouldn' try to destroy the build request in the except block
22:28:14 edleafe why not?
22:28:20 mriedem because if we're in a cell conductor we don't have access to the api db
22:28:32 mriedem and rather than BuildRequestNotFound, we'll get CantStartEngineError
22:28:41 openstack Launchpad bug 1733933 in OpenStack Compute (nova) "nova-conductor is masking error when rescheduling" [Low,Triaged]
22:28:41 mriedem ala https://bugs.launchpad.net/nova/+bug/1733933
22:29:00 edleafe ah. I don't know how you keep all that straight
22:29:27 mriedem well, that's also what failed here http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cond-cell1.txt.gz#_Dec_12_22_23_48_751837
22:29:41 mriedem er this i mean http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cond-cell1.txt.gz#_Dec_12_22_23_48_940236
22:29:52 mriedem File "/opt/stack/new/nova/nova/conductor/manager.py", line 478, in _destroy_build_request
22:30:26 mriedem i'll leave a note in the code so we don't forget
22:34:07 mriedem edleafe: ok commented. after thinking through that, it might be best to leave L578
22:34:17 mriedem because host_lists is going to be per-instance right?
22:34:31 mriedem although, if we're rescheduling, it will be a single entry yeah?
22:35:23 edleafe yeah, always single on reschedule
22:36:34 edleafe also, the populate_retry method pops the exc_reason from the filter_properties, so it isn't available later in build_instances
22:36:56 edleafe why not check above as you mentioned before?
22:37:04 mriedem we could - i left yet another comment

Earlier   Later