Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-08
07:01:47 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi(N-R-P): Add API to support compute node resource provider update and create https://review.openstack.org/521041
08:30:55 ivve hi guys, im having some of a strange issue that im trying to solve.. i have a few computes and trying to safetly disable and then migrate off all vms on a compute. and get this message from conductor: https://hastebin.com/hifusejuva.sql
08:31:42 ivve all computes have a full pass on virt-host-validate and im running virt_type = kvm in nova
08:31:51 ivve all are on baremetal
08:32:25 ivve also, livemigration of single vms works perfectly
08:33:42 ivve im guessing next step is conductor debug
08:36:21 ivve here is the debug. im working through it now
08:36:23 ivve https://hastebin.com/uyiwogodup.sql
08:41:42 fanzhang ivve looks like the hosts and nodes have been filtered by scheduler, and you actually get no valid hosts. I'm suggesting that you should pay attention to nova-scheduler log to find out what really happens while select_destinations
08:43:10 ivve fanzhang: aye .. ill check that again. it looked fine the first time around
08:45:28 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return all resources in provider_summaries https://review.openstack.org/558045
08:45:29 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Consider nested RPs in get_all_with_shared https://review.openstack.org/556450
08:45:30 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return resource providers without inventories https://review.openstack.org/559554
08:45:30 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480
08:45:30 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Support shared and nested allocation candidates https://review.openstack.org/556514
08:45:31 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Consult the dictionary for root provider ids https://review.openstack.org/559555
08:51:36 ivve fanzhang: it is ignoring host02 which is the host with nova service disabled. then host02 and host03 is up and should be able to answer the call
08:55:03 ivve https://hastebin.com/kakibikuna.scala
08:57:56 ivve is it just me or is the filter not working properly or is one host missing in the filter?
09:04:11 ivve meaning ComputeFilter
09:12:02 fanzhang since you have 3 instances to build in one request_spec ( correct me if I am wrong ), there should a loop, for each vm, nova-scheduler has to select, filter, weigh the destinations to get an available host, I mean the log should be much larger than this one. Take mine https://paste.ubuntu.com/p/JRjfxXJj3c/ for example. Oh, do you log debug info or not?
09:12:41 ivve debug is true
09:13:30 fanzhang yeah, just saw that
09:13:34 ivve the host that is disabled that im trying to migrate contains 1 instance
09:13:46 ivve ram/cpu/disk is not an issue
09:14:11 ivve this is a small test environment but 16cpu/48gb and 4.5tb ceph space is available
09:14:20 ivve instances are quite small 2cpu/4gbram i think
09:14:52 ivve i have verified this problem in two openstacks (they are both deployed with osa stable/pike)
09:15:34 ivve it filters host02 which is fine (since its disabled). but what happens to host01 and host03..
09:15:59 fanzhang or maybe you can try to get inventory and usage info of your resource provider, aka, compute node, host01 and host03, to see if there's any data errors. I am not sure it could help.
09:16:13 ivve none of the hosts are in any aggregate but these are the enabled filters:
09:16:18 ivve RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateCoreFilter,AggregateDiskFilter
09:16:37 ivve perhaps its the aggragatecorefilter? im checking
09:17:33 ivve perhaps this is the issue. none of the hosts are in aggregates
09:17:39 ivve ill try to move them to one
09:18:27 fanzhang oh right, that could be the issue.
09:20:44 ivve not much diffrent :(
09:20:51 ivve https://hastebin.com/fanamiyili.scala
09:21:00 ivve i put them in AZ and aggs
09:21:06 ivve still fails
09:21:07 ivve :/
09:23:17 ivve all are created from the same image, no affinity stuff going on, no server group.. agg and AZ is fixed and looks ok so not that
09:23:29 ivve ram/cpu filters out of question
09:24:17 ivve i have a different lab environment with 7 computes, it behaves the same
09:26:04 ivve all environments have ceph
09:26:08 fanzhang do you get any logs like "Starting with 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70" ? I think your log is not as detailed as mine above. And I have no more moves to help you out here, I guess. Sorry for that.
09:26:31 ivve thats the thing, nothing like that
09:27:14 ivve grepping for "get_filter"
09:27:18 ivve 0 hits :(
09:27:58 fanzhang how many controller nodes do you have?
09:28:00 ivve this is only the scheduler log
09:28:01 ivve 3
09:28:05 ivve im tailing all 3
09:28:13 fanzhang so check the scheduler log on the other two
09:30:01 ivve im only pasting the log that answers the call
09:30:06 ivve uhm, the node
09:36:48 ivve fanzhang: here is the full tail - https://hastebin.com/imugowosoy.scala
09:37:28 openstackgerrit XiaojueGuan proposed openstack/nova master: fix the lose of content in the document whose url is: https://docs.openstack.org/newton/config-reference/compute/schedulers.html https://review.openstack.org/559560
09:38:08 fanzhang ivve looks like you don't grep using the same req-id?
09:38:36 ivve this is the full command
09:38:49 ivve tail -f infra1-nova-scheduler-container-213eba01/nova-scheduler.log infra2-nova-scheduler-container-432537dc/nova-scheduler.log infra3-nova-scheduler-container-0801807a/nova-scheduler.log
09:39:11 ivve i wait for a while until a few "checks" has passed, then i engage a host migration
09:39:17 ivve and that is what comes out
09:40:09 fanzhang well, you can grep by req-id of migration requet, maybe it's much more clear
09:40:36 ivve i guess
09:40:46 ivve and then go for all logs, i suppose
09:42:38 fanzhang good luck, :) I have to go now.
09:43:17 ivve https://hastebin.com/ezitiperig.scala
09:46:27 fanzhang ivve oh hi, I quick checked the log before leaving, so you forced a host while migrating ?
09:46:51 ivve i disabled it
09:47:15 ivve and using the migrate host function to "evacuate" instances to other computes
09:48:35 ivve the horizon function known as "migrate host" -> Migrate all instances from a host with disabled nova-compute service
09:49:09 ivve instead of manually migrating every single instance or doing it by script
09:49:44 ivve i want the filters in scheduler to assign a proper host for all the instances
09:55:51 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi(N-R-P):Get vgpu info from `allocations` https://review.openstack.org/521717
09:59:16 openstackgerrit XiaojueGuan proposed openstack/nova master: fix the lose of content in the document whose url is: https://docs.openstack.org/newton/config-reference/compute/schedulers.html https://review.openstack.org/559560
10:53:20 openstackgerrit Tetsuro Nakamura proposed openstack/nova-specs master: Return all resources in provider_summaries spec https://review.openstack.org/559466
22:14:11 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608
22:14:40 openstackgerrit Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483
22:16:42 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.61 - List/Show all server migration types https://review.openstack.org/430839
#openstack-nova - 2018-04-09
01:05:37 openstackgerrit jichenjc proposed openstack/nova stable/queens: Avoid showing password in log https://review.openstack.org/559603
01:06:00 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add microversion to support extra_specs in flavor API. https://review.openstack.org/554159
01:37:46 openstackgerrit jichenjc proposed openstack/nova master: Avoid live migrate to same host https://review.openstack.org/542689
01:38:02 openstackgerrit jichenjc proposed openstack/nova master: Avoid raise InstanceNotRunning exception https://review.openstack.org/541152
02:10:02 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add host/hostId to instance action events API https://review.openstack.org/557145
02:44:18 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi(N-R-P):Get vgpu info from `allocations` https://review.openstack.org/521717
02:46:00 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add microversion to support extra_specs in flavor API. https://review.openstack.org/554159
02:55:04 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/api/openstack/compute/test_hosts.py https://review.openstack.org/559266
02:56:43 openstackgerrit jichenjc proposed openstack/nova master: make metadata doc up to date https://review.openstack.org/559613
04:24:32 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480
04:24:32 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Support shared and nested allocation candidates https://review.openstack.org/556514
04:24:33 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Consult the dictionary for root provider ids https://review.openstack.org/559555
04:24:33 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return resource providers without inventories https://review.openstack.org/559554
06:12:19 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (1) https://review.openstack.org/558704
06:22:58 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Improve getting nested providers with capacity https://review.openstack.org/559638
06:28:15 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (2) https://review.openstack.org/558993
06:28:30 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (3) https://review.openstack.org/559258
06:47:30 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in test_user_data.py https://review.openstack.org/559264
06:58:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/api/*/test_instance_actions.py https://review.openstack.org/559269
07:09:04 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix the request context in ServiceFixture https://review.openstack.org/558160

Earlier   Later