Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-06
22:45:01 mriedem dansmith: i know you want to know this, but i figured out the bug
22:49:53 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - Add a microversion for consumer generation https://review.openstack.org/558958
22:55:46 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
22:55:46 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
23:35:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771
23:37:32 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771
#openstack-nova - 2018-04-07
00:40:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't persist RequestSpec.retry https://review.openstack.org/559447
01:29:29 openstackgerrit Merged openstack/nova master: DRY up test_rollback_live_migration_set_migration_status https://review.openstack.org/555489
02:34:55 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
02:34:55 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
02:52:14 openstackgerrit Merged openstack/nova master: Avoid unnecessary port update during live migration https://review.openstack.org/556332
05:27:12 openstackgerrit Tetsuro Nakamura proposed openstack/nova-specs master: Return all resources in provider_summaries spec https://review.openstack.org/559466
05:58:12 openstackgerrit Tetsuro Nakamura proposed openstack/nova-specs master: Enable NUMA Features for Libvirt/QEMU Driver https://review.openstack.org/533077
06:20:37 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return all resources in provider_summaries https://review.openstack.org/558045
06:51:57 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/548772
09:19:28 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Support shared and nested allocation candidates https://review.openstack.org/556514
09:19:28 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Consider nested RPs in get_all_with_shared https://review.openstack.org/556450
09:19:29 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480
12:43:44 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Add request_id to instance action notifications https://review.openstack.org/553288
13:38:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't persist RequestSpec.retry https://review.openstack.org/559447
13:38:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771
19:33:09 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
19:33:10 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
22:12:46 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
23:37:33 openstackgerrit Eric Fried proposed openstack/nova master: add lower-constraints job https://review.openstack.org/555961
23:44:44 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
#openstack-nova - 2018-04-08
00:52:07 openstackgerrit Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757
00:53:18 openstackgerrit Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757
01:10:23 openstackgerrit Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757
01:18:30 openstackgerrit Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757
04:45:34 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
04:45:35 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
06:42:23 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi(N-R-P): Add API to support compute node resource provider update and create https://review.openstack.org/521041
07:01:47 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi(N-R-P): Add API to support compute node resource provider update and create https://review.openstack.org/521041
08:30:55 ivve hi guys, im having some of a strange issue that im trying to solve.. i have a few computes and trying to safetly disable and then migrate off all vms on a compute. and get this message from conductor: https://hastebin.com/hifusejuva.sql
08:31:42 ivve all computes have a full pass on virt-host-validate and im running virt_type = kvm in nova
08:31:51 ivve all are on baremetal
08:32:25 ivve also, livemigration of single vms works perfectly
08:33:42 ivve im guessing next step is conductor debug
08:36:21 ivve here is the debug. im working through it now
08:36:23 ivve https://hastebin.com/uyiwogodup.sql
08:41:42 fanzhang ivve looks like the hosts and nodes have been filtered by scheduler, and you actually get no valid hosts. I'm suggesting that you should pay attention to nova-scheduler log to find out what really happens while select_destinations
08:43:10 ivve fanzhang: aye .. ill check that again. it looked fine the first time around
08:45:28 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return all resources in provider_summaries https://review.openstack.org/558045
08:45:29 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Consider nested RPs in get_all_with_shared https://review.openstack.org/556450
08:45:30 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return resource providers without inventories https://review.openstack.org/559554
08:45:30 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480
08:45:30 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Support shared and nested allocation candidates https://review.openstack.org/556514
08:45:31 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Consult the dictionary for root provider ids https://review.openstack.org/559555
08:51:36 ivve fanzhang: it is ignoring host02 which is the host with nova service disabled. then host02 and host03 is up and should be able to answer the call
08:55:03 ivve https://hastebin.com/kakibikuna.scala
08:57:56 ivve is it just me or is the filter not working properly or is one host missing in the filter?
09:04:11 ivve meaning ComputeFilter
09:12:02 fanzhang since you have 3 instances to build in one request_spec ( correct me if I am wrong ), there should a loop, for each vm, nova-scheduler has to select, filter, weigh the destinations to get an available host, I mean the log should be much larger than this one. Take mine https://paste.ubuntu.com/p/JRjfxXJj3c/ for example. Oh, do you log debug info or not?
09:12:41 ivve debug is true
09:13:30 fanzhang yeah, just saw that
09:13:34 ivve the host that is disabled that im trying to migrate contains 1 instance
09:13:46 ivve ram/cpu/disk is not an issue
09:14:11 ivve this is a small test environment but 16cpu/48gb and 4.5tb ceph space is available
09:14:20 ivve instances are quite small 2cpu/4gbram i think
09:14:52 ivve i have verified this problem in two openstacks (they are both deployed with osa stable/pike)
09:15:34 ivve it filters host02 which is fine (since its disabled). but what happens to host01 and host03..
09:15:59 fanzhang or maybe you can try to get inventory and usage info of your resource provider, aka, compute node, host01 and host03, to see if there's any data errors. I am not sure it could help.
09:16:13 ivve none of the hosts are in any aggregate but these are the enabled filters:
09:16:18 ivve RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateCoreFilter,AggregateDiskFilter
09:16:37 ivve perhaps its the aggragatecorefilter? im checking
09:17:33 ivve perhaps this is the issue. none of the hosts are in aggregates
09:17:39 ivve ill try to move them to one
09:18:27 fanzhang oh right, that could be the issue.
09:20:44 ivve not much diffrent :(
09:20:51 ivve https://hastebin.com/fanamiyili.scala
09:21:00 ivve i put them in AZ and aggs
09:21:06 ivve still fails
09:21:07 ivve :/
09:23:17 ivve all are created from the same image, no affinity stuff going on, no server group.. agg and AZ is fixed and looks ok so not that
09:23:29 ivve ram/cpu filters out of question
09:24:17 ivve i have a different lab environment with 7 computes, it behaves the same
09:26:04 ivve all environments have ceph
09:26:08 fanzhang do you get any logs like "Starting with 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70" ? I think your log is not as detailed as mine above. And I have no more moves to help you out here, I guess. Sorry for that.
09:26:31 ivve thats the thing, nothing like that
09:27:14 ivve grepping for "get_filter"
09:27:18 ivve 0 hits :(
09:27:58 fanzhang how many controller nodes do you have?
09:28:00 ivve this is only the scheduler log
09:28:01 ivve 3
09:28:05 ivve im tailing all 3
09:28:13 fanzhang so check the scheduler log on the other two
09:30:01 ivve im only pasting the log that answers the call
09:30:06 ivve uhm, the node
09:36:48 ivve fanzhang: here is the full tail - https://hastebin.com/imugowosoy.scala
09:37:28 openstackgerrit XiaojueGuan proposed openstack/nova master: fix the lose of content in the document whose url is: https://docs.openstack.org/newton/config-reference/compute/schedulers.html https://review.openstack.org/559560
09:38:08 fanzhang ivve looks like you don't grep using the same req-id?
09:38:36 ivve this is the full command
09:38:49 ivve tail -f infra1-nova-scheduler-container-213eba01/nova-scheduler.log infra2-nova-scheduler-container-432537dc/nova-scheduler.log infra3-nova-scheduler-container-0801807a/nova-scheduler.log
09:39:11 ivve i wait for a while until a few "checks" has passed, then i engage a host migration
09:39:17 ivve and that is what comes out
09:40:09 fanzhang well, you can grep by req-id of migration requet, maybe it's much more clear
09:40:36 ivve i guess
09:40:46 ivve and then go for all logs, i suppose
09:42:38 fanzhang good luck, :) I have to go now.

Earlier   Later