Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-19
22:54:56 mriedem so you don't need help debugging placement failures now?
22:55:20 tonyb mriedem: No I don't think so for this thing but in gernal it'd be good to know how to do it
22:55:22 mriedem n-sch logs will tell you for a given request which hosts are being filtered
22:55:45 mriedem i don't think we log the allocation_candidates response since it could be huge
22:57:28 tonyb mriedem: Okay what I'm seeing 'Got no allocation candidates from the Placement API' so I think that measn I the filters don't run which is why I'm not seeing the output I'm expecting in select_destindations?
22:58:13 openstackgerrit Merged openstack/python-novaclient master: Fix inconsistency https://review.openstack.org/572770
22:59:03 mriedem correct
22:59:17 mriedem tonyb: what's the scenario? normal server create? or a rebuild, or force_hosts?
22:59:36 mriedem could just be there are no hosts with available capacity for the flavor being used
23:01:52 tonyb mriedem: Lots of scenarios: tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest is the one I picked at random
23:02:39 tonyb mriedem: I'm off reading the libvirt code for baselineCPU as that's certainly part of the problem
23:02:39 mriedem tonyb: so some tripleo ci job shit the bed or what?
23:03:10 tonyb mriedem: oooo No I'm looking at the pkvm thing from thismornings meeting
23:03:31 mriedem oh, don't worry about tempest in that case,
23:03:39 mriedem the n-cpu logs just show that the host baselineCPU stuff is f'ed
23:04:01 tonyb mriedem: Yeah I finally got there, and hence readin the libvirt code ;P
23:04:50 mriedem ok cool. point taken though - if people though debugging scheduler failures were hard before placement, it will be worse, because the filtering for some things is done in sql now
23:04:55 mriedem *thought
23:05:21 tonyb mriedem: Yup. I agree.
23:05:58 mriedem efried: haven't we mused about some kind of "tell me why this request doesn't fit" type tool?
23:06:24 efried mriedem: We have. It would be insanely complicated to write, which is why we haven't gotten beyond musing.
23:06:45 jlk :9
23:06:55 mriedem jlk: i thought you left
23:06:57 jlk I remember spending a lot of time debugging selection, when introducing more complex rules
23:07:02 tonyb mriedem: if a live system can't use psc-placement and some gray-matter to work (well guess) why?
23:07:08 jlk mriedem: I did, but I'm still dealing with libvirt and KVM and stuff.
23:07:10 efried mriedem: Insanely complicated in the sense that we would have to rewrite pretty much every line of logic in the GET /a_c flow, splitting monolithic SQL statements into smaller components, etc.
23:07:19 jlk I'm ... "borrowing" some things from Nova
23:07:30 cfriesen_ could you split it up into the subqueries at least, to see which ones are ruling out significant numbers of hosts?
23:07:37 mriedem jlk: uh, for k8s?
23:07:40 jlk no
23:07:50 jlk an in-house bare metal cloud I'm teaching how to do VMs
23:08:16 jlk k8s will run on a VM on this, instead of on bare metal, but we have a bunch of other bare metal consumers that would be better serviced by VMs, even if the VM takes up the whole machine.
23:08:23 mriedem i know you can enable debugging at the sql level, but that would suck
23:09:03 cfriesen_ efried: I mean when the query comes back with no hosts you could re-run the various sub-queries
23:09:46 openstackgerrit Eric Fried proposed openstack/nova master: Check provider generation and retry on conflict https://review.openstack.org/556669
23:11:13 efried cfriesen_: The sub-queries that would matter - at least for resource availability - are pretty well mashed together in monolithic sql statements at the moment.
23:12:06 cfriesen_ if we ever put hugepages and cpu pinning and pci devices and provider networks in placement I think we'd probably need some sort of useful logging on failure to find a candidate.
23:13:11 cfriesen_ when you get into stuff like "you wanted a pci device attached to this network, but there's not enough 1G hugepages on any numa node associated with those PCI devices"
23:13:18 efried Some of the logic splitup for nrp and granular will actually help with that, much as it got hated on during review.
23:14:00 efried but for any serious debugging, other than sql dumps as mriedem says, it's gonna need a crap ton of refactoring.
23:14:20 efried I'm Audi 2000. See y'all tomorrow.
23:19:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Add queued_for_delete field to InstanceMapping object https://review.openstack.org/566795
23:19:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Online migration tool for populating queued-for-delete https://review.openstack.org/582536
23:19:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Update queued-for-delete from the ComputeAPI during deletion/restoration https://review.openstack.org/566813
23:19:12 openstackgerrit Matt Riedemann proposed openstack/nova master: [POC] Graceful handling of nova-list when a cell is down https://review.openstack.org/567785
23:23:24 openstackgerrit Matt Riedemann proposed openstack/nova master: Online migration tool for populating queued-for-delete https://review.openstack.org/582536
23:23:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Update queued-for-delete from the ComputeAPI during deletion/restoration https://review.openstack.org/566813
23:23:26 openstackgerrit Matt Riedemann proposed openstack/nova master: [POC] Graceful handling of nova-list when a cell is down https://review.openstack.org/567785
23:23:45 mriedem melwitt: +2s on those first 3 ^
23:24:54 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110
23:25:09 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (6) https://review.openstack.org/574113
23:43:11 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (7) https://review.openstack.org/574974
23:43:14 oomichi takashin: hi, can you check https://review.openstack.org/#/c/578040/ again ?
23:43:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311
23:44:07 takashin oomichi: okay. I will review it again.
23:44:22 oomichi thanks :)
23:44:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Use consumer generation in _heal_allocations_for_instance https://review.openstack.org/577905
#openstack-nova - 2018-07-20
00:46:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.openstack.org/575581
00:46:21 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.openstack.org/576017
00:46:37 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.openstack.org/576018
01:05:49 oomichi takashin: Can you check my comment on https://review.openstack.org/#/c/571992 ?
01:06:07 takashin oomichi: okay
01:07:23 takashin oomichi: Thank you for your review. I will fix them.
01:11:52 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Add regression test for bug 1781880 https://review.openstack.org/583927
01:11:53 openstack bug 1781880 in OpenStack Compute (nova) "nova service-list for new compute service causes TypeError with servicegroup_driver mc " [Medium,In progress] https://launchpad.net/bugs/1781880 - Assigned to Zhenyu Zheng (zhengzhenyu)
01:12:30 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Fix nova service-list for new compute service causes TypeError https://review.openstack.org/583936
01:12:56 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Fix nova service-list for new compute service causes TypeError https://review.openstack.org/583936
01:21:04 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Remove stevedore extensions server_create method https://review.openstack.org/583882
01:27:56 gmann mriedem_away: we have now tempest-scenario-all job which will cover all scenario tests including slow. can i update this one to use that job - https://review.openstack.org/#/c/567697 ?
01:34:29 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Use Tempest scenario job to run all scenario tests including slow https://review.openstack.org/567697
01:51:08 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.openstack.org/576019
01:51:28 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.openstack.org/576020
01:51:46 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.openstack.org/576027
02:08:18 openstackgerrit melanie witt proposed openstack/nova master: DNM: Don't overwrite greenthread-local context in target_cell https://review.openstack.org/582332
02:09:16 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge extended availability zone controller https://review.openstack.org/584197
02:13:31 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge extended availability zone controller https://review.openstack.org/502859
02:16:54 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge extended availability zone response into server controller https://review.openstack.org/502859
02:19:00 openstackgerrit fupingxie proposed openstack/nova master: Delete allocations when it is re-allocated https://review.openstack.org/582899
02:42:00 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Report 0 root_gb in resource tracker if instance is bfv. https://review.openstack.org/584204
02:55:43 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.openstack.org/576031
02:56:04 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299
02:56:34 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.openstack.org/576344
03:35:57 openstackgerrit fupingxie proposed openstack/nova master: Trivial: Fix spelling error https://review.openstack.org/584212
04:14:48 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Consider forbidden traits in early exit of _get_by_one_request https://review.openstack.org/584218
04:55:44 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge config drive extension response into server controller https://review.openstack.org/584223
04:59:24 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.openstack.org/576673
04:59:34 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.openstack.org/576676
05:00:11 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.openstack.org/576689
05:10:58 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.openstack.org/576709
05:32:18 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Merge config drive extension response into server controller https://review.openstack.org/584223
05:48:51 openstackgerrit fupingxie proposed openstack/nova master: Delete allocations when it is re-allocated https://review.openstack.org/582899
06:44:29 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Update installation guide to be more clear about cellsv2 https://review.openstack.org/584244
07:21:15 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Consider forbidden traits in early exit of _get_by_one_request https://review.openstack.org/584218
08:19:21 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Report 0 root_gb in resource tracker if instance is bfv. https://review.openstack.org/584204
08:31:59 openstackgerrit zhufl proposed openstack/nova master: Fix missing print format error https://review.openstack.org/584277
08:39:17 openstackgerrit Lei Zhang proposed openstack/nova master: Add method to get cpu traits https://review.openstack.org/560317
08:49:40 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Add regression test for bug 1781880 https://review.openstack.org/583927
08:49:41 openstack bug 1781880 in OpenStack Compute (nova) "nova service-list for new compute service causes TypeError with servicegroup_driver mc " [Medium,In progress] https://launchpad.net/bugs/1781880 - Assigned to Zhenyu Zheng (zhengzhenyu)
09:20:20 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Update launch-from-volume doc to latest version. https://review.openstack.org/572241

Earlier   Later