Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-19
22:15:00 openstackgerrit do3meli proposed openstack/nova master: docs: add nova host-evacuate command to evacuate documentation https://review.openstack.org/578040
22:15:54 mriedem tonyb: well, it was originally failing because the CI is configured with cpu_mode='none' and so i believe the model put into the xml was ppc64le, so that was changed to mode=host-model and model=power8 (per someone from libvirt/qemu that knows about this),
22:16:02 mriedem but now it fails on something else, which cfriesen_ said might be:
22:16:03 mriedem (3:45:04 PM) cfriesen_: https://github.com/libvirt/libvirt/blob/master/src/conf/cpu_conf.c#L317
22:16:03 mriedem (3:44:06 PM) cfriesen_: mriedem: looks like its if you try to parse XML with a <host><cpu> section that then tries to specify a "mode". That's only allowed for XML describing guest cpus.
22:16:41 mriedem tonyb: i was suggesting the powerkvm ci could just set cpu_mode=host-model and cpu_model=power8, but that's basically what the code is now doing as a workaround
22:26:40 cfriesen_ mriedem: if you want to specify cpu_model=power8, wouldn't you want a cpu_mode of custom?
22:28:17 mriedem if a tree falls in the woods
22:28:55 mriedem cfriesen_: yeah that makes more sense
22:29:14 cfriesen_ mriedem: hmm...https://bugzilla.redhat.com/show_bug.cgi?id=1237025 seems to think mode of "host-model" and "model" of "power8" is valid. weird.
22:29:15 openstack bugzilla.redhat.com bug 1237025 in libvirt "Guest can not start with different combinations of <cpu> mode and <model>" [Medium,Closed: notabug] - Assigned to abologna
22:29:33 cfriesen_ wonder if this is a bizarre powerpc-ism
22:30:12 mriedem jlk: said s390x causes all the problems,
22:30:15 mriedem but power is right up there
22:32:29 cfriesen_ mriedem: https://libvirt.org/formatdomain.html#elementsCPU under the host-model section documents PowerPC weirdness.
22:33:09 mriedem "Specifying CPU model is not supported either"
22:34:01 mriedem Since 1.2.11 PowerISA allows processors to run VMs in binary compatibility mode supporting an older version of ISA. Libvirt on PowerPC architecture uses the host-model to signify a guest mode CPU running in binary compatibility mode
22:34:10 cfriesen_ yeah, that's the interesting bit
22:34:24 mriedem ii libvirt-bin 4.0.0-1ubuntu8.3~cloud0
22:34:25 jlk of course Power would override that to means omething else
22:36:08 cfriesen_ I don't get why they wouldn't just use a mode of "custom" with a model of "power8" (or whatever) instead.
22:36:39 mriedem could try that
22:36:48 mriedem kind of throwing things at the wall at this point until something works
22:43:47 openstackgerrit Eric Fried proposed openstack/nova master: Check provider generation and retry on conflict https://review.openstack.org/556669
22:52:41 tonyb mriedem: Yeah setting power8 is wrong long term but shoudl be fine for now (but I wouldn't merge it that way)
22:53:36 tonyb mriedem: How do I debug placement failures? or at least see what allocation_candidates returned
22:54:34 tonyb Oh nm /me finds the 'renew until 27/07/18 08:05:33
22:54:40 tonyb ' in n-cpu
22:54:56 mriedem so you don't need help debugging placement failures now?
22:55:20 tonyb mriedem: No I don't think so for this thing but in gernal it'd be good to know how to do it
22:55:22 mriedem n-sch logs will tell you for a given request which hosts are being filtered
22:55:45 mriedem i don't think we log the allocation_candidates response since it could be huge
22:57:28 tonyb mriedem: Okay what I'm seeing 'Got no allocation candidates from the Placement API' so I think that measn I the filters don't run which is why I'm not seeing the output I'm expecting in select_destindations?
22:58:13 openstackgerrit Merged openstack/python-novaclient master: Fix inconsistency https://review.openstack.org/572770
22:59:03 mriedem correct
22:59:17 mriedem tonyb: what's the scenario? normal server create? or a rebuild, or force_hosts?
22:59:36 mriedem could just be there are no hosts with available capacity for the flavor being used
23:01:52 tonyb mriedem: Lots of scenarios: tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest is the one I picked at random
23:02:39 mriedem tonyb: so some tripleo ci job shit the bed or what?
23:02:39 tonyb mriedem: I'm off reading the libvirt code for baselineCPU as that's certainly part of the problem
23:03:10 tonyb mriedem: oooo No I'm looking at the pkvm thing from thismornings meeting
23:03:31 mriedem oh, don't worry about tempest in that case,
23:03:39 mriedem the n-cpu logs just show that the host baselineCPU stuff is f'ed
23:04:01 tonyb mriedem: Yeah I finally got there, and hence readin the libvirt code ;P
23:04:50 mriedem ok cool. point taken though - if people though debugging scheduler failures were hard before placement, it will be worse, because the filtering for some things is done in sql now
23:04:55 mriedem *thought
23:05:21 tonyb mriedem: Yup. I agree.
23:05:58 mriedem efried: haven't we mused about some kind of "tell me why this request doesn't fit" type tool?
23:06:24 efried mriedem: We have. It would be insanely complicated to write, which is why we haven't gotten beyond musing.
23:06:45 jlk :9
23:06:55 mriedem jlk: i thought you left
23:06:57 jlk I remember spending a lot of time debugging selection, when introducing more complex rules
23:07:02 tonyb mriedem: if a live system can't use psc-placement and some gray-matter to work (well guess) why?
23:07:08 jlk mriedem: I did, but I'm still dealing with libvirt and KVM and stuff.
23:07:10 efried mriedem: Insanely complicated in the sense that we would have to rewrite pretty much every line of logic in the GET /a_c flow, splitting monolithic SQL statements into smaller components, etc.
23:07:19 jlk I'm ... "borrowing" some things from Nova
23:07:30 cfriesen_ could you split it up into the subqueries at least, to see which ones are ruling out significant numbers of hosts?
23:07:37 mriedem jlk: uh, for k8s?
23:07:40 jlk no
23:07:50 jlk an in-house bare metal cloud I'm teaching how to do VMs
23:08:16 jlk k8s will run on a VM on this, instead of on bare metal, but we have a bunch of other bare metal consumers that would be better serviced by VMs, even if the VM takes up the whole machine.
23:08:23 mriedem i know you can enable debugging at the sql level, but that would suck
23:09:03 cfriesen_ efried: I mean when the query comes back with no hosts you could re-run the various sub-queries
23:09:46 openstackgerrit Eric Fried proposed openstack/nova master: Check provider generation and retry on conflict https://review.openstack.org/556669
23:11:13 efried cfriesen_: The sub-queries that would matter - at least for resource availability - are pretty well mashed together in monolithic sql statements at the moment.
23:12:06 cfriesen_ if we ever put hugepages and cpu pinning and pci devices and provider networks in placement I think we'd probably need some sort of useful logging on failure to find a candidate.
23:13:11 cfriesen_ when you get into stuff like "you wanted a pci device attached to this network, but there's not enough 1G hugepages on any numa node associated with those PCI devices"
23:13:18 efried Some of the logic splitup for nrp and granular will actually help with that, much as it got hated on during review.
23:14:00 efried but for any serious debugging, other than sql dumps as mriedem says, it's gonna need a crap ton of refactoring.
23:14:20 efried I'm Audi 2000. See y'all tomorrow.
23:19:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Add queued_for_delete field to InstanceMapping object https://review.openstack.org/566795
23:19:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Online migration tool for populating queued-for-delete https://review.openstack.org/582536
23:19:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Update queued-for-delete from the ComputeAPI during deletion/restoration https://review.openstack.org/566813
23:19:12 openstackgerrit Matt Riedemann proposed openstack/nova master: [POC] Graceful handling of nova-list when a cell is down https://review.openstack.org/567785
23:23:24 openstackgerrit Matt Riedemann proposed openstack/nova master: Online migration tool for populating queued-for-delete https://review.openstack.org/582536
23:23:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Update queued-for-delete from the ComputeAPI during deletion/restoration https://review.openstack.org/566813
23:23:26 openstackgerrit Matt Riedemann proposed openstack/nova master: [POC] Graceful handling of nova-list when a cell is down https://review.openstack.org/567785
23:23:45 mriedem melwitt: +2s on those first 3 ^
23:24:54 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110
23:25:09 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (6) https://review.openstack.org/574113
23:43:11 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (7) https://review.openstack.org/574974
23:43:14 oomichi takashin: hi, can you check https://review.openstack.org/#/c/578040/ again ?
23:43:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311
23:44:07 takashin oomichi: okay. I will review it again.
23:44:22 oomichi thanks :)
23:44:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Use consumer generation in _heal_allocations_for_instance https://review.openstack.org/577905
#openstack-nova - 2018-07-20
00:46:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.openstack.org/575581
00:46:21 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.openstack.org/576017
00:46:37 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.openstack.org/576018
01:05:49 oomichi takashin: Can you check my comment on https://review.openstack.org/#/c/571992 ?
01:06:07 takashin oomichi: okay
01:07:23 takashin oomichi: Thank you for your review. I will fix them.
01:11:52 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Add regression test for bug 1781880 https://review.openstack.org/583927
01:11:53 openstack bug 1781880 in OpenStack Compute (nova) "nova service-list for new compute service causes TypeError with servicegroup_driver mc " [Medium,In progress] https://launchpad.net/bugs/1781880 - Assigned to Zhenyu Zheng (zhengzhenyu)
01:12:30 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Fix nova service-list for new compute service causes TypeError https://review.openstack.org/583936
01:12:56 openstackgerrit Zhenyu Zheng proposed openstack/nova master: Fix nova service-list for new compute service causes TypeError https://review.openstack.org/583936
01:21:04 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Remove stevedore extensions server_create method https://review.openstack.org/583882
01:27:56 gmann mriedem_away: we have now tempest-scenario-all job which will cover all scenario tests including slow. can i update this one to use that job - https://review.openstack.org/#/c/567697 ?
01:34:29 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Use Tempest scenario job to run all scenario tests including slow https://review.openstack.org/567697
01:51:08 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.openstack.org/576019

Earlier   Later