Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-19
18:58:32 mriedem it's like gd time travel
18:58:53 mriedem efried: do you want to continue talking about this or want me to review https://review.openstack.org/#/c/556669/ ?
18:58:58 mriedem because i need to get in the zone
18:59:44 efried mriedem: zone away. I'm going to write a patch to clean this shit up, based on now being able to assume we're talking to a queens minimum.
19:08:19 jaypipes mriedem: cool, will look shortly.
19:54:17 openstackgerrit Chris Dent proposed openstack/nova master: WIP POC: Use os-resource-classes in placement https://review.openstack.org/584084
19:54:18 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Move resource_class_cache into placement hierarchy https://review.openstack.org/584085
19:54:19 openstackgerrit Chris Dent proposed openstack/nova master: [placement] ensure_rc_cache only at start of process https://review.openstack.org/584086
20:20:38 openstackgerrit Merged openstack/nova master: Implement migrate_instance_start method for neutron https://review.openstack.org/556334
20:24:57 mriedem kashyap: what does this mean? " libvirtError: unsupported configuration: Attribute mode is only allowed for guest CPU"
20:26:01 mriedem https://www.redhat.com/archives/libvir-list/2012-January/msg00232.html
20:26:57 openstackgerrit Artom Lifshitz proposed openstack/nova master: DNM: extra logging for 1775947 https://review.openstack.org/584032
20:43:56 cfriesen_ mriedem: looks like its if you try to parse XML with a <host><cpu> section that then tries to specify a "mode". That's only allowed for XML describing guest cpus.
20:44:54 cfriesen_ https://github.com/libvirt/libvirt/blob/master/src/conf/cpu_conf.c#L317
20:51:25 melwitt nova meeting in 9 minutes
20:55:01 openstackgerrit Eric Fried proposed openstack/nova master: Check provider generation and retry on conflict https://review.openstack.org/556669
21:31:40 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in test_compute_api.py (4) https://review.openstack.org/568462
21:31:58 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104
21:32:17 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106
21:46:25 mriedem gonna rebase and address comments in the handling a down cell series
21:49:41 melwitt tonyb: https://review.openstack.org/#/c/560317/ is failing powerkvm CI and we need help figuring out why
21:49:54 tonyb melwitt: okay I'll look at it
21:50:20 melwitt the powerkvm CI owner is on PTO this week, we found out that's why we didn't get response to pings
22:05:04 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153
22:07:24 melwitt mriedem: os-traits release proposed https://review.openstack.org/584130
22:07:28 melwitt fyi
22:08:51 openstackgerrit Takashi NATSUME proposed openstack/nova-specs master: Create specs directory for Stein https://review.openstack.org/573602
22:12:29 openstackgerrit do3meli proposed openstack/nova master: docs: add nova host-evacuate command to evacuate documentation https://review.openstack.org/578040
22:15:00 openstackgerrit do3meli proposed openstack/nova master: docs: add nova host-evacuate command to evacuate documentation https://review.openstack.org/578040
22:15:54 mriedem tonyb: well, it was originally failing because the CI is configured with cpu_mode='none' and so i believe the model put into the xml was ppc64le, so that was changed to mode=host-model and model=power8 (per someone from libvirt/qemu that knows about this),
22:16:02 mriedem but now it fails on something else, which cfriesen_ said might be:
22:16:03 mriedem (3:45:04 PM) cfriesen_: https://github.com/libvirt/libvirt/blob/master/src/conf/cpu_conf.c#L317
22:16:03 mriedem (3:44:06 PM) cfriesen_: mriedem: looks like its if you try to parse XML with a <host><cpu> section that then tries to specify a "mode". That's only allowed for XML describing guest cpus.
22:16:41 mriedem tonyb: i was suggesting the powerkvm ci could just set cpu_mode=host-model and cpu_model=power8, but that's basically what the code is now doing as a workaround
22:26:40 cfriesen_ mriedem: if you want to specify cpu_model=power8, wouldn't you want a cpu_mode of custom?
22:28:17 mriedem if a tree falls in the woods
22:28:55 mriedem cfriesen_: yeah that makes more sense
22:29:14 cfriesen_ mriedem: hmm...https://bugzilla.redhat.com/show_bug.cgi?id=1237025 seems to think mode of "host-model" and "model" of "power8" is valid. weird.
22:29:15 openstack bugzilla.redhat.com bug 1237025 in libvirt "Guest can not start with different combinations of <cpu> mode and <model>" [Medium,Closed: notabug] - Assigned to abologna
22:29:33 cfriesen_ wonder if this is a bizarre powerpc-ism
22:30:12 mriedem jlk: said s390x causes all the problems,
22:30:15 mriedem but power is right up there
22:32:29 cfriesen_ mriedem: https://libvirt.org/formatdomain.html#elementsCPU under the host-model section documents PowerPC weirdness.
22:33:09 mriedem "Specifying CPU model is not supported either"
22:34:01 mriedem Since 1.2.11 PowerISA allows processors to run VMs in binary compatibility mode supporting an older version of ISA. Libvirt on PowerPC architecture uses the host-model to signify a guest mode CPU running in binary compatibility mode
22:34:10 cfriesen_ yeah, that's the interesting bit
22:34:24 mriedem ii libvirt-bin 4.0.0-1ubuntu8.3~cloud0
22:34:25 jlk of course Power would override that to means omething else
22:36:08 cfriesen_ I don't get why they wouldn't just use a mode of "custom" with a model of "power8" (or whatever) instead.
22:36:39 mriedem could try that
22:36:48 mriedem kind of throwing things at the wall at this point until something works
22:43:47 openstackgerrit Eric Fried proposed openstack/nova master: Check provider generation and retry on conflict https://review.openstack.org/556669
22:52:41 tonyb mriedem: Yeah setting power8 is wrong long term but shoudl be fine for now (but I wouldn't merge it that way)
22:53:36 tonyb mriedem: How do I debug placement failures? or at least see what allocation_candidates returned
22:54:34 tonyb Oh nm /me finds the 'renew until 27/07/18 08:05:33
22:54:40 tonyb ' in n-cpu
22:54:56 mriedem so you don't need help debugging placement failures now?
22:55:20 tonyb mriedem: No I don't think so for this thing but in gernal it'd be good to know how to do it
22:55:22 mriedem n-sch logs will tell you for a given request which hosts are being filtered
22:55:45 mriedem i don't think we log the allocation_candidates response since it could be huge
22:57:28 tonyb mriedem: Okay what I'm seeing 'Got no allocation candidates from the Placement API' so I think that measn I the filters don't run which is why I'm not seeing the output I'm expecting in select_destindations?
22:58:13 openstackgerrit Merged openstack/python-novaclient master: Fix inconsistency https://review.openstack.org/572770
22:59:03 mriedem correct
22:59:17 mriedem tonyb: what's the scenario? normal server create? or a rebuild, or force_hosts?
22:59:36 mriedem could just be there are no hosts with available capacity for the flavor being used
23:01:52 tonyb mriedem: Lots of scenarios: tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest is the one I picked at random
23:02:39 mriedem tonyb: so some tripleo ci job shit the bed or what?
23:02:39 tonyb mriedem: I'm off reading the libvirt code for baselineCPU as that's certainly part of the problem
23:03:10 tonyb mriedem: oooo No I'm looking at the pkvm thing from thismornings meeting
23:03:31 mriedem oh, don't worry about tempest in that case,
23:03:39 mriedem the n-cpu logs just show that the host baselineCPU stuff is f'ed
23:04:01 tonyb mriedem: Yeah I finally got there, and hence readin the libvirt code ;P
23:04:50 mriedem ok cool. point taken though - if people though debugging scheduler failures were hard before placement, it will be worse, because the filtering for some things is done in sql now
23:04:55 mriedem *thought
23:05:21 tonyb mriedem: Yup. I agree.
23:05:58 mriedem efried: haven't we mused about some kind of "tell me why this request doesn't fit" type tool?
23:06:24 efried mriedem: We have. It would be insanely complicated to write, which is why we haven't gotten beyond musing.
23:06:45 jlk :9
23:06:55 mriedem jlk: i thought you left
23:06:57 jlk I remember spending a lot of time debugging selection, when introducing more complex rules
23:07:02 tonyb mriedem: if a live system can't use psc-placement and some gray-matter to work (well guess) why?
23:07:08 jlk mriedem: I did, but I'm still dealing with libvirt and KVM and stuff.
23:07:10 efried mriedem: Insanely complicated in the sense that we would have to rewrite pretty much every line of logic in the GET /a_c flow, splitting monolithic SQL statements into smaller components, etc.
23:07:19 jlk I'm ... "borrowing" some things from Nova
23:07:30 cfriesen_ could you split it up into the subqueries at least, to see which ones are ruling out significant numbers of hosts?
23:07:37 mriedem jlk: uh, for k8s?
23:07:40 jlk no
23:07:50 jlk an in-house bare metal cloud I'm teaching how to do VMs
23:08:16 jlk k8s will run on a VM on this, instead of on bare metal, but we have a bunch of other bare metal consumers that would be better serviced by VMs, even if the VM takes up the whole machine.
23:08:23 mriedem i know you can enable debugging at the sql level, but that would suck
23:09:03 cfriesen_ efried: I mean when the query comes back with no hosts you could re-run the various sub-queries
23:09:46 openstackgerrit Eric Fried proposed openstack/nova master: Check provider generation and retry on conflict https://review.openstack.org/556669
23:11:13 efried cfriesen_: The sub-queries that would matter - at least for resource availability - are pretty well mashed together in monolithic sql statements at the moment.
23:12:06 cfriesen_ if we ever put hugepages and cpu pinning and pci devices and provider networks in placement I think we'd probably need some sort of useful logging on failure to find a candidate.
23:13:11 cfriesen_ when you get into stuff like "you wanted a pci device attached to this network, but there's not enough 1G hugepages on any numa node associated with those PCI devices"
23:13:18 efried Some of the logic splitup for nrp and granular will actually help with that, much as it got hated on during review.
23:14:00 efried but for any serious debugging, other than sql dumps as mriedem says, it's gonna need a crap ton of refactoring.
23:14:20 efried I'm Audi 2000. See y'all tomorrow.
23:19:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Add queued_for_delete field to InstanceMapping object https://review.openstack.org/566795
23:19:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Online migration tool for populating queued-for-delete https://review.openstack.org/582536

Earlier   Later