Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-20
14:55:25 efried sorry, "command line" was a bogus phrase
14:55:29 mriedem stlog.py:38}}
14:55:29 mriedem Sep 06 13:02:32.181234 ubuntu-xenial-ovh-bhs1-0001824138 devstack@placement-api.service[6991]: DEBUG nova.api.openstack.placement.requestlog [req-68f9280e-61b2-446e-8622-a64e27e7c617 req-df696f1f-3435-4b6c-8a3b-c6dc1c58c1ba service placement] Starting request: 158.69.79.172 "GET /placement/allocation_candidates?limit=1000&resources=MEMORY_MB%3A64%2CVCPU%3A1" {{(pid=6993) __call__ /opt/stack/nova/nova/api/openstack/placement/r
14:55:54 mriedem must be a bfv instance ^ because there is no DISK_GB requested
14:56:35 mnaser yup /allocation_candidates?limit=1000&resources=DISK_GB%3A10%2CMEMORY_MB%3A2048%2CVCPU%3A4
14:56:36 mnaser lets see
14:58:20 mnaser openstack allocation candidate list --resource DISK_GB=10 --resource MEMORY_MB=2048 --resource VCPU=4 => Operation or argument is not supported with version 1.0
14:58:25 mnaser mriedem: do i need some special env variable?
14:58:49 mnaser --os-placement-api-version 1.10
14:58:49 mnaser sigh
14:58:52 mnaser ill read docs from now on
14:59:47 gibi FYI, I've added the bp use-nested-allocation-candidates to the 3rd runway slot
14:59:51 gibi First part of https://blueprints.launchpad.net/nova/+spec/use-nested-allocation-candidates (gibi) [END: 2018-10-04]
15:00:01 cdent i guess I need to learn osc-placement so I can parse this stuff. I'm still back on raw curl which is nice and comfy and explicit
15:00:05 mnaser yeah it's returning one host (the host itself) which gets filtered out in a live migration, i think i'll have to write up the clean up / audit stuff
15:00:36 mnaser cdent: it's nice to not have to write it all out esp to grab a token and then build out a curl request
15:01:26 mriedem mnaser: yeah osc is opt-in per request for microversions, unlike nova cli
15:01:36 mriedem you could just set an env var if you want the latest,
15:01:49 mriedem export OS_PLACEMENT_API_VERSION=<whatever you have that is the latest>
15:03:06 cdent mnaser: yeah, totally agree that if you haven't built the curl-based habits the osc way is good. It's just that since I've been curl-ing against placement for such a long time I've got habits (including the token getting and using parts) that are rather built in. I'm complaining about myself here, not other people.
15:03:22 mnaser cdent: yeah i have a lot of terrible old habits
15:03:32 mnaser cdent: like using LOG.debug() instead of an actual debugger
15:03:50 cdent dood! LOG.debug() is too fancy. sys.stderr.write() 4evah
15:04:07 mnaser System.out.println()
15:04:11 mnaser at least we're not in that darkest timeline
15:06:40 tobias-urdin when live migrating is the vcpu_pin_set option a 1:1 mapping for source and dest host? I start at the same lowest possible core but some machines have different amount of vcpus available, if I migrate an instances that has vcpu 79 but source host only has 60 vcpu will it reallocate where to place that or error out?
15:07:37 mnaser i think that's a libvirt decision
15:08:33 tobias-urdin you are probably correct, will trace some more
15:08:54 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549
15:09:25 mriedem i didn't think live migration + cpu pinning was supported yet?
15:09:26 mriedem cfriesen: ^?
15:12:10 cfriesen mriedem: tobias-urdin: live migration doesn't currently recalculate CPUs on the dest node. This is one of the things covered under the "numa-aware live migration" blueprint.
15:12:52 tobias-urdin cfriesen: roger that, thanks :)
15:13:49 cfriesen tobias-urdin: the live migration *might* succeed, but it might end up putting multiple guest CPUs on the same host CPU, or it might fail due to different host topology.
15:14:15 tobias-urdin ok, that explains why it worked atleast thanks
15:14:46 zzzeek jaypipes-ooo: think fast! what's your favorite innodb_flush_method setting ! don't think!
15:16:43 jaypipes-ooo zzzeek: direct?
15:16:50 zzzeek jaypipes-ooo: OK
15:17:22 zzzeek jaypipes-ooo: we're adding the option to tripleo
15:17:46 jaypipes-ooo zzzeek: but only if there is a battery-backed write cache on the disks.
15:17:55 jaypipes-ooo zzzeek: otherwise, keep it normal.
15:18:39 openstackgerrit Merged openstack/nova stable/rocky: Fix nova-status "_check_resource_providers" check https://review.openstack.org/600098
15:19:11 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549
15:20:07 zzzeek jaypipes-ooo: OK is that impacting the durability of the logs only? usually b.c. we use galera, we assume write sets have also gone to other nodes
15:20:33 zzzeek jaypipes-ooo: so w/ these kinds of options, we have to worry about all galera nodes power off simultaneously, but if only one machine, then not
15:20:42 openstackgerrit Balazs Gibizer proposed openstack/nova master: Run negative server moving tests with nested RPs https://review.openstack.org/604125
15:20:53 zzzeek jaypipes-ooo: we certainly arent changing this setting by default
15:21:08 zzzeek jaypipes-ooo: Im just trying to create my own importance by curating what option we expose :)
15:21:44 jaypipes-ooo zzzeek: :) just leave it defaulted to normal then. It's mostly micro-tuning for write-heavy environments anyway.
15:21:54 jaypipes-ooo zzzeek: which openstack's DBs definitely are *not*.
15:22:04 zzzeek jaypipes-ooo: yes. I just need to have an intelligent answer for why someone would want to set this
15:22:19 zzzeek jaypipes-ooo: it's usually folks with old OSP and struggling w/ huge keystone token table
15:22:50 jaypipes-ooo zzzeek: ack. there's a solution to that, though... TRUNCATE TABLE tokens; :P
15:22:54 zzzeek jaypipes-ooo: agree
15:23:42 jaypipes-ooo zzzeek: yeah, overall I'd advise not mucking with many of the innodb config settings other than innodb_buffer_pool_size.
15:23:54 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549
15:24:06 zzzeek jaypipes-ooo: it's an argument im slwoly losing due to the occasional angry customer
15:24:11 zzzeek with a mysql DBA
15:24:33 zzzeek jaypipes-ooo: which of course only matters to me if I have to be on the phone w/ them on friday night b.c. they lost their DB
15:24:37 jaypipes-ooo zzzeek: really no need considering nova, ironic, etc's database schemas are so poor anyway (lots of "blobject" storage and bloated not-really-relational-relations schema that those settings won't IMHO make much of a diff.
15:24:45 zzzeek jaypipes-ooo: agree
15:39:58 openstackgerrit Merged openstack/nova master: libvirt: Use 'virt' as the default machine type for ARMv7 https://review.openstack.org/602592
15:40:08 openstackgerrit Merged openstack/nova master: Fix evacuate logging https://review.openstack.org/593055
15:40:14 openstackgerrit Merged openstack/nova stable/queens: Fix soft deleting vm fails after "nova resize" vm https://review.openstack.org/603340
15:50:54 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604134
15:51:58 openstackgerrit Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658
15:51:59 openstackgerrit Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829
15:56:56 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604138
15:58:48 mnaser woo
15:58:54 mnaser i got a script going and it helped figure out the issue too
15:59:49 mnaser it's really not cleaned up but does the job, here's a placement 'audit' script: http://paste.openstack.org/show/730450/
16:01:15 mriedem cool, throw that into "nova-manage placement audit" or something
16:02:07 mriedem should be able to use nova.scheduler.client.report.SchedulerReportClient for the HTTP calls
16:02:18 mnaser yeah, and probably can use nova objects to avoid http calls?
16:02:38 mriedem yeah maybe
16:03:20 mnaser oh boi
16:03:57 mnaser i think i'd say its a bug if a field is missing in nova api if it's empty/blank?
16:04:12 mriedem depends
16:04:13 mriedem which field
16:04:24 mnaser with_servers=True
16:04:31 mnaser the servers: [] field is omitted if no servers are there
16:04:40 mnaser so rather than servers: []. its just not there
16:04:46 mriedem welcome to the compute api
16:04:53 mnaser i.e.: {u'status': u'disabled', u'state': u'down', u'id': u'13b1a4b6-d242-44c5-828e-e6f0a94284da', u'hypervisor_hostname': u'foobar'}
16:05:51 mriedem i believe that's how https://developer.openstack.org/api-ref/compute/#list-hypervisor-servers worked
16:05:58 mnaser https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L90-L92
16:06:05 mnaser if servers with servers = [] evaluates to false?
16:06:11 mriedem yes
16:06:50 mnaser i mean that could have been omitted but i dunno if thats an api bug or something we're expecting
16:07:14 mriedem that's just how that's always worked,
16:07:21 mriedem and this is probably not the only instance of it
16:07:30 mriedem where a field is omitted in the response if it doesn't have any value
16:07:52 mriedem in this case it was just an oversight when i rewrote os-hypervisors in 2.53
16:07:59 mriedem b/c it's using the same view builder code
16:08:07 mnaser its nice that it gave compute node uuids
16:08:08 mnaser it made this a lot easier
16:08:29 mriedem yup, that's part of why we had to do 2.53
16:08:53 mriedem https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-pike
16:08:59 mriedem it wasn't a fun microversion to write let me tell you
16:09:24 mriedem gmann does have a spec started for fixing random warts like this https://review.openstack.org/#/c/603969/

Earlier   Later