| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-20 | |||
| 14:53:05 | mriedem | because i know of another bug related to that... | |
| 14:53:15 | mnaser | nope im not passing one | |
| 14:53:30 | mriedem | ok, you just expect a certain host to be picked b/c of weights and capacity etc? | |
| 14:53:54 | mnaser | yeah, it's not even hitting weight/capacity, placement is not returning a host that has capacity | |
| 14:54:08 | mriedem | i think we'd have to add something like this https://github.com/openstack/nova/blob/master/nova/config.py#L39 | |
| 14:54:09 | efried | Does the placement log not show the full command line? | |
| 14:54:11 | mnaser | which tells me there's probably incorrect allocations in placement that make that host look 'overloaded' | |
| 14:54:22 | mnaser | ah that's a good alternative, checking uwsgi logs on placement side | |
| 14:54:30 | mriedem | there should be the same request id on both sides | |
| 14:54:46 | mriedem | that's your workaround for now | |
| 14:54:53 | efried | The placement logs by default won't show the payloads (request or response) but the queryparams are there I thought. | |
| 14:55:01 | mriedem | yes they should be | |
| 14:55:09 | mnaser | i should be able to rerun that http request | |
| 14:55:24 | mriedem | http://logs.openstack.org/75/470775/38/check/tempest-full/a97793a/controller/logs/screen-placement-api.txt.gz#_Sep_06_13_02_32_181234 | |
| 14:55:25 | efried | sorry, "command line" was a bogus phrase | |
| 14:55:29 | mriedem | Sep 06 13:02:32.181234 ubuntu-xenial-ovh-bhs1-0001824138 devstack@placement-api.service[6991]: DEBUG nova.api.openstack.placement.requestlog [req-68f9280e-61b2-446e-8622-a64e27e7c617 req-df696f1f-3435-4b6c-8a3b-c6dc1c58c1ba service placement] Starting request: 158.69.79.172 "GET /placement/allocation_candidates?limit=1000&resources=MEMORY_MB%3A64%2CVCPU%3A1" {{(pid=6993) __call__ /opt/stack/nova/nova/api/openstack/placement/r | |
| 14:55:29 | mriedem | stlog.py:38}} | |
| 14:55:54 | mriedem | must be a bfv instance ^ because there is no DISK_GB requested | |
| 14:56:35 | mnaser | yup /allocation_candidates?limit=1000&resources=DISK_GB%3A10%2CMEMORY_MB%3A2048%2CVCPU%3A4 | |
| 14:56:36 | mnaser | lets see | |
| 14:58:20 | mnaser | openstack allocation candidate list --resource DISK_GB=10 --resource MEMORY_MB=2048 --resource VCPU=4 => Operation or argument is not supported with version 1.0 | |
| 14:58:25 | mnaser | mriedem: do i need some special env variable? | |
| 14:58:49 | mnaser | sigh | |
| 14:58:49 | mnaser | --os-placement-api-version 1.10 | |
| 14:58:52 | mnaser | ill read docs from now on | |
| 14:59:47 | gibi | FYI, I've added the bp use-nested-allocation-candidates to the 3rd runway slot | |
| 14:59:51 | gibi | First part of https://blueprints.launchpad.net/nova/+spec/use-nested-allocation-candidates (gibi) [END: 2018-10-04] | |
| 15:00:01 | cdent | i guess I need to learn osc-placement so I can parse this stuff. I'm still back on raw curl which is nice and comfy and explicit | |
| 15:00:05 | mnaser | yeah it's returning one host (the host itself) which gets filtered out in a live migration, i think i'll have to write up the clean up / audit stuff | |
| 15:00:36 | mnaser | cdent: it's nice to not have to write it all out esp to grab a token and then build out a curl request | |
| 15:01:26 | mriedem | mnaser: yeah osc is opt-in per request for microversions, unlike nova cli | |
| 15:01:36 | mriedem | you could just set an env var if you want the latest, | |
| 15:01:49 | mriedem | export OS_PLACEMENT_API_VERSION=<whatever you have that is the latest> | |
| 15:03:06 | cdent | mnaser: yeah, totally agree that if you haven't built the curl-based habits the osc way is good. It's just that since I've been curl-ing against placement for such a long time I've got habits (including the token getting and using parts) that are rather built in. I'm complaining about myself here, not other people. | |
| 15:03:22 | mnaser | cdent: yeah i have a lot of terrible old habits | |
| 15:03:32 | mnaser | cdent: like using LOG.debug() instead of an actual debugger | |
| 15:03:50 | cdent | dood! LOG.debug() is too fancy. sys.stderr.write() 4evah | |
| 15:04:07 | mnaser | System.out.println() | |
| 15:04:11 | mnaser | at least we're not in that darkest timeline | |
| 15:06:40 | tobias-urdin | when live migrating is the vcpu_pin_set option a 1:1 mapping for source and dest host? I start at the same lowest possible core but some machines have different amount of vcpus available, if I migrate an instances that has vcpu 79 but source host only has 60 vcpu will it reallocate where to place that or error out? | |
| 15:07:37 | mnaser | i think that's a libvirt decision | |
| 15:08:33 | tobias-urdin | you are probably correct, will trace some more | |
| 15:08:54 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549 | |
| 15:09:25 | mriedem | i didn't think live migration + cpu pinning was supported yet? | |
| 15:09:26 | mriedem | cfriesen: ^? | |
| 15:12:10 | cfriesen | mriedem: tobias-urdin: live migration doesn't currently recalculate CPUs on the dest node. This is one of the things covered under the "numa-aware live migration" blueprint. | |
| 15:12:52 | tobias-urdin | cfriesen: roger that, thanks :) | |
| 15:13:49 | cfriesen | tobias-urdin: the live migration *might* succeed, but it might end up putting multiple guest CPUs on the same host CPU, or it might fail due to different host topology. | |
| 15:14:15 | tobias-urdin | ok, that explains why it worked atleast thanks | |
| 15:14:46 | zzzeek | jaypipes-ooo: think fast! what's your favorite innodb_flush_method setting ! don't think! | |
| 15:16:43 | jaypipes-ooo | zzzeek: direct? | |
| 15:16:50 | zzzeek | jaypipes-ooo: OK | |
| 15:17:22 | zzzeek | jaypipes-ooo: we're adding the option to tripleo | |
| 15:17:46 | jaypipes-ooo | zzzeek: but only if there is a battery-backed write cache on the disks. | |
| 15:17:55 | jaypipes-ooo | zzzeek: otherwise, keep it normal. | |
| 15:18:39 | openstackgerrit | Merged openstack/nova stable/rocky: Fix nova-status "_check_resource_providers" check https://review.openstack.org/600098 | |
| 15:19:11 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549 | |
| 15:20:07 | zzzeek | jaypipes-ooo: OK is that impacting the durability of the logs only? usually b.c. we use galera, we assume write sets have also gone to other nodes | |
| 15:20:33 | zzzeek | jaypipes-ooo: so w/ these kinds of options, we have to worry about all galera nodes power off simultaneously, but if only one machine, then not | |
| 15:20:42 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Run negative server moving tests with nested RPs https://review.openstack.org/604125 | |
| 15:20:53 | zzzeek | jaypipes-ooo: we certainly arent changing this setting by default | |
| 15:21:08 | zzzeek | jaypipes-ooo: Im just trying to create my own importance by curating what option we expose :) | |
| 15:21:44 | jaypipes-ooo | zzzeek: :) just leave it defaulted to normal then. It's mostly micro-tuning for write-heavy environments anyway. | |
| 15:21:54 | jaypipes-ooo | zzzeek: which openstack's DBs definitely are *not*. | |
| 15:22:04 | zzzeek | jaypipes-ooo: yes. I just need to have an intelligent answer for why someone would want to set this | |
| 15:22:19 | zzzeek | jaypipes-ooo: it's usually folks with old OSP and struggling w/ huge keystone token table | |
| 15:22:50 | jaypipes-ooo | zzzeek: ack. there's a solution to that, though... TRUNCATE TABLE tokens; :P | |
| 15:22:54 | zzzeek | jaypipes-ooo: agree | |
| 15:23:42 | jaypipes-ooo | zzzeek: yeah, overall I'd advise not mucking with many of the innodb config settings other than innodb_buffer_pool_size. | |
| 15:23:54 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549 | |
| 15:24:06 | zzzeek | jaypipes-ooo: it's an argument im slwoly losing due to the occasional angry customer | |
| 15:24:11 | zzzeek | with a mysql DBA | |
| 15:24:33 | zzzeek | jaypipes-ooo: which of course only matters to me if I have to be on the phone w/ them on friday night b.c. they lost their DB | |
| 15:24:37 | jaypipes-ooo | zzzeek: really no need considering nova, ironic, etc's database schemas are so poor anyway (lots of "blobject" storage and bloated not-really-relational-relations schema that those settings won't IMHO make much of a diff. | |
| 15:24:45 | zzzeek | jaypipes-ooo: agree | |
| 15:39:58 | openstackgerrit | Merged openstack/nova master: libvirt: Use 'virt' as the default machine type for ARMv7 https://review.openstack.org/602592 | |
| 15:40:08 | openstackgerrit | Merged openstack/nova master: Fix evacuate logging https://review.openstack.org/593055 | |
| 15:40:14 | openstackgerrit | Merged openstack/nova stable/queens: Fix soft deleting vm fails after "nova resize" vm https://review.openstack.org/603340 | |
| 15:50:54 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604134 | |
| 15:51:58 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 15:51:59 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 15:56:56 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604138 | |
| 15:58:48 | mnaser | woo | |
| 15:58:54 | mnaser | i got a script going and it helped figure out the issue too | |
| 15:59:49 | mnaser | it's really not cleaned up but does the job, here's a placement 'audit' script: http://paste.openstack.org/show/730450/ | |
| 16:01:15 | mriedem | cool, throw that into "nova-manage placement audit" or something | |
| 16:02:07 | mriedem | should be able to use nova.scheduler.client.report.SchedulerReportClient for the HTTP calls | |
| 16:02:18 | mnaser | yeah, and probably can use nova objects to avoid http calls? | |
| 16:02:38 | mriedem | yeah maybe | |
| 16:03:20 | mnaser | oh boi | |
| 16:03:57 | mnaser | i think i'd say its a bug if a field is missing in nova api if it's empty/blank? | |
| 16:04:12 | mriedem | depends | |
| 16:04:13 | mriedem | which field | |
| 16:04:24 | mnaser | with_servers=True | |
| 16:04:31 | mnaser | the servers: [] field is omitted if no servers are there | |
| 16:04:40 | mnaser | so rather than servers: []. its just not there | |
| 16:04:46 | mriedem | welcome to the compute api | |
| 16:04:53 | mnaser | i.e.: {u'status': u'disabled', u'state': u'down', u'id': u'13b1a4b6-d242-44c5-828e-e6f0a94284da', u'hypervisor_hostname': u'foobar'} | |
| 16:05:51 | mriedem | i believe that's how https://developer.openstack.org/api-ref/compute/#list-hypervisor-servers worked | |
| 16:05:58 | mnaser | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L90-L92 | |