Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-02
20:02:06 mriedem since there would be no need for rootwrap
20:02:12 efried right
20:02:15 efried esberglu ^
20:02:20 dansmith except for starting privsep
20:02:35 efried oh
20:02:44 dansmith I can't remember how we settled on that.. it might be hard-coded in privsep stuff now or something
20:03:27 mriedem i thought we did a hard-coded thing to get around grenade
20:03:32 efried Basically, trying to figure out if my driver needs that opt set to True if the only root-y stuff it does is via privsep now.
20:03:36 mriedem which was the great schizm of 2015
20:04:49 mriedem https://github.com/openstack/oslo.rootwrap/commit/37c2a041d33f0fdce7ce2832398c1f60f3ee8703
20:04:51 mriedem is what i'm thinking of
20:05:09 mriedem which is different from what dansmith is talking about, i think
20:09:05 esberglu efried: I will push a run through without use_rootwrap_daemon and confirm
20:09:44 efried esberglu Sounds like a plan. Could consider splitting that bit out of the change; I'd be +2 on the other parts.
20:20:34 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Do not set allocation.id in AllocationList.create_all() https://review.openstack.org/530794
20:49:23 openstackgerrit Eric Fried proposed openstack/nova master: Move aggregates from report client to ProviderTree https://review.openstack.org/521685
20:49:24 openstackgerrit Eric Fried proposed openstack/nova master: Track provider traits in report client https://review.openstack.org/521686
20:49:24 openstackgerrit Eric Fried proposed openstack/nova master: Track associated sharing RPs in report client https://review.openstack.org/526539
20:49:25 openstackgerrit Eric Fried proposed openstack/nova master: Raise on API errors getting aggregates/traits https://review.openstack.org/526540
20:49:25 openstackgerrit Eric Fried proposed openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756
20:49:26 openstackgerrit Eric Fried proposed openstack/nova master: Track tree-associated providers in report client https://review.openstack.org/526541
20:49:26 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098
20:49:27 openstackgerrit Eric Fried proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187
20:49:27 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246
20:49:36 efried Rebase only (some manual) ^^
20:59:58 openstackgerrit Jackie Truong proposed openstack/python-novaclient master: Microversion 2.59 - Add trusted_image_certificates https://review.openstack.org/500396
21:10:13 openstackgerrit Jay Pipes proposed openstack/nova-specs master: Support aggregate affinity filters/weighers https://review.openstack.org/529135
21:26:32 mnaser what are the implications of using cpu_allocation_ratio on compute with the placement api?
21:27:23 mnaser i'm testing around cpu_allocation_ratio=2.0 on a compute node but i'm not sure if its properly reporting that number to the placement as im getting no valid host
21:28:51 jaypipes mnaser: easy enough to check... in the API database (the one placement uses, do this: SELECT * FROM inventories WHERE allocation_ratio > 1.0 AND resource_class = 0;
21:29:01 mnaser jaypipes: gah thah was an obvious one
21:29:07 jaypipes mnaser: :)
21:29:09 mnaser and i was trying to think how i was going to query the placement api
21:29:15 mnaser and making curl calls
21:29:16 mnaser :p
21:29:29 jaypipes mnaser: you could do that, too, but wouldn't be as fun ;)
21:29:44 jaypipes mnaser: FYI, VCPU == resource_class "0"
21:29:47 mnaser jaypipes: true
21:29:57 jaypipes MEMORY_MB == 1
21:29:59 jaypipes DISK_GB == 2
21:30:36 mnaser ok the allocation ratio is indeed there
21:30:45 mnaser i guess ill have to run the scheduler in verbose
21:31:32 mnaser i could swear scheduling failures were reported as warnings in the scheduler
21:31:50 mriedem "and i was trying to think how i was going to query the placement api" - osc-placement plugin has some placement CLI now
21:31:58 mnaser oh really
21:32:06 mriedem pretty minimal at this point
21:32:22 mriedem supports working on resource providers and inventories records
21:32:24 jaypipes mnaser: no, but in debug/verbose mode, you will see some log messages like "Filter XXX: Started with 10 hosts, finished with 6 hosts" or something like that.
21:33:10 dansmith mnaser: like a (statistically-consistent) boss
21:33:29 jaypipes mnaser: however it's just as good to look in the placement-api logs for the requests to GET /allocation_candidates and then I can give you the SQL to run that represents that particular request.
21:33:50 mnaser 3 schedulers, 1 running verbose, 3 build attempts and i am 0-3
21:34:15 mnaser oh duh because verbose=True is a default now and I should be debug=True
21:34:24 jaypipes heh
21:34:59 mnaser um
21:35:09 mnaser "Got no allocation candidates from the Placement API. This may be a temporary occurrence as compute nodes start up and begin reporting inventory to the Placement service."
21:35:23 mnaser i might have a bit more on my hands than i expected
21:35:40 mnaser shouldn't that be WARN instead of DEBUG?
21:36:26 jaypipes mnaser: there was a deliberate use of DEBUG for most/all messages in the scheduler code paths due to concern about performance.
21:36:28 cdent mnaser: that's what I was going to suggest: first make sure you're getting any allocations candidates, you can contrust queries direct to /allocation_candidates: https://developer.openstack.org/api-ref/placement/#allocation-candidates
21:36:45 cdent s/contrust/construct/
21:36:48 mnaser jaypipes: i see
21:36:54 jaypipes mnaser: what was the GET /allocation_candidates HTTP request you see in your logs?\
21:37:03 mnaser cdent: ok, i'll try to do that, might be a bit difficult considering the volume of traffic at the placement api
21:37:06 jaypipes placement-api logs, that is.
21:37:14 mnaser because i'm pretty sure this is some upgrade-leftover
21:37:33 mnaser let me check
21:38:22 mnaser "GET /allocation_candidates?resources=MEMORY_MB%3A65536%2CVCPU%3A64" status: 200 len: 53 microversion: 1.10
21:39:17 cdent mnaser: you can also make similar queries to /resource_providers to confirm what you think should be there is in fact there (basically a shorter version of what an /allocation_candidates query might report. you want 64 vcpus?
21:39:56 mnaser so right now i have an empty 32 physical core machine and im trying to launch 64 cores on it (with allocation ratio set to 2) -- im aware of performance implications but this is just to test out oversubscription
21:40:52 jaypipes ahhhhhhhhh
21:41:13 jaypipes mnaser: so, you will notice that max_unit is == 32 for your VCPU inventories.
21:41:28 mnaser for that host correct
21:41:41 jaypipes mnaser: this is to prevent someone from attempting to launch an instance that consumes more VCPU than the physical number of CPUs on the host.
21:41:58 jaypipes mnaser: allocation_ratio does not affect this constraint.
21:42:29 mnaser so, no single VM can have above max_unit, but the total vcpus is limited to max_unit * allocation_ratio ?
21:42:45 jaypipes mnaser: allocation_ratio only allows over-subscription of resources when min_unit <= $REQUESTED_AMOUNT <= max_unit
21:42:58 jaypipes mnaser: that is correct.
21:43:21 mnaser so everything was probably working and my lazy (launch one big instance to test overcommit) was at fault :D
21:43:29 jaypipes heh, yeah :)
21:43:32 cdent Is bauzas back yet? He'd be relishing this moment (even though I still think we made the right decision)
21:43:34 mnaser let me double check this by launching a few instances
21:44:59 mnaser and indeed, that was it
21:45:06 mnaser 2x 32c was ok
21:46:10 mnaser thanks jaypipes and everyone else for the input
21:47:36 cdent mnaser, while you're hear can you clarify something for me. above you said "might be a bit difficult considering the volume of traffic at the placement api". What did you mean by that? How is/was the traffic at the api impacting your ability to make requests of it, in this case?
21:47:44 cdent sigh: s/hear/here/
21:48:09 mnaser cdent: oh, i was doing some testing on a host aggregate on our public cloud, so the volume of traffic hitting the placement api is pretty significant
21:48:24 mnaser from compute nodes reporting inventory to VMs getting booted its pretty busy and gets churned a lot
21:49:26 cdent Okay, my suggestion was, effectively, to construst the query that the scheduler was doing, yourself, so you could isolate the "no candidates returned" problem, without having to boot anything
21:50:19 mnaser cdent: yeah, i have to admit the placement api is still one of the things that i'm not as comfortable with as the rest of nova.. but this little experience added a bunch of stuff to my 'toolkit'
21:50:38 cdent I'm the opposite :)
21:52:19 mnaser :D
22:24:16 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257
22:24:17 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Add Neutron port capabilities to devspec in request https://review.openstack.org/451777
22:24:17 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Format NIC features using os-traits definitions https://review.openstack.org/466051
22:28:07 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: Read Neutron port 'binding_profile' during boot https://review.openstack.org/507481
22:36:02 eandersson Is services versions supposed to be upgraded when you upgrade the version of Openstack?
23:35:20 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM driver: ovs vif https://review.openstack.org/422512
23:45:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Workaround CellDatabase fixture limitation when creating computes https://review.openstack.org/524264
23:45:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Add multi-cell negative test for cold migration with target host https://review.openstack.org/524027

Earlier   Later