| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-02 | |||
| 21:29:57 | jaypipes | MEMORY_MB == 1 | |
| 21:29:59 | jaypipes | DISK_GB == 2 | |
| 21:30:36 | mnaser | ok the allocation ratio is indeed there | |
| 21:30:45 | mnaser | i guess ill have to run the scheduler in verbose | |
| 21:31:32 | mnaser | i could swear scheduling failures were reported as warnings in the scheduler | |
| 21:31:50 | mriedem | "and i was trying to think how i was going to query the placement api" - osc-placement plugin has some placement CLI now | |
| 21:31:58 | mnaser | oh really | |
| 21:32:06 | mriedem | pretty minimal at this point | |
| 21:32:22 | mriedem | supports working on resource providers and inventories records | |
| 21:32:24 | jaypipes | mnaser: no, but in debug/verbose mode, you will see some log messages like "Filter XXX: Started with 10 hosts, finished with 6 hosts" or something like that. | |
| 21:33:10 | dansmith | mnaser: like a (statistically-consistent) boss | |
| 21:33:29 | jaypipes | mnaser: however it's just as good to look in the placement-api logs for the requests to GET /allocation_candidates and then I can give you the SQL to run that represents that particular request. | |
| 21:33:50 | mnaser | 3 schedulers, 1 running verbose, 3 build attempts and i am 0-3 | |
| 21:34:15 | mnaser | oh duh because verbose=True is a default now and I should be debug=True | |
| 21:34:24 | jaypipes | heh | |
| 21:34:59 | mnaser | um | |
| 21:35:09 | mnaser | "Got no allocation candidates from the Placement API. This may be a temporary occurrence as compute nodes start up and begin reporting inventory to the Placement service." | |
| 21:35:23 | mnaser | i might have a bit more on my hands than i expected | |
| 21:35:40 | mnaser | shouldn't that be WARN instead of DEBUG? | |
| 21:36:26 | jaypipes | mnaser: there was a deliberate use of DEBUG for most/all messages in the scheduler code paths due to concern about performance. | |
| 21:36:28 | cdent | mnaser: that's what I was going to suggest: first make sure you're getting any allocations candidates, you can contrust queries direct to /allocation_candidates: https://developer.openstack.org/api-ref/placement/#allocation-candidates | |
| 21:36:45 | cdent | s/contrust/construct/ | |
| 21:36:48 | mnaser | jaypipes: i see | |
| 21:36:54 | jaypipes | mnaser: what was the GET /allocation_candidates HTTP request you see in your logs?\ | |
| 21:37:03 | mnaser | cdent: ok, i'll try to do that, might be a bit difficult considering the volume of traffic at the placement api | |
| 21:37:06 | jaypipes | placement-api logs, that is. | |
| 21:37:14 | mnaser | because i'm pretty sure this is some upgrade-leftover | |
| 21:37:33 | mnaser | let me check | |
| 21:38:22 | mnaser | "GET /allocation_candidates?resources=MEMORY_MB%3A65536%2CVCPU%3A64" status: 200 len: 53 microversion: 1.10 | |
| 21:39:17 | cdent | mnaser: you can also make similar queries to /resource_providers to confirm what you think should be there is in fact there (basically a shorter version of what an /allocation_candidates query might report. you want 64 vcpus? | |
| 21:39:56 | mnaser | so right now i have an empty 32 physical core machine and im trying to launch 64 cores on it (with allocation ratio set to 2) -- im aware of performance implications but this is just to test out oversubscription | |
| 21:40:52 | jaypipes | ahhhhhhhhh | |
| 21:41:13 | jaypipes | mnaser: so, you will notice that max_unit is == 32 for your VCPU inventories. | |
| 21:41:28 | mnaser | for that host correct | |
| 21:41:41 | jaypipes | mnaser: this is to prevent someone from attempting to launch an instance that consumes more VCPU than the physical number of CPUs on the host. | |
| 21:41:58 | jaypipes | mnaser: allocation_ratio does not affect this constraint. | |
| 21:42:29 | mnaser | so, no single VM can have above max_unit, but the total vcpus is limited to max_unit * allocation_ratio ? | |
| 21:42:45 | jaypipes | mnaser: allocation_ratio only allows over-subscription of resources when min_unit <= $REQUESTED_AMOUNT <= max_unit | |
| 21:42:58 | jaypipes | mnaser: that is correct. | |
| 21:43:21 | mnaser | so everything was probably working and my lazy (launch one big instance to test overcommit) was at fault :D | |
| 21:43:29 | jaypipes | heh, yeah :) | |
| 21:43:32 | cdent | Is bauzas back yet? He'd be relishing this moment (even though I still think we made the right decision) | |
| 21:43:34 | mnaser | let me double check this by launching a few instances | |
| 21:44:59 | mnaser | and indeed, that was it | |
| 21:45:06 | mnaser | 2x 32c was ok | |
| 21:46:10 | mnaser | thanks jaypipes and everyone else for the input | |
| 21:47:36 | cdent | mnaser, while you're hear can you clarify something for me. above you said "might be a bit difficult considering the volume of traffic at the placement api". What did you mean by that? How is/was the traffic at the api impacting your ability to make requests of it, in this case? | |
| 21:47:44 | cdent | sigh: s/hear/here/ | |
| 21:48:09 | mnaser | cdent: oh, i was doing some testing on a host aggregate on our public cloud, so the volume of traffic hitting the placement api is pretty significant | |
| 21:48:24 | mnaser | from compute nodes reporting inventory to VMs getting booted its pretty busy and gets churned a lot | |
| 21:49:26 | cdent | Okay, my suggestion was, effectively, to construst the query that the scheduler was doing, yourself, so you could isolate the "no candidates returned" problem, without having to boot anything | |
| 21:50:19 | mnaser | cdent: yeah, i have to admit the placement api is still one of the things that i'm not as comfortable with as the rest of nova.. but this little experience added a bunch of stuff to my 'toolkit' | |
| 21:50:38 | cdent | I'm the opposite :) | |
| 21:52:19 | mnaser | :D | |
| 22:24:16 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257 | |
| 22:24:17 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Add Neutron port capabilities to devspec in request https://review.openstack.org/451777 | |
| 22:24:17 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Format NIC features using os-traits definitions https://review.openstack.org/466051 | |
| 22:28:07 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Read Neutron port 'binding_profile' during boot https://review.openstack.org/507481 | |
| 22:36:02 | eandersson | Is services versions supposed to be upgraded when you upgrade the version of Openstack? | |
| 23:35:20 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM driver: ovs vif https://review.openstack.org/422512 | |
| 23:45:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Workaround CellDatabase fixture limitation when creating computes https://review.openstack.org/524264 | |
| 23:45:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add multi-cell negative test for cold migration with target host https://review.openstack.org/524027 | |
| #openstack-nova - 2018-01-03 | |||
| 01:11:11 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM driver: ovs vif https://review.openstack.org/422512 | |
| 01:25:51 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: SEA https://review.openstack.org/523216 | |
| 01:47:12 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Use neutron port_list when filtering instance by ip https://review.openstack.org/525505 | |
| 02:02:25 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: manager: more detailed info of unsupported compute driver https://review.openstack.org/530701 | |
| 02:07:12 | openstackgerrit | Wangpan proposed openstack/nova master: Handle glance exception during rotating instance backup https://review.openstack.org/523395 | |
| 06:21:26 | openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova master: Use method validate_integer from oslo.utils https://review.openstack.org/528620 | |
| 07:01:14 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: SEA https://review.openstack.org/523216 | |
| 07:55:08 | Roamer` | johnthetubaguy, when you get back, could you please take a look at https://review.openstack.org/140733/ - is there a chance that you might remove your procedural -2 from a previous release cycle and possibly consider the almost trivial StorPool Nova volume attachment driver for inclusion in Queens? Thanks a lot in advance! Right now, with our Cinder and os-brick drivers merged, it's a bit weird - | |
| 07:55:14 | Roamer` | the main use case for the Cinder driver still requires manual patching :) | |
| 09:08:08 | openstackgerrit | Merged openstack/nova master: Deprecate API extensions policies https://review.openstack.org/526258 | |
| 09:11:52 | stephenfin | mdbooth: Done | |
| 09:32:59 | openstackgerrit | Rajesh Tailor proposed openstack/nova master: Allow force-delete even if task_state is not None https://review.openstack.org/530879 | |
| 10:05:11 | mdbooth | stephenfin: Thanks | |
| 10:34:19 | ralonsoh_ | stephenfin: hello, about https://review.openstack.org/#/c/449257/52/nova/pci/request.py | |
| 10:34:30 | stephenfin | ralonsoh_: Shoot | |
| 10:34:50 | ralonsoh_ | stephenfin: if I accept this suggestion, I need to refactor the whole patch | |
| 10:35:11 | ralonsoh_ | stephenfin: the scope of this feature is to store a list of strings in spec | |
| 10:35:25 | stephenfin | ralonsoh_: Yup, afraid so :( I realize | |
| 10:35:44 | stephenfin | I realize that, but as soon as we create the object that way, we're stuck with it | |
| 10:35:44 | ralonsoh_ | stephenfin: not to modify how InstacePCIRequestSpec was storing the other parameters | |
| 10:36:17 | ralonsoh_ | stephenfin: but "spec" was storing parameters in key=value fashion | |
| 10:36:52 | stephenfin | Right, but you're already modifying how the other parameters were stored | |
| 10:37:26 | stephenfin | Previously they were stored in a 'ListOfDictOfNullableStringsField'. Now they're stored in the 'InstancePCIRequestSpec' object | |
| 10:37:28 | ralonsoh_ | in a generic way. But there were no specific fields for those parameters (product, vendor, etc) | |
| 10:37:47 | ralonsoh_ | ok, I'll submit this modification next release | |
| 10:38:04 | ralonsoh_ | First, I'll submit again the spec | |
| 10:38:39 | stephenfin | ralonsoh_: I realize it's a good chunk of rework, but it just seems like a bad idea to not do this when we're already doing the work here | |
| 10:38:49 | stephenfin | I could help in the rework, but if I do then I can't review :/ | |
| 10:39:02 | stephenfin | ralonsoh_: So that is your call | |
| 10:39:23 | ralonsoh_ | ok, I'll abandond the patch for now | |
| 10:39:30 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ocata: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528226 | |
| 10:40:54 | stephenfin | ralonsoh_: I wouldn't, personally. Better to keep it as -W so we can see the reasoning when it's reworked | |
| 10:41:00 | stephenfin | But that's me | |
| 10:41:49 | ralonsoh_ | and then there will be another reason to refactor it again | |
| 10:43:56 | stephenfin | ralonsoh_: That hardly seems fair :( I'm not asking for nit-picky stuff here and I (hope) I've explained the rationale for the change as much as possible (plus offered to help where possible) | |
| 11:33:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Modify PciDevice.uuid generation code https://review.openstack.org/530487 | |
| 11:33:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add an online migration for PciDevice.uuid https://review.openstack.org/530905 | |
| 13:02:25 | openstackgerrit | Silvan Kaiser proposed openstack/nova master: Added mount fstype based validation of Quobyte mounts https://review.openstack.org/522245 | |