| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-29 | |||
| 13:49:52 | zigo | But then? How come placement didn't find resources? | |
| 13:50:08 | mriedem | let's make sure the compute node in nova is matching that same uuid | |
| 13:50:13 | mriedem | from nova cli, | |
| 13:50:15 | mriedem | as admin, | |
| 13:50:20 | mriedem | nova hypervisor-list | |
| 13:50:51 | zigo | I see my host, and it's up and enabled. | |
| 13:50:59 | zigo | I did all this before asking! :P | |
| 13:51:03 | mriedem | does it's id equal f9716941-356f-4a2e-b5ea-31c3c1630892 ? | |
| 13:51:10 | zigo | Yup. | |
| 13:51:29 | mriedem | ok what type of flavor did you use when you tried creating the server? | |
| 13:51:32 | zigo | Is it normal that I get allocation_ratio 0.0 for all resources? | |
| 13:51:41 | mriedem | so, | |
| 13:51:54 | mriedem | that sounds exactly like the same problem the xenserver CI guys are having in the ML right now | |
| 13:51:57 | mriedem | and no, | |
| 13:52:01 | mriedem | cpu should be 16.0, | |
| 13:52:02 | zigo | 256 RAM, 5 GB HDD, 1 VCPU | |
| 13:52:05 | mriedem | ram and disk should also be > 0 | |
| 13:52:14 | mriedem | jaypipes: efried: naichuans: ^ | |
| 13:53:49 | mriedem | zigo: using libvirt right? | |
| 13:54:24 | zigo | mriedem: Yeah, normal qemu in a virtualbox right now. | |
| 13:54:38 | zigo | mriedem: Also using it in the OpenStack CI with puppet-openstack stuff ... | |
| 13:55:15 | zigo | mriedem: Is there a way to force something in the allocation_ratio to fix things? | |
| 13:55:36 | mriedem | yes via osc-placement, but i'd have to look it up quick | |
| 13:56:10 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-class-set | |
| 13:56:57 | mriedem | so: openstack resource provider inventory class set --allocation_ratio 16.0 --total 4 f9716941-356f-4a2e-b5ea-31c3c1630892 VCPU | |
| 13:57:45 | mriedem | cdent: ever remember talking about making allocation_ratio a minimum of 1.0 in placement? | |
| 13:57:52 | mriedem | allowing anything to set that to 0.0 seems like a bad idea | |
| 13:59:00 | gibi | mriedem: having < 1.0 could make sense for handling overhead, but I agree that 0.0 doesn't make sense | |
| 13:59:18 | zigo | mriedem: The max_unit for VCPUs looks funny now: http://paste.openstack.org/show/729052/ | |
| 13:59:21 | cdent | mriedem: the only vague tickle I have in my mind was "if you have a float, how do you set a min_unit that is the min-est" | |
| 13:59:35 | zigo | Shouldn't it be num-of-vcpu * allocation_ratio ? | |
| 13:59:45 | cdent | What i'm not clear on is how/what is coming along with these 0s | |
| 14:00:09 | mriedem | cdent: i guess it couldn't be done with jsonschema and would have to be in python | |
| 14:00:19 | mriedem | if allocation_ratio <= 0.0: 400 | |
| 14:00:30 | mriedem | or use a custom jsonschema validator | |
| 14:00:42 | zigo | Can I consider there's a bug somewhere, and wait for the fix? :) | |
| 14:00:55 | mriedem | so clearly something in nova is creating resource providers with 0.0 allocation ratios from nova.conf, which is the default, but it should be getting the values from the ComputeNode object | |
| 14:01:00 | cdent | I can't remember any specific conversation about this, but I wonder if 0 was allowed as a way to disable inventory? | |
| 14:01:03 | mriedem | zigo: yes | |
| 14:01:17 | mriedem | zigo: naichuans: if you haven't already, it would be great to have a nova bug for tracking this | |
| 14:01:30 | cdent | (while still keeping a record of the "actual" inventory) | |
| 14:01:32 | mriedem | yeah idk, maybe jaypipes remmebers | |
| 14:02:09 | mriedem | so https://github.com/openstack/nova/blob/6522ea3ecfe99cca3fb33258b11e5a1f34e6e8f0/nova/compute/resource_tracker.py#L84 in the RT is what is supposed to set the default allocation ratio for each inventory on the compute node provider | |
| 14:02:24 | mriedem | assuming the virt driver doesn't provide an allocation ratio override, which neither libvirt nor xenapi do | |
| 14:02:24 | zigo | mriedem: cdent: What is max_unit supposed to represent? Isn't it total * allocation_ratio? | |
| 14:02:43 | jaypipes | mriedem: sorry, reading back was on a call | |
| 14:02:55 | cdent | zigo: no, it is an expression of the largest amount any individual resource allocation can allocate | |
| 14:02:57 | jaypipes | zigo: max_unit is not total * allocation_ratio, no | |
| 14:03:07 | cdent | so in the case of something like disk or vcpu it is generally the max physical amount | |
| 14:03:24 | cdent | as you don't want a single instance to have more than is physically available, regardless of allocation ratio | |
| 14:03:37 | zigo | Ah, great, meaning I can have hosts with up to 2 billion VCUPs ! :) | |
| 14:04:13 | zigo | I can start writing another bug, ok. | |
| 14:04:44 | cdent | total: the real physical amount, allocation ratio: multiplier for over or under commit, min_unit: smallest individual request, max_unit: largest individual request, reserved: what the system is using for itself | |
| 14:06:24 | mriedem | i wonder if there is a bug in ProviderTree.update_inventory where it's not updating the allocation_ratio when we create/update the provider in placement | |
| 14:07:23 | mriedem | zigo: you're testing with rocky right? | |
| 14:10:55 | zigo | https://bugs.launchpad.net/nova/+bug/1789654 | |
| 14:10:55 | openstack | Launchpad bug 1789654 in OpenStack Compute (nova) "placement allocation_ratio initialized with 0.0" [Undecided,New] | |
| 14:10:59 | zigo | mriedem: Correct ! | |
| 14:11:15 | zigo | mriedem: Fresh packages just built this and last week in a record time. | |
| 14:11:24 | zigo | It took me less than 2 weeks, this time ... :P | |
| 14:11:36 | openstackgerrit | Merged openstack/osc-placement master: Update reno for stable/rocky https://review.openstack.org/586115 | |
| 14:13:34 | mriedem | so i thought we used to log something when inventory changed on a provider... | |
| 14:13:36 | mriedem | not seeing that | |
| 14:13:51 | mriedem | from one of the failed xenserver ci logs, the RP is created here http://logs.openstack.org/41/590041/17/check/tempest-full/b3f9ddd/controller/logs/screen-n-cpu.txt.gz#_Aug_27_14_18_25_580517 | |
| 14:14:00 | mriedem | then the generation changes twice but we don't log why | |
| 14:14:36 | zigo | mriedem: If you want the full logs and everything, you can find it here: https://review.openstack.org/#/c/597175/ | |
| 14:14:58 | mriedem | any of those failed jobs? | |
| 14:14:58 | zigo | That commit just switches repo from queens to rocky for Debian and puppet-openstack. | |
| 14:15:02 | zigo | Yep. | |
| 14:15:05 | zigo | The first one for example. | |
| 14:15:33 | mriedem | yup so in this case we create the RP here http://logs.openstack.org/75/597175/1/check/puppet-openstack-integration-4-scenario001-tempest-debian-stable-luminous/fd38fcf/logs/nova/nova-compute.txt.gz#_2018-08-28_17_09_40_686 | |
| 14:16:48 | mriedem | here we POST to placement http://logs.openstack.org/75/597175/1/check/puppet-openstack-integration-4-scenario001-tempest-debian-stable-luminous/fd38fcf/logs/nova/nova-placement-api.txt.gz#_2018-08-28_17_09_40_683 | |
| 14:16:57 | zigo | mriedem: What I can do is artificially change the default 0.0 allocation ratio in the package. Would you advise me to do that? | |
| 14:17:18 | mriedem | here we add inventory http://logs.openstack.org/75/597175/1/check/puppet-openstack-integration-4-scenario001-tempest-debian-stable-luminous/fd38fcf/logs/nova/nova-placement-api.txt.gz#_2018-08-28_17_09_40_794 | |
| 14:18:08 | mriedem | zigo: i would probably not recommend that at this time no | |
| 14:18:19 | mriedem | i personally would like to figure out what we're dealing with first | |
| 14:18:29 | cdent | agreed | |
| 14:18:50 | cdent | as far as I can tell the nova-compute is never logging when it sets inventory? | |
| 14:18:59 | mriedem | no, it used to | |
| 14:20:12 | openstackgerrit | Merged openstack/osc-placement master: Add image link in README.rst https://review.openstack.org/586839 | |
| 14:20:25 | efried | stephenfin: Done (though I'm not a stable core): https://review.openstack.org/597421 | |
| 14:20:52 | stephenfin | efried: Ah, indeed. Good enough though | |
| 14:21:51 | dansmith | mriedem: we did discuss having the compute node not override the allocation ratio set on a provider, only use that value when creating it | |
| 14:21:55 | dansmith | did that ever happen? | |
| 14:22:07 | dansmith | I don't see those extra values in conf/compute | |
| 14:22:18 | mriedem | i don't remember that happening no | |
| 14:22:38 | dansmith | https://review.openstack.org/#/c/552105/ | |
| 14:23:21 | mriedem | yup, i have unanswered questions in there | |
| 14:23:36 | dansmith | yep, just offering evidence that it didn't | |
| 14:24:05 | mriedem | efried: would be helpful if provider tree's _update_generation method took an "operation" param or something, | |
| 14:24:12 | mriedem | i.e. we're updating the generation b/c inventory was updated or something | |
| 14:24:22 | sean-k-mooney | mriedem: i might add a live migration item to the nova-neutron ptg slot. | |
| 14:24:45 | sean-k-mooney | mriedem: i have stated to address the neutron bugs i have found but looks like there are other nova ones too | |
| 14:24:53 | efried | kosamara: I started going through the spec to make some content, but found it's really going to be easier if we can both post our updates to gerrit. | |
| 14:25:08 | efried | kosamara: I also wanted to know how much you'd looked at the cyborg project, if at all. | |
| 14:25:24 | mriedem | zigo: let me know if/when you have a nova bug posted and i can push some debug patches | |
| 14:25:59 | mriedem | so, | |
| 14:26:11 | efried | mriedem: I would expect the report client side to be logging what it's doing there. | |
| 14:26:12 | zigo | mriedem: Yeah, here: https://bugs.launchpad.net/nova/+bug/1789654 | |
| 14:26:12 | openstack | Launchpad bug 1789654 in OpenStack Compute (nova) "placement allocation_ratio initialized with 0.0" [Undecided,New] | |