| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 14:59:47 | sfinucan | It's the easiest way to check that out for things like cpu_thread_policy, where we can't do a simple "have I X free CPUs" check | |
| 14:59:51 | dansmith | edleafe: we're claiming before we know that, that's the whole point | |
| 15:00:02 | bauzas | okay, here is a thinh | |
| 15:00:04 | edleafe | dansmith: it would already be accounted for | |
| 15:00:09 | dansmith | cdent: oh, that's what I initially was saying, yeah | |
| 15:00:12 | bauzas | given all the nasty and fancy things we do with RT | |
| 15:00:15 | edleafe | in the RP's reserved | |
| 15:00:17 | bauzas | I have a though | |
| 15:00:38 | bauzas | why not just posting an allocation without all that stuff, and do modify the allocation at the RT state ? | |
| 15:00:43 | dansmith | cdent: not claim the flavor+overhead, but claim flavor, and require reserved to be set high enough to cover it | |
| 15:00:53 | mriedem | bauzas: that's one of the options i said in the spec | |
| 15:00:56 | dansmith | bauzas: because that's terrible | |
| 15:01:00 | edleafe | I see chasing the last tiny bit of resource is going to make the code an even bigger mess than it is now | |
| 15:01:01 | bauzas | ie. we claim for the resouces we know, but we later heal that when we claim | |
| 15:01:17 | bauzas | when we /RT/.claim | |
| 15:01:21 | cdent | dansmith: yes, but doing that in an semi-automated way based on the initial calculated (and config'd) inventory. Updating it at boot time. | |
| 15:01:32 | bauzas | either way, placement doesn't know a bit of that NFV gangband | |
| 15:01:38 | dansmith | cdent: well, whether or not we do that is a detail I guess | |
| 15:01:53 | bauzas | so I think NFV claims would still be a 2-phase step | |
| 15:02:07 | dansmith | cdent: however, I think I lost sight of that as a simple option early on | |
| 15:02:14 | bauzas | ie. rounding the allocation by the scheduler, and refine that allocation as soon as we have a better vision | |
| 15:02:18 | dansmith | cdent: that's probably the way to go, yeah, and do the more complex thing if that ends up being to naive for some reason | |
| 15:02:24 | cdent | dansmith: yes | |
| 15:02:39 | cdent | otherwise we run into some very complex wangling for unclear gains | |
| 15:02:41 | dansmith | cdent: I lost sight of it when we started talking about claiming the flavor+overhead in one go | |
| 15:02:48 | dansmith | cdent: which means we couldn't rely on reserved | |
| 15:03:07 | bauzas | soooo | |
| 15:03:36 | mriedem | dansmith: cdent: so would that be based on reserved_host_disk_mb and reserved_host_memory_mb ? | |
| 15:03:38 | bauzas | could we save my life, and assume that we would POST an allocation that is rounded, and later refine it within the instance_claim phase, exactly like we already do ? | |
| 15:04:09 | bauzas | 90% of users would have benefits, and things wouldn't change (even improve) for NFV users | |
| 15:04:10 | cdent | bauzas: we slightly increase the risk of retry/error if the second claim can't claim | |
| 15:04:22 | dansmith | mriedem: cdent: and vcpu | |
| 15:04:24 | cdent | seems better to not allocation overhead per instance | |
| 15:04:37 | bauzas | cdent: like we already do :) | |
| 15:04:41 | cdent | yes | |
| 15:04:46 | cdent | and instead cover it with reserved | |
| 15:04:48 | cdent | it is safer | |
| 15:04:52 | dansmith | mriedem: cdent: meaning, we'd have to always have at least one reserved vcpu | |
| 15:04:59 | cdent | more likely to have successful claims more often | |
| 15:05:08 | mriedem | dansmith: today in _compute_node_to_inventory_dict in the scheduler report client we don't reserve any vcpu | |
| 15:05:15 | dansmith | right | |
| 15:05:19 | cdent | dansmith: I believe you, but why? | |
| 15:05:33 | dansmith | cdent: because libvirt needs to have vcpu+1 if certain flavor configs are set | |
| 15:05:39 | cdent | ah, okay | |
| 15:06:08 | dansmith | cdent: if you ask for dedicated io threads or something, then it requires another vcpu dedicated to you to handle that | |
| 15:06:12 | mriedem | dansmith: cdent: thinking ahead, _compute_node_to_inventory_dict is only used for the old style driver.get_available_resource call, | |
| 15:06:24 | dansmith | cdent: which is really expensive, and probably unfortunate for people that don't want that, so we'd need a way to not reserve the last vcpu on a system for people that don't want it | |
| 15:06:28 | mriedem | with get_inventory() we could make the reservation calculation in the virt drivers that actually do this overhead thing | |
| 15:06:55 | cdent | yes | |
| 15:07:06 | dansmith | sure, but libvirt needs to not do that if the operator doesn't want it, | |
| 15:07:17 | dansmith | because they may have no flavors configured for this dedicated thing, | |
| 15:07:23 | dansmith | and we don't want to prevent them from using their last vcpu :) | |
| 15:07:45 | mriedem | are you talking to me or cdent? | |
| 15:07:49 | cdent | how much does that matter (the last vcpu) | |
| 15:07:56 | cdent | mriedem: yes, about get_inventory, I like get_inventory | |
| 15:07:56 | dansmith | you | |
| 15:08:08 | dansmith | heh | |
| 15:08:09 | dansmith | cdent: vcpus are few | |
| 15:08:10 | dansmith | cdent: compared to memory or disk | |
| 15:08:15 | bauzas | sorry, in a meeting atm | |
| 15:08:24 | dansmith | cdent: imagine a cloud of 4-core ARM boxes | |
| 15:08:32 | dansmith | where you lose one core per system | |
| 15:08:54 | cdent | are we losing a pcpu or a vcpu? | |
| 15:09:04 | mriedem | dansmith: "libvirt needs to not do that if the operator doesn't want it," - we just take that into account in the libvirt get_inventory() method don't we? | |
| 15:09:15 | mriedem | when determining the reserved value for the VCPU resource provider inventory? | |
| 15:09:16 | cdent | (physical, not pinned) | |
| 15:09:16 | dansmith | mriedem: yeah, it just has to be configurable | |
| 15:09:25 | dansmith | mriedem: and we don't have that currently, AFAIK | |
| 15:09:32 | dansmith | mriedem: we have reserved disk and memory only right? | |
| 15:09:39 | mriedem | dansmith: as far as i know | |
| 15:09:46 | dansmith | cdent: pcpu in the case of \this dedicated thing | |
| 15:09:46 | mriedem | but i learn new terrible things every day :) | |
| 15:10:38 | cdent | dansmith: gotchya | |
| 15:11:00 | namnh | Hi everyone, I am reading the rolling upgrade in Nova, there is a point that make me confuse. According to the docs, we have to gracefully shutdown all Nova services except nova-compute. In my understanding, this will ensures no old server (like nova-api, nova-conductor, nova-scheduler) won't interact with new schema DB. | |
| 15:11:12 | namnh | But in the docs was written "Install the code for the next version of Nova, either in a venv or a separate control plane node, including all the python dependencies. Using the newly installed nova code, run the DB sync. (nova-manage db sync; nova-manage api_db sync)." it means there will have period, old services interact with new schema DB. Could you please explain for me this point? | |
| 15:11:29 | namnh | Here is the docs: https://docs.openstack.org/developer/nova/upgrade.html#minimal-downtime-upgrade-process | |
| 15:11:36 | edleafe | dansmith: so those who are configuring flavors this way are already sacrificed a pcpu now, right? IOW, this wouldn't change it? | |
| 15:11:59 | edleafe | s/sacrificed/sacrificiing | |
| 15:12:14 | dansmith | namnh: nova compute does not talk to the db directly, so it won't touch the different schema | |
| 15:12:34 | dansmith | edleafe: they are, but only at claim time, not at reserved-for-the-future time | |
| 15:12:55 | dansmith | edleafe: since that is not config, but a flavor setting, we need a config to know if we _should_ reserve that or not | |
| 15:13:08 | edleafe | dansmith: ok, but in your 4 core example, they can only use 3 in either case | |
| 15:13:13 | dansmith | and of course, we should validate all this with the people that wrote this code :) | |
| 15:13:25 | dansmith | edleafe: but people that don't use that dedicated thing.. those are the ones we careabout | |
| 15:13:41 | dansmith | edleafe: because we don't want to reserve (and thus never schedule to) the fourth cpu | |
| 15:14:10 | edleafe | dansmith: so maybe add a config option to compute: WASTE_PCPU = True | |
| 15:14:17 | dansmith | edleafe: exactly what I'm saying | |
| 15:14:17 | edleafe | :) | |
| 15:14:58 | edleafe | and these are a small minority of deployments? | |
| 15:16:05 | namnh | dansmith: Yes, I know. we just have to gracefully shutdown all nova server *except* nova-compute. And in my understading, we have to do that to ensure no nova service (like nova-api, nova-condutor, nova-scheduler) can interact with new schema DB | |
| 15:16:08 | dansmith | I would definitely imagine so | |
| 15:16:12 | dansmith | edleafe: it's an expensive change to make RT performance better | |
| 15:16:13 | namnh | dansmith: is that right? | |
| 15:16:34 | namnh | s/server/services | |
| 15:16:41 | gcb | mriedem, for the first issue you mentioned in the email , we can revert the change of file https://review.openstack.org/#/c/457188/3/nova/tests/unit/virt/ironic/test_driver.py | |
| 15:16:52 | dansmith | namnh: no, you can apply the schema change before you shut down the control services | |
| 15:17:21 | mriedem | the schema changes are always additive so it shouldn't matter right? | |
| 15:17:24 | dansmith | namnh: that's why the docs say "or venv". You can apply the schema before you do anything, then shut down and upgrade your control services at the same time, then start on computes | |
| 15:17:27 | dansmith | mriedem: right | |
| 15:17:29 | gcb | mriedem, I have no idea why it fails randomly now | |