| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 15:04:37 | bauzas | cdent: like we already do :) | |
| 15:04:41 | cdent | yes | |
| 15:04:46 | cdent | and instead cover it with reserved | |
| 15:04:48 | cdent | it is safer | |
| 15:04:52 | dansmith | mriedem: cdent: meaning, we'd have to always have at least one reserved vcpu | |
| 15:04:59 | cdent | more likely to have successful claims more often | |
| 15:05:08 | mriedem | dansmith: today in _compute_node_to_inventory_dict in the scheduler report client we don't reserve any vcpu | |
| 15:05:15 | dansmith | right | |
| 15:05:19 | cdent | dansmith: I believe you, but why? | |
| 15:05:33 | dansmith | cdent: because libvirt needs to have vcpu+1 if certain flavor configs are set | |
| 15:05:39 | cdent | ah, okay | |
| 15:06:08 | dansmith | cdent: if you ask for dedicated io threads or something, then it requires another vcpu dedicated to you to handle that | |
| 15:06:12 | mriedem | dansmith: cdent: thinking ahead, _compute_node_to_inventory_dict is only used for the old style driver.get_available_resource call, | |
| 15:06:24 | dansmith | cdent: which is really expensive, and probably unfortunate for people that don't want that, so we'd need a way to not reserve the last vcpu on a system for people that don't want it | |
| 15:06:28 | mriedem | with get_inventory() we could make the reservation calculation in the virt drivers that actually do this overhead thing | |
| 15:06:55 | cdent | yes | |
| 15:07:06 | dansmith | sure, but libvirt needs to not do that if the operator doesn't want it, | |
| 15:07:17 | dansmith | because they may have no flavors configured for this dedicated thing, | |
| 15:07:23 | dansmith | and we don't want to prevent them from using their last vcpu :) | |
| 15:07:45 | mriedem | are you talking to me or cdent? | |
| 15:07:49 | cdent | how much does that matter (the last vcpu) | |
| 15:07:56 | dansmith | you | |
| 15:07:56 | cdent | mriedem: yes, about get_inventory, I like get_inventory | |
| 15:08:08 | dansmith | heh | |
| 15:08:09 | dansmith | cdent: vcpus are few | |
| 15:08:10 | dansmith | cdent: compared to memory or disk | |
| 15:08:15 | bauzas | sorry, in a meeting atm | |
| 15:08:24 | dansmith | cdent: imagine a cloud of 4-core ARM boxes | |
| 15:08:32 | dansmith | where you lose one core per system | |
| 15:08:54 | cdent | are we losing a pcpu or a vcpu? | |
| 15:09:04 | mriedem | dansmith: "libvirt needs to not do that if the operator doesn't want it," - we just take that into account in the libvirt get_inventory() method don't we? | |
| 15:09:15 | mriedem | when determining the reserved value for the VCPU resource provider inventory? | |
| 15:09:16 | dansmith | mriedem: yeah, it just has to be configurable | |
| 15:09:16 | cdent | (physical, not pinned) | |
| 15:09:25 | dansmith | mriedem: and we don't have that currently, AFAIK | |
| 15:09:32 | dansmith | mriedem: we have reserved disk and memory only right? | |
| 15:09:39 | mriedem | dansmith: as far as i know | |
| 15:09:46 | mriedem | but i learn new terrible things every day :) | |
| 15:09:46 | dansmith | cdent: pcpu in the case of \this dedicated thing | |
| 15:10:38 | cdent | dansmith: gotchya | |
| 15:11:00 | namnh | Hi everyone, I am reading the rolling upgrade in Nova, there is a point that make me confuse. According to the docs, we have to gracefully shutdown all Nova services except nova-compute. In my understanding, this will ensures no old server (like nova-api, nova-conductor, nova-scheduler) won't interact with new schema DB. | |
| 15:11:12 | namnh | But in the docs was written "Install the code for the next version of Nova, either in a venv or a separate control plane node, including all the python dependencies. Using the newly installed nova code, run the DB sync. (nova-manage db sync; nova-manage api_db sync)." it means there will have period, old services interact with new schema DB. Could you please explain for me this point? | |
| 15:11:29 | namnh | Here is the docs: https://docs.openstack.org/developer/nova/upgrade.html#minimal-downtime-upgrade-process | |
| 15:11:36 | edleafe | dansmith: so those who are configuring flavors this way are already sacrificed a pcpu now, right? IOW, this wouldn't change it? | |
| 15:11:59 | edleafe | s/sacrificed/sacrificiing | |
| 15:12:14 | dansmith | namnh: nova compute does not talk to the db directly, so it won't touch the different schema | |
| 15:12:34 | dansmith | edleafe: they are, but only at claim time, not at reserved-for-the-future time | |
| 15:12:55 | dansmith | edleafe: since that is not config, but a flavor setting, we need a config to know if we _should_ reserve that or not | |
| 15:13:08 | edleafe | dansmith: ok, but in your 4 core example, they can only use 3 in either case | |
| 15:13:13 | dansmith | and of course, we should validate all this with the people that wrote this code :) | |
| 15:13:25 | dansmith | edleafe: but people that don't use that dedicated thing.. those are the ones we careabout | |
| 15:13:41 | dansmith | edleafe: because we don't want to reserve (and thus never schedule to) the fourth cpu | |
| 15:14:10 | edleafe | dansmith: so maybe add a config option to compute: WASTE_PCPU = True | |
| 15:14:17 | edleafe | :) | |
| 15:14:17 | dansmith | edleafe: exactly what I'm saying | |
| 15:14:58 | edleafe | and these are a small minority of deployments? | |
| 15:16:05 | namnh | dansmith: Yes, I know. we just have to gracefully shutdown all nova server *except* nova-compute. And in my understading, we have to do that to ensure no nova service (like nova-api, nova-condutor, nova-scheduler) can interact with new schema DB | |
| 15:16:08 | dansmith | I would definitely imagine so | |
| 15:16:12 | dansmith | edleafe: it's an expensive change to make RT performance better | |
| 15:16:13 | namnh | dansmith: is that right? | |
| 15:16:34 | namnh | s/server/services | |
| 15:16:41 | gcb | mriedem, for the first issue you mentioned in the email , we can revert the change of file https://review.openstack.org/#/c/457188/3/nova/tests/unit/virt/ironic/test_driver.py | |
| 15:16:52 | dansmith | namnh: no, you can apply the schema change before you shut down the control services | |
| 15:17:21 | mriedem | the schema changes are always additive so it shouldn't matter right? | |
| 15:17:24 | dansmith | namnh: that's why the docs say "or venv". You can apply the schema before you do anything, then shut down and upgrade your control services at the same time, then start on computes | |
| 15:17:27 | dansmith | mriedem: right | |
| 15:17:29 | gcb | mriedem, I have no idea why it fails randomly now | |
| 15:17:38 | mriedem | gcb: melwitt is going to re-write the test | |
| 15:18:39 | gcb | mriedem, good to know that, for the second issue, when the issue occurred ? From the Monday ? | |
| 15:18:45 | mriedem | namnh: see the 3rd bullet there which says "At this point, new columns and tables may exist in the database. These DB schema changes are done in a way that both the N and N+1 release can perform operations against the same schema." | |
| 15:19:07 | mriedem | namnh: meaning the schema changes are all additive and won't break old code running against them, | |
| 15:19:13 | mriedem | like any new column is always nullable | |
| 15:19:18 | mriedem | and we don't drop columns | |
| 15:20:14 | namnh | dansmith: so why we have to shutdown all nova services (except nova-compute)? due to RPC or something? | |
| 15:20:56 | dansmith | namnh: you only have to shut them down when you upgrade them, and all of them have to be shutdown together | |
| 15:21:13 | dansmith | namnh: what you can't have is nova-api and nova-scheduler on different *code* versions accessing the database at the same time | |
| 15:22:08 | mriedem | "For maximum safety (no failed API operations)," | |
| 15:22:40 | dansmith | it says that? | |
| 15:22:41 | mriedem | namnh: the control services have code in place to check versions of the computes since we support mixed level computes, but we don't for the control services | |
| 15:22:43 | mriedem | dansmith: yeah | |
| 15:22:44 | dansmith | that's not really the reason | |
| 15:22:55 | mriedem | see step 2 bullet 1 | |
| 15:23:26 | dansmith | maybe that's easier to understand than the real reason, I dunno | |
| 15:24:17 | namnh | dansmith mriedem: if possible, could you give me an example for error when nova-api and nova-scheduler on different *code* versions accessing the database at the same time? | |
| 15:24:38 | dansmith | namnh: data corruption | |
| 15:24:48 | dansmith | seriously, don't do it | |
| 15:26:07 | namnh | dansmith: ok, I understood. Thanks for your time. | |
| 15:28:14 | namnh | mriedem: I just have understood. thanks | |
| 15:45:22 | dr_gogeta86 | hi | |
| 15:45:41 | dr_gogeta86 | how can I force a compute node up | |
| 15:45:41 | dr_gogeta86 | ? | |
| 15:45:52 | dr_gogeta86 | i've also rebooted many times but is always down | |
| 15:47:10 | efried | andymccr https://review.openstack.org/#/c/455710/12 - Pretty. So pretty. | |
| 15:47:36 | imacdonn | dr_gogeta86, I think you're supposed to ask questions like that (i.e. how to make it work, not regarding code development) in #openstack | |
| 15:47:57 | imacdonn | see topic ;) | |
| 15:48:01 | dr_gogeta86 | sorry | |
| 15:48:04 | imacdonn | np | |
| 15:48:05 | efried | dr_gogeta86 Yeah, what imacdonn said - but, check your n-cpu log | |
| 15:48:31 | efried | Sounds like compute service is failing to start or check in. | |
| 15:49:03 | cdent | efried: "save, like, five whole lines of code" <- my favorite review comment today, can hear it in my head | |