| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 15:06:28 | mriedem | with get_inventory() we could make the reservation calculation in the virt drivers that actually do this overhead thing | |
| 15:06:55 | cdent | yes | |
| 15:07:06 | dansmith | sure, but libvirt needs to not do that if the operator doesn't want it, | |
| 15:07:17 | dansmith | because they may have no flavors configured for this dedicated thing, | |
| 15:07:23 | dansmith | and we don't want to prevent them from using their last vcpu :) | |
| 15:07:45 | mriedem | are you talking to me or cdent? | |
| 15:07:49 | cdent | how much does that matter (the last vcpu) | |
| 15:07:56 | dansmith | you | |
| 15:07:56 | cdent | mriedem: yes, about get_inventory, I like get_inventory | |
| 15:08:08 | dansmith | heh | |
| 15:08:09 | dansmith | cdent: vcpus are few | |
| 15:08:10 | dansmith | cdent: compared to memory or disk | |
| 15:08:15 | bauzas | sorry, in a meeting atm | |
| 15:08:24 | dansmith | cdent: imagine a cloud of 4-core ARM boxes | |
| 15:08:32 | dansmith | where you lose one core per system | |
| 15:08:54 | cdent | are we losing a pcpu or a vcpu? | |
| 15:09:04 | mriedem | dansmith: "libvirt needs to not do that if the operator doesn't want it," - we just take that into account in the libvirt get_inventory() method don't we? | |
| 15:09:15 | mriedem | when determining the reserved value for the VCPU resource provider inventory? | |
| 15:09:16 | dansmith | mriedem: yeah, it just has to be configurable | |
| 15:09:16 | cdent | (physical, not pinned) | |
| 15:09:25 | dansmith | mriedem: and we don't have that currently, AFAIK | |
| 15:09:32 | dansmith | mriedem: we have reserved disk and memory only right? | |
| 15:09:39 | mriedem | dansmith: as far as i know | |
| 15:09:46 | mriedem | but i learn new terrible things every day :) | |
| 15:09:46 | dansmith | cdent: pcpu in the case of \this dedicated thing | |
| 15:10:38 | cdent | dansmith: gotchya | |
| 15:11:00 | namnh | Hi everyone, I am reading the rolling upgrade in Nova, there is a point that make me confuse. According to the docs, we have to gracefully shutdown all Nova services except nova-compute. In my understanding, this will ensures no old server (like nova-api, nova-conductor, nova-scheduler) won't interact with new schema DB. | |
| 15:11:12 | namnh | But in the docs was written "Install the code for the next version of Nova, either in a venv or a separate control plane node, including all the python dependencies. Using the newly installed nova code, run the DB sync. (nova-manage db sync; nova-manage api_db sync)." it means there will have period, old services interact with new schema DB. Could you please explain for me this point? | |
| 15:11:29 | namnh | Here is the docs: https://docs.openstack.org/developer/nova/upgrade.html#minimal-downtime-upgrade-process | |
| 15:11:36 | edleafe | dansmith: so those who are configuring flavors this way are already sacrificed a pcpu now, right? IOW, this wouldn't change it? | |
| 15:11:59 | edleafe | s/sacrificed/sacrificiing | |
| 15:12:14 | dansmith | namnh: nova compute does not talk to the db directly, so it won't touch the different schema | |
| 15:12:34 | dansmith | edleafe: they are, but only at claim time, not at reserved-for-the-future time | |
| 15:12:55 | dansmith | edleafe: since that is not config, but a flavor setting, we need a config to know if we _should_ reserve that or not | |
| 15:13:08 | edleafe | dansmith: ok, but in your 4 core example, they can only use 3 in either case | |
| 15:13:13 | dansmith | and of course, we should validate all this with the people that wrote this code :) | |
| 15:13:25 | dansmith | edleafe: but people that don't use that dedicated thing.. those are the ones we careabout | |
| 15:13:41 | dansmith | edleafe: because we don't want to reserve (and thus never schedule to) the fourth cpu | |
| 15:14:10 | edleafe | dansmith: so maybe add a config option to compute: WASTE_PCPU = True | |
| 15:14:17 | edleafe | :) | |
| 15:14:17 | dansmith | edleafe: exactly what I'm saying | |
| 15:14:58 | edleafe | and these are a small minority of deployments? | |
| 15:16:05 | namnh | dansmith: Yes, I know. we just have to gracefully shutdown all nova server *except* nova-compute. And in my understading, we have to do that to ensure no nova service (like nova-api, nova-condutor, nova-scheduler) can interact with new schema DB | |
| 15:16:08 | dansmith | I would definitely imagine so | |
| 15:16:12 | dansmith | edleafe: it's an expensive change to make RT performance better | |
| 15:16:13 | namnh | dansmith: is that right? | |
| 15:16:34 | namnh | s/server/services | |
| 15:16:41 | gcb | mriedem, for the first issue you mentioned in the email , we can revert the change of file https://review.openstack.org/#/c/457188/3/nova/tests/unit/virt/ironic/test_driver.py | |
| 15:16:52 | dansmith | namnh: no, you can apply the schema change before you shut down the control services | |
| 15:17:21 | mriedem | the schema changes are always additive so it shouldn't matter right? | |
| 15:17:24 | dansmith | namnh: that's why the docs say "or venv". You can apply the schema before you do anything, then shut down and upgrade your control services at the same time, then start on computes | |
| 15:17:27 | dansmith | mriedem: right | |
| 15:17:29 | gcb | mriedem, I have no idea why it fails randomly now | |
| 15:17:38 | mriedem | gcb: melwitt is going to re-write the test | |
| 15:18:39 | gcb | mriedem, good to know that, for the second issue, when the issue occurred ? From the Monday ? | |
| 15:18:45 | mriedem | namnh: see the 3rd bullet there which says "At this point, new columns and tables may exist in the database. These DB schema changes are done in a way that both the N and N+1 release can perform operations against the same schema." | |
| 15:19:07 | mriedem | namnh: meaning the schema changes are all additive and won't break old code running against them, | |
| 15:19:13 | mriedem | like any new column is always nullable | |
| 15:19:18 | mriedem | and we don't drop columns | |
| 15:20:14 | namnh | dansmith: so why we have to shutdown all nova services (except nova-compute)? due to RPC or something? | |
| 15:20:56 | dansmith | namnh: you only have to shut them down when you upgrade them, and all of them have to be shutdown together | |
| 15:21:13 | dansmith | namnh: what you can't have is nova-api and nova-scheduler on different *code* versions accessing the database at the same time | |
| 15:22:08 | mriedem | "For maximum safety (no failed API operations)," | |
| 15:22:40 | dansmith | it says that? | |
| 15:22:41 | mriedem | namnh: the control services have code in place to check versions of the computes since we support mixed level computes, but we don't for the control services | |
| 15:22:43 | mriedem | dansmith: yeah | |
| 15:22:44 | dansmith | that's not really the reason | |
| 15:22:55 | mriedem | see step 2 bullet 1 | |
| 15:23:26 | dansmith | maybe that's easier to understand than the real reason, I dunno | |
| 15:24:17 | namnh | dansmith mriedem: if possible, could you give me an example for error when nova-api and nova-scheduler on different *code* versions accessing the database at the same time? | |
| 15:24:38 | dansmith | namnh: data corruption | |
| 15:24:48 | dansmith | seriously, don't do it | |
| 15:26:07 | namnh | dansmith: ok, I understood. Thanks for your time. | |
| 15:28:14 | namnh | mriedem: I just have understood. thanks | |
| 15:45:22 | dr_gogeta86 | hi | |
| 15:45:41 | dr_gogeta86 | how can I force a compute node up | |
| 15:45:41 | dr_gogeta86 | ? | |
| 15:45:52 | dr_gogeta86 | i've also rebooted many times but is always down | |
| 15:47:10 | efried | andymccr https://review.openstack.org/#/c/455710/12 - Pretty. So pretty. | |
| 15:47:36 | imacdonn | dr_gogeta86, I think you're supposed to ask questions like that (i.e. how to make it work, not regarding code development) in #openstack | |
| 15:47:57 | imacdonn | see topic ;) | |
| 15:48:01 | dr_gogeta86 | sorry | |
| 15:48:04 | imacdonn | np | |
| 15:48:05 | efried | dr_gogeta86 Yeah, what imacdonn said - but, check your n-cpu log | |
| 15:48:31 | efried | Sounds like compute service is failing to start or check in. | |
| 15:49:03 | cdent | efried: "save, like, five whole lines of code" <- my favorite review comment today, can hear it in my head | |
| 15:49:15 | efried | cdent :) | |
| 15:51:42 | dr_gogeta86 | efried, imacdonn tnx | |
| 15:51:55 | mriedem | cdent: to answer your question in the spec, i don't think we account for overheads when commiting the quota reservation | |
| 15:52:08 | mriedem | cdent: because the quota reservation is created in the api based on the flavor, | |
| 15:52:16 | cdent | mriedem: that's what I figured, thus my concern | |
| 15:52:21 | cdent | thanks for confirming it | |
| 15:52:22 | mriedem | cdent: then the reservation id for the quota is passed through to compute | |
| 15:52:30 | mriedem | and if the claim is good, we commit the reservation on the compute | |
| 15:52:34 | mriedem | but we don't modify it at all | |
| 15:53:07 | cdent | so if we make allocations be the true count, if they include overhead, user is going to be doing WTF sometimes | |
| 15:53:26 | mriedem | now that i say this, i don't see us passing reservations to build_and_run_instance | |
| 15:53:31 | mriedem | so i'm digging | |
| 15:53:31 | cdent | gyres within gyres | |
| 15:54:23 | cdent | what rough quota, its limit come round at last | |