| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 15:09:39 | mriedem | dansmith: as far as i know | |
| 15:09:46 | mriedem | but i learn new terrible things every day :) | |
| 15:09:46 | dansmith | cdent: pcpu in the case of \this dedicated thing | |
| 15:10:38 | cdent | dansmith: gotchya | |
| 15:11:00 | namnh | Hi everyone, I am reading the rolling upgrade in Nova, there is a point that make me confuse. According to the docs, we have to gracefully shutdown all Nova services except nova-compute. In my understanding, this will ensures no old server (like nova-api, nova-conductor, nova-scheduler) won't interact with new schema DB. | |
| 15:11:12 | namnh | But in the docs was written "Install the code for the next version of Nova, either in a venv or a separate control plane node, including all the python dependencies. Using the newly installed nova code, run the DB sync. (nova-manage db sync; nova-manage api_db sync)." it means there will have period, old services interact with new schema DB. Could you please explain for me this point? | |
| 15:11:29 | namnh | Here is the docs: https://docs.openstack.org/developer/nova/upgrade.html#minimal-downtime-upgrade-process | |
| 15:11:36 | edleafe | dansmith: so those who are configuring flavors this way are already sacrificed a pcpu now, right? IOW, this wouldn't change it? | |
| 15:11:59 | edleafe | s/sacrificed/sacrificiing | |
| 15:12:14 | dansmith | namnh: nova compute does not talk to the db directly, so it won't touch the different schema | |
| 15:12:34 | dansmith | edleafe: they are, but only at claim time, not at reserved-for-the-future time | |
| 15:12:55 | dansmith | edleafe: since that is not config, but a flavor setting, we need a config to know if we _should_ reserve that or not | |
| 15:13:08 | edleafe | dansmith: ok, but in your 4 core example, they can only use 3 in either case | |
| 15:13:13 | dansmith | and of course, we should validate all this with the people that wrote this code :) | |
| 15:13:25 | dansmith | edleafe: but people that don't use that dedicated thing.. those are the ones we careabout | |
| 15:13:41 | dansmith | edleafe: because we don't want to reserve (and thus never schedule to) the fourth cpu | |
| 15:14:10 | edleafe | dansmith: so maybe add a config option to compute: WASTE_PCPU = True | |
| 15:14:17 | edleafe | :) | |
| 15:14:17 | dansmith | edleafe: exactly what I'm saying | |
| 15:14:58 | edleafe | and these are a small minority of deployments? | |
| 15:16:05 | namnh | dansmith: Yes, I know. we just have to gracefully shutdown all nova server *except* nova-compute. And in my understading, we have to do that to ensure no nova service (like nova-api, nova-condutor, nova-scheduler) can interact with new schema DB | |
| 15:16:08 | dansmith | I would definitely imagine so | |
| 15:16:12 | dansmith | edleafe: it's an expensive change to make RT performance better | |
| 15:16:13 | namnh | dansmith: is that right? | |
| 15:16:34 | namnh | s/server/services | |
| 15:16:41 | gcb | mriedem, for the first issue you mentioned in the email , we can revert the change of file https://review.openstack.org/#/c/457188/3/nova/tests/unit/virt/ironic/test_driver.py | |
| 15:16:52 | dansmith | namnh: no, you can apply the schema change before you shut down the control services | |
| 15:17:21 | mriedem | the schema changes are always additive so it shouldn't matter right? | |
| 15:17:24 | dansmith | namnh: that's why the docs say "or venv". You can apply the schema before you do anything, then shut down and upgrade your control services at the same time, then start on computes | |
| 15:17:27 | dansmith | mriedem: right | |
| 15:17:29 | gcb | mriedem, I have no idea why it fails randomly now | |
| 15:17:38 | mriedem | gcb: melwitt is going to re-write the test | |
| 15:18:39 | gcb | mriedem, good to know that, for the second issue, when the issue occurred ? From the Monday ? | |
| 15:18:45 | mriedem | namnh: see the 3rd bullet there which says "At this point, new columns and tables may exist in the database. These DB schema changes are done in a way that both the N and N+1 release can perform operations against the same schema." | |
| 15:19:07 | mriedem | namnh: meaning the schema changes are all additive and won't break old code running against them, | |
| 15:19:13 | mriedem | like any new column is always nullable | |
| 15:19:18 | mriedem | and we don't drop columns | |
| 15:20:14 | namnh | dansmith: so why we have to shutdown all nova services (except nova-compute)? due to RPC or something? | |
| 15:20:56 | dansmith | namnh: you only have to shut them down when you upgrade them, and all of them have to be shutdown together | |
| 15:21:13 | dansmith | namnh: what you can't have is nova-api and nova-scheduler on different *code* versions accessing the database at the same time | |
| 15:22:08 | mriedem | "For maximum safety (no failed API operations)," | |
| 15:22:40 | dansmith | it says that? | |
| 15:22:41 | mriedem | namnh: the control services have code in place to check versions of the computes since we support mixed level computes, but we don't for the control services | |
| 15:22:43 | mriedem | dansmith: yeah | |
| 15:22:44 | dansmith | that's not really the reason | |
| 15:22:55 | mriedem | see step 2 bullet 1 | |
| 15:23:26 | dansmith | maybe that's easier to understand than the real reason, I dunno | |
| 15:24:17 | namnh | dansmith mriedem: if possible, could you give me an example for error when nova-api and nova-scheduler on different *code* versions accessing the database at the same time? | |
| 15:24:38 | dansmith | namnh: data corruption | |
| 15:24:48 | dansmith | seriously, don't do it | |
| 15:26:07 | namnh | dansmith: ok, I understood. Thanks for your time. | |
| 15:28:14 | namnh | mriedem: I just have understood. thanks | |
| 15:45:22 | dr_gogeta86 | hi | |
| 15:45:41 | dr_gogeta86 | how can I force a compute node up | |
| 15:45:41 | dr_gogeta86 | ? | |
| 15:45:52 | dr_gogeta86 | i've also rebooted many times but is always down | |
| 15:47:10 | efried | andymccr https://review.openstack.org/#/c/455710/12 - Pretty. So pretty. | |
| 15:47:36 | imacdonn | dr_gogeta86, I think you're supposed to ask questions like that (i.e. how to make it work, not regarding code development) in #openstack | |
| 15:47:57 | imacdonn | see topic ;) | |
| 15:48:01 | dr_gogeta86 | sorry | |
| 15:48:04 | imacdonn | np | |
| 15:48:05 | efried | dr_gogeta86 Yeah, what imacdonn said - but, check your n-cpu log | |
| 15:48:31 | efried | Sounds like compute service is failing to start or check in. | |
| 15:49:03 | cdent | efried: "save, like, five whole lines of code" <- my favorite review comment today, can hear it in my head | |
| 15:49:15 | efried | cdent :) | |
| 15:51:42 | dr_gogeta86 | efried, imacdonn tnx | |
| 15:51:55 | mriedem | cdent: to answer your question in the spec, i don't think we account for overheads when commiting the quota reservation | |
| 15:52:08 | mriedem | cdent: because the quota reservation is created in the api based on the flavor, | |
| 15:52:16 | cdent | mriedem: that's what I figured, thus my concern | |
| 15:52:21 | cdent | thanks for confirming it | |
| 15:52:22 | mriedem | cdent: then the reservation id for the quota is passed through to compute | |
| 15:52:30 | mriedem | and if the claim is good, we commit the reservation on the compute | |
| 15:52:34 | mriedem | but we don't modify it at all | |
| 15:53:07 | cdent | so if we make allocations be the true count, if they include overhead, user is going to be doing WTF sometimes | |
| 15:53:26 | mriedem | now that i say this, i don't see us passing reservations to build_and_run_instance | |
| 15:53:31 | mriedem | so i'm digging | |
| 15:53:31 | cdent | gyres within gyres | |
| 15:54:23 | cdent | what rough quota, its limit come round at last | |
| 15:56:48 | mriedem | cdent: so i was totally wrong | |
| 15:56:52 | mriedem | we commit the reservations in the api | |
| 15:56:59 | mriedem | in the _provision_instances method | |
| 15:57:19 | cdent | mriedem: that sounds like only partially wrong | |
| 15:57:29 | mriedem | either way we don't consider the overhead in the quota | |
| 15:57:33 | mriedem | quota usage i mean | |
| 15:57:46 | mriedem | had to fix that before melwitt dropped the 'quota usage' hammer on me | |
| 15:58:22 | melwitt | lol | |
| 15:59:05 | cdent | so many hammers, so few channels | |
| 16:00:00 | mriedem | ok, so yeah if we don't account for the overhead in the quota usage, | |
| 16:00:03 | mriedem | but we did in the allocation, | |
| 16:00:19 | mriedem | and we start using allocations (quota usage) for counting quotas stuff that mel is doing, | |
| 16:00:23 | mriedem | we're in a bind | |
| 16:01:01 | mriedem | because horizon shows me that i have 4GB of memory quota left, but the allocation is really 4GB + 256MB RAM | |
| 16:01:20 | mriedem | however, that's already the state we're in today | |
| 16:01:32 | mriedem | you're getting that extra overhead from the flavor w/o paying for it | |
| 16:02:04 | mriedem | if we account for the overhead in the allocations, and we use allocations for reporting quota, then we actually fix that hole don't we? | |
| 16:02:11 | mriedem | so you can bill your customers based on real usage? | |
| 16:02:45 | cdent | people will not like that change | |
| 16:02:51 | mriedem | define people | |
| 16:02:58 | cdent | end users | |
| 16:03:12 | cdent | "yesterday i was able to do X but today I can do x-1" | |