| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 15:17:24 | dansmith | namnh: that's why the docs say "or venv". You can apply the schema before you do anything, then shut down and upgrade your control services at the same time, then start on computes | |
| 15:17:27 | dansmith | mriedem: right | |
| 15:17:29 | gcb | mriedem, I have no idea why it fails randomly now | |
| 15:17:38 | mriedem | gcb: melwitt is going to re-write the test | |
| 15:18:39 | gcb | mriedem, good to know that, for the second issue, when the issue occurred ? From the Monday ? | |
| 15:18:45 | mriedem | namnh: see the 3rd bullet there which says "At this point, new columns and tables may exist in the database. These DB schema changes are done in a way that both the N and N+1 release can perform operations against the same schema." | |
| 15:19:07 | mriedem | namnh: meaning the schema changes are all additive and won't break old code running against them, | |
| 15:19:13 | mriedem | like any new column is always nullable | |
| 15:19:18 | mriedem | and we don't drop columns | |
| 15:20:14 | namnh | dansmith: so why we have to shutdown all nova services (except nova-compute)? due to RPC or something? | |
| 15:20:56 | dansmith | namnh: you only have to shut them down when you upgrade them, and all of them have to be shutdown together | |
| 15:21:13 | dansmith | namnh: what you can't have is nova-api and nova-scheduler on different *code* versions accessing the database at the same time | |
| 15:22:08 | mriedem | "For maximum safety (no failed API operations)," | |
| 15:22:40 | dansmith | it says that? | |
| 15:22:41 | mriedem | namnh: the control services have code in place to check versions of the computes since we support mixed level computes, but we don't for the control services | |
| 15:22:43 | mriedem | dansmith: yeah | |
| 15:22:44 | dansmith | that's not really the reason | |
| 15:22:55 | mriedem | see step 2 bullet 1 | |
| 15:23:26 | dansmith | maybe that's easier to understand than the real reason, I dunno | |
| 15:24:17 | namnh | dansmith mriedem: if possible, could you give me an example for error when nova-api and nova-scheduler on different *code* versions accessing the database at the same time? | |
| 15:24:38 | dansmith | namnh: data corruption | |
| 15:24:48 | dansmith | seriously, don't do it | |
| 15:26:07 | namnh | dansmith: ok, I understood. Thanks for your time. | |
| 15:28:14 | namnh | mriedem: I just have understood. thanks | |
| 15:45:22 | dr_gogeta86 | hi | |
| 15:45:41 | dr_gogeta86 | how can I force a compute node up | |
| 15:45:41 | dr_gogeta86 | ? | |
| 15:45:52 | dr_gogeta86 | i've also rebooted many times but is always down | |
| 15:47:10 | efried | andymccr https://review.openstack.org/#/c/455710/12 - Pretty. So pretty. | |
| 15:47:36 | imacdonn | dr_gogeta86, I think you're supposed to ask questions like that (i.e. how to make it work, not regarding code development) in #openstack | |
| 15:47:57 | imacdonn | see topic ;) | |
| 15:48:01 | dr_gogeta86 | sorry | |
| 15:48:04 | imacdonn | np | |
| 15:48:05 | efried | dr_gogeta86 Yeah, what imacdonn said - but, check your n-cpu log | |
| 15:48:31 | efried | Sounds like compute service is failing to start or check in. | |
| 15:49:03 | cdent | efried: "save, like, five whole lines of code" <- my favorite review comment today, can hear it in my head | |
| 15:49:15 | efried | cdent :) | |
| 15:51:42 | dr_gogeta86 | efried, imacdonn tnx | |
| 15:51:55 | mriedem | cdent: to answer your question in the spec, i don't think we account for overheads when commiting the quota reservation | |
| 15:52:08 | mriedem | cdent: because the quota reservation is created in the api based on the flavor, | |
| 15:52:16 | cdent | mriedem: that's what I figured, thus my concern | |
| 15:52:21 | cdent | thanks for confirming it | |
| 15:52:22 | mriedem | cdent: then the reservation id for the quota is passed through to compute | |
| 15:52:30 | mriedem | and if the claim is good, we commit the reservation on the compute | |
| 15:52:34 | mriedem | but we don't modify it at all | |
| 15:53:07 | cdent | so if we make allocations be the true count, if they include overhead, user is going to be doing WTF sometimes | |
| 15:53:26 | mriedem | now that i say this, i don't see us passing reservations to build_and_run_instance | |
| 15:53:31 | mriedem | so i'm digging | |
| 15:53:31 | cdent | gyres within gyres | |
| 15:54:23 | cdent | what rough quota, its limit come round at last | |
| 15:56:48 | mriedem | cdent: so i was totally wrong | |
| 15:56:52 | mriedem | we commit the reservations in the api | |
| 15:56:59 | mriedem | in the _provision_instances method | |
| 15:57:19 | cdent | mriedem: that sounds like only partially wrong | |
| 15:57:29 | mriedem | either way we don't consider the overhead in the quota | |
| 15:57:33 | mriedem | quota usage i mean | |
| 15:57:46 | mriedem | had to fix that before melwitt dropped the 'quota usage' hammer on me | |
| 15:58:22 | melwitt | lol | |
| 15:59:05 | cdent | so many hammers, so few channels | |
| 16:00:00 | mriedem | ok, so yeah if we don't account for the overhead in the quota usage, | |
| 16:00:03 | mriedem | but we did in the allocation, | |
| 16:00:19 | mriedem | and we start using allocations (quota usage) for counting quotas stuff that mel is doing, | |
| 16:00:23 | mriedem | we're in a bind | |
| 16:01:01 | mriedem | because horizon shows me that i have 4GB of memory quota left, but the allocation is really 4GB + 256MB RAM | |
| 16:01:20 | mriedem | however, that's already the state we're in today | |
| 16:01:32 | mriedem | you're getting that extra overhead from the flavor w/o paying for it | |
| 16:02:04 | mriedem | if we account for the overhead in the allocations, and we use allocations for reporting quota, then we actually fix that hole don't we? | |
| 16:02:11 | mriedem | so you can bill your customers based on real usage? | |
| 16:02:45 | cdent | people will not like that change | |
| 16:02:51 | mriedem | define people | |
| 16:02:58 | cdent | end users | |
| 16:03:12 | cdent | "yesterday i was able to do X but today I can do x-1" | |
| 16:03:30 | mriedem | i imagine cloud providers would like it though | |
| 16:03:39 | mriedem | more cash for the utility company | |
| 16:03:43 | cdent | yeah, like tron or whatever, I fight for the users | |
| 16:04:30 | mriedem | cdent: just once, join evil corp | |
| 16:04:30 | melwitt | "we fixed the glitch" | |
| 16:04:59 | mriedem | bauzas: dansmith: jaypipes: i left a summary in the spec | |
| 16:05:02 | mriedem | https://review.openstack.org/#/c/437424/ | |
| 16:05:06 | cdent | mriedem: I'm not sure I have any soul left to sell | |
| 16:05:11 | jaypipes | mriedem: danke | |
| 16:05:34 | mriedem | at a high level i feel like we should be accounting for the overhead (real usage) in the allocations | |
| 16:05:52 | mriedem | which would also benefit the initial select_destinations since placement has the whole story for actual allocations on a provider | |
| 16:05:59 | hypothermic_cat | lyarwood: hi, are you around? | |
| 16:06:44 | lyarwood | hypothermic_cat: hey, what's up? | |
| 16:06:51 | mriedem | i agree that over-estimating the reserved inventory buffer is an easier solution on us, but it also means operators have to be taking that into consideration now when configuring the comptues and doing capacity planning | |
| 16:07:11 | cdent | only if we don't automate it (through magic yet to be determined) | |
| 16:07:14 | dansmith | mriedem: unless we inflate the reserved amount ourselves | |
| 16:07:15 | dansmith | right | |
| 16:07:27 | mriedem | yeah i don't know how we do that yet | |
| 16:07:50 | bauzas | mriedem: okay, will read it | |
| 16:07:52 | mriedem | this is definitely fodder for the forum session on claims in the scheduler | |
| 16:07:56 | bauzas | and will try to make a new rev soon-ish | |
| 16:07:58 | melwitt | I don't remember what the "overhead" is. I've seen it before in the RT | |
| 16:08:10 | mriedem | melwitt: see estimate_instance_overhead in the virt drivers | |
| 16:08:19 | melwitt | or the virt driver | |
| 16:08:43 | bauzas | melwitt: don't put your foot into that ... :p | |
| 16:08:55 | mriedem | it's related to quota so melwitt has to put both feet in | |
| 16:08:56 | melwitt | I have before but then I forgot, so I guess I'm safe | |
| 16:08:58 | bauzas | or take the left one, if you want to be lucky | |