| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-19 | |||
| 12:56:44 | mriedem | bauzas: were you witholding a +W on https://review.openstack.org/#/c/610088/ for some reason? | |
| 12:56:46 | SteelyDan | mriedem: does this ring any bells? http://logs.openstack.org/58/591658/12/check/tempest-full-py3/5224550/controller/logs/screen-n-api.txt.gz?level=TRACE#_Oct_18_20_49_20_086138 | |
| 12:56:47 | mriedem | even though you were +2? | |
| 12:57:10 | SteelyDan | there's a ton of unrelated red in that log, so ignore the rest, but the FK error there doesn't seem related to the patch | |
| 12:57:20 | SteelyDan | and there are also rabbit connection failures later in the log | |
| 12:58:14 | mriedem | hmm, no, also seems weird that we'd get cell0 FK errors for a resize operation.... | |
| 12:58:24 | mriedem | which shouldn't have anything to do with cell0 | |
| 12:58:48 | SteelyDan | it's an action, | |
| 12:58:50 | mriedem | unless it's trying to record an action in the api, | |
| 12:58:54 | SteelyDan | but yeah that clearly looks like we're talking to the wrong db | |
| 12:58:56 | mriedem | and defaulting to the cell0 db connectoin in nova.conf, | |
| 12:59:06 | mriedem | but when we look up the instance we should target the context to cell1 | |
| 12:59:21 | SteelyDan | oh I bet I know | |
| 12:59:26 | mriedem | so, it looks like it's probably shitting b/c it's trying to create an action in cell0 for an instance in cell1 | |
| 12:59:39 | SteelyDan | this changes it so we're not permanently targeting the context.. so the resize goes on to be untargeted | |
| 13:00:22 | SteelyDan | didn't think of that until you made the connection to cell0 | |
| 13:00:34 | SteelyDan | that's also probably why we can't connect to fake:/// or whatever rabbit | |
| 13:00:44 | mriedem | uh huh | |
| 13:02:51 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 13:02:52 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 13:16:23 | bauzas | mriedem: just a miss I guess | |
| 13:16:24 | bauzas | fixed | |
| 13:18:06 | mriedem | danke | |
| 13:18:19 | bauzas | bitte | |
| 13:30:37 | openstackgerrit | Daniel Abad proposed openstack/nova master: Fix ironic client ironic_url deprecation warning https://review.openstack.org/611872 | |
| 14:01:00 | melwitt | ||
| 14:18:29 | openstackgerrit | Stephen Finucane proposed openstack/osc-placement master: Enforce key-value'ness for 'allocation candidate list --resource' https://review.openstack.org/611883 | |
| 14:18:30 | openstackgerrit | Stephen Finucane proposed openstack/osc-placement master: tox: Hide deprecation warnings from stdlib https://review.openstack.org/611884 | |
| 14:49:43 | spatel | I got this error when i reboot one of my instance any idea what is this? http://paste.openstack.org/show/732501/ | |
| 15:27:25 | cfriesen | spatel: looks like corrupt filesystem | |
| 15:28:07 | cfriesen | or at least corrupt something on disk | |
| 15:31:48 | mriedem_afk | i've triaged https://bugs.launchpad.net/nova/+bug/1798805 but not sure if we would ever do anything about it | |
| 15:31:48 | openstack | Launchpad bug 1798805 in OpenStack Compute (nova) "Nova scheduler schedules VMs on nodes where nova-compute is down" [Wishlist,Triaged] | |
| 15:33:07 | finucannot | bauzas: If I delete a compute nodes RP, it'll be recreated, right? | |
| 15:34:04 | SteelyDan | mriedem_afk: I think that means disable in the "break its needs" sort of sense | |
| 15:34:31 | SteelyDan | I'm strongly in favor of keeping disable purely for scheduler reasons, but I think it's saying if the compute is actually down, we shouldn't take action on instances | |
| 15:34:35 | SteelyDan | which is probably reasonable | |
| 15:35:01 | SteelyDan | or make our super awesome bug-free state sync periodic power the instance on when the compute comes back | |
| 15:36:03 | imacdonn | I kinda sorta wish there was a "don't schedule anything new" status, which is different from "don't try to interact with me at all" | |
| 15:36:26 | SteelyDan | imacdonn: that's what disable means | |
| 15:36:29 | SteelyDan | it's the only thing it means | |
| 15:36:43 | imacdonn | the former, you mean | |
| 15:36:59 | SteelyDan | it means don't schedule anything there | |
| 15:37:04 | imacdonn | right | |
| 15:37:31 | SteelyDan | you're saying you wish there was a "kneecap this compute" | |
| 15:37:33 | SteelyDan | API? | |
| 15:37:45 | imacdonn | IMO, it should really mean "don't talk that node at all right now", and there should be some other way to tell the scheduler not to pout anything NEW there | |
| 15:38:23 | finucannot | mriedem_afk, awaugama: More investigation needed but I think there's something wrong with those foo_allocation_ratio options. I configured them on the compute node, waited ages aaand...nada. Deleting the RP fixed things | |
| 15:38:27 | imacdonn | (or maybe the opposite, so keep the existing meaning of "disable") ... but I think they are use-cases for each | |
| 15:38:37 | SteelyDan | we're not changing the existing meaning for disable | |
| 15:38:57 | SteelyDan | we could add another state, but it just means for every single operation we do another db hit to check the state of the host | |
| 15:39:03 | finucannot | mriedem_afk, awaugama: At least, assuming my understanding of how that's _supposed_ to work is correct. It it Friday so maybe it's not | |
| 15:39:24 | imacdonn | "hard_disabled" ? :) | |
| 15:39:31 | SteelyDan | no. | |
| 15:42:13 | imacdonn | possible use-case: node is being taken down for hardware maintenance. operator has shut down the instances on it. We don't want to allow the user to star their instances back up until the maintenance is completed | |
| 15:42:46 | imacdonn | (we do this all the time, in a private cloud context) | |
| 15:44:37 | spatel | cfriesen: I have other VMs running on same compute node they are fine.. very strange issue | |
| 15:45:31 | SteelyDan | imacdonn: yeah I'm sure everyone does.. I get the use case, it's unfortunate to need to check the status on every call, but I get it | |
| 15:53:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api-ref: 'os-hypervisors' doesn't reflect overcommit ratio https://review.openstack.org/611604 | |
| 16:07:26 | cfriesen | imacdonn: so lock the instances? | |
| 16:09:01 | cfriesen | imacdonn: or stop the nova-compute process? | |
| 16:09:08 | imacdonn | cfriesen: I guess, but the owner can unlock it? | |
| 16:09:47 | cfriesen | imacdonn: you could always make lock/unlock admin-only. | |
| 16:09:57 | imacdonn | cfriesen: I guess stopping nova-compute may cause the bug that mriedem_afk was reviewing ... although there are some open questions in the bug | |
| 16:16:10 | cfriesen | I'm not sure it's really a bug...more like a feature to more gracefully handle cases where things get "stuck" in what was supposed to be a transitional state. | |
| 16:52:27 | openstackgerrit | Merged openstack/nova master: Fix deprecated base64.decodestring warning https://review.openstack.org/610401 | |
| 17:00:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fail to live migration if instance has a NUMA topology https://review.openstack.org/611088 | |
| 17:05:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fail to live migration if instance has a NUMA topology https://review.openstack.org/611088 | |
| 17:23:29 | awaugama | finucannot, was at lunch. so you're able to reproduce that setting the value on the compute node doesn't change anything in placement? | |
| 17:29:21 | sean-k-mooney | awaugama: which setting is that? | |
| 17:30:12 | awaugama | cpu_allocation_ratio I believe | |
| 17:30:45 | awaugama | yeah | |
| 17:30:46 | sean-k-mooney | awaugama: there is a spec current for changing how we adress this for stien | |
| 17:31:11 | awaugama | I'm not aware of one | |
| 17:31:38 | sean-k-mooney | awaugama: there are 2 but i belive we have setteled on one ill see if i can find it | |
| 17:31:46 | awaugama | cool | |
| 17:32:57 | sean-k-mooney | awaugama: this is one of the specs https://review.openstack.org/#/c/552105/ | |
| 17:33:25 | sean-k-mooney | i think that is the most recent one | |
| 17:35:03 | sean-k-mooney | this is the other one https://review.openstack.org/#/c/544683/ but im not sure if it will still be requried | |
| 17:35:11 | awaugama | checking | |
| 17:36:02 | awaugama | yeah that looks like what finucannot was talking about being messed up | |
| 17:36:24 | sean-k-mooney | well that depends on what you were expecting | |
| 17:36:36 | sean-k-mooney | awaugama: what was your initall issue | |
| 17:37:07 | awaugama | setting cpu_allocation_ratio to 1 in nova.conf changed the value on the compute node (even on compute node startup it showed it was changed) but placement still has it set to 16 | |
| 17:37:54 | sean-k-mooney | awaugama: right i belive currently it will only use the value if it is first creating the provider | |
| 17:38:16 | sean-k-mooney | if the provider exists it will not update it i think... | |
| 17:38:57 | awaugama | that seems counter-intuitive | |
| 17:39:25 | sean-k-mooney | awaugama: this will be changed going forward to allow the cpu_allocation_ratio confige option to allow specify the placement value too and cpu_inital_allocation_ratio to specify what it shoudl be for newly creeted resouce providers | |
| 17:39:52 | sean-k-mooney | awaugama: the current behavior i belive is intened to allow you to mainge the allocation ratios via the api instead of the config | |
| 17:40:26 | awaugama | interesting. finucannot and mriedem_afk ^ | |
| 17:40:39 | awaugama | i need to step away for a few minutes, brb | |
| 17:41:43 | sean-k-mooney | awaugama: finucannot is likely not around anymore. | |
| 17:42:12 | sean-k-mooney | awaugama: he is usually enjoying his weekend by now :) | |
| 17:50:39 | awaugama | sean-k-mooney, yeah, I figure he'll get the scrollback at somepoint | |
| 17:57:41 | sean-k-mooney | awaugama: the inital allocation ratios spec is likely the one that is most relevent to you https://review.openstack.org/#/c/552105/ | |
| 18:04:31 | spatel | sean-k-mooney: hey! how are you doing :) | |
| 18:04:54 | sean-k-mooney | spatel: quite well thank you. how are you :) | |
| 18:04:57 | spatel | after longtime i am seeing you online or may be i was not paying attention | |
| 18:05:13 | sean-k-mooney | i was on company traing most of this week | |
| 18:05:21 | spatel | I am great!! and my openstack also going great | |