| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-03 | |||
| 18:16:11 | mriedem | *it | |
| 18:16:17 | eandersson | It's just with 800+ nodes it gets a bit spammy :D | |
| 18:16:19 | melwitt | Tahvok: ah, k. yeah. so when we released 5.1.1, that means it was tagged as 15.1.1 and pushed to pypi etc as that version | |
| 18:16:24 | mriedem | i had a patch for oslo.log at one point to be able to drop stuff like that in the scheduler to AUDIT log | |
| 18:16:26 | mriedem | below DEBUG | |
| 18:16:43 | mriedem | eandersson: is it debug today? | |
| 18:16:53 | eandersson | INFO | |
| 18:16:56 | melwitt | Tahvok: this was the release patch. I'm not sure if/how openstack-ansible consumes that https://review.openstack.org/564044 | |
| 18:17:02 | eandersson | but we are still on mitaka (so might have changed) | |
| 18:17:22 | mriedem | eandersson: which specific message? | |
| 18:17:40 | mriedem | Successfully synced instances from host ? | |
| 18:18:04 | mriedem | yeah those should probably be debug | |
| 18:18:14 | eandersson | Yea | |
| 18:18:27 | mriedem | the "received <operation> from unknown host" is probably ok at INFO | |
| 18:18:42 | mriedem | but all the other "OH HI I DID SOMETHING I"M SUPPOSED TO DO!!!" is debug | |
| 18:18:45 | mriedem | *should be debug | |
| 18:18:48 | eandersson | I can throw up a quick commit on gerrit | |
| 18:18:52 | mriedem | ack | |
| 18:18:58 | mriedem | just don't get any on me | |
| 18:20:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | |
| 18:20:02 | mriedem | melwitt: this time with 10x more bdmturducken ^ call in the next 20 minutes and we'll double your bdms | |
| 18:20:31 | melwitt | and the peasants rejoiced | |
| 18:21:02 | Tahvok | melwitt: mriedem thanks a lot for helping with the issue! And didn't know what was going on... | |
| 18:21:16 | Tahvok | We've had it for a few months now... | |
| 18:21:21 | mriedem | i can't say that's it, but it sounds right | |
| 18:21:34 | mriedem | yeah it was a regression in ocata i believe | |
| 18:21:51 | melwitt | np, I'd been chasing that bug for a long time to get the fix out | |
| 18:49:58 | mriedem | so uh, i don't think this is ever going to be true https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3196 | |
| 18:50:06 | mriedem | is it possible to reboot a rescued instance? | |
| 18:50:09 | mriedem | the api doesn't allow it | |
| 18:50:39 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/vm_states.py#L69-L72 | |
| 18:51:15 | mriedem | is there some other place in nova where we internally reboot an instance when it's rescued? | |
| 18:52:49 | melwitt | I know not | |
| 18:53:13 | mriedem | git blame history here i go | |
| 18:54:01 | mriedem | https://github.com/openstack/nova/commit/39ffe806cdfe97721fee5e48c8591c4c3b52766b | |
| 18:54:22 | openstackgerrit | Merged openstack/nova master: support multiple member_of qparams https://review.openstack.org/561315 | |
| 18:54:25 | mriedem | https://github.com/openstack/nova/blob/39ffe806cdfe97721fee5e48c8591c4c3b52766b/nova/compute/api.py#L1702 | |
| 18:54:32 | mriedem | wtf, you couldn't reboot a rescue vm | |
| 18:55:57 | melwitt | what? that patch is about the vm_state, just making sure it stays "rescued" after the reboot. right? | |
| 18:56:01 | mriedem | https://review.openstack.org/#/q/topic:bug/1170237+(status:open+OR+status:merged) | |
| 18:56:18 | mriedem | yeah, but if you look at the reboot api code in that same hash, we don't allow rebooting a rescued vm | |
| 18:56:28 | melwitt | oh :\ | |
| 18:56:35 | mriedem | i mean, we don't allow rebooting an instance with vm_state=RESCUE | |
| 18:56:37 | mriedem | https://review.openstack.org/#/q/topic:bug/1170237+(status:open+OR+status:merged) | |
| 18:56:49 | mriedem | looks like some rax/xen thing | |
| 18:56:54 | mriedem | that likely never made the api change upstream | |
| 18:56:57 | mriedem | gd rax | |
| 18:57:25 | melwitt | guh, weird. so we have patches that say "now you can reboot a rescued instance" but you can't because the API kicks you out | |
| 18:57:38 | mriedem | the rax api code probably lets you | |
| 18:57:41 | melwitt | right | |
| 18:57:51 | mriedem | where was that one guy that still works at rax? | |
| 18:58:01 | mriedem | the guy always in korea.. | |
| 18:58:08 | melwitt | tbh, I don't know whether we're suppose to be able to reboot rescued instances. I'm not that familiar with the "rescue" function | |
| 18:58:18 | mriedem | neither am i, | |
| 18:58:23 | mriedem | but i can read code :) | |
| 18:58:25 | mriedem | that's how i found this | |
| 18:58:52 | melwitt | well, yeah. I mean whether or not it makes sense. I would guess from the lack of user complaints that it's not usual to try to reboot one | |
| 18:59:06 | mriedem | mikal: johnthetubaguy: when you're around, maybe you can sort this out - did rax have a proprietary change to allow rebooting rescued instances? https://review.openstack.org/#/q/topic:bug/1170237+(status:open+OR+status:merged) | |
| 18:59:24 | mriedem | because the upstream api doesn't allow that | |
| 18:59:36 | mriedem | see https://github.com/openstack/nova/blob/master/nova/compute/vm_states.py#L69-L72 | |
| 19:09:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix being able to hard reboot a pausing instance https://review.openstack.org/566143 | |
| 19:26:58 | openstackgerrit | Oliver Walsh proposed openstack/nova master: Fix handling of connect issues in _ensure_resource_provider https://review.openstack.org/566148 | |
| 19:28:08 | mriedem | owalsh: i already have a patch up for that | |
| 19:28:13 | mriedem | beat you by a few hours | |
| 19:28:27 | mriedem | https://review.openstack.org/#/c/566096/ | |
| 19:29:18 | owalsh | mriedem: snap | |
| 19:32:27 | owalsh | mriedem: worse on pike, it cache None | |
| 19:32:42 | mriedem | yeah it was a backport regression | |
| 19:33:58 | melwitt | oh, so _that's_ where the None RP comes from. geesh | |
| 19:34:22 | melwitt | I remember we reverted a thing while we were trying to get backports done for the stable releases | |
| 19:34:48 | mriedem | yeah that was ocata | |
| 19:35:00 | mriedem | i suspect the fleetify devstack stuff in pike+ was hiding it for us | |
| 19:35:01 | mriedem | somehow | |
| 19:35:28 | mriedem | although even in ocata devstack i thought we started the compute last | |
| 19:44:00 | mriedem | dansmith: what do you think a safe batch size is for this heal allocations CLI? at first i default to CONF.api.max_limit but that's 1000 which seems way too big https://review.openstack.org/#/c/565886/5/nova/cmd/manage.py@1776 - map_instances defaults to 50 | |
| 19:44:05 | mriedem | so was thinking about using 50 | |
| 19:44:21 | dansmith | yeah 1000 is too much | |
| 19:44:24 | dansmith | 50 is probably good | |
| 19:44:25 | mriedem | online_data_migrations also does 50 | |
| 19:44:32 | dansmith | yup | |
| 19:44:54 | melwitt | did we do an audit of other uses of safe_connect where None can be returned? | |
| 19:45:11 | mriedem | melwitt: jaypipes has been working on untangling that | |
| 19:45:18 | melwitt | k, cool | |
| 19:59:36 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix wrong arguments for 'detach_volume' https://review.openstack.org/566152 | |
| 20:06:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-manage placement heal_allocations CLI https://review.openstack.org/565886 | |
| 20:11:38 | mriedem | owalsh: if you want to do the backport to queens and pike for https://review.openstack.org/#/c/566096/ that would speed things along so i can +2 the backports | |
| 20:11:52 | mriedem | owalsh: beware: the provider tree stuff in rocky will likely mean merge conflicts for the backports | |
| 20:16:59 | arvindn05 | dansmith: looks like there was agreement on the rebuild instance with traits thread. Can you send out an update on ML on the final approach? | |
| 20:17:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153 | |
| 20:17:43 | dansmith | arvindn05: I started it a bit ago but got distracted.. if there is agreement you're not blocked right? | |
| 20:23:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | |
| 20:24:17 | arvindn05 | dansmith: i am blocked at this point because my approved patch https://review.openstack.org/#/c/560596/ is also holding based on the decision on the rebuild issue | |
| 20:25:15 | arvindn05 | my next patches would be update the spec with decision on rebuild and propose the code patch for the same | |
| 20:25:24 | dansmith | arvindn05: you mean you're blocked because your patches don't do what mriedem wants yeah? | |
| 20:25:48 | dansmith | arvindn05: if you'd do what we said in the meeting this morning, then he'd remove his -W and everything would move along, AFAICT | |
| 20:26:53 | arvindn05 | dansmith: yup. but i was unfortunately not in the meeting and not entirely sure what approach was decided | |
| 20:27:12 | dansmith | if only there was a log... | |
| 20:27:34 | arvindn05 | dansmith: is it fair to summarize it as we want to go check the allocations route | |
| 20:27:50 | arvindn05 | (8:25:17 AM) efried: arvindn05: That was the impression I got. But yeah, let's see what dansmith has to say. | |
| 20:27:59 | dansmith | arvindn05: yes | |