| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-07 | |||
| 16:24:35 | mriedem | or talked with artom | |
| 16:32:26 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Ignore MoxStubout deprecation warnings https://review.openstack.org/623545 | |
| 16:52:42 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623546 | |
| 16:56:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623547 | |
| 17:09:02 | artom | mriedem, bauzas, are we seriously OK with the length of those methods? I won't be a pain just for the hell of it, and I understand that sometimes old code grows beyond what we want, but brand new code? | |
| 17:09:02 | nicolasbock | Hi, I have a server on Newton that after a migration attempt shows `task_state = resize_finish`, `status = RESIZE`. I can't resize confirm or start the server. I don't see any obvious error messages in the nova-compute logs on either the old nor the new hypervisor. | |
| 17:09:08 | artom | I dunno, I just have trouble with that | |
| 17:09:24 | nicolasbock | The resized disks have the same md5sum on either hypervisor | |
| 17:09:31 | artom | If everyone's overruling me there's not much I can do, but I wanted to at least make that point | |
| 17:09:46 | mriedem | artom: i'm not disagreeing with your comments | |
| 17:10:00 | nicolasbock | Basically, I am wondering what the problem could be? | |
| 17:10:21 | artom | mriedem, ok, cool, thanks :) | |
| 17:10:45 | mriedem | nicolasbock: looks like something probably failed during the resize (maybe a db change?) where the task_state was not set to None | |
| 17:10:52 | mriedem | so check the compute and conductor logs for errors | |
| 17:11:22 | nicolasbock | the logs on the destination ? | |
| 17:11:29 | nicolasbock | And thanks mriedem | |
| 17:12:07 | mriedem | yeah finish resize happens on the dest machine | |
| 17:13:08 | mriedem | db errors would be in conductor logs though | |
| 17:13:25 | nicolasbock | The compute only says: `nova-compute.log:2018-12-07 17:08:04.883 26498 INFO nova.compute.manager [-] [instance: 585b884f-29be-4bb1-8f95-8f5e4c02127d] During sync_power_state the instance has a pending task (resize_finish). Skip.` | |
| 17:14:11 | mriedem | this is where that task_state would be set https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3913 | |
| 17:14:20 | nicolasbock | I am going through the conductor logs now | |
| 17:14:21 | mriedem | and then task_state=None when done https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3946 | |
| 17:14:39 | mriedem | if something blew up in there, this should reset the task_state to None https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3954 | |
| 17:14:45 | mriedem | but if you're having db problems, it's a crapshoot | |
| 17:15:45 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623552 | |
| 17:20:29 | nicolasbock | I don't see any DB errors either. Strange. | |
| 17:50:42 | aspiers | mriedem_lunch: thanks a lot for the quick reply - will get you those confirmations about volume-backed and config drive ASAP | |
| 18:10:57 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional recreate test for bug 1799727 https://review.openstack.org/614563 | |
| 18:10:57 | openstack | bug 1799727 in OpenStack Compute (nova) rocky "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem) | |
| 18:11:03 | openstackgerrit | Merged openstack/nova stable/rocky: Provide allocation_ratio/reserved amounts from update_provider_tree() https://review.openstack.org/614564 | |
| 18:11:10 | openstackgerrit | Merged openstack/nova stable/rocky: Add regression test for bug 1796737 https://review.openstack.org/614587 | |
| 18:11:10 | openstack | bug 1796737 in OpenStack Compute (nova) rocky "resize: hypervisor local_gb_used still reports usage even with volume-backed instances after fix for bug 1469179" [Medium,In progress] https://launchpad.net/bugs/1796737 - Assigned to Matt Riedemann (mriedem) | |
| 18:11:15 | openstackgerrit | Merged openstack/nova stable/rocky: Properly track local root disk usage during moves https://review.openstack.org/614588 | |
| 18:11:22 | openstackgerrit | Merged openstack/nova stable/rocky: Time how long select_destinations() takes in conductor https://review.openstack.org/608575 | |
| 18:11:28 | openstackgerrit | Merged openstack/nova stable/rocky: Refix disk size during live migration with disk over-commit https://review.openstack.org/602477 | |
| 18:11:35 | openstackgerrit | Merged openstack/nova stable/rocky: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/610163 | |
| 18:12:12 | melwitt | stephenfin: been meaning to ask you, the nova-next job stopped testing vnc console with TLS several months ago, see http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/screen-n-novnc-cell1.txt.gz#_Dec_04_01_01_59_749371 I tried digging into it and couldn't find what's going wrong, how it can't find the cert file. wanted to give you a heads up in case you had any clues | |
| 18:21:31 | sean-k-mooney | melwitt: i wonder if devstack change its default or something and its not enableing tls anymore | |
| 18:23:28 | melwitt | sean-k-mooney: yeah, it's supposed to set everything up so long as the tls-proxy service is enabled https://github.com/openstack-dev/devstack/blob/78a564bb0304b6f930e1491e7e116a0a0f6d9ab6/stack.sh#L848 | |
| 18:24:21 | melwitt | and in the example log I linked, it is enabled http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/devstacklog.txt.gz#_2018-12-04_00_45_46_475 | |
| 18:24:55 | melwitt | I looked around devstack months ago looking for a change but didn't find anything. I could have missed it | |
| 18:53:37 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP Reject migration requests when src compute is down https://review.openstack.org/623489 | |
| 18:57:14 | openstackgerrit | Jay Pipes proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.openstack.org/623557 | |
| 18:57:15 | openstackgerrit | Jay Pipes proposed openstack/nova master: single pass instance info fetch in host manager https://review.openstack.org/623558 | |
| 18:58:25 | leakypipes | belmoreira: ^^ would be great to get some initial feedback from you on these performance patches to the scheduler. See commit message from https://review.openstack.org/#/c/623558/ for details. | |
| 18:58:30 | leakypipes | melwitt: ^^ | |
| 19:00:39 | melwitt | kewl | |
| 19:02:41 | openstackgerrit | Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646 | |
| 19:09:14 | leakypipes | topic name, that is... | |
| 19:09:42 | mriedem | call me ishaeml | |
| 19:09:46 | mriedem | *ishmael | |
| 19:11:27 | mriedem | dansmith: you may enjoy https://review.openstack.org/#/c/615347 and it also resolves a functional race bug | |
| 19:11:32 | mriedem | http://status.openstack.org/elastic-recheck/#1800472 | |
| 19:39:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Drop nova-multiattach job https://review.openstack.org/623568 | |
| 19:45:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop nova-multiattach job https://review.openstack.org/606981 | |
| 19:48:29 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Add descriptions about microversions https://review.openstack.org/619144 | |
| 19:57:32 | cfriesen | is there an equivalent to "nova service-list" in OSC, which would show the state of each compute node's service separately? | |
| 19:58:27 | cfriesen | never mind, found "openstack compute service list" | |
| 20:10:51 | dansmith | mriedem: see comment | |
| 20:10:58 | dansmith | +3 on the code part | |
| 20:18:38 | radez | dansmith: thx, we're working on multi-tenancy with ironic trying to test using networking-ansible to isolate networks | |
| 20:19:06 | radez | this works fine, though in our CI we're setting up two tenants and trying to deploy an instance using ironic in each of the tenants | |
| 20:19:09 | dansmith | radez: on master or rocky? | |
| 20:19:15 | radez | rocky | |
| 20:19:28 | radez | when we schedule the first node it deploys just fine | |
| 20:19:41 | dansmith | okay just to be clear.. nova with compute_driver=ironic, right? | |
| 20:20:12 | radez | yes i believe so , let me dousble check that, I've been relying on ooo to setup ironic for me | |
| 20:21:04 | dansmith | just want to make it clear you're using nova to schedule an instance that will be running on a node in ironic.. the way you worded it above made me first think you were just talking about standalone ironic | |
| 20:22:14 | radez | sry, you're correct we're using nova to schedule and ironic node | |
| 20:22:23 | dansmith | okay, sorry, continue :) | |
| 20:25:21 | radez | gathering my details here to try and depict this correctly | |
| 20:25:46 | radez | so we deploy the first node and that goes swimmingly. comes active we can connect to it. | |
| 20:25:52 | radez | this is in one tenant | |
| 20:26:40 | radez | in the second tenant we try and deploy another node and the nova logs apprear to indicate that when it goes to get available nodes to deploy to it returns 1 node but it returns the already allocated node instead of the available node | |
| 20:27:01 | radez | lemme gather a few logs that depict this | |
| 20:27:07 | dansmith | are you sure it's selecting the same *node* and not the same nova-compute? | |
| 20:27:35 | dansmith | because if you have only one machine running nova-compute, that compute service will be responsible for all the ironic nodes | |
| 20:28:14 | dansmith | I assume that's not the problem because you would likely not even notice and not be here with an issue, but just ... in case it's relevant | |
| 20:28:46 | dansmith | also, while you're collecting logs, this shouldn't have anything to do with the multi-tenancy part, as the scheduling, picking a node, reserving a node, etc doesn't really factor tenants into the calculus | |
| 20:29:06 | radez | I *think* that's not the case because the ironic node uuids and the nova instance uuids match up in the logs | |
| 20:29:35 | radez | fair enough, just laying out the context incase that was helpful | |
| 20:29:47 | dansmith | okay, they should only really line up in the compute log I think, probably not in the scheduling log, fwiw | |
| 20:29:53 | dansmith | ack, yep, just letting you know | |
| 20:31:25 | dansmith | radez: my tacos are just about warmed, so keep collecting logs and dumping info here and I'll reply when my fingers are clean :) | |
| 20:31:44 | dansmith | also, mriedem is here, he's real smart, and he was just telling me he was bored and looking for a good goose to chase :P | |
| 20:32:24 | radez | lol, cool I just got a call I need to take that will distract me for about 30 mins. I'll finish collecting my logs and ping one of you in a bit, sry about that.... I appreciate your help! | |
| 20:33:17 | mriedem | what's good for the goose is good for the gander | |
| 20:34:00 | dansmith | radez: okay, also keep in mind it's friday afternoon so probably not many hours left in the day (and I'm sure you're ahead of the rest of us in EST) :D | |
| 20:42:01 | dansmith | taco consumption complete | |
| 20:43:01 | mriedem | dansmith: replied on the reno thing - i'm ok with dropping that if it would just cause more concern than it's worth | |
| 20:43:18 | mriedem | that compat was really kind of out the window since pike i think | |
| 20:43:21 | mriedem | when the api was made cells aware | |
| 20:43:59 | dansmith | mriedem: yeah, so I'd just drop it if you're cool with it | |
| 20:44:13 | dansmith | hah | |
| 20:44:23 | mriedem | i'm very cool | |
| 20:44:24 | dansmith | zuul says 135h remaining on my patch I submitted at like 7am this morning | |
| 20:45:04 | mriedem | https://review.openstack.org/#/c/602804/ queued for at least 36 hours, hit a job timeout | |
| 20:46:36 | dansmith | good lord | |
| 20:46:57 | dansmith | maybe we should start enacting a french work-week to avoid overloading the gate | |
| 20:46:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347 | |