| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-07 | |||
| 16:19:17 | mriedem | if we're meh on that now, then sure we can remove it | |
| 16:19:17 | cdent | mriedem, jaypipes: based on the commit message on https://review.openstack.org/493536 it looks gratuitous | |
| 16:19:53 | mriedem | if i'm going to remove the lock, i might as well also make it a singleton in the api since we don't need 70 of these created when it's used in 3 places | |
| 16:20:01 | jaypipes | mriedem: from the commit message: "I've made the _client method synchronized so that in the unlikely event that the resource tracker is trying to do its update job while some other thing is happening, we won't waste the client. This may not be necessary, but probably doesn't harm anything." | |
| 16:20:25 | mriedem | i know, i read it | |
| 16:20:39 | cdent | given that I never synchronize anything, I'm guess somewhere along the way it was suggested as a "maybe you should..." and I just did, without any real data | |
| 16:21:20 | jaypipes | cdent: either that, or copy/pasta from somewhere in the resource tracker way back when... not sure, either way, I'm pretty sure it's unnecessary. | |
| 16:21:48 | cdent | we'll find out soon enough, I reckon | |
| 16:22:58 | melwitt | bauzas: reminder that review comments on the reshaper patch await your reply https://review.openstack.org/599208 | |
| 16:23:01 | cdent | generally speaking, unless we can come up with a really good reason for a lock, we should not | |
| 16:23:35 | bauzas | melwitt: ok, I was thinking those details were pretty nitty, but I'll then provide a new revision | |
| 16:24:15 | mriedem | bauzas: if you thought that, you could have said so weeks ago... | |
| 16:24:34 | melwitt | bauzas: ok. you don't necessarily have to push a new revision but could make a comment in response so we know your thoughts | |
| 16:24:35 | mriedem | or talked with artom | |
| 16:32:26 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Ignore MoxStubout deprecation warnings https://review.openstack.org/623545 | |
| 16:52:42 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623546 | |
| 16:56:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623547 | |
| 17:09:02 | artom | mriedem, bauzas, are we seriously OK with the length of those methods? I won't be a pain just for the hell of it, and I understand that sometimes old code grows beyond what we want, but brand new code? | |
| 17:09:02 | nicolasbock | Hi, I have a server on Newton that after a migration attempt shows `task_state = resize_finish`, `status = RESIZE`. I can't resize confirm or start the server. I don't see any obvious error messages in the nova-compute logs on either the old nor the new hypervisor. | |
| 17:09:08 | artom | I dunno, I just have trouble with that | |
| 17:09:24 | nicolasbock | The resized disks have the same md5sum on either hypervisor | |
| 17:09:31 | artom | If everyone's overruling me there's not much I can do, but I wanted to at least make that point | |
| 17:09:46 | mriedem | artom: i'm not disagreeing with your comments | |
| 17:10:00 | nicolasbock | Basically, I am wondering what the problem could be? | |
| 17:10:21 | artom | mriedem, ok, cool, thanks :) | |
| 17:10:45 | mriedem | nicolasbock: looks like something probably failed during the resize (maybe a db change?) where the task_state was not set to None | |
| 17:10:52 | mriedem | so check the compute and conductor logs for errors | |
| 17:11:22 | nicolasbock | the logs on the destination ? | |
| 17:11:29 | nicolasbock | And thanks mriedem | |
| 17:12:07 | mriedem | yeah finish resize happens on the dest machine | |
| 17:13:08 | mriedem | db errors would be in conductor logs though | |
| 17:13:25 | nicolasbock | The compute only says: `nova-compute.log:2018-12-07 17:08:04.883 26498 INFO nova.compute.manager [-] [instance: 585b884f-29be-4bb1-8f95-8f5e4c02127d] During sync_power_state the instance has a pending task (resize_finish). Skip.` | |
| 17:14:11 | mriedem | this is where that task_state would be set https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3913 | |
| 17:14:20 | nicolasbock | I am going through the conductor logs now | |
| 17:14:21 | mriedem | and then task_state=None when done https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3946 | |
| 17:14:39 | mriedem | if something blew up in there, this should reset the task_state to None https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3954 | |
| 17:14:45 | mriedem | but if you're having db problems, it's a crapshoot | |
| 17:15:45 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623552 | |
| 17:20:29 | nicolasbock | I don't see any DB errors either. Strange. | |
| 17:50:42 | aspiers | mriedem_lunch: thanks a lot for the quick reply - will get you those confirmations about volume-backed and config drive ASAP | |
| 18:10:57 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional recreate test for bug 1799727 https://review.openstack.org/614563 | |
| 18:10:57 | openstack | bug 1799727 in OpenStack Compute (nova) rocky "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem) | |
| 18:11:03 | openstackgerrit | Merged openstack/nova stable/rocky: Provide allocation_ratio/reserved amounts from update_provider_tree() https://review.openstack.org/614564 | |
| 18:11:10 | openstackgerrit | Merged openstack/nova stable/rocky: Add regression test for bug 1796737 https://review.openstack.org/614587 | |
| 18:11:10 | openstack | bug 1796737 in OpenStack Compute (nova) rocky "resize: hypervisor local_gb_used still reports usage even with volume-backed instances after fix for bug 1469179" [Medium,In progress] https://launchpad.net/bugs/1796737 - Assigned to Matt Riedemann (mriedem) | |
| 18:11:15 | openstackgerrit | Merged openstack/nova stable/rocky: Properly track local root disk usage during moves https://review.openstack.org/614588 | |
| 18:11:22 | openstackgerrit | Merged openstack/nova stable/rocky: Time how long select_destinations() takes in conductor https://review.openstack.org/608575 | |
| 18:11:28 | openstackgerrit | Merged openstack/nova stable/rocky: Refix disk size during live migration with disk over-commit https://review.openstack.org/602477 | |
| 18:11:35 | openstackgerrit | Merged openstack/nova stable/rocky: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/610163 | |
| 18:12:12 | melwitt | stephenfin: been meaning to ask you, the nova-next job stopped testing vnc console with TLS several months ago, see http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/screen-n-novnc-cell1.txt.gz#_Dec_04_01_01_59_749371 I tried digging into it and couldn't find what's going wrong, how it can't find the cert file. wanted to give you a heads up in case you had any clues | |
| 18:21:31 | sean-k-mooney | melwitt: i wonder if devstack change its default or something and its not enableing tls anymore | |
| 18:23:28 | melwitt | sean-k-mooney: yeah, it's supposed to set everything up so long as the tls-proxy service is enabled https://github.com/openstack-dev/devstack/blob/78a564bb0304b6f930e1491e7e116a0a0f6d9ab6/stack.sh#L848 | |
| 18:24:21 | melwitt | and in the example log I linked, it is enabled http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/devstacklog.txt.gz#_2018-12-04_00_45_46_475 | |
| 18:24:55 | melwitt | I looked around devstack months ago looking for a change but didn't find anything. I could have missed it | |
| 18:53:37 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP Reject migration requests when src compute is down https://review.openstack.org/623489 | |
| 18:57:14 | openstackgerrit | Jay Pipes proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.openstack.org/623557 | |
| 18:57:15 | openstackgerrit | Jay Pipes proposed openstack/nova master: single pass instance info fetch in host manager https://review.openstack.org/623558 | |
| 18:58:25 | leakypipes | belmoreira: ^^ would be great to get some initial feedback from you on these performance patches to the scheduler. See commit message from https://review.openstack.org/#/c/623558/ for details. | |
| 18:58:30 | leakypipes | melwitt: ^^ | |
| 19:00:39 | melwitt | kewl | |
| 19:02:41 | openstackgerrit | Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646 | |
| 19:09:14 | leakypipes | topic name, that is... | |
| 19:09:42 | mriedem | call me ishaeml | |
| 19:09:46 | mriedem | *ishmael | |
| 19:11:27 | mriedem | dansmith: you may enjoy https://review.openstack.org/#/c/615347 and it also resolves a functional race bug | |
| 19:11:32 | mriedem | http://status.openstack.org/elastic-recheck/#1800472 | |
| 19:39:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Drop nova-multiattach job https://review.openstack.org/623568 | |
| 19:45:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop nova-multiattach job https://review.openstack.org/606981 | |
| 19:48:29 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Add descriptions about microversions https://review.openstack.org/619144 | |
| 19:57:32 | cfriesen | is there an equivalent to "nova service-list" in OSC, which would show the state of each compute node's service separately? | |
| 19:58:27 | cfriesen | never mind, found "openstack compute service list" | |
| 20:10:51 | dansmith | mriedem: see comment | |
| 20:10:58 | dansmith | +3 on the code part | |
| 20:18:38 | radez | dansmith: thx, we're working on multi-tenancy with ironic trying to test using networking-ansible to isolate networks | |
| 20:19:06 | radez | this works fine, though in our CI we're setting up two tenants and trying to deploy an instance using ironic in each of the tenants | |
| 20:19:09 | dansmith | radez: on master or rocky? | |
| 20:19:15 | radez | rocky | |
| 20:19:28 | radez | when we schedule the first node it deploys just fine | |
| 20:19:41 | dansmith | okay just to be clear.. nova with compute_driver=ironic, right? | |
| 20:20:12 | radez | yes i believe so , let me dousble check that, I've been relying on ooo to setup ironic for me | |
| 20:21:04 | dansmith | just want to make it clear you're using nova to schedule an instance that will be running on a node in ironic.. the way you worded it above made me first think you were just talking about standalone ironic | |
| 20:22:14 | radez | sry, you're correct we're using nova to schedule and ironic node | |
| 20:22:23 | dansmith | okay, sorry, continue :) | |
| 20:25:21 | radez | gathering my details here to try and depict this correctly | |
| 20:25:46 | radez | so we deploy the first node and that goes swimmingly. comes active we can connect to it. | |
| 20:25:52 | radez | this is in one tenant | |
| 20:26:40 | radez | in the second tenant we try and deploy another node and the nova logs apprear to indicate that when it goes to get available nodes to deploy to it returns 1 node but it returns the already allocated node instead of the available node | |
| 20:27:01 | radez | lemme gather a few logs that depict this | |
| 20:27:07 | dansmith | are you sure it's selecting the same *node* and not the same nova-compute? | |
| 20:27:35 | dansmith | because if you have only one machine running nova-compute, that compute service will be responsible for all the ironic nodes | |
| 20:28:14 | dansmith | I assume that's not the problem because you would likely not even notice and not be here with an issue, but just ... in case it's relevant | |
| 20:28:46 | dansmith | also, while you're collecting logs, this shouldn't have anything to do with the multi-tenancy part, as the scheduling, picking a node, reserving a node, etc doesn't really factor tenants into the calculus | |
| 20:29:06 | radez | I *think* that's not the case because the ironic node uuids and the nova instance uuids match up in the logs | |
| 20:29:35 | radez | fair enough, just laying out the context incase that was helpful | |
| 20:29:47 | dansmith | okay, they should only really line up in the compute log I think, probably not in the scheduling log, fwiw | |
| 20:29:53 | dansmith | ack, yep, just letting you know | |
| 20:31:25 | dansmith | radez: my tacos are just about warmed, so keep collecting logs and dumping info here and I'll reply when my fingers are clean :) | |
| 20:31:44 | dansmith | also, mriedem is here, he's real smart, and he was just telling me he was bored and looking for a good goose to chase :P | |
| 20:32:24 | radez | lol, cool I just got a call I need to take that will distract me for about 30 mins. I'll finish collecting my logs and ping one of you in a bit, sry about that.... I appreciate your help! | |
| 20:33:17 | mriedem | what's good for the goose is good for the gander | |