| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-07 | |||
| 17:14:20 | nicolasbock | I am going through the conductor logs now | |
| 17:14:21 | mriedem | and then task_state=None when done https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3946 | |
| 17:14:39 | mriedem | if something blew up in there, this should reset the task_state to None https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3954 | |
| 17:14:45 | mriedem | but if you're having db problems, it's a crapshoot | |
| 17:15:45 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623552 | |
| 17:20:29 | nicolasbock | I don't see any DB errors either. Strange. | |
| 17:50:42 | aspiers | mriedem_lunch: thanks a lot for the quick reply - will get you those confirmations about volume-backed and config drive ASAP | |
| 18:10:57 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional recreate test for bug 1799727 https://review.openstack.org/614563 | |
| 18:10:57 | openstack | bug 1799727 in OpenStack Compute (nova) rocky "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem) | |
| 18:11:03 | openstackgerrit | Merged openstack/nova stable/rocky: Provide allocation_ratio/reserved amounts from update_provider_tree() https://review.openstack.org/614564 | |
| 18:11:10 | openstackgerrit | Merged openstack/nova stable/rocky: Add regression test for bug 1796737 https://review.openstack.org/614587 | |
| 18:11:10 | openstack | bug 1796737 in OpenStack Compute (nova) rocky "resize: hypervisor local_gb_used still reports usage even with volume-backed instances after fix for bug 1469179" [Medium,In progress] https://launchpad.net/bugs/1796737 - Assigned to Matt Riedemann (mriedem) | |
| 18:11:15 | openstackgerrit | Merged openstack/nova stable/rocky: Properly track local root disk usage during moves https://review.openstack.org/614588 | |
| 18:11:22 | openstackgerrit | Merged openstack/nova stable/rocky: Time how long select_destinations() takes in conductor https://review.openstack.org/608575 | |
| 18:11:28 | openstackgerrit | Merged openstack/nova stable/rocky: Refix disk size during live migration with disk over-commit https://review.openstack.org/602477 | |
| 18:11:35 | openstackgerrit | Merged openstack/nova stable/rocky: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/610163 | |
| 18:12:12 | melwitt | stephenfin: been meaning to ask you, the nova-next job stopped testing vnc console with TLS several months ago, see http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/screen-n-novnc-cell1.txt.gz#_Dec_04_01_01_59_749371 I tried digging into it and couldn't find what's going wrong, how it can't find the cert file. wanted to give you a heads up in case you had any clues | |
| 18:21:31 | sean-k-mooney | melwitt: i wonder if devstack change its default or something and its not enableing tls anymore | |
| 18:23:28 | melwitt | sean-k-mooney: yeah, it's supposed to set everything up so long as the tls-proxy service is enabled https://github.com/openstack-dev/devstack/blob/78a564bb0304b6f930e1491e7e116a0a0f6d9ab6/stack.sh#L848 | |
| 18:24:21 | melwitt | and in the example log I linked, it is enabled http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/devstacklog.txt.gz#_2018-12-04_00_45_46_475 | |
| 18:24:55 | melwitt | I looked around devstack months ago looking for a change but didn't find anything. I could have missed it | |
| 18:53:37 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP Reject migration requests when src compute is down https://review.openstack.org/623489 | |
| 18:57:14 | openstackgerrit | Jay Pipes proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.openstack.org/623557 | |
| 18:57:15 | openstackgerrit | Jay Pipes proposed openstack/nova master: single pass instance info fetch in host manager https://review.openstack.org/623558 | |
| 18:58:25 | leakypipes | belmoreira: ^^ would be great to get some initial feedback from you on these performance patches to the scheduler. See commit message from https://review.openstack.org/#/c/623558/ for details. | |
| 18:58:30 | leakypipes | melwitt: ^^ | |
| 19:00:39 | melwitt | kewl | |
| 19:02:41 | openstackgerrit | Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646 | |
| 19:09:14 | leakypipes | topic name, that is... | |
| 19:09:42 | mriedem | call me ishaeml | |
| 19:09:46 | mriedem | *ishmael | |
| 19:11:27 | mriedem | dansmith: you may enjoy https://review.openstack.org/#/c/615347 and it also resolves a functional race bug | |
| 19:11:32 | mriedem | http://status.openstack.org/elastic-recheck/#1800472 | |
| 19:39:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Drop nova-multiattach job https://review.openstack.org/623568 | |
| 19:45:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop nova-multiattach job https://review.openstack.org/606981 | |
| 19:48:29 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Add descriptions about microversions https://review.openstack.org/619144 | |
| 19:57:32 | cfriesen | is there an equivalent to "nova service-list" in OSC, which would show the state of each compute node's service separately? | |
| 19:58:27 | cfriesen | never mind, found "openstack compute service list" | |
| 20:10:51 | dansmith | mriedem: see comment | |
| 20:10:58 | dansmith | +3 on the code part | |
| 20:18:38 | radez | dansmith: thx, we're working on multi-tenancy with ironic trying to test using networking-ansible to isolate networks | |
| 20:19:06 | radez | this works fine, though in our CI we're setting up two tenants and trying to deploy an instance using ironic in each of the tenants | |
| 20:19:09 | dansmith | radez: on master or rocky? | |
| 20:19:15 | radez | rocky | |
| 20:19:28 | radez | when we schedule the first node it deploys just fine | |
| 20:19:41 | dansmith | okay just to be clear.. nova with compute_driver=ironic, right? | |
| 20:20:12 | radez | yes i believe so , let me dousble check that, I've been relying on ooo to setup ironic for me | |
| 20:21:04 | dansmith | just want to make it clear you're using nova to schedule an instance that will be running on a node in ironic.. the way you worded it above made me first think you were just talking about standalone ironic | |
| 20:22:14 | radez | sry, you're correct we're using nova to schedule and ironic node | |
| 20:22:23 | dansmith | okay, sorry, continue :) | |
| 20:25:21 | radez | gathering my details here to try and depict this correctly | |
| 20:25:46 | radez | so we deploy the first node and that goes swimmingly. comes active we can connect to it. | |
| 20:25:52 | radez | this is in one tenant | |
| 20:26:40 | radez | in the second tenant we try and deploy another node and the nova logs apprear to indicate that when it goes to get available nodes to deploy to it returns 1 node but it returns the already allocated node instead of the available node | |
| 20:27:01 | radez | lemme gather a few logs that depict this | |
| 20:27:07 | dansmith | are you sure it's selecting the same *node* and not the same nova-compute? | |
| 20:27:35 | dansmith | because if you have only one machine running nova-compute, that compute service will be responsible for all the ironic nodes | |
| 20:28:14 | dansmith | I assume that's not the problem because you would likely not even notice and not be here with an issue, but just ... in case it's relevant | |
| 20:28:46 | dansmith | also, while you're collecting logs, this shouldn't have anything to do with the multi-tenancy part, as the scheduling, picking a node, reserving a node, etc doesn't really factor tenants into the calculus | |
| 20:29:06 | radez | I *think* that's not the case because the ironic node uuids and the nova instance uuids match up in the logs | |
| 20:29:35 | radez | fair enough, just laying out the context incase that was helpful | |
| 20:29:47 | dansmith | okay, they should only really line up in the compute log I think, probably not in the scheduling log, fwiw | |
| 20:29:53 | dansmith | ack, yep, just letting you know | |
| 20:31:25 | dansmith | radez: my tacos are just about warmed, so keep collecting logs and dumping info here and I'll reply when my fingers are clean :) | |
| 20:31:44 | dansmith | also, mriedem is here, he's real smart, and he was just telling me he was bored and looking for a good goose to chase :P | |
| 20:32:24 | radez | lol, cool I just got a call I need to take that will distract me for about 30 mins. I'll finish collecting my logs and ping one of you in a bit, sry about that.... I appreciate your help! | |
| 20:33:17 | mriedem | what's good for the goose is good for the gander | |
| 20:34:00 | dansmith | radez: okay, also keep in mind it's friday afternoon so probably not many hours left in the day (and I'm sure you're ahead of the rest of us in EST) :D | |
| 20:42:01 | dansmith | taco consumption complete | |
| 20:43:01 | mriedem | dansmith: replied on the reno thing - i'm ok with dropping that if it would just cause more concern than it's worth | |
| 20:43:18 | mriedem | that compat was really kind of out the window since pike i think | |
| 20:43:21 | mriedem | when the api was made cells aware | |
| 20:43:59 | dansmith | mriedem: yeah, so I'd just drop it if you're cool with it | |
| 20:44:13 | dansmith | hah | |
| 20:44:23 | mriedem | i'm very cool | |
| 20:44:24 | dansmith | zuul says 135h remaining on my patch I submitted at like 7am this morning | |
| 20:45:04 | mriedem | https://review.openstack.org/#/c/602804/ queued for at least 36 hours, hit a job timeout | |
| 20:46:36 | dansmith | good lord | |
| 20:46:57 | dansmith | maybe we should start enacting a french work-week to avoid overloading the gate | |
| 20:46:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347 | |
| 20:46:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove "API Service Version" upgrade check https://review.openstack.org/615348 | |
| 21:01:53 | radez | ok, I'm back now, sry about that | |
| 21:02:00 | radez | dansmith: mriedem: http://paste.openstack.org/show/736843/ | |
| 21:02:25 | radez | this shows the uuids of the bm nodes and the nova instances that seems to be behaving like this | |
| 21:03:20 | radez | and the nova logs where nova seems to assign the already assigned bm node to the new nova instance | |
| 21:03:51 | radez | let me know if there's more logs I can get to help | |
| 21:04:41 | dansmith | radez: so these nodes should all exist in placement with a single inventory item for the resource/node class, | |
| 21:04:51 | dansmith | which means they should never get past the scheduler with two instances on the same node | |
| 21:05:12 | radez | translation was helpful thx :) | |
| 21:05:31 | radez | [instance: a840afbc-1314-44d6-83b3-c20b790b9322] Claim successful on node 3c3eb5ee-a358-421a-b2e9-7fd9a017ba13 | |
| 21:06:34 | radez | hm, well now I think I've given you the logs that show that the associated ironic node and nova instance are properly associated together | |
| 21:07:42 | mriedem | do the flavors have cpu/ram/disk set to anything !0? | |
| 21:07:44 | radez | oh wait, sry I'm getting my self confused... right so does that log line I just pasted here show that nova is trying to | |
| 21:07:46 | dansmith | radez: what log is this? just nova compute? | |
| 21:08:06 | radez | assign ironic node 3c3eb5ee to nova instance a840afbc? | |
| 21:08:34 | radez | yea nova-compute log | |
| 21:08:43 | dansmith | radez: so probably want to look at the scheduler log | |
| 21:09:00 | dansmith | radez: scheduler is what decides which node it is going to tell compute to use for each instance | |
| 21:10:10 | radez | k lemme find a840afbc in the schedule logs | |
| 21:13:17 | radez | http://paste.openstack.org/show/736844/ | |