| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-22 | |||
| 16:21:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log why rescheduling is disabled https://review.openstack.org/638699 | |
| 16:21:15 | mriedem | leakypipes: dansmith: some fun from the past ^ | |
| 16:26:42 | sean-k-mooney | kashyap: wouldnt that show up in the audit.log | |
| 16:27:31 | kashyap | sean-k-mooney: No idea what AppArmour does; let me see, though | |
| 16:27:37 | kashyap | I know how to troubleshoot SELinuz | |
| 16:28:32 | sean-k-mooney | both used to show up in demesg then they moved to journalctl but were also loged to the audit.log under /var/log i think | |
| 16:31:01 | sean-k-mooney | this is also a kernel bug in ubuntu 18.04 that causes nested virt to not work but if that iss the issue the core dump in dmesg telegraphfs the issue rather quirckly. | |
| 16:32:09 | kashyap | sean-k-mooney: Where is the audit.log here: http://logs.openstack.org/48/631948/9/check/tempest-full-py3/e2ae3fb/controller/logs/ | |
| 16:32:10 | sean-k-mooney | kashyap: this is the ubuntu issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1773184 | |
| 16:32:11 | openstack | Launchpad bug 1773184 in linux (Ubuntu Bionic) "Starting a KVM guest in a guest (nested VM) crash the kernel" [High,Triaged] | |
| 16:32:31 | dansmith | bland? | |
| 16:32:35 | dansmith | blonde? | |
| 16:32:59 | kashyap | :D | |
| 16:33:13 | sean-k-mooney | i dont think its copied | |
| 16:33:13 | kashyap | Got dansmithed ... s/blnd/blind :D | |
| 16:33:33 | kashyap | Let me check with Clark | |
| 16:34:17 | giblet | mriedem: replied to your question in https://review.openstack.org/#/c/569459 | |
| 16:35:04 | kashyap | sean-k-mooney: BTW, upstream KVM list has a huge nested virt patch series; so no idea if that fixes | |
| 16:35:10 | kashyap | sean-k-mooney: I've seen that stack trace, though | |
| 16:35:13 | sean-k-mooney | kashyap: i have it https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1773184 on one of my servers byt the way so i know that iss an issue with the ubunut cloud images. that said i think its realteded to my bios/microcode | |
| 16:35:16 | openstack | Launchpad bug 1773184 in linux (Ubuntu Bionic) "Starting a KVM guest in a guest (nested VM) crash the kernel" [High,Triaged] | |
| 16:36:06 | sean-k-mooney | kashyap: well for me swaping out the vanial 4.20 kernel in the l1 guest allowed the l2 guest to boot | |
| 16:36:39 | sean-k-mooney | if i use centos on the same host for the l1 guest spanwnign the l2 guest crashes the l1 guest and it reboot | |
| 16:37:28 | kashyap | Okay, my cardinal rule when setting up nested virt for dev envs: ensure L0, L1 (and even L2) kernels to be as similiar (and as newer as they can) | |
| 16:37:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use Selection object to fill request group mapping https://review.openstack.org/638711 | |
| 16:37:54 | sean-k-mooney | i was useing the same kernel for l0 and l1 originally | |
| 16:37:59 | kashyap | sean-k-mooney: The test combination explosion is one of the difficult things | |
| 16:38:02 | sean-k-mooney | with a cirros image for l2 | |
| 16:38:10 | sean-k-mooney | ya | |
| 16:38:27 | dansmith | mriedem: got it | |
| 16:38:55 | sean-k-mooney | i honestly have never had issue with nested vert in the past | |
| 16:40:01 | kashyap | sean-k-mooney: The "past" is a fuzzy word. Precisely for these reasons I used to track all the kernel bugs I filed for nVMX: https://kashyapc.fedorapeople.org/kernel-kvm-bugs.txt | |
| 16:40:18 | kashyap | (They're all fixed, though. And it hasn't been updated in ages) | |
| 16:45:08 | mriedem | the more i dig into https://bugs.launchpad.net/nova/+bug/1790204 the more it's a spec at this point | |
| 16:45:09 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged] | |
| 16:47:14 | mriedem | i imagine this is all dead code since queens or at least rocky https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L140 | |
| 16:47:43 | mriedem | because we should never have to double up allocations on the same consumer (instance) during scheduling since we moved the allocations on the source node to the migration record, | |
| 16:47:43 | sean-k-mooney | kashyap: let me rephsase this is the first time i have ever personally it a nested vert bug in the 5-6 years i have used nested virt by defualt | |
| 16:47:47 | mriedem | except maybe evacuate... | |
| 16:48:00 | kashyap | sean-k-mooney: Ah, okay. :-) | |
| 16:49:00 | melwitt | o/ | |
| 16:59:56 | cdent | fried_rice: my turn to ask you for a reminder: | |
| 17:00:20 | cdent | resource provider A represents a compute node, currently idle | |
| 17:00:58 | cdent | third part system makes allocations against that compute node (consuming capacity in placement but not in _reality_) | |
| 17:01:31 | cdent | the compute node will or will not correct that? | |
| 17:01:57 | cdent | I feel like we've gone back and forth on that a bit and I forget where we are now | |
| 17:02:11 | cdent | I'd just test it, but I currently have no cloud | |
| 17:03:30 | fried_rice | cdent: If the allocation is made against a real instance UUID, I think nova will "heal" it at some point. Otherwise, we don't muck with allocations that aren't related to instances afaik. | |
| 17:03:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Dedupe compute install guides https://review.openstack.org/638716 | |
| 17:03:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Dedupe controller install guides https://review.openstack.org/638715 | |
| 17:04:17 | fried_rice | cdent: mriedem might have to weigh in on the circumstances under which healing occurs at all. It may be just at the behest of a nova-manage command. | |
| 17:04:41 | cdent | i've managed to make myself curious enough that I'm going to go spin up the devstack | |
| 17:05:04 | fried_rice | cdent: Now, oob inventory/trait/agg changes against providers owned by the compute node will definitely be "healed" next time upt runs. | |
| 17:05:32 | fried_rice | which actually may not be as soon as we would like - was having this conversation with aspiers the other day. | |
| 17:06:36 | cdent | yeah, to inventory/trait stuff | |
| 17:07:01 | cdent | what's the issue on the "soon"? | |
| 17:09:25 | mriedem | there is no auto-healing of allocations by nova-compute | |
| 17:09:30 | kashyap | mriedem: I think I found the sucker in my case; spent the last hour-ish duking aroud the system. This looks suspicious: | |
| 17:09:34 | kashyap | --- | |
| 17:09:35 | kashyap | Feb 21 17:14:13 ubuntu-bionic-inap-mtl01-0002851272 kernel: traps: qemu-system-x86[31240] general protection ip:5600cbedaf78 sp:7f2dba1ebf00 error:0 in qemu-system-x86_64[5600cb81c000+8d2000] | |
| 17:09:39 | kashyap | --- | |
| 17:10:01 | mriedem | cdent: fried_rice: coincidentally, read the commit message on https://review.openstack.org/#/c/638700/ for a history lesson | |
| 17:10:22 | aspiers | cdent: IIUC, the sync from provider tree to placement is essentially one way, so if something gets removed from placement then it won't get replaced until the provider cache is reset | |
| 17:10:44 | fried_rice | cdent: It came up when aspiers was fixing the "automatic traits from capabilities" patch https://review.openstack.org/#/c/538498/ | |
| 17:10:45 | cdent | mriedem: I read that a bit earlier today and the aforementioned lack of brain made all the characters move around | |
| 17:10:45 | fried_rice | But we enabled switching off periodic refresh. So that might never happen. | |
| 17:10:45 | fried_rice | because one of the test scenarios needs to be: "user" adds or removes a compute-owned capability trait; compute restores it. | |
| 17:11:15 | aspiers | cdent: https://review.openstack.org/#/c/538498/15/nova/tests/functional/test_servers.py@2294 | |
| 17:11:34 | mriedem | tl;dr while we had computes older than pike, the compute service would overwrite instance allocations created by the scheduler | |
| 17:12:00 | cdent | I'm confused by this notion of user removes compute-owned capability trait. I'm guessing that "don't do that" is considered insufficient? | |
| 17:12:35 | fried_rice | cdent: yeah :) | |
| 17:12:37 | mriedem | users most often probably don't know what they should and shouldn't do | |
| 17:12:52 | aspiers | cdent: it's due to this optimisation https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1047 | |
| 17:13:19 | cdent | i guess I was hoping for "most users probably don't know how to delete a trait" | |
| 17:13:31 | mriedem | cdent: if any code is going to mess with your externally mucked allocation stuff, it's like this https://github.com/openstack/nova/blob/27617ee1931b3240dbd0ad4c7d8ffd64cc202bc9/nova/compute/resource_tracker.py#L1241 | |
| 17:13:31 | aspiers | cdent: see also https://pasteboard.co/I1Tbzp2.png (which is linked from the commit message) | |
| 17:14:05 | mriedem | cdent: which would get you an INFO message https://github.com/openstack/nova/blob/27617ee1931b3240dbd0ad4c7d8ffd64cc202bc9/nova/compute/resource_tracker.py#L1293 | |
| 17:14:11 | cdent | mriedem: yeah, was just looking at that method | |
| 17:14:47 | cdent | aspiers: yeah, that's why I was trying to suggest that the os-traits should grow bigger, faster | |
| 17:14:58 | fried_rice | mriedem: we still haven't solved the resize-to-same-host doubling though, right? | |
| 17:15:15 | aspiers | cdent: makes sense | |
| 17:15:55 | fried_rice | mriedem: which is what your last paragraph is talking about | |
| 17:16:11 | fried_rice | cdent: It doesn't matter if they're in os-traits or not. | |
| 17:16:58 | cdent | they can't be deleted if they are in os-traits | |
| 17:17:13 | cdent | (that was all I was getting at, nothing more) | |
| 17:17:16 | fried_rice | not deleted from placement; deleted from the compute node. | |
| 17:17:17 | aspiers | I think they can be deleted from a resource provider | |
| 17:17:21 | fried_rice | yeah | |
| 17:17:42 | fried_rice | aspiers worded it better, "delete from placement" is too vague. | |
| 17:17:44 | aspiers | just not from global traits list | |
| 17:17:56 | cdent | oh, from the rp, I see | |
| 17:18:03 | cdent | yeah, just don't do that :) | |
| 17:18:05 | aspiers | fried_rice: the ambiguity only just became clear in my head ;-/ | |
| 17:18:29 | aspiers | hence why it's still present in that Venn diagram I just linked | |
| 17:18:39 | cdent | I honestly think that we need to make the system break when people break it | |
| 17:18:46 | cdent | not try to fix it for them | |
| 17:18:53 | aspiers | shit, actually that diagram is wrong isn't it :-/ | |
| 17:19:25 | cdent | we want failures to show as soon as possible | |
| 17:19:40 | aspiers | yeah I'm totally OK with the "if you do that you get to keep all the broken pieces" approach | |
| 17:19:40 | cdent | not be in a situation where every 5 minutes system a removes a trait and every 7th minute something puts it back | |