| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-01 | |||
| 17:24:32 | openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646 | |
| 17:24:33 | openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647 | |
| 17:24:34 | openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: New dict format of allocations (v1.11, v1.12) https://review.openstack.org/542819 | |
| 17:24:35 | openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Transactionally update allocations (v1.13) https://review.openstack.org/546674 | |
| 17:24:36 | openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Add nested resource providers (v1.14) https://review.openstack.org/546675 | |
| 17:24:37 | openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Limit allocation candidates (v1.15, v1.16) https://review.openstack.org/548043 | |
| 17:24:38 | openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Allocation candidates parameter: required (v1.17) https://review.openstack.org/548326 | |
| 17:25:47 | melwitt | mriedem: I noticed yesterday that it appears nova-next isn't running console proxy with SSL anymore. says "no cert file" but I see the cert stuff being set up in the devstack log. don't yet know what's wrong http://logs.openstack.org/49/479949/71/check/nova-next/7751dc4/logs/devstacklog.txt.gz#_2018-05-01_15_14_45_231 | |
| 17:27:52 | mriedem | hmm yeah http://logs.openstack.org/49/479949/71/check/nova-next/7751dc4/logs/local.conf.txt.gz has NOVA_CONSOLE_PROXY_COMPUTE_TLS=True | |
| 17:28:33 | mriedem | does the config have to be in the cell? | |
| 17:28:39 | mriedem | if so, it's probably writing to the wrong nova config | |
| 17:29:02 | mriedem | [vnc] is configured properly here http://logs.openstack.org/49/479949/71/check/nova-next/7751dc4/logs/etc/nova/nova.conf.txt.gz | |
| 17:29:12 | mriedem | and here http://logs.openstack.org/49/479949/71/check/nova-next/7751dc4/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 17:29:58 | melwitt | I would think the config doesn't have the be in the cell until my changes (and the above log I linked is on a change unrelated to my per cell proxy stuff) | |
| 17:29:59 | mriedem | melwitt: i wonder if it's something with using the queens UCA | |
| 17:30:01 | mriedem | libvirt might have changed | |
| 17:30:05 | mriedem | re: https://review.openstack.org/#/c/396186/7/lib/nova_plugins/functions-libvirt | |
| 17:30:42 | melwitt | maybe, guh I hope not. I have no clue how that stuff works | |
| 17:30:59 | melwitt | but this used to work and I can't think of anything that's changed | |
| 17:31:00 | mriedem | this is where that check happens http://logs.openstack.org/49/479949/71/check/nova-next/7751dc4/logs/devstacklog.txt.gz#_2018-05-01_15_05_38_543 | |
| 17:31:48 | melwitt | yeah, saw that too | |
| 17:33:53 | melwitt | "no cert file" sounds like it can't find the cert file. it deploys them to /etc/pki/libvirt-vnc/ | |
| 17:34:28 | melwitt | oh wait, but the config says /etc/pki/nova-novnc/ | |
| 17:35:11 | mriedem | looking in the libvirtd logs, i can see qemu being executed with "tls,x509verify=/etc/pki/libvirt-vnc" | |
| 17:36:46 | melwitt | okay, so I wonder if these are wrong in nova.conf? not sure | |
| 17:36:46 | melwitt | [vnc] | |
| 17:36:47 | melwitt | vencrypt_ca_certs = /etc/pki/nova-novnc/ca-cert.pem | |
| 17:37:14 | mriedem | don't know, but that wouldn't really explain how it was working before | |
| 17:37:27 | mriedem | probably need to ask stephenfin or danpb | |
| 17:37:38 | melwitt | yeah, true. I don't even know where teh nova-novnc dir comes from | |
| 17:37:58 | mriedem | it's created by devstack https://review.openstack.org/#/c/396186/7/lib/nova | |
| 17:38:57 | melwitt | fwiw, I was the one who got the devstack patch to actually run with SSL because there were minor things wrong in the devstack changes originally because things had changed over time (the series has been around forever) | |
| 17:39:31 | melwitt | because I didn't want to approve the patches until there was some demonstration of the proxy running with SSL enabled | |
| 17:43:40 | melwitt | wonder if the chown user:group needs to be updated again, that's what I had to change last time https://review.openstack.org/#/c/396186/7/lib/nova_plugins/functions-libvirt@156 | |
| 17:44:06 | mriedem | that's what i was thinking, | |
| 17:44:11 | mriedem | because i had to deal with that for the multiattach stuff | |
| 17:45:11 | mriedem | melwitt: this https://github.com/openstack-dev/devstack/blob/master/stackrc#L630 | |
| 17:45:46 | mriedem | could add a debug patch to make sure that libvirt-qemu exists | |
| 17:46:38 | melwitt | hm, yeah. this would be the qemu user and group. originally it was just qemu:qemu and I had to update it to libvirt-qemu:libvirt-qemu | |
| 17:50:18 | mriedem | melwitt: you should kick an intern to make sure this article from 2011 is up to date https://wiki.libvirt.org/page/VNCTLSSetup | |
| 17:51:19 | melwitt | heh, that's a good doc | |
| 17:54:03 | openstackgerrit | Aditya Vaja proposed openstack/nova master: remove IVS plug/unplug as they're moved to separate plugin https://review.openstack.org/534371 | |
| 17:57:20 | wolverineav | stephenfin: mriedem - updated the review with patch addressing comments ^^ take a look when you get a chance :) | |
| 18:06:24 | melwitt | based on this old bug, it's recommended to use qemu:qemu but it depends on what the distro has done https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/522619 | |
| 18:06:24 | openstack | Launchpad bug 522619 in libvirt (Ubuntu) "libvirt launced kvm / qemu system processes run as root by default" [Wishlist,Fix released] | |
| 18:08:35 | openstackgerrit | Aditya Vaja proposed openstack/nova master: remove IVS plug/unplug as they're moved to separate plugin https://review.openstack.org/534371 | |
| 18:10:33 | mriedem | melwitt: so probably the thing to do is check /etc/group like in stackrc and fallback to qemu:qemu if libvirt-qemu doesn't exist | |
| 18:15:12 | larsks | dansmith: a hack: https://github.com/larsks/os-placement-tools | |
| 18:17:16 | mriedem | tssurya: comments inline | |
| 18:17:31 | tssurya | mriedem: thanks | |
| 18:17:47 | openstackgerrit | Julia Kreger proposed openstack/nova master: ironic: add instance_uuid before any other spawn activity https://review.openstack.org/563722 | |
| 18:19:21 | mriedem | larsks: "--fox" typo | |
| 18:19:58 | larsks | mriedem: hah, you fell for my clever plan. | |
| 18:20:03 | larsks | Also, thanks, fixed it. | |
| 18:20:15 | mriedem | is this for busted migrations? | |
| 18:20:53 | larsks | mriedem: well, maybe. We hit a situation on Friday at boston university where we were unable to schedule new instances on an empty compute node. It turns out there were a number of stale allocations there (and elsewhere in the cluster). | |
| 18:20:59 | larsks | Maybe they were caused by busted migrations? | |
| 18:21:15 | larsks | But this fixed the immediate problem regardless of the cause. | |
| 18:21:16 | mriedem | what version of nova? | |
| 18:21:33 | larsks | It is now pike, but it was upgraded from ocata. And possibly from earlier. | |
| 18:21:42 | mriedem | pike GA or latest stable release? | |
| 18:21:53 | mriedem | post pike GA we found a number of bugs in places where we didn't cleanup allocations properly | |
| 18:21:55 | mriedem | those should all be fixed now | |
| 18:21:57 | larsks | Uh...whatever red hat is distributing in rhel-osp 12. | |
| 18:22:03 | mriedem | heh, fair enough | |
| 18:23:37 | mriedem | https://github.com/larsks/os-placement-tools/blob/master/check_placement.py#L16 is nice | |
| 18:24:11 | larsks | Thanks! | |
| 18:24:14 | dansmith | mriedem: nope | |
| 18:24:41 | dansmith | mriedem: it does coincide with me ruminating about deleting duplicate allocations in -placement last week though | |
| 18:25:27 | mriedem | wonder if anyone ever started on that osc-placement change to orchestrate setting inventory on an entire placement provider aggregate set | |
| 18:25:39 | mriedem | unrelated to this besides placement tooling | |
| 18:25:39 | dansmith | larsks: that looks awesome | |
| 18:25:59 | dansmith | larsks: you should check for instance state and only agree to --fix those in ACTIVE though | |
| 18:26:13 | dansmith | larsks: before ocata (ish?) you could have legit reasons for two allocations for one instance, because that's how we did migrations | |
| 18:26:28 | dansmith | but if the instance is ACTIVE and has two, that'll always need fixing I think | |
| 18:26:35 | dansmith | mriedem: right? | |
| 18:26:43 | mriedem | and/or don't touch any instances with a non-null task_state | |
| 18:26:57 | mriedem | before pike the computes would heal allocations | |
| 18:27:06 | dansmith | yeah | |
| 18:27:23 | mriedem | but yeah don't fix an instance that's undergoing a task_state transition | |
| 18:27:36 | mriedem | yee be warned | |
| 18:31:38 | larsks | So noted. | |
| 18:32:43 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add Project object https://review.openstack.org/565401 | |
| 18:32:44 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add User object to placement modeling https://review.openstack.org/565402 | |
| 18:32:45 | openstackgerrit | Jay Pipes proposed openstack/nova master: add Consumer object https://review.openstack.org/565403 | |
| 18:32:46 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add create() methods to Project, User and Consumer https://review.openstack.org/565404 | |
| 18:32:47 | openstackgerrit | Jay Pipes proposed openstack/nova master: move consumer ensure to API layer https://review.openstack.org/565405 | |
| 18:32:48 | openstackgerrit | Jay Pipes proposed openstack/nova master: remove Allocation.project_id & Allocation.user_id https://review.openstack.org/565406 | |
| 18:32:49 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework allocation handler _allocations_dict() https://review.openstack.org/565407 | |
| 18:32:50 | openstackgerrit | Jay Pipes proposed openstack/nova master: increment consumer generation when allocs change https://review.openstack.org/565513 | |
| 18:36:02 | melwitt | looks like libvirt-qemu is the right user and group, actually logged in the devstack log http://logs.openstack.org/49/479949/71/check/nova-next/7751dc4/logs/devstacklog.txt.gz#_2018-05-01_15_04_42_683 | |
| 18:36:26 | mriedem | huh | |
| 18:36:29 | mriedem | nuts | |
| 18:36:54 | melwitt | yeah. back to the drawing board | |
| 18:42:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate the nova-consoleauth service https://review.openstack.org/565367 | |
| 18:43:37 | mriedem | alright https://review.openstack.org/#/q/topic:bp/convert-consoles-to-objects+status:open is ready to go | |
| 18:43:44 | mriedem | and it's the last day in the runway | |
| 18:45:45 | melwitt | so close ... | |
| 18:46:04 | mriedem | well, efried and dansmith can make that magic happen | |
| 18:46:24 | mriedem | *or | |