| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-01 | |||
| 18:19:58 | larsks | mriedem: hah, you fell for my clever plan. | |
| 18:20:03 | larsks | Also, thanks, fixed it. | |
| 18:20:15 | mriedem | is this for busted migrations? | |
| 18:20:53 | larsks | mriedem: well, maybe. We hit a situation on Friday at boston university where we were unable to schedule new instances on an empty compute node. It turns out there were a number of stale allocations there (and elsewhere in the cluster). | |
| 18:20:59 | larsks | Maybe they were caused by busted migrations? | |
| 18:21:15 | larsks | But this fixed the immediate problem regardless of the cause. | |
| 18:21:16 | mriedem | what version of nova? | |
| 18:21:33 | larsks | It is now pike, but it was upgraded from ocata. And possibly from earlier. | |
| 18:21:42 | mriedem | pike GA or latest stable release? | |
| 18:21:53 | mriedem | post pike GA we found a number of bugs in places where we didn't cleanup allocations properly | |
| 18:21:55 | mriedem | those should all be fixed now | |
| 18:21:57 | larsks | Uh...whatever red hat is distributing in rhel-osp 12. | |
| 18:22:03 | mriedem | heh, fair enough | |
| 18:23:37 | mriedem | https://github.com/larsks/os-placement-tools/blob/master/check_placement.py#L16 is nice | |
| 18:24:11 | larsks | Thanks! | |
| 18:24:14 | dansmith | mriedem: nope | |
| 18:24:41 | dansmith | mriedem: it does coincide with me ruminating about deleting duplicate allocations in -placement last week though | |
| 18:25:27 | mriedem | wonder if anyone ever started on that osc-placement change to orchestrate setting inventory on an entire placement provider aggregate set | |
| 18:25:39 | mriedem | unrelated to this besides placement tooling | |
| 18:25:39 | dansmith | larsks: that looks awesome | |
| 18:25:59 | dansmith | larsks: you should check for instance state and only agree to --fix those in ACTIVE though | |
| 18:26:13 | dansmith | larsks: before ocata (ish?) you could have legit reasons for two allocations for one instance, because that's how we did migrations | |
| 18:26:28 | dansmith | but if the instance is ACTIVE and has two, that'll always need fixing I think | |
| 18:26:35 | dansmith | mriedem: right? | |
| 18:26:43 | mriedem | and/or don't touch any instances with a non-null task_state | |
| 18:26:57 | mriedem | before pike the computes would heal allocations | |
| 18:27:06 | dansmith | yeah | |
| 18:27:23 | mriedem | but yeah don't fix an instance that's undergoing a task_state transition | |
| 18:27:36 | mriedem | yee be warned | |
| 18:31:38 | larsks | So noted. | |
| 18:32:43 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add Project object https://review.openstack.org/565401 | |
| 18:32:44 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add User object to placement modeling https://review.openstack.org/565402 | |
| 18:32:45 | openstackgerrit | Jay Pipes proposed openstack/nova master: add Consumer object https://review.openstack.org/565403 | |
| 18:32:46 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add create() methods to Project, User and Consumer https://review.openstack.org/565404 | |
| 18:32:47 | openstackgerrit | Jay Pipes proposed openstack/nova master: move consumer ensure to API layer https://review.openstack.org/565405 | |
| 18:32:48 | openstackgerrit | Jay Pipes proposed openstack/nova master: remove Allocation.project_id & Allocation.user_id https://review.openstack.org/565406 | |
| 18:32:49 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework allocation handler _allocations_dict() https://review.openstack.org/565407 | |
| 18:32:50 | openstackgerrit | Jay Pipes proposed openstack/nova master: increment consumer generation when allocs change https://review.openstack.org/565513 | |
| 18:36:02 | melwitt | looks like libvirt-qemu is the right user and group, actually logged in the devstack log http://logs.openstack.org/49/479949/71/check/nova-next/7751dc4/logs/devstacklog.txt.gz#_2018-05-01_15_04_42_683 | |
| 18:36:26 | mriedem | huh | |
| 18:36:29 | mriedem | nuts | |
| 18:36:54 | melwitt | yeah. back to the drawing board | |
| 18:42:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate the nova-consoleauth service https://review.openstack.org/565367 | |
| 18:43:37 | mriedem | alright https://review.openstack.org/#/q/topic:bp/convert-consoles-to-objects+status:open is ready to go | |
| 18:43:44 | mriedem | and it's the last day in the runway | |
| 18:45:45 | melwitt | so close ... | |
| 18:46:04 | mriedem | well, efried and dansmith can make that magic happen | |
| 18:46:24 | mriedem | *or | |
| 18:46:27 | melwitt | note that once the "convert websocketproxy" change is approved, we need to ask frickler to lift the -2 on the first devstack sandwich change so it can go through | |
| 18:46:33 | mriedem | yup | |
| 18:46:36 | efried | mriedem: subtle. | |
| 18:54:33 | melwitt | dansmith: do you remember why we have to "enable before stop" for the conductor fleet? I had the same thing for the console proxies https://review.openstack.org/#/c/484973/12/lib/nova@1078 | |
| 18:56:50 | dansmith | if the thing isn't enabled, then stop_service won't do anything, | |
| 18:57:08 | dansmith | but I thnk that conductor case was just for when we were moving to super conductor | |
| 18:57:19 | dansmith | although maybe it's still required since we generate the service id for services in the cell? | |
| 19:00:28 | melwitt | I wasn't sure why we can't assume they're enabled from the earlier call to start_nova_console_proxies | |
| 19:01:11 | dansmith | idk, I'm kinda into something else at the moment so I can't look deeply, | |
| 19:01:21 | dansmith | but what I said is what I remember off the top of my head for that stuff | |
| 19:02:45 | melwitt | yeah, np, thanks for that. not looking for a deep answer, wish I had written a comment on here from back when someone explained it | |
| 19:05:18 | openstackgerrit | Eric Fried proposed openstack/nova master: placement: Granular GET /allocation_candidates https://review.openstack.org/517757 | |
| 19:05:27 | melwitt | I think it might be because start_* and stop_* aren't going to happen in the same run (stack.sh vs unstack.sh) and there's not going to be any saving of state about the cell services between those runs | |
| 19:06:20 | melwitt | so during the unstack, you can't stop the per cell processes unless you enable them first | |
| 19:07:41 | dansmith | that's what I'm saying, | |
| 19:07:49 | dansmith | because we generate their service names from the cell we're in | |
| 19:07:52 | dansmith | n-cond-cell1, etc | |
| 19:07:55 | melwitt | right | |
| 19:08:37 | melwitt | what I was missing is that there's no saving of state after a stack.sh so there's no way the teardown would "save" the fact that the services were created and enabled by the stack.sh run | |
| 19:21:02 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: update add-consumer-generation to focus on API https://review.openstack.org/565565 | |
| 20:00:49 | openstackgerrit | Oliver Walsh proposed openstack/nova master: libvirt: fix setting tx_queue_size when rx_queue_size is not set https://review.openstack.org/565573 | |
| 20:07:39 | efried | owalsh: Do we need a bug for ^ ? | |
| 20:13:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanup placement policy generator docs https://review.openstack.org/565225 | |
| 20:13:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578 | |
| 20:15:07 | mriedem | efried: i thought about that, but it's not even tagged in rc1 | |
| 20:15:21 | mriedem | so not really a good chance of someone hitting that besides red hat QE | |
| 20:15:54 | efried | mriedem: Sure, it's only been broken for a week, but still, paperwork. | |
| 20:16:03 | efried | mriedem: If you of all people don't need a bug, I'm sure not gonna push for it :) | |
| 20:16:38 | owalsh | awww but I love paperwork :-( | |
| 20:18:03 | owalsh | efried, mriedem: yea, I'm lazy and it's not released yet but can raise one if you think it's necessary | |
| 20:19:20 | openstackgerrit | Merged openstack/nova master: Add user_id to RequestSpec https://review.openstack.org/565340 | |
| 20:20:56 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Plumb trusted_certs through libvirt driver image paths https://review.openstack.org/561262 | |
| 20:20:57 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 20:20:58 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add notification support for trusted_certs https://review.openstack.org/563269 | |
| 20:34:01 | melwitt | hm, looks like all of the third party CIs suddenly are failing | |
| 20:35:00 | mriedem | all or just powervm? | |
| 20:35:04 | melwitt | hyperv CI can't resolve a hostname, some others like powervm and virtuozzo, the log files aren't found | |
| 20:35:14 | mriedem | i blame zuul | |
| 20:35:16 | melwitt | and it seems like this has happened starting today | |
| 20:43:52 | randomhack_ | was this bug ever solved? https://bugs.launchpad.net/nova/+bug/1633033. ran into this today on a Newton in | |
| 20:43:52 | openstack | Launchpad bug 1633033 in OpenStack Compute (nova) "live migration with encrypted volume fails" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 20:44:54 | randomhack_ | love migration of an instance with encrypted volume resulted in volume being mounted without decryption and corrupted the file system | |
| 20:45:42 | randomhack_ | s/love/live | |
| 20:46:49 | randomhack_ | known limitation of encrypted iscsi volumes? | |
| 20:48:15 | mriedem | randomhack_: lyarwood would know but it's really late for him (UK) | |
| 20:49:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425 | |
| 20:49:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578 | |
| 20:49:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanup placement policy generator docs https://review.openstack.org/565225 | |
| 20:49:24 | randomhack_ | mriedem: thanks i’ll check tomorrow earlier | |
| 20:49:32 | melwitt | yeah. IIRC, it will work as of pike when using qemu native luks decryption https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/libvirt-qemu-native-luks.html | |
| 20:49:59 | randomhack_ | melwitt: thanks! | |
| 20:50:04 | melwitt | because I thought lyarwood had a depends-on patch that ran live migration on top of the series | |