| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-18 | |||
| 20:27:18 | mriedem | but, brain is mush | |
| 20:27:32 | mriedem | i would realize it at 3am most likely | |
| 20:44:48 | openstackgerrit | sean mooney proposed openstack/nova master: Add regression test for bug 1781710 https://review.openstack.org/583339 | |
| 20:44:50 | openstack | bug 1781710 in OpenStack Compute (nova) "ServersOnMultiNodesTest.test_create_server_with_scheduler_hint_group_anti_affinity failing with "Servers are on the same host"" [High,Fix released] https://launchpad.net/bugs/1781710 - Assigned to Matt Riedemann (mriedem) | |
| 20:47:34 | sean-k-mooney | by the way have people hit this. | |
| 20:47:36 | sean-k-mooney | RuntimeError: On Python 3, zvm sdk supports to Python 3.5 | |
| 20:48:21 | sean-k-mooney | im currently on centos 7.5 and i install python 3.6. we dont support 3.4 upstream right | |
| 20:49:16 | sean-k-mooney | centos only ship 3.4 and 3.6 in the package manager | |
| 20:50:33 | sean-k-mooney | efried: do you know what are the changes of getting 3.6 support in the zvm sdk | |
| 20:50:59 | efried | sean-k-mooney: You mean so you can tox with the effin thing? | |
| 20:51:12 | sean-k-mooney | yes | |
| 20:51:29 | sean-k-mooney | right now i cand run pep8 or any of the python3 based envs | |
| 20:51:46 | sean-k-mooney | unless in install 3.5 that is or downgrade to 3.4 | |
| 20:51:49 | efried | sean-k-mooney: Sorry, no idea. I got around it by sudo pip installing zVMCloudConnector on my system (not the venv) | |
| 20:52:04 | efried | sean-k-mooney: Oh, maybe 3.6 support is a different issue. | |
| 20:52:50 | sean-k-mooney | there is a check for the python 3 version. i symlinked python3 to 3.6 inst normally 3.4 | |
| 20:52:57 | sean-k-mooney | on centos anyway. | |
| 20:53:12 | sean-k-mooney | i can just swap the symlink back to 3.4 for now | |
| 20:55:07 | sean-k-mooney | are the gates still running ubuntu 16.04? | |
| 20:55:25 | sean-k-mooney | ubuntu 18.04 also ships 3.6 by defualt i think now | |
| 20:57:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715 | |
| 20:57:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739 | |
| 20:57:15 | mriedem | we're still testing against 16.04 in the gate for rocky so that's what i use | |
| 20:58:15 | sean-k-mooney | well untill the zvm sdk is drop as a depency or upgraded to support 3.6 we wont be able to move nova gate jobs to 18.04 | |
| 20:58:44 | sean-k-mooney | at least given the python3 first goal | |
| 20:58:54 | mriedem | alert the TC! | |
| 20:59:30 | mriedem | that's this thing right? https://github.com/mfcloud/python-zvm-sdk/blob/master/setup.py | |
| 20:59:31 | efried | sean-k-mooney: I wish dansmith or mikal or someone had warned us about problems like this. http://lists.openstack.org/pipermail/openstack-dev/2018-April/129294.html | |
| 20:59:57 | mriedem | https://github.com/mfcloud/python-zvm-sdk/blob/master/setup.py#L39 | |
| 21:00:13 | efried | http://lists.openstack.org/pipermail/openstack-dev/2018-April/129350.html | |
| 21:00:50 | mriedem | i guess this is where we'd report a bug that it doesn't work with 3.6 https://bugs.launchpad.net/python-zvm-sdk/+bugs | |
| 21:01:00 | mriedem | and then harass the shit out of jichenjc to get that fixed and released | |
| 21:01:29 | efried | support 3.7 while we're at it, so we don't get behind immediately. | |
| 21:01:45 | sean-k-mooney | mriedem: ya the weird thing is the error message seams to indicate it support 3.5 but only declares 2.7 support in that file | |
| 21:01:57 | mriedem | yeah it definitely supports 3.5 https://github.com/mfcloud/python-zvm-sdk/blob/master/tox.ini#L27 | |
| 21:02:04 | mriedem | what blows up with 3.6? | |
| 21:02:09 | mriedem | it just doesn't install/ | |
| 21:02:10 | mriedem | ? | |
| 21:02:23 | sean-k-mooney | yep tox fails while pip installing it | |
| 21:02:30 | mriedem | what's the error? | |
| 21:02:56 | sean-k-mooney | one sec | |
| 21:04:29 | sean-k-mooney | http://paste.openstack.org/show/726237/ | |
| 21:06:04 | mriedem | bingo https://github.com/mfcloud/python-zvm-sdk/blob/master/zvmsdk/version.py#L28 | |
| 21:07:02 | sean-k-mooney | you know if you replace that 5 with a 6 it will proably work | |
| 21:07:28 | mriedem | try it | |
| 21:07:55 | mriedem | or just try doing that and add a tox36 target and run it's unit tests | |
| 21:08:08 | mriedem | this could be your first (and only ever) zvmcloudconnector patch :) | |
| 21:09:31 | sean-k-mooney | some might say its one too many. | |
| 21:09:54 | sean-k-mooney | that said i just downloaded and ran its py27 unitests and one of them still failed | |
| 21:09:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715 | |
| 21:09:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739 | |
| 21:10:28 | mriedem | let me state one more time, for the record, for this week, that i hate all of the request spec compat code in conductor | |
| 21:11:12 | sean-k-mooney | mriedem: duely noted. | |
| 21:11:57 | sean-k-mooney | i just found another bug in there test code... i am going to go down a rabbit whole with this i can tell | |
| 21:12:31 | _ix | Afternoon folks. I realize this channel is less support oriented, but there's no response in #openstack. I'm having an issue in our pike deployment with resource allocations still existing on one compute, but the vm itself resides on another. It's not reflecting additional usage on the original VM, but the scheduler still won't allow me to instantiate there. | |
| 21:12:39 | _ix | How can I clear the allocation from the original hypervisor? | |
| 21:13:05 | _ix | > There are allocations remaining against the source host that might need to be removed | |
| 21:14:21 | sean-k-mooney | _ix: did you migrate or resize the vm at some point? if you have not completed the migrations then it will have both allocations if i rembere correctly | |
| 21:14:39 | _ix | I did, but the migration failed. | |
| 21:15:01 | _ix | I believe it was the case where a live migration was ongoing, and the targeted compute's services were unavailable or became unavailable. | |
| 21:15:24 | _ix | From a user's perspective, there doesn't seem to be a way to back out of the migration. | |
| 21:15:40 | sean-k-mooney | _ix so the vm migrated but the migration timeded out becase the node became unableale? | |
| 21:16:23 | _ix | The vm stayed on the source but received allocation on the target. | |
| 21:16:24 | sean-k-mooney | _ix: well migration is an admin action so "end users" should not be aware of them in general | |
| 21:16:33 | _ix | That's a fair point. | |
| 21:17:10 | sean-k-mooney | mriedem: jaypipes efried do you know how to clean up the stale allocation? | |
| 21:17:29 | _ix | How can I force the issue? I've manually added allocations records to reflect reality in the nova_api.allocations table and marked the failed migration as deleted in nova.migrations. | |
| 21:17:51 | efried | sean-k-mooney: I haven't been following the conversation. You want to manually remove an allocation from placement? | |
| 21:18:02 | _ix | Yes please. | |
| 21:18:15 | sean-k-mooney | efried: live migration failed but allocation on dest remain | |
| 21:18:32 | efried | mriedem is the osc-placement expert. I can only tell you how to do it programmatically :) | |
| 21:19:48 | _ix | It's an open bug, but it's such an edge case, I totally understand why this might lead to something out-of-state: https://bugs.launchpad.net/nova/+bug/1753676 | |
| 21:19:48 | openstack | Launchpad bug 1753676 in OpenStack Compute (nova) "Live migration not working as Expected when Restarting nova-compute service while migration from source node" [Undecided,Confirmed] | |
| 21:20:23 | _ix | I'm pleased to report live migrations tend to work quite well and fast, I might add. | |
| 21:20:34 | sean-k-mooney | _ix: ya migration in pike was kind of buggy. dansmith was working on fixing that i think | |
| 21:20:56 | mriedem | yeah osc-placement | |
| 21:21:16 | mriedem | https://docs.openstack.org/osc-placement/latest/index.html | |
| 21:21:29 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-set specifically | |
| 21:21:46 | _ix | mriedem: Thanks! | |
| 21:21:51 | mriedem | if the consumer (instance in this case) shows having allocations against multiple resource providers (compute nodes), then determine the one you want to keep and the one you want to remove, and update the allocations | |
| 21:22:04 | mriedem | you can view the allocations with https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-aggregate-list | |
| 21:22:07 | mriedem | oops | |
| 21:22:12 | sean-k-mooney | mriedem: would the consumer not be the migration in this case | |
| 21:22:13 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-show | |
| 21:22:18 | mriedem | sean-k-mooney: not in pike | |
| 21:22:25 | sean-k-mooney | oh ok | |
| 21:22:27 | mriedem | that change happened in queens | |
| 21:22:41 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/migration-allocations.html | |
| 21:22:55 | mriedem | also, pike GA had a few bugs with cleaning up allocations during failures, | |
| 21:23:01 | mriedem | which we fixed in stable point releases of pike | |
| 21:23:07 | mriedem | _ix: which version of pike are you running? latest stable i hope | |
| 21:23:18 | mriedem | https://docs.openstack.org/releasenotes/nova/pike.html | |
| 21:24:20 | _ix | mriedem: Latest stable, I believe. | |
| 21:24:42 | _ix | I must be missing a client. What enables the placement hierarchy? | |
| 21:25:11 | mriedem | _ix: https://pypi.org/project/osc-placement/ | |
| 21:25:26 | mriedem | it's an osc plugin | |
| 21:25:56 | _ix | I wish I were more familiar. I'll do some reading. Thanks very much! | |
| 21:27:47 | mriedem | np | |
| 21:28:21 | mriedem | placement is a new big required thing to a lot of people in pike so it's ok to ask operational questions here | |
| 21:36:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add regression test for bug 1781710 https://review.openstack.org/583339 | |