Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-13
11:55:15 jangutter https://lwn.net/ml/python-dev/bf0bf5b8-1ef0-556b-632c-eaebfb325496@python.org/
11:55:24 jaypipes jangutter: what the actual f!?
11:55:26 jaypipes :)
11:55:38 jaypipes it's a conspiracy!
11:56:45 jangutter jaypipes: I guess they'll have to adopt the much superior 'import gutter' module.
11:57:18 jaypipes lol
11:58:06 kashyap Hehe
11:58:12 frickler woa, being called a dead battery sounds really harsh to me
12:04:15 openstackgerrit Harald Jensås proposed openstack/nova master: cell_mapping - wrap IPv6 address in format_url https://review.opendev.org/664559
12:06:14 ohwhyosa Hiya there!
12:06:47 ohwhyosa I was advised to come here to ask about an issue, I'll try to summarize it as well as I can!
12:07:06 ohwhyosa So, I have a 3-nodes openstack-ansible deployment
12:07:34 ohwhyosa infra1 (10.0.0.11) compute1(10.0.0.9) and storage1(10.0.0.10)
12:08:11 ohwhyosa It was kinda working on the first deployment, using spice (though it worked horribly) and without multidomain suport
12:08:30 ohwhyosa So I nuked the lxc containers, deployed again with novnc and multidomain support
12:08:53 ohwhyosa I nuked them using lxc-containers-destroy.yml
12:10:14 ohwhyosa when it was all up again, I couldn't create instances (it complained no hosts were available, though that was a lie, there were plenty resources)
12:11:13 ohwhyosa Looking at the logs, placement said it couldn't assign resources to that node beacsue "hostname" already existed
12:11:31 ohwhyosa I changed the hostname, rebooted everything, and now I can create instances
12:11:33 sean-k-mooney ohwhyosa: so i think i know what the issue is
12:12:17 sean-k-mooney let me see if i can find the bug
12:12:51 ohwhyosa But when I try to access the console, I get "Invalid token, expired or invalid token"... I've checked the logs, and even took the liberty of modifying the python file to provide more logging information...and the token string is empty (not when it hashes it, though, because it does still hash an empty string)
12:12:52 sean-k-mooney basically im guessing teh lxv-contaiern-destroy destored the compute node containers but not placment right?
12:13:19 ohwhyosa It should have destroyed everything, I think I only have containers on infra1
12:13:41 ohwhyosa but could be, that would explain why the hostname was already registered
12:13:50 sean-k-mooney is infra1 where your contol plain services run
12:13:56 ohwhyosa Yup
12:14:01 sean-k-mooney ok
12:14:23 sean-k-mooney what i think happend is that we destroyted the compute agent and the compute service entry for that agent
12:14:45 sean-k-mooney but im assuming we could not destroy the plamcenet resouce providers because there were allcoation against them
12:14:59 ohwhyosa I'm checking, and all the console_auth_tokens are on the nova database, cell0 has that table empty, in case that is of importance
12:15:02 sean-k-mooney so when you redeployed a new compute service was created
12:15:14 ohwhyosa Hmmm makes sense as a protective measure
12:15:42 sean-k-mooney and that had a new uuid and when it tried to create a new inventory in placment it failed because the old RP still existd wiht a differnet uuid but the same name
12:16:17 ohwhyosa Yup, it did complain about uuid (though that problem "vanishged" after changing hostname...which I now realize might not have been a great idea)
12:16:27 ohwhyosa But didn't lxc-containers-destroy nuke the DB as well?
12:16:58 sean-k-mooney ohwhyosa: if you check placement for a resouce provider with the onld hostname it should have allocation against it
12:17:14 sean-k-mooney if that is the case its the same issue we are currently trying to fix
12:18:40 ohwhyosa Command would be openstack allocation candidate list?
12:19:21 sean-k-mooney am im not sure if we can do this vai the openstack cli but no i dont think so
12:20:12 ohwhyosa sean-k-mooney: How should I check?
12:20:18 sean-k-mooney you need to hit this endpoint https://developer.openstack.org/api-ref/placement/?expanded=list-resource-provider-allocations-detail#list-resource-provider-allocations
12:20:38 sean-k-mooney but ill see if there is a cli you can use one second
12:20:43 ohwhyosa Thanks! brb, will check
12:21:33 sean-k-mooney ohwhyosa: this is the mailing list post related to this issue by the way http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007097.html
12:22:45 sean-k-mooney ohwhyosa: openstack resource provider allocation show
12:23:34 sean-k-mooney actully no that is not per RP its per consumer which is not what you want
12:24:14 ohwhyosa sean-k-mooney: that command returns "is not an openstack command"
12:24:30 ohwhyosa sean-k-mooney: It recommends resource member list, service provider list and service list
12:24:51 sean-k-mooney you would need a new enough version of osc-placmenet for it to be there
12:25:01 sean-k-mooney but it also is incorrect for what you want to do
12:28:00 ohwhyosa Hmmm according to openstack endpoints list tells me that placement api is at 10.0.0.23:8780
12:28:21 ohwhyosa But nmap tells me that port is closed and thus curl connectionr efused
12:28:55 sean-k-mooney that is strange
12:29:40 sean-k-mooney do you need to run the command within one of the lxc continers
12:29:52 ohwhyosa the openstack one?
12:30:15 sean-k-mooney yes
12:30:26 sean-k-mooney im not sure how osa setup your networking
12:30:46 ohwhyosa Yup, I use it from the utility container
12:31:32 ohwhyosa Okay, now I have access to the api
12:31:42 ohwhyosa I was using the external_lb_vip u.u
12:31:51 sean-k-mooney ah
12:31:59 sean-k-mooney well i would normally suggest doing
12:32:01 sean-k-mooney curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://10.0.0.23:8780/resource_providers/${uuid}/allocations" | python -mjson.tool
12:32:29 sean-k-mooney assuming you have a token and know the uuid of the RP with the old host name
12:33:41 sean-k-mooney you can get a token with "OS_TOKEN=$(openstack token issue -c id -f value)"
12:35:33 sean-k-mooney you can get teh uuid with "uuid=$(resource provider list --name ${old_hostname} -c uuid -f value)"
12:37:48 sean-k-mooney sorry "uuid=$(openstack resource provider list --name ${old_hostname} -c uuid -f value)"
12:39:10 ohwhyosa resource member or service provider?
12:39:17 ohwhyosa resource provider complains it doesn't exists
12:39:22 ohwhyosa and thanks a lot, I'm learning a ton
12:40:06 sean-k-mooney do you have osc-placement installed? and if so what version
12:41:27 ohwhyosa I believe I don't, I don't seem to be able to use any of its commands
12:42:34 sean-k-mooney ok you could install it but since we are using curl anyway this will fined the RP for you
12:42:36 sean-k-mooney curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers/?name=${old_hostname}" | python -mjson.tool
12:43:00 sean-k-mooney you need to replace 192.168.100.253/placement with your placemetn endpoint
12:45:13 ohwhyosa Not found (neither with the new nor the old hostname)
12:45:23 sean-k-mooney ya i had a typo
12:45:25 sean-k-mooney curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers?name=${old_hostname}" | python -mjson.tool
12:45:42 sean-k-mooney you do not want the / at the end of the url i think that was the error
12:46:25 ohwhyosa Yup!
12:46:34 ohwhyosa Indeed I have both the old name and the new one
12:47:22 sean-k-mooney cool so if you grab the uuid for the old name and append <uuid>/allocations to that url
12:47:34 sean-k-mooney then it should not be empty on the old host name correct?
12:47:54 sean-k-mooney e.g. curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers/${uuid}/allocations" | python -mjson.tool
12:48:33 ohwhyosa {
12:48:33 ohwhyosa "resource_provider_generation": 2,
12:48:33 ohwhyosa "allocations": {}
12:48:33 ohwhyosa }
12:48:46 sean-k-mooney huh i was expecting that to be non empty
12:48:57 sean-k-mooney well the issue is that that RP was not deleted
12:49:40 sean-k-mooney i guess the lxc container destory playbook may not have actully deleted it properly
12:49:57 ohwhyosa It has self link, inventories link and usages link when getting the uuid
12:50:58 ohwhyosa Hmmm could be, should I just delete it?
12:51:16 ohwhyosa And should that fix the novnc console also?
12:51:28 sean-k-mooney you have changed the hostname so it should not be an issue.
12:51:40 sean-k-mooney i think the novnc console issue is seperate
12:52:21 sean-k-mooney the fact the RP exsited is why you could not originally recreate the invenoties when the compute agent started up
12:52:30 ohwhyosa It does have a populated resource inventory though
12:52:35 ohwhyosa Oooooh get it!

Earlier   Later