| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-13 | |||
| 12:12:51 | ohwhyosa | But when I try to access the console, I get "Invalid token, expired or invalid token"... I've checked the logs, and even took the liberty of modifying the python file to provide more logging information...and the token string is empty (not when it hashes it, though, because it does still hash an empty string) | |
| 12:12:52 | sean-k-mooney | basically im guessing teh lxv-contaiern-destroy destored the compute node containers but not placment right? | |
| 12:13:19 | ohwhyosa | It should have destroyed everything, I think I only have containers on infra1 | |
| 12:13:41 | ohwhyosa | but could be, that would explain why the hostname was already registered | |
| 12:13:50 | sean-k-mooney | is infra1 where your contol plain services run | |
| 12:13:56 | ohwhyosa | Yup | |
| 12:14:01 | sean-k-mooney | ok | |
| 12:14:23 | sean-k-mooney | what i think happend is that we destroyted the compute agent and the compute service entry for that agent | |
| 12:14:45 | sean-k-mooney | but im assuming we could not destroy the plamcenet resouce providers because there were allcoation against them | |
| 12:14:59 | ohwhyosa | I'm checking, and all the console_auth_tokens are on the nova database, cell0 has that table empty, in case that is of importance | |
| 12:15:02 | sean-k-mooney | so when you redeployed a new compute service was created | |
| 12:15:14 | ohwhyosa | Hmmm makes sense as a protective measure | |
| 12:15:42 | sean-k-mooney | and that had a new uuid and when it tried to create a new inventory in placment it failed because the old RP still existd wiht a differnet uuid but the same name | |
| 12:16:17 | ohwhyosa | Yup, it did complain about uuid (though that problem "vanishged" after changing hostname...which I now realize might not have been a great idea) | |
| 12:16:27 | ohwhyosa | But didn't lxc-containers-destroy nuke the DB as well? | |
| 12:16:58 | sean-k-mooney | ohwhyosa: if you check placement for a resouce provider with the onld hostname it should have allocation against it | |
| 12:17:14 | sean-k-mooney | if that is the case its the same issue we are currently trying to fix | |
| 12:18:40 | ohwhyosa | Command would be openstack allocation candidate list? | |
| 12:19:21 | sean-k-mooney | am im not sure if we can do this vai the openstack cli but no i dont think so | |
| 12:20:12 | ohwhyosa | sean-k-mooney: How should I check? | |
| 12:20:18 | sean-k-mooney | you need to hit this endpoint https://developer.openstack.org/api-ref/placement/?expanded=list-resource-provider-allocations-detail#list-resource-provider-allocations | |
| 12:20:38 | sean-k-mooney | but ill see if there is a cli you can use one second | |
| 12:20:43 | ohwhyosa | Thanks! brb, will check | |
| 12:21:33 | sean-k-mooney | ohwhyosa: this is the mailing list post related to this issue by the way http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007097.html | |
| 12:22:45 | sean-k-mooney | ohwhyosa: openstack resource provider allocation show | |
| 12:23:34 | sean-k-mooney | actully no that is not per RP its per consumer which is not what you want | |
| 12:24:14 | ohwhyosa | sean-k-mooney: that command returns "is not an openstack command" | |
| 12:24:30 | ohwhyosa | sean-k-mooney: It recommends resource member list, service provider list and service list | |
| 12:24:51 | sean-k-mooney | you would need a new enough version of osc-placmenet for it to be there | |
| 12:25:01 | sean-k-mooney | but it also is incorrect for what you want to do | |
| 12:28:00 | ohwhyosa | Hmmm according to openstack endpoints list tells me that placement api is at 10.0.0.23:8780 | |
| 12:28:21 | ohwhyosa | But nmap tells me that port is closed and thus curl connectionr efused | |
| 12:28:55 | sean-k-mooney | that is strange | |
| 12:29:40 | sean-k-mooney | do you need to run the command within one of the lxc continers | |
| 12:29:52 | ohwhyosa | the openstack one? | |
| 12:30:15 | sean-k-mooney | yes | |
| 12:30:26 | sean-k-mooney | im not sure how osa setup your networking | |
| 12:30:46 | ohwhyosa | Yup, I use it from the utility container | |
| 12:31:32 | ohwhyosa | Okay, now I have access to the api | |
| 12:31:42 | ohwhyosa | I was using the external_lb_vip u.u | |
| 12:31:51 | sean-k-mooney | ah | |
| 12:31:59 | sean-k-mooney | well i would normally suggest doing | |
| 12:32:01 | sean-k-mooney | curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://10.0.0.23:8780/resource_providers/${uuid}/allocations" | python -mjson.tool | |
| 12:32:29 | sean-k-mooney | assuming you have a token and know the uuid of the RP with the old host name | |
| 12:33:41 | sean-k-mooney | you can get a token with "OS_TOKEN=$(openstack token issue -c id -f value)" | |
| 12:35:33 | sean-k-mooney | you can get teh uuid with "uuid=$(resource provider list --name ${old_hostname} -c uuid -f value)" | |
| 12:37:48 | sean-k-mooney | sorry "uuid=$(openstack resource provider list --name ${old_hostname} -c uuid -f value)" | |
| 12:39:10 | ohwhyosa | resource member or service provider? | |
| 12:39:17 | ohwhyosa | resource provider complains it doesn't exists | |
| 12:39:22 | ohwhyosa | and thanks a lot, I'm learning a ton | |
| 12:40:06 | sean-k-mooney | do you have osc-placement installed? and if so what version | |
| 12:41:27 | ohwhyosa | I believe I don't, I don't seem to be able to use any of its commands | |
| 12:42:34 | sean-k-mooney | ok you could install it but since we are using curl anyway this will fined the RP for you | |
| 12:42:36 | sean-k-mooney | curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers/?name=${old_hostname}" | python -mjson.tool | |
| 12:43:00 | sean-k-mooney | you need to replace 192.168.100.253/placement with your placemetn endpoint | |
| 12:45:13 | ohwhyosa | Not found (neither with the new nor the old hostname) | |
| 12:45:23 | sean-k-mooney | ya i had a typo | |
| 12:45:25 | sean-k-mooney | curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers?name=${old_hostname}" | python -mjson.tool | |
| 12:45:42 | sean-k-mooney | you do not want the / at the end of the url i think that was the error | |
| 12:46:25 | ohwhyosa | Yup! | |
| 12:46:34 | ohwhyosa | Indeed I have both the old name and the new one | |
| 12:47:22 | sean-k-mooney | cool so if you grab the uuid for the old name and append <uuid>/allocations to that url | |
| 12:47:34 | sean-k-mooney | then it should not be empty on the old host name correct? | |
| 12:47:54 | sean-k-mooney | e.g. curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers/${uuid}/allocations" | python -mjson.tool | |
| 12:48:33 | ohwhyosa | { | |
| 12:48:33 | ohwhyosa | "resource_provider_generation": 2, | |
| 12:48:33 | ohwhyosa | "allocations": {} | |
| 12:48:33 | ohwhyosa | } | |
| 12:48:46 | sean-k-mooney | huh i was expecting that to be non empty | |
| 12:48:57 | sean-k-mooney | well the issue is that that RP was not deleted | |
| 12:49:40 | sean-k-mooney | i guess the lxc container destory playbook may not have actully deleted it properly | |
| 12:49:57 | ohwhyosa | It has self link, inventories link and usages link when getting the uuid | |
| 12:50:58 | ohwhyosa | Hmmm could be, should I just delete it? | |
| 12:51:16 | ohwhyosa | And should that fix the novnc console also? | |
| 12:51:28 | sean-k-mooney | you have changed the hostname so it should not be an issue. | |
| 12:51:40 | sean-k-mooney | i think the novnc console issue is seperate | |
| 12:52:21 | sean-k-mooney | the fact the RP exsited is why you could not originally recreate the invenoties when the compute agent started up | |
| 12:52:30 | ohwhyosa | It does have a populated resource inventory though | |
| 12:52:35 | ohwhyosa | Oooooh get it! | |
| 12:52:37 | ohwhyosa | Thanks a ton | |
| 12:53:21 | sean-k-mooney | ya the invetory and RP are just left over form the last time the compute node ran update_provider_tree before you redeploy | |
| 12:54:04 | sean-k-mooney | you should clean up the RPs so that the schduler does not try to select them but the nova filters shoudl eliminate them anyway | |
| 12:54:32 | sean-k-mooney | before you try to delete them can you first check if there is a compute service with the old name still | |
| 12:54:51 | ohwhyosa | sorry, i deleted it before reading this | |
| 12:55:02 | ohwhyosa | How can I check if there is a service with the old name? | |
| 12:56:06 | sean-k-mooney | openstack compute service list --host ${old_hostname} --service nova-compute | |
| 12:56:38 | ohwhyosa | Yep, there is one! enabled and down | |
| 12:56:58 | sean-k-mooney | ya its down because it has not recived a heartbeat since you redeployed. | |
| 12:57:28 | ohwhyosa | I should disable and delete it right? | |
| 12:57:48 | ohwhyosa | So, only two issues left with nova for now hahaha the novnc proxy and the err 32 broken pipe | |
| 12:57:49 | sean-k-mooney | if you do openstack compute service delete <service uuid> it will clean up the old placmenet RP too | |
| 12:58:08 | ohwhyosa | Oh! So it does already what I did via the api! | |
| 12:58:25 | sean-k-mooney | ya | |
| 12:58:38 | sean-k-mooney | the playbook you ran jsut nuked the containers | |
| 12:58:51 | sean-k-mooney | it did not also clean up the compute resouces in nova or placmenet | |
| 12:59:25 | sean-k-mooney | normally if it reused the same host name it should have reused the same compute service but im guessing somthing went wrong | |
| 13:00:49 | ohwhyosa | Good, good, note to self then, delete all the resources before nuking containers | |
| 13:01:33 | sean-k-mooney | well i would check with the openstack-ansible people on how they expect it to be done | |
| 13:01:52 | sean-k-mooney | e.g. im sure they have a playbook or docs for reporviioning node somewhere | |
| 13:02:10 | sean-k-mooney | or they will be able to tell you what the workflow shoud be with there tooling | |