| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-13 | |||
| 12:22:45 | sean-k-mooney | ohwhyosa: openstack resource provider allocation show | |
| 12:23:34 | sean-k-mooney | actully no that is not per RP its per consumer which is not what you want | |
| 12:24:14 | ohwhyosa | sean-k-mooney: that command returns "is not an openstack command" | |
| 12:24:30 | ohwhyosa | sean-k-mooney: It recommends resource member list, service provider list and service list | |
| 12:24:51 | sean-k-mooney | you would need a new enough version of osc-placmenet for it to be there | |
| 12:25:01 | sean-k-mooney | but it also is incorrect for what you want to do | |
| 12:28:00 | ohwhyosa | Hmmm according to openstack endpoints list tells me that placement api is at 10.0.0.23:8780 | |
| 12:28:21 | ohwhyosa | But nmap tells me that port is closed and thus curl connectionr efused | |
| 12:28:55 | sean-k-mooney | that is strange | |
| 12:29:40 | sean-k-mooney | do you need to run the command within one of the lxc continers | |
| 12:29:52 | ohwhyosa | the openstack one? | |
| 12:30:15 | sean-k-mooney | yes | |
| 12:30:26 | sean-k-mooney | im not sure how osa setup your networking | |
| 12:30:46 | ohwhyosa | Yup, I use it from the utility container | |
| 12:31:32 | ohwhyosa | Okay, now I have access to the api | |
| 12:31:42 | ohwhyosa | I was using the external_lb_vip u.u | |
| 12:31:51 | sean-k-mooney | ah | |
| 12:31:59 | sean-k-mooney | well i would normally suggest doing | |
| 12:32:01 | sean-k-mooney | curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://10.0.0.23:8780/resource_providers/${uuid}/allocations" | python -mjson.tool | |
| 12:32:29 | sean-k-mooney | assuming you have a token and know the uuid of the RP with the old host name | |
| 12:33:41 | sean-k-mooney | you can get a token with "OS_TOKEN=$(openstack token issue -c id -f value)" | |
| 12:35:33 | sean-k-mooney | you can get teh uuid with "uuid=$(resource provider list --name ${old_hostname} -c uuid -f value)" | |
| 12:37:48 | sean-k-mooney | sorry "uuid=$(openstack resource provider list --name ${old_hostname} -c uuid -f value)" | |
| 12:39:10 | ohwhyosa | resource member or service provider? | |
| 12:39:17 | ohwhyosa | resource provider complains it doesn't exists | |
| 12:39:22 | ohwhyosa | and thanks a lot, I'm learning a ton | |
| 12:40:06 | sean-k-mooney | do you have osc-placement installed? and if so what version | |
| 12:41:27 | ohwhyosa | I believe I don't, I don't seem to be able to use any of its commands | |
| 12:42:34 | sean-k-mooney | ok you could install it but since we are using curl anyway this will fined the RP for you | |
| 12:42:36 | sean-k-mooney | curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers/?name=${old_hostname}" | python -mjson.tool | |
| 12:43:00 | sean-k-mooney | you need to replace 192.168.100.253/placement with your placemetn endpoint | |
| 12:45:13 | ohwhyosa | Not found (neither with the new nor the old hostname) | |
| 12:45:23 | sean-k-mooney | ya i had a typo | |
| 12:45:25 | sean-k-mooney | curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers?name=${old_hostname}" | python -mjson.tool | |
| 12:45:42 | sean-k-mooney | you do not want the / at the end of the url i think that was the error | |
| 12:46:25 | ohwhyosa | Yup! | |
| 12:46:34 | ohwhyosa | Indeed I have both the old name and the new one | |
| 12:47:22 | sean-k-mooney | cool so if you grab the uuid for the old name and append <uuid>/allocations to that url | |
| 12:47:34 | sean-k-mooney | then it should not be empty on the old host name correct? | |
| 12:47:54 | sean-k-mooney | e.g. curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers/${uuid}/allocations" | python -mjson.tool | |
| 12:48:33 | ohwhyosa | { | |
| 12:48:33 | ohwhyosa | "resource_provider_generation": 2, | |
| 12:48:33 | ohwhyosa | "allocations": {} | |
| 12:48:33 | ohwhyosa | } | |
| 12:48:46 | sean-k-mooney | huh i was expecting that to be non empty | |
| 12:48:57 | sean-k-mooney | well the issue is that that RP was not deleted | |
| 12:49:40 | sean-k-mooney | i guess the lxc container destory playbook may not have actully deleted it properly | |
| 12:49:57 | ohwhyosa | It has self link, inventories link and usages link when getting the uuid | |
| 12:50:58 | ohwhyosa | Hmmm could be, should I just delete it? | |
| 12:51:16 | ohwhyosa | And should that fix the novnc console also? | |
| 12:51:28 | sean-k-mooney | you have changed the hostname so it should not be an issue. | |
| 12:51:40 | sean-k-mooney | i think the novnc console issue is seperate | |
| 12:52:21 | sean-k-mooney | the fact the RP exsited is why you could not originally recreate the invenoties when the compute agent started up | |
| 12:52:30 | ohwhyosa | It does have a populated resource inventory though | |
| 12:52:35 | ohwhyosa | Oooooh get it! | |
| 12:52:37 | ohwhyosa | Thanks a ton | |
| 12:53:21 | sean-k-mooney | ya the invetory and RP are just left over form the last time the compute node ran update_provider_tree before you redeploy | |
| 12:54:04 | sean-k-mooney | you should clean up the RPs so that the schduler does not try to select them but the nova filters shoudl eliminate them anyway | |
| 12:54:32 | sean-k-mooney | before you try to delete them can you first check if there is a compute service with the old name still | |
| 12:54:51 | ohwhyosa | sorry, i deleted it before reading this | |
| 12:55:02 | ohwhyosa | How can I check if there is a service with the old name? | |
| 12:56:06 | sean-k-mooney | openstack compute service list --host ${old_hostname} --service nova-compute | |
| 12:56:38 | ohwhyosa | Yep, there is one! enabled and down | |
| 12:56:58 | sean-k-mooney | ya its down because it has not recived a heartbeat since you redeployed. | |
| 12:57:28 | ohwhyosa | I should disable and delete it right? | |
| 12:57:48 | ohwhyosa | So, only two issues left with nova for now hahaha the novnc proxy and the err 32 broken pipe | |
| 12:57:49 | sean-k-mooney | if you do openstack compute service delete <service uuid> it will clean up the old placmenet RP too | |
| 12:58:08 | ohwhyosa | Oh! So it does already what I did via the api! | |
| 12:58:25 | sean-k-mooney | ya | |
| 12:58:38 | sean-k-mooney | the playbook you ran jsut nuked the containers | |
| 12:58:51 | sean-k-mooney | it did not also clean up the compute resouces in nova or placmenet | |
| 12:59:25 | sean-k-mooney | normally if it reused the same host name it should have reused the same compute service but im guessing somthing went wrong | |
| 13:00:49 | ohwhyosa | Good, good, note to self then, delete all the resources before nuking containers | |
| 13:01:33 | sean-k-mooney | well i would check with the openstack-ansible people on how they expect it to be done | |
| 13:01:52 | sean-k-mooney | e.g. im sure they have a playbook or docs for reporviioning node somewhere | |
| 13:02:10 | sean-k-mooney | or they will be able to tell you what the workflow shoud be with there tooling | |
| 13:02:30 | sean-k-mooney | if they dont you can file a bug and im sure they would be happy to add some | |
| 13:03:03 | ohwhyosa | Yup! I'll ask them, to be fair the whole #openstack-* ircs are full of helpful people, thanks a couple tons | |
| 13:03:05 | sean-k-mooney | ohwhyosa: mnaser is really approachable although he is also quite busy but im sure he and the other osa folk will be able to help too | |
| 13:03:47 | mnaser | yeah i don't think we have a 'off-board compute' playbook | |
| 13:03:55 | mnaser | whichgoes and cleans up placement and what not | |
| 13:04:21 | sean-k-mooney | mnaser: do you have a playbook that will reconfigure exisitng nodes? | |
| 13:04:29 | ohwhyosa | Thanks! mnaser so you recommend deleting the service via cli? | |
| 13:05:11 | mnaser | i mean usually the idea is that we run things and they automatically converge again.. in theory | |
| 13:13:26 | ohwhyosa | mnaser: Hmmm converge as in work again right? So, even if I hadn't changed the hostnamed, should it have started working again? | |
| 13:15:24 | ohwhyosa | Also, https://pastebin.com/vnmSEKDW --> This is the broken pipe error, in case you want to take a look, and this | |
| 13:18:28 | ohwhyosa | And this https://pastebin.com/A3VrPvy9 is the noVNC issue | |
| 13:18:40 | ohwhyosa | The problem, though, is that the "token" var is an empty string | |
| 13:18:51 | mriedem | amorin: do you plan on backporting this? https://review.opendev.org/#/c/660761/ | |
| 13:19:04 | ohwhyosa | I don't really know where is it querying from, it does mention something in the docs about the token being in the cookies | |
| 13:19:14 | ohwhyosa | But it is inthe url | |
| 13:19:50 | mriedem | need some stable cores on these +2ed rocky changes https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/rocky+label:Code-Review=2 | |
| 13:20:04 | lyarwood | ack, I can look shortly | |
| 13:20:31 | lyarwood | mriedem: https://review.opendev.org/#/c/663677/ would you mind looking at that in return | |
| 13:21:34 | mriedem | sure | |
| 13:22:32 | amorin | mriedem: yes | |
| 13:25:22 | amorin | mriedem: however, I think I will abandon this one: https://review.opendev.org/#/c/660982/ | |
| 13:25:37 | amorin | we dont need to force refresh, nova DB is always correct in my tests | |
| 13:25:44 | amorin | we dont need to ask neutron | |
| 13:25:48 | amorin | what do you think? | |