| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-13 | |||
| 07:18:55 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Validate requested host/node during servers create https://review.opendev.org/661237 | |
| 07:22:12 | bhagyashris | sean-k-mooney: Hi, | |
| 07:31:03 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Make evacuation respects anti-affinity rule https://review.opendev.org/649963 | |
| 07:35:35 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: libvirt: Use SATA bus for cdrom devices when using Q35 machine type https://review.opendev.org/663677 | |
| 07:36:46 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969 | |
| 07:41:39 | mkrai_ | Hi Nova Developers, I request you to provide your input on https://etherpad.openstack.org/p/ironic-nova-reset-configuration | |
| 07:44:56 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Specify availability_zone to unshelve https://review.opendev.org/663851 | |
| 07:55:30 | yaawang | sean-k-mooney: johnthetubaguy Hi, I've updated this spec, please take a look if you have time. https://review.opendev.org/#/c/651681 | |
| 07:59:01 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Specify availability_zone to unshelve https://review.opendev.org/663851 | |
| 08:17:27 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens https://review.opendev.org/637716 | |
| 08:17:59 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.opendev.org/641200 | |
| 08:26:09 | openstackgerrit | ya.wang proposed openstack/nova-specs master: Re-propose the spec to allow specifying a list of CPU models https://review.opendev.org/642030 | |
| 08:55:26 | hjensas | Hi Nova, anyone who can review (my first?) ever nova change - https://review.opendev.org/664559 | |
| 10:31:32 | gibi | hjensas: I'm on it. thanks for working on it | |
| 11:12:41 | gibi | hjensas: I've replied to you in https://review.opendev.org/664559 | |
| 11:38:12 | hjensas | gibi: thanks, I'll take look. | |
| 11:55:04 | jangutter | jaypipes: Did you know about this? If PEP 594 gets adopted you won't be able to do 'import pipes' in Python? THEY'RE REMOVING YOU! | |
| 11:55:15 | jangutter | https://lwn.net/ml/python-dev/bf0bf5b8-1ef0-556b-632c-eaebfb325496@python.org/ | |
| 11:55:24 | jaypipes | jangutter: what the actual f!? | |
| 11:55:26 | jaypipes | :) | |
| 11:55:38 | jaypipes | it's a conspiracy! | |
| 11:56:45 | jangutter | jaypipes: I guess they'll have to adopt the much superior 'import gutter' module. | |
| 11:57:18 | jaypipes | lol | |
| 11:58:06 | kashyap | Hehe | |
| 11:58:12 | frickler | woa, being called a dead battery sounds really harsh to me | |
| 12:04:15 | openstackgerrit | Harald Jensås proposed openstack/nova master: cell_mapping - wrap IPv6 address in format_url https://review.opendev.org/664559 | |
| 12:06:14 | ohwhyosa | Hiya there! | |
| 12:06:47 | ohwhyosa | I was advised to come here to ask about an issue, I'll try to summarize it as well as I can! | |
| 12:07:06 | ohwhyosa | So, I have a 3-nodes openstack-ansible deployment | |
| 12:07:34 | ohwhyosa | infra1 (10.0.0.11) compute1(10.0.0.9) and storage1(10.0.0.10) | |
| 12:08:11 | ohwhyosa | It was kinda working on the first deployment, using spice (though it worked horribly) and without multidomain suport | |
| 12:08:30 | ohwhyosa | So I nuked the lxc containers, deployed again with novnc and multidomain support | |
| 12:08:53 | ohwhyosa | I nuked them using lxc-containers-destroy.yml | |
| 12:10:14 | ohwhyosa | when it was all up again, I couldn't create instances (it complained no hosts were available, though that was a lie, there were plenty resources) | |
| 12:11:13 | ohwhyosa | Looking at the logs, placement said it couldn't assign resources to that node beacsue "hostname" already existed | |
| 12:11:31 | ohwhyosa | I changed the hostname, rebooted everything, and now I can create instances | |
| 12:11:33 | sean-k-mooney | ohwhyosa: so i think i know what the issue is | |
| 12:12:17 | sean-k-mooney | let me see if i can find the bug | |
| 12:12:51 | ohwhyosa | But when I try to access the console, I get "Invalid token, expired or invalid token"... I've checked the logs, and even took the liberty of modifying the python file to provide more logging information...and the token string is empty (not when it hashes it, though, because it does still hash an empty string) | |
| 12:12:52 | sean-k-mooney | basically im guessing teh lxv-contaiern-destroy destored the compute node containers but not placment right? | |
| 12:13:19 | ohwhyosa | It should have destroyed everything, I think I only have containers on infra1 | |
| 12:13:41 | ohwhyosa | but could be, that would explain why the hostname was already registered | |
| 12:13:50 | sean-k-mooney | is infra1 where your contol plain services run | |
| 12:13:56 | ohwhyosa | Yup | |
| 12:14:01 | sean-k-mooney | ok | |
| 12:14:23 | sean-k-mooney | what i think happend is that we destroyted the compute agent and the compute service entry for that agent | |
| 12:14:45 | sean-k-mooney | but im assuming we could not destroy the plamcenet resouce providers because there were allcoation against them | |
| 12:14:59 | ohwhyosa | I'm checking, and all the console_auth_tokens are on the nova database, cell0 has that table empty, in case that is of importance | |
| 12:15:02 | sean-k-mooney | so when you redeployed a new compute service was created | |
| 12:15:14 | ohwhyosa | Hmmm makes sense as a protective measure | |
| 12:15:42 | sean-k-mooney | and that had a new uuid and when it tried to create a new inventory in placment it failed because the old RP still existd wiht a differnet uuid but the same name | |
| 12:16:17 | ohwhyosa | Yup, it did complain about uuid (though that problem "vanishged" after changing hostname...which I now realize might not have been a great idea) | |
| 12:16:27 | ohwhyosa | But didn't lxc-containers-destroy nuke the DB as well? | |
| 12:16:58 | sean-k-mooney | ohwhyosa: if you check placement for a resouce provider with the onld hostname it should have allocation against it | |
| 12:17:14 | sean-k-mooney | if that is the case its the same issue we are currently trying to fix | |
| 12:18:40 | ohwhyosa | Command would be openstack allocation candidate list? | |
| 12:19:21 | sean-k-mooney | am im not sure if we can do this vai the openstack cli but no i dont think so | |
| 12:20:12 | ohwhyosa | sean-k-mooney: How should I check? | |
| 12:20:18 | sean-k-mooney | you need to hit this endpoint https://developer.openstack.org/api-ref/placement/?expanded=list-resource-provider-allocations-detail#list-resource-provider-allocations | |
| 12:20:38 | sean-k-mooney | but ill see if there is a cli you can use one second | |
| 12:20:43 | ohwhyosa | Thanks! brb, will check | |
| 12:21:33 | sean-k-mooney | ohwhyosa: this is the mailing list post related to this issue by the way http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007097.html | |
| 12:22:45 | sean-k-mooney | ohwhyosa: openstack resource provider allocation show | |
| 12:23:34 | sean-k-mooney | actully no that is not per RP its per consumer which is not what you want | |
| 12:24:14 | ohwhyosa | sean-k-mooney: that command returns "is not an openstack command" | |
| 12:24:30 | ohwhyosa | sean-k-mooney: It recommends resource member list, service provider list and service list | |
| 12:24:51 | sean-k-mooney | you would need a new enough version of osc-placmenet for it to be there | |
| 12:25:01 | sean-k-mooney | but it also is incorrect for what you want to do | |
| 12:28:00 | ohwhyosa | Hmmm according to openstack endpoints list tells me that placement api is at 10.0.0.23:8780 | |
| 12:28:21 | ohwhyosa | But nmap tells me that port is closed and thus curl connectionr efused | |
| 12:28:55 | sean-k-mooney | that is strange | |
| 12:29:40 | sean-k-mooney | do you need to run the command within one of the lxc continers | |
| 12:29:52 | ohwhyosa | the openstack one? | |
| 12:30:15 | sean-k-mooney | yes | |
| 12:30:26 | sean-k-mooney | im not sure how osa setup your networking | |
| 12:30:46 | ohwhyosa | Yup, I use it from the utility container | |
| 12:31:32 | ohwhyosa | Okay, now I have access to the api | |
| 12:31:42 | ohwhyosa | I was using the external_lb_vip u.u | |
| 12:31:51 | sean-k-mooney | ah | |
| 12:31:59 | sean-k-mooney | well i would normally suggest doing | |
| 12:32:01 | sean-k-mooney | curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://10.0.0.23:8780/resource_providers/${uuid}/allocations" | python -mjson.tool | |
| 12:32:29 | sean-k-mooney | assuming you have a token and know the uuid of the RP with the old host name | |
| 12:33:41 | sean-k-mooney | you can get a token with "OS_TOKEN=$(openstack token issue -c id -f value)" | |
| 12:35:33 | sean-k-mooney | you can get teh uuid with "uuid=$(resource provider list --name ${old_hostname} -c uuid -f value)" | |
| 12:37:48 | sean-k-mooney | sorry "uuid=$(openstack resource provider list --name ${old_hostname} -c uuid -f value)" | |
| 12:39:10 | ohwhyosa | resource member or service provider? | |
| 12:39:17 | ohwhyosa | resource provider complains it doesn't exists | |
| 12:39:22 | ohwhyosa | and thanks a lot, I'm learning a ton | |
| 12:40:06 | sean-k-mooney | do you have osc-placement installed? and if so what version | |
| 12:41:27 | ohwhyosa | I believe I don't, I don't seem to be able to use any of its commands | |
| 12:42:34 | sean-k-mooney | ok you could install it but since we are using curl anyway this will fined the RP for you | |
| 12:42:36 | sean-k-mooney | curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers/?name=${old_hostname}" | python -mjson.tool | |
| 12:43:00 | sean-k-mooney | you need to replace 192.168.100.253/placement with your placemetn endpoint | |
| 12:45:13 | ohwhyosa | Not found (neither with the new nor the old hostname) | |
| 12:45:23 | sean-k-mooney | ya i had a typo | |
| 12:45:25 | sean-k-mooney | curl -s -H "X-Auth-Token: ${OS_TOKEN}" "http://192.168.100.253/placement/resource_providers?name=${old_hostname}" | python -mjson.tool | |
| 12:45:42 | sean-k-mooney | you do not want the / at the end of the url i think that was the error | |
| 12:46:25 | ohwhyosa | Yup! | |
| 12:46:34 | ohwhyosa | Indeed I have both the old name and the new one | |
| 12:47:22 | sean-k-mooney | cool so if you grab the uuid for the old name and append <uuid>/allocations to that url | |