| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-11-21 | |||
| 14:22:03 | sean-k-mooney | allow the compute service to start up and regeister its self | |
| 14:22:17 | sean-k-mooney | then make sure the instance alines and them migrate it | |
| 14:23:25 | sean-k-mooney | admin1: i want to make it very clear however that the hostname changing is one of the most distructive things that can happen to the nova/placment dbs and is very non trivial to recover form | |
| 14:23:58 | sean-k-mooney | if you remove the placment rp with/without the fqdn | |
| 14:24:05 | sean-k-mooney | it will allow the compute service to start | |
| 14:24:28 | sean-k-mooney | if you ensure the instnace.host matches the running compute service you should then be able to manage it and migrate it | |
| 14:24:43 | admin1 | is there an api way to delete the entry from placement | |
| 14:24:46 | admin1 | instead from db | |
| 14:24:50 | admin1 | cli way | |
| 14:24:56 | sean-k-mooney | yes if it has no allocations | |
| 14:25:20 | sean-k-mooney | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-delete | |
| 14:25:58 | sean-k-mooney | you might have to do https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete to delete the allcotion for the vm first | |
| 14:26:38 | sean-k-mooney | once the compute service is running and regesterd in placment again | |
| 14:27:04 | sean-k-mooney | update the instanace.host value to match the running service | |
| 14:27:25 | sean-k-mooney | optionally run https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement-heal-allocations | |
| 14:27:33 | sean-k-mooney | for the singel instnace | |
| 14:27:37 | sean-k-mooney | and then migrate it | |
| 14:28:07 | sean-k-mooney | heal allocations will restorte the allocatiosn in palcment that you deleted to allow you to delete the resouce provider | |
| 14:28:27 | sean-k-mooney | cold migration will fix the alloction on the destination in either case when you confirm the migration | |
| 14:29:24 | admin1 | UUID of the consumer -- is the UUID of the vm ? | |
| 14:29:31 | sean-k-mooney | yep | |
| 14:29:41 | sean-k-mooney | in this case at least | |
| 14:30:02 | sean-k-mooney | if you are cold migrating a vm it will also have a second allocation using the migration uuid | |
| 14:32:02 | admin1 | resource provider allocation show UUID ( of h20 ) shows blank, but delete gives Resource provider has allocations | |
| 14:33:36 | admin1 | so there could be some more allocations in the old uuid .. but not present int he virsh list that i can see | |
| 14:34:07 | sean-k-mooney | were you able to delete the fqdn version | |
| 14:34:14 | admin1 | yes | |
| 14:34:18 | admin1 | fqdn one is gone | |
| 14:34:23 | sean-k-mooney | and the vm has the fqdn currently | |
| 14:34:51 | sean-k-mooney | if so can you check what virsh hostname outputs | |
| 14:35:01 | admin1 | in the instances.node , its set to fqdn | |
| 14:36:06 | sean-k-mooney | is it the hostname or hostname.fqdn | |
| 14:36:23 | sean-k-mooney | or i guess hostname.domain | |
| 14:36:57 | admin1 | i was able to delete both now | |
| 14:37:33 | sean-k-mooney | oh ok good | |
| 14:37:45 | sean-k-mooney | so the compute agent should now be able to start | |
| 14:37:50 | admin1 | the GUI hypervisors showed the instances .. | |
| 14:38:17 | admin1 | it registered itself now .. | |
| 14:38:25 | sean-k-mooney | ya if you have db currption like this its hard to resolve | |
| 14:38:58 | sean-k-mooney | so if its regeisterd its self you just need to ensure the instance.host and service.host agree and you shoudl be able to migrate | |
| 14:39:12 | admin1 | now when i try to migrate, using openstack server migrate, it says compute host h20 could not be found | |
| 14:39:20 | admin1 | so i guess its trying to refer to some other h20 | |
| 14:40:06 | sean-k-mooney | you see the compute service in openstack compute service list right and its up | |
| 14:40:30 | sean-k-mooney | oh did you update the compute service mappings in the api deb | |
| 14:40:45 | admin1 | not the last part .. | |
| 14:40:45 | sean-k-mooney | you need to run nova-mange cell_v2 discover_hosts i think | |
| 14:41:06 | admin1 | that would be from inside the nova venv ? | |
| 14:41:06 | sean-k-mooney | the new comptue service recorred need to be mapped to the correct cell | |
| 14:41:19 | sean-k-mooney | ideally form one of the contoller with db access | |
| 14:41:23 | admin1 | ok | |
| 14:41:39 | admin1 | from the os utils as admin, or as nova in the nova venv | |
| 14:41:58 | sean-k-mooney | nova-mange uses the credential in the nova.conf | |
| 14:42:04 | admin1 | got it | |
| 14:42:11 | sean-k-mooney | so you should use the config the conductor uses | |
| 14:42:19 | sean-k-mooney | or ap | |
| 14:42:23 | sean-k-mooney | *or api | |
| 14:42:51 | sean-k-mooney | https://docs.openstack.org/nova/latest/cli/nova-manage.html#cell-v2-discover-hosts | |
| 14:50:09 | admin1 | sean-k-mooney , thank you .. finally its migrating to another host | |
| 14:50:33 | sean-k-mooney | admin1: once you have completed that and confirmed the migration | |
| 14:50:41 | admin1 | i read the spec .. having a uuid that is associated with the server and not associated with hostname will fix issues like this in future | |
| 14:50:44 | sean-k-mooney | i woudl advise checking if any other computes have had a host name change | |
| 14:51:15 | sean-k-mooney | admin1: that not really what the sepc is goign to do | |
| 14:51:36 | admin1 | in my case, grafana showed fqdn while others were non-fqdn, so another collegue decided to change the fqdn to just hostname only and not full fqdn to make the graphs sane | |
| 14:51:37 | sean-k-mooney | admin1: the spec will record the compute node uuid in a file so we can detech if the hostname changes | |
| 14:52:10 | sean-k-mooney | admin1: ya if they did that everywhere they woudl have severly currpted the db | |
| 14:52:32 | sean-k-mooney | you woudl need to cold migrate every workload on teh affected hosts to resolve it | |
| 14:52:58 | sean-k-mooney | its much much better to correct the hostname if no new instnace have been created | |
| 14:54:21 | admin1 | yeah .. we normally ensure all is what is needed before deploying the first vm | |
| 14:54:43 | admin1 | but in this case, since the server was replaced and pxe put the fqdn, it slipped | |
| 14:54:55 | sean-k-mooney | ack | |
| 14:54:58 | admin1 | and it was corrected only after the first vm was deployed | |
| 14:55:19 | sean-k-mooney | i would advise autiting the rest to make sure there are no other in this state | |
| 14:55:40 | sean-k-mooney | the longer you hosts like this the harder it is to fix | |
| 15:13:23 | opendevreview | Merged openstack/os-vif stable/zed: Move mtu update request into ovsdb transaction https://review.opendev.org/c/openstack/os-vif/+/863993 | |
| 16:23:16 | opendevreview | sean mooney proposed openstack/nova-specs master: add spec for fqdn in hostname https://review.opendev.org/c/openstack/nova-specs/+/862626 | |
| 16:24:19 | sean-k-mooney | gibi: dansmith melwitt ^ that hopefully has adressed all the outstanding comments | |
| 16:26:10 | admin1 | as an operator, i would prefer to not have fqdn but just hostnames .. because when it comes to monitoring and graphs, with fqdn it clutters the whole graph .. h20.location.dev.cloud.domain.com where the location.dev.cloud.domain.com is redundant in all | |
| 16:26:58 | sean-k-mooney | admin1: that spec is for vms | |
| 16:27:12 | sean-k-mooney | and on the compute nodes you are free to use either | |
| 16:27:21 | sean-k-mooney | i prefer using hostnames on the computes nodes too | |
| 16:30:32 | sean-k-mooney | admin1: for what its worth nova did not test and recommended against using fqdns for the compute node hostname for a very long time | |
| 16:30:54 | sean-k-mooney | some installer implemtned it anyway and we kind of got stuck with supporting it | |
| 16:31:31 | sean-k-mooney | unfortunetly if you want to use tls i think (not 100% sure) a fqdn is required for the certs | |
| 16:31:48 | sean-k-mooney | that is the main reason they started changing to FQDNs as far as i am aware | |
| 17:08:29 | opendevreview | sean mooney proposed openstack/nova stable/xena: Record SRIOV PF MAC in the binding profile https://review.opendev.org/c/openstack/nova/+/864933 | |
| 17:08:30 | opendevreview | sean mooney proposed openstack/nova stable/xena: Remove double mocking https://review.opendev.org/c/openstack/nova/+/864934 | |
| 17:08:31 | opendevreview | sean mooney proposed openstack/nova stable/xena: Remove double mocking... again https://review.opendev.org/c/openstack/nova/+/864935 | |
| 17:08:32 | opendevreview | sean mooney proposed openstack/nova stable/xena: Add compute restart capability for libvirt func tests https://review.opendev.org/c/openstack/nova/+/864936 | |
| 17:08:33 | opendevreview | sean mooney proposed openstack/nova stable/xena: enable blocked VDPA move operations https://review.opendev.org/c/openstack/nova/+/864937 | |
| 18:18:27 | opendevreview | Merged openstack/os-vif stable/yoga: Move mtu update request into ovsdb transaction https://review.opendev.org/c/openstack/os-vif/+/863994 | |
| 20:59:53 | opendevreview | Merged openstack/nova stable/zed: Handle "no RAM info was set" migration case https://review.opendev.org/c/openstack/nova/+/860732 | |
| 21:50:43 | opendevreview | Merged openstack/nova master: Update contributor guide for 2023.1 Antelope https://review.opendev.org/c/openstack/nova/+/858238 | |
| 22:08:42 | opendevreview | Ghanshyam proposed openstack/nova master: Update gate jobs as per the 2023.1 cycle testing runtime https://review.opendev.org/c/openstack/nova/+/861111 | |
| 22:08:57 | opendevreview | Ghanshyam proposed openstack/os-vif master: Update gate jobs as per the 2023.1 cycle testing runtime https://review.opendev.org/c/openstack/os-vif/+/861468 | |
| #openstack-nova - 2022-11-22 | |||
| 01:19:59 | opendevreview | xielijie proposed openstack/nova master: Persist connection_info to bdm table https://review.opendev.org/c/openstack/nova/+/865214 | |
| 02:22:13 | opendevreview | Jorhson Deng proposed openstack/nova master: Use the page's memory not host_cell.memory for small pagesize https://review.opendev.org/c/openstack/nova/+/864812 | |
| 03:05:39 | opendevreview | Jorhson Deng proposed openstack/nova master: Use the page's memory not host_cell.memory for small pagesize https://review.opendev.org/c/openstack/nova/+/864812 | |
| 06:30:53 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/wallaby: [stable-only] Use os-brick from source in wallaby https://review.opendev.org/c/openstack/nova/+/865134 | |
| 06:31:13 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/wallaby: Cleanup old resize instances dir before resize https://review.opendev.org/c/openstack/nova/+/864691 | |
| 07:52:28 | opendevreview | Jorhson Deng proposed openstack/nova master: Use the page's memory not host_cell.memory for small pagesize https://review.opendev.org/c/openstack/nova/+/864812 | |