| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-16 | |||
| 19:51:14 | mriedem | efried: some rambling in https://review.opendev.org/#/c/687416/ if it means anything to you, probably not worth pursuing | |
| 20:07:45 | efried | mriedem: separate pursuit though, yah? | |
| 20:08:07 | efried | I was also considering making False the default throughout, since that's the more "modern" thing, but that would obv be a separate change as well. | |
| 20:08:14 | mriedem | efried: yeah | |
| 20:44:31 | KeithMnemonic | frickler: we found the issue, they needed -on-shared-storage. | |
| 21:37:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete source allocations in move_allocations if target no longer exists https://review.opendev.org/689049 | |
| 22:05:27 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Update keypairs in saving an instance object https://review.opendev.org/683043 | |
| 22:05:46 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Move Destination object tests to their own test class https://review.opendev.org/683017 | |
| 23:24:46 | openstackgerrit | Merged openstack/nova stable/train: Fix unit of hw_rng:rate_period https://review.opendev.org/688083 | |
| 23:52:48 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635 | |
| #openstack-nova - 2019-10-17 | |||
| 00:02:09 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Remove the todo in the migrations spec https://review.opendev.org/689056 | |
| 00:43:51 | openstackgerrit | Merged openstack/nova-specs master: Remove the todo in the migrations spec https://review.opendev.org/689056 | |
| 02:14:32 | SonPham | hi. how to install python-novaclient package from github. I installed from git but horizon error | |
| 03:07:26 | openstackgerrit | Liang Fang proposed openstack/nova-specs master: Support volume local cache https://review.opendev.org/689070 | |
| 04:14:06 | LiangFang | hi cores, I want to propose a spec about volume local cache. Can anybody be my Feature liaison？Thank you so much. spec: Support volume local cache https://review.opendev.org/689070 | |
| 04:14:15 | LiangFang | I'm new to Nova | |
| 06:58:08 | eandersson | VMs stuck in BUILD is such a pain :'( | |
| 07:02:52 | gibi_off | eandersson: can I do something to help with that? | |
| 07:03:26 | eandersson | gibi, been fighting these for a while in Rocky. We got a few issus fixed upstream. | |
| 07:03:49 | eandersson | We first though these were due to restarted computes | |
| 07:03:57 | eandersson | https://review.opendev.org/#/c/687535/ | |
| 07:04:08 | eandersson | but I think the issues are due to race conditions in this case | |
| 07:04:18 | eandersson | I see two instances scheduled at the same second on the same compute | |
| 07:04:30 | eandersson | I see an allocation, but both VMs are stuck in building / scheduling | |
| 07:04:50 | eandersson | with no compute visible in the instance info | |
| 07:04:59 | eandersson | (had to track the compute down using the db) | |
| 07:05:40 | gibi | eandersson: interesting. So you see both server having allocation in placement or only one of the server has allocation? | |
| 07:05:58 | eandersson | both have allocation in placement | |
| 07:06:09 | eandersson | both are stuck in BUILD | |
| 07:06:40 | gibi | and none of the servers has instance.host updated to point to the compute? | |
| 07:07:23 | eandersson | correct | |
| 07:08:24 | eandersson | I see on the compute 2x Final resource view, image xxx at /var/lib | |
| 07:08:27 | eandersson | and then nothing else | |
| 07:17:47 | eandersson | hmm maybe the compute is bad in this case | |
| 07:17:51 | eandersson | I just got 3 more stuck on that host | |
| 07:18:07 | eandersson | but don't understand why they never fail | |
| 07:18:16 | gibi | eandersson: I guess instance.task_state is None. Unfortunately there is no logs coming out from the compute between the build request reaching the compute and setting the instance.vm_state to BUILDING and between the instance_claim that will set instance.host | |
| 07:20:20 | eandersson | btw we are running the very latest rocky | |
| 07:21:01 | gibi | eandersson: did you happen to see instance.create.start versioned notification or compute.instance.create.start legacy notification for these servers? | |
| 07:21:14 | eandersson | I can check designate | |
| 07:21:43 | eandersson | or nvm they only catch end | |
| 07:23:06 | gibi | eandersson: does pci devices or sriov ports are requested for these servers? | |
| 07:24:35 | eandersson | nothing special | |
| 07:25:41 | gibi | eandersson: the two Final resource view logged considers the allocation of the servers? | |
| 07:26:23 | eandersson | Sorry, don't fully understand that | |
| 07:27:00 | eandersson | last part | |
| 07:27:03 | gibi | you mentioned that you saw two "Final resouce view " logs from the compute | |
| 07:27:18 | gibi | that contains how much vcpu and ram is used on the compute | |
| 07:27:42 | gibi | does that usage contains the usage of your two servers stuck in build? | |
| 07:29:25 | eandersson | actually does not look like it adds up | |
| 07:29:59 | eandersson | in fact when they are stuck I dont see disk in allocations | |
| 07:31:08 | eandersson | It honestly looks like allocations is just wrong | |
| 07:31:12 | eandersson | (in the db) | |
| 07:32:20 | eandersson | I see 3 items in the db, but only two vms on the box (and nothing stuck in building atm) | |
| 07:35:11 | gibi | eandersson: do you see logs like " Lock "compute_resources" acquired by "nova.compute.resource_tracker.instance_claim"" ? | |
| 07:36:49 | gibi | or in general any 'Lock "compute_resources"' | |
| 07:36:55 | eandersson | nothing ;'( | |
| 07:41:30 | eandersson | It's possible that the allocations are a legacy of some failed cold migrations tbh | |
| 07:41:46 | eandersson | but not sure why they would cause a deadlock | |
| 07:45:10 | eandersson | nvm the unaccounted for allocation was another vm stuck in bad state | |
| 07:45:15 | eandersson | just forgot --all-projects | |
| 07:46:21 | eandersson | stuck in building for over 24 hours :p | |
| 07:47:20 | eandersson | also nvm the disk allocation issue... my query had LIMIT 10 on it :D | |
| 07:48:08 | eandersson | Let me try to schedule a VM manually to that host and see if it works | |
| 07:54:21 | gibi | eandersson: I suggest to add oslo_concurrency=DEBUG to the [DEFAULT]/default_log_levels config of the nova-compute service because if you see Final resource view logs then you should see logs about the " Lock "compute_resources" as well | |
| 07:54:31 | eandersson | Sure I can do that now | |
| 07:54:39 | eandersson | Can reproduce this issue 100% on this host | |
| 07:55:34 | gibi | so the periodic jobs can run and update the resource view but no new instances get the chance to claim resoruces | |
| 07:57:00 | gibi | both uses the same compute_resources lock so I don't see how one of them can progress and not the other | |
| 07:57:18 | eandersson | > Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" : | |
| 07:57:57 | eandersson | > Running periodic task ComputeManager._poll_unconfirmed_resizes run_periodic_tasks | |
| 07:59:46 | eandersson | > Compute_service record updated for computexxx:computexxx _update_available_resource | |
| 07:59:51 | eandersson | > Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" | |
| 07:59:55 | eandersson | These are the last two lines | |
| 08:00:10 | gibi | eandersson: now when you boot a VM do you see Lock "compute_resources" acquired by "nova.compute.resource_tracker.instance_claim" ? | |
| 08:01:16 | eandersson | I don't see instance_claim | |
| 08:02:18 | gibi | then somehow the build request cannot grab the compute_resources lock | |
| 08:02:47 | eandersson | > > Instance x has been scheduled to this compute host, the scheduler has made an allocation against this compute node but the instance has yet to start. Skipping heal of allocation: | |
| 08:03:00 | eandersson | > _remove_deleted_instances_allocations | |
| 08:04:31 | gibi | eandersson: that is logged because instance.host is not set | |
| 08:06:40 | gibi | eandersson: do you see logs like Lock "805e10fe-2601-4849-9593-3a83f2875bfb" acquired by "nova.compute.manager._locked_do_build_and_run_instance" where the lock name is the uuid of the server being built? | |
| 08:07:17 | eandersson | nothing with locked_do_build | |
| 08:09:17 | gibi | eandersson: do you have max_concurrent_builds configured ? | |
| 08:09:43 | eandersson | nope | |
| 08:12:48 | eandersson | I don't see any lock not getting released | |
| 08:13:38 | eandersson | unless it was held before the vm was created | |
| 08:14:33 | gibi | this is strange. The instance.uuid lock is grabbed here https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f692f3/nova/compute/manager.py#L2039 and inside that lock we set the instance.vm_state from SCHEDULING to BUILDING here https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f692f3/nova/compute/manager.py#L2132 | |
| 08:15:17 | gibi | just to be sure does the instance.vm_state is in BUILDING ? | |
| 08:15:27 | gibi | or it is still in SCHEDULING ? | |
| 08:15:44 | gibi | eandersson: nvm | |
| 08:15:56 | gibi | the SCHEDULING was the task_state | |
| 08:15:58 | eandersson | > OS-EXT-STS:task_state | scheduling | |
| 08:16:04 | eandersson | > OS-EXT-STS:vm_state | building | |
| 08:17:04 | eandersson | btw restarting the nova-compute does nothing | |
| 08:17:22 | eandersson | but deleting the vm works fine | |
| 08:18:25 | gibi | eandersson: if restarting the compute does not fix the issue then it cannot be a lock as that would be cleaned up by the restart. also it cannot be that your compute run out of RPC workers to process incomming messages either | |
| 08:20:10 | gibi | but it feels like the build request does not reach the compute | |
| 08:20:45 | eandersson | I can see the build request in the db at least | |
| 08:21:24 | gibi | eandersson: does other computes using the same message bus (rabbit) work properly? | |
| 08:22:18 | eandersson | yep I see messaging flowing between nova and nova-compute | |