Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-12
09:54:19 gibi masterpe: and then you tried to live migrate to compute2
09:54:30 masterpe yes
09:55:51 openstackgerrit John Garbutt proposed openstack/nova master: Add tests for API using unified limits driver https://review.opendev.org/712140
09:55:56 gibi masterpe: the "allocations remaining" logs ae coming from the compute2 log?
09:56:08 masterpe yes
09:56:31 openstackgerrit John Garbutt proposed openstack/nova master: Make unified limits APIs return reserved of 0 https://review.opendev.org/712141
09:56:40 gibi masterpe: what nova versions do you use?
09:58:03 masterpe I use it from openstack-ansible tag 18.1.12
09:58:10 masterpe it is comming from source
09:59:21 gibi based on https://releases.openstack.org/teams/openstackansible.html that means you use Rocky
09:59:47 masterpe nova-compute --version gives me 18.2.3
09:59:54 masterpe yes
10:00:31 gibi did you check the allocations table in the placement database? (I'm not sure how ansible deploys placement in rocky)
10:02:00 masterpe there is no placement database
10:02:27 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Enforce api and db limits https://review.opendev.org/712142
10:02:50 openstackgerrit John Garbutt proposed openstack/nova master: Add unified limits configuration https://review.opendev.org/712137
10:02:50 gibi masterpe: then I guess there is a nova_api database with an allocations table
10:03:48 gibi also you can try to look at the allocations from the placement aPI
10:04:05 gibi $ openstack resource provider list --name compute2
10:04:12 gibi grab the uuid from the output
10:04:13 masterpe ah yes, thanks. There is a allocations table in the database nova and nova_api
10:04:29 gibi openstack resource provider allocation show <uuid>
10:06:14 gibi do you see allocations on your emptied compute?
10:07:34 masterpe on the openstack cli I don't see a resource provider
10:17:02 gibi 'openstack resource provider list' without the name filter should list your compute2 maybe you use fqdn and that makes the name compute2.example.tld
10:22:25 gibi masterpe: when you recovered compute2 did you restarted the nova-compute service? If yes did you see logs like https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L729 during the startup of the service?
10:23:40 masterpe The system was completly powerd off, after evacuated I started the server
10:25:23 gibi during the service startup nova-compute should have cleaned up the allocations of the evacuated instances and log the above linket INFO log
10:53:22 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: WIP Microversion 2.83: Add volume-patch CLI https://review.opendev.org/712651
11:06:15 brinzhang gmann: would you have time to review os-instance-actions --- add default policy patch? as I said before, the cmd/test_policy for test_filter_rules_non_admin and test_filter_rules_instance_owner, I think the default policy need to be update.
12:31:46 openstackgerrit John Garbutt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/712142
12:37:15 openstackgerrit John Garbutt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/712139
12:37:35 openstackgerrit John Garbutt proposed openstack/nova master: Add tests for the UnifiedLimits quota driver https://review.opendev.org/712138
12:37:41 openstackgerrit John Garbutt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/712139
12:37:50 openstackgerrit John Garbutt proposed openstack/nova master: Add tests for API using unified limits driver https://review.opendev.org/712140
12:38:13 openstackgerrit John Garbutt proposed openstack/nova master: Make unified limits APIs return reserved of 0 https://review.opendev.org/712141
12:38:19 openstackgerrit sean mooney proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341
12:38:20 openstackgerrit sean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029
12:38:55 openstackgerrit John Garbutt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/712142
12:39:23 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Update quota_class APIs for db and api limits https://review.opendev.org/712143
12:41:59 openstackgerrit sean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029
13:12:33 openstackgerrit John Garbutt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/712143
13:17:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Ensures that COMPUTE_RESOURCE_SEMAPHORE usage is fair https://review.opendev.org/712674
13:17:45 gibi melwitt: ^^
13:19:31 gibi melwitt: I created some test and I'm +2 on Jason's patch
13:22:25 masterpe gibi: I have just rebooted the compute node and search in the nova-compute.log, but I could not find any "Cleaning up allocations" message
13:22:39 gibi masterpe: interesting
13:25:36 gibi masterpe: did you managed to find the allocations in placement either via the CLI or in the db?
13:27:12 gibi masterpe: are the evacuated instances still exists (not deleted) on compute3 ?
13:27:17 masterpe cli does not work, I will search in the DB.
13:27:43 masterpe yes
13:31:51 dansmith gibi: I think it's safe to merge the vmware deprecation package now.. quite a bit of support on it
13:33:06 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Microversion 2.83: Add volume-patch CLI https://review.opendev.org/712651
13:34:21 brinzhang dansmith: can you leave some comments in https://review.opendev.org/#/c/694428/3/nova/objects/instance_action.py@201
13:34:48 brinzhang dansmith: I am confusing of mriedem said in comments ^^
13:34:59 gibi dansmith: done
13:35:17 dansmith brinzhang: yeah I remember not fully grasping that as well, but I will read and try to grok
13:35:19 dansmith gibi: thanks
13:35:34 gibi dansmith: thanks for proposing the deprecation
13:35:41 dansmith np
13:36:44 brinzhang dansmith: thanks, I know little about serialize_args.
13:37:55 masterpe gibi: yes the instance are still on compute3
13:38:59 gibi masterpe: was the evacuation successfull? what does nova instance-action-list <uuid> shows regarding the evacuation?
13:40:09 lyarwood sean-k-mooney / stephenfin / kashyap: https://review.opendev.org/#/c/696834/ - would you be able to hit this today if you have time?
13:41:00 kashyap lyarwood: I have already read bits of the patch yesterday, will respond
13:41:51 masterpe nova instance-action-list b4a3a998-e8b4-482c-ae4d-dddaee707ff1 ( the uuid is from a instance )
13:41:57 masterpe evacuate | req-552e7f33-8f94-4bc3-a992-8f59b1f92f11 | - | 2020-03-06T17:26:14.000000
13:42:10 lyarwood kashyap: thanks
13:43:11 gibi masterpe: yeah so the evacuation was successful. I have no further idea how https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L729 is not triggered.
13:44:20 gibi masterpe: if you find the offending allocations against compute2 in allocations table of the nova_api db then you can deleted them manually. But I have no idea how to prevent them to happen
13:44:22 dansmith brinzhang: so, the thing I don't get about what he said is the "pass a format_exc_val kwarg" which isn't really an option here
13:44:48 masterpe I can not find that code in the file
13:45:04 dansmith brinzhang: but we can make serialize_args pass the cleaned-up exception message-or-type thing for exc_val
13:48:28 masterpe gibi: In my manager.py the commit https://opendev.org/openstack/nova/commit/09de94e39bbcfc7f8130638e73a8248e49cb6ab7 is missing, so probebly the file is from before that.
13:50:05 brinzhang dansmith: what do you mean? Can you elaborate? I still don't quite understand. In other words, I don't know what I should do about @serialize_args
13:50:14 gibi masterpe: ahh. you have 18.2.3 and that patch is in 18.3
13:50:17 dansmith brinzhang: I'm cooking up a patch for you
13:50:23 gibi masterpe: that explains what you see
13:50:44 brinzhang dansmith: thanks
13:51:28 masterpe I see 4 enty's
13:51:28 masterpe when I do select * from allocations where consumer_id = 'b4a3a998-e8b4-482c-ae4d-dddaee707ff1';
13:51:45 masterpe the collum resource_provider_id is the compute node?
13:52:35 gibi masterpe: yes, the resource_provider_id is the compute node uuid
13:52:55 gibi masterpe: most probably you have allocation for that instance on two computes, compute2 and compute3
13:53:13 gibi the bugfix that is missing from your rocky version is fixing that
13:53:37 gibi if you could upgrade to 18.3 and restart the compute that would also fix your deployment as well
14:03:22 dansmith gibi: fwiw, normally I can make the 1600 meeting but not today because I already have a thing I have to do
14:03:31 dansmith er, s/fwiw/fyi/
14:03:43 gibi dansmith: ack, thanks for the heads up
14:04:23 masterpe The id in resource_providers is linked with resource_provider_id of allocations
14:05:15 gibi masterpe: yes sorry, the resource_provider_id is an internal id pointing to the resource_providers table and there is an uuid in that table that points to the compute node
14:05:49 masterpe resource_provider_id has id 9
14:06:24 masterpe so I can delete all records in allocations with resource_provider_id = 9?
14:08:51 gibi masterpe: if you evacuated all the servers from compute2 then yes
14:09:37 masterpe yes compute2 is emty
14:23:05 dSrinivas Hi all, while attaching the volume to the Nova intance it is failing with this errorhttp://paste.openstack.org/show/790596/ can anyone please me
14:25:11 dSrinivas For FC not for iscsi
14:31:18 sean-k-mooney so that look like an os_brick issue but have you check if the FC deivce is mounted properly on the host at the expected path
14:31:54 dSrinivas FC devices are Online
14:32:23 dSrinivas It is attaching the volume and again it is deleting after 20seconds

Earlier   Later