| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-11 | |||
| 21:29:36 | melwitt | *devstack with ceph plugin | |
| 21:29:42 | sean-k-mooney | ah | |
| 21:29:47 | melwitt | I had a hell of a time last time. this time all smooth | |
| 21:30:17 | melwitt | cause if it fails in the middle for any reason, restarting it didn't seem to work. just keeps tripping over itself | |
| 21:30:31 | melwitt | restarting meaning run ./stack.sh again | |
| 21:30:54 | sean-k-mooney | did you run clean.sh or unstack before runing stack again | |
| 21:31:05 | melwitt | I didn't | |
| 21:31:20 | sean-k-mooney | ya thats proably your issue | |
| 21:31:53 | melwitt | do those still work ok? that was why I hesitated, didn't want to make it worse. but, I ended up having to start all over from scratch anyway so I should have tried it | |
| 21:32:19 | sean-k-mooney | unstack works fine. clean more or less works fine but ocationally it misses files | |
| 21:32:31 | melwitt | ok, good to know | |
| 21:32:39 | sean-k-mooney | unstack is much safer to run | |
| 21:32:42 | sean-k-mooney | then clean | |
| 21:32:44 | dansmith | yeah I try to fix bugs in ./clean.sh when I find them | |
| 21:32:46 | dansmith | like in cyborg lately | |
| 21:33:41 | sean-k-mooney | if you just run stack twice you can get into weird race condition where agents are running but you recrate the db and things break | |
| 21:40:24 | sean-k-mooney | actully speaking of cyborg i should rebase my patch for there devstack plugin | |
| #openstack-nova - 2020-03-12 | |||
| 00:44:20 | brinzhang | gmann: I already granular os-instance-actions GET API, you can review https://review.opendev.org/#/c/711791/3 | |
| 01:29:18 | gmann | brinzhang: we need to add in same patch when adding the new defaults. | |
| 01:30:27 | gmann | like this -https://review.opendev.org/#/c/706672/7/nova/policies/attach_interfaces.py | |
| 01:31:09 | brinzhang | gmann: I was already separate, and the granular beased on this patch, does need I rollback? | |
| 01:32:50 | brinzhang | gmann: split it from the base patch, I think it's clear to me to know what happened | |
| 01:33:27 | gmann | brinzhang: yeah it will be easy for otherwise we endup doing half things per patches. also if you do in single patch we can see tests better way about what behavior is changed | |
| 01:34:40 | brinzhang | gmann: ok, I can combine them. | |
| 01:36:12 | gmann | thanks. because it end up changing the policy twice. first in default role patch and then again changing in new patch./ | |
| 01:38:20 | brinzhang | gmann: about you comments of https://review.opendev.org/#/c/706470/10/nova/tests/unit/cmd/test_policy.py | |
| 01:38:53 | brinzhang | I think I should keep my changes, otherwise the policy seem cannot access request | |
| 01:42:18 | gmann | I do not think it will make any difference if so then something wrong which we need to debug | |
| 01:42:38 | brinzhang | gmann: ok, let me try | |
| 01:42:39 | gmann | because old rules should keep working as it is. | |
| 03:36:06 | openstackgerrit | Kevin Zhao proposed openstack/nova master: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/712607 | |
| 04:24:57 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470 | |
| 04:40:06 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470 | |
| 04:46:13 | brinzhang | gmann: I was merged these patches, and leave comments in that failed case, would please review this after you online https://review.opendev.org/706470 | |
| 05:57:51 | openstackgerrit | Kevin Zhao proposed openstack/nova master: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/712607 | |
| 07:08:31 | dSrinivas | Hi All, i am getting this error http://paste.openstack.org/show/790571/ from March 3 in nova-compute logs in our CI FC HBA's are Online and Connectivity exists between vm and Storage | |
| 08:07:08 | openstackgerrit | Merged openstack/nova stable/queens: Add retry_on_deadlock to migration_update DB API https://review.opendev.org/712212 | |
| 08:28:37 | gibi | morning | |
| 09:25:24 | lyarwood | https://review.opendev.org/#/q/status:open+topic:bug/1861071 - I'd appreciate reviews of this bugfix series from any non-RH cores with bandwidth if possible, there's a followup series on top of this as well as I'd like to backport this. | |
| 09:26:16 | gibi | lyarwood: ack, added to my growing review queue | |
| 09:27:42 | lyarwood | gibi: awesome thanks :) anything I can help with in return please let me know | |
| 09:28:52 | gibi | lyarwood: sure. I have two patch series to be backported furter, as soon I as I did the backport I will ping you | |
| 09:33:28 | lyarwood | gibi: ack thanks | |
| 09:42:40 | masterpe | Hi, when I try to live migrate a instance I get the following error: No valid host was found. Unable to move instance ae88f5d0-28be-4234-86bf-d89a33a2380b to host compute2. There is not enough capacity on the host for the instance | |
| 09:43:53 | masterpe | compute2 has been host-evacuate, after that I deleted all the instance config files of the compute node | |
| 09:46:21 | masterpe | The nova logs gets filled with messgae: Instance 9111eb48-2291-457c-88e2-57f5c4491aa4 has been moved to another host compute3 (compute3.example.tld). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 4, u'MEMORY_MB': 16384}}.: InstanceNotFound_Remote: Instance 0b02dcda-b283-435c-9986-891323411973 could not be found. | |
| 09:46:56 | masterpe | The DB table allocations is empty | |
| 09:52:12 | gibi | masterpe: did nova-compute service on compute2 was recovered after you evacuated everything out of compute2? | |
| 09:53:02 | masterpe | sorry? | |
| 09:53:10 | gibi | I think the nova-compute on compute2 should clean up the allocations | |
| 09:53:49 | gibi | masterpe: I assume that when compute2 was host-evacuated the nova-compute service on that host was down | |
| 09:54:09 | gibi | masterpe: and then you recovered that service | |
| 09:54:19 | gibi | masterpe: and then you tried to live migrate to compute2 | |
| 09:54:30 | masterpe | yes | |
| 09:55:51 | openstackgerrit | John Garbutt proposed openstack/nova master: Add tests for API using unified limits driver https://review.opendev.org/712140 | |
| 09:55:56 | gibi | masterpe: the "allocations remaining" logs ae coming from the compute2 log? | |
| 09:56:08 | masterpe | yes | |
| 09:56:31 | openstackgerrit | John Garbutt proposed openstack/nova master: Make unified limits APIs return reserved of 0 https://review.opendev.org/712141 | |
| 09:56:40 | gibi | masterpe: what nova versions do you use? | |
| 09:58:03 | masterpe | I use it from openstack-ansible tag 18.1.12 | |
| 09:58:10 | masterpe | it is comming from source | |
| 09:59:21 | gibi | based on https://releases.openstack.org/teams/openstackansible.html that means you use Rocky | |
| 09:59:47 | masterpe | nova-compute --version gives me 18.2.3 | |
| 09:59:54 | masterpe | yes | |
| 10:00:31 | gibi | did you check the allocations table in the placement database? (I'm not sure how ansible deploys placement in rocky) | |
| 10:02:00 | masterpe | there is no placement database | |
| 10:02:27 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Enforce api and db limits https://review.opendev.org/712142 | |
| 10:02:50 | openstackgerrit | John Garbutt proposed openstack/nova master: Add unified limits configuration https://review.opendev.org/712137 | |
| 10:02:50 | gibi | masterpe: then I guess there is a nova_api database with an allocations table | |
| 10:03:48 | gibi | also you can try to look at the allocations from the placement aPI | |
| 10:04:05 | gibi | $ openstack resource provider list --name compute2 | |
| 10:04:12 | gibi | grab the uuid from the output | |
| 10:04:13 | masterpe | ah yes, thanks. There is a allocations table in the database nova and nova_api | |
| 10:04:29 | gibi | openstack resource provider allocation show <uuid> | |
| 10:06:14 | gibi | do you see allocations on your emptied compute? | |
| 10:07:34 | masterpe | on the openstack cli I don't see a resource provider | |
| 10:17:02 | gibi | 'openstack resource provider list' without the name filter should list your compute2 maybe you use fqdn and that makes the name compute2.example.tld | |
| 10:22:25 | gibi | masterpe: when you recovered compute2 did you restarted the nova-compute service? If yes did you see logs like https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L729 during the startup of the service? | |
| 10:23:40 | masterpe | The system was completly powerd off, after evacuated I started the server | |
| 10:25:23 | gibi | during the service startup nova-compute should have cleaned up the allocations of the evacuated instances and log the above linket INFO log | |
| 10:53:22 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: WIP Microversion 2.83: Add volume-patch CLI https://review.opendev.org/712651 | |
| 11:06:15 | brinzhang | gmann: would you have time to review os-instance-actions --- add default policy patch? as I said before, the cmd/test_policy for test_filter_rules_non_admin and test_filter_rules_instance_owner, I think the default policy need to be update. | |
| 12:31:46 | openstackgerrit | John Garbutt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/712142 | |
| 12:37:15 | openstackgerrit | John Garbutt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/712139 | |
| 12:37:35 | openstackgerrit | John Garbutt proposed openstack/nova master: Add tests for the UnifiedLimits quota driver https://review.opendev.org/712138 | |
| 12:37:41 | openstackgerrit | John Garbutt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/712139 | |
| 12:37:50 | openstackgerrit | John Garbutt proposed openstack/nova master: Add tests for API using unified limits driver https://review.opendev.org/712140 | |
| 12:38:13 | openstackgerrit | John Garbutt proposed openstack/nova master: Make unified limits APIs return reserved of 0 https://review.opendev.org/712141 | |
| 12:38:19 | openstackgerrit | sean mooney proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 12:38:20 | openstackgerrit | sean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | |
| 12:38:55 | openstackgerrit | John Garbutt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/712142 | |
| 12:39:23 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Update quota_class APIs for db and api limits https://review.opendev.org/712143 | |
| 12:41:59 | openstackgerrit | sean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | |
| 13:12:33 | openstackgerrit | John Garbutt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/712143 | |
| 13:17:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ensures that COMPUTE_RESOURCE_SEMAPHORE usage is fair https://review.opendev.org/712674 | |
| 13:17:45 | gibi | melwitt: ^^ | |
| 13:19:31 | gibi | melwitt: I created some test and I'm +2 on Jason's patch | |
| 13:22:25 | masterpe | gibi: I have just rebooted the compute node and search in the nova-compute.log, but I could not find any "Cleaning up allocations" message | |
| 13:22:39 | gibi | masterpe: interesting | |
| 13:25:36 | gibi | masterpe: did you managed to find the allocations in placement either via the CLI or in the db? | |