| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-14 | |||
| 20:30:54 | dansmith | sean-k-mooney: replied. thanks for reviewing and for your initial help in that of course | |
| 20:33:00 | sean-k-mooney | no worries | |
| 20:36:34 | sean-k-mooney | dansmith: ya makes sense to me | |
| 20:36:55 | sean-k-mooney | im glad you finally got it working by the way. | |
| 20:37:06 | sean-k-mooney | its always nice when the ci verifys something like this for us | |
| 20:37:47 | sean-k-mooney | the glance folks might even want to take a look for there own testing | |
| 21:36:25 | openstack | bug 1550919 in OpenStack Compute (nova) rocky "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Lee Yarwood (lyarwood) | |
| 21:36:25 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional test for bug 1550919 https://review.opendev.org/732707 | |
| 21:54:44 | dansmith | sean-k-mooney: yeah, they'll have a lot more surface they can test once my two devstack patches land.. I hope they will make use of that, but.. who knows | |
| 22:24:03 | sean-k-mooney | dansmith: well if we are testing it its better then nothing | |
| 23:20:46 | openstackgerrit | Merged openstack/nova stable/rocky: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/732708 | |
| 23:37:53 | mnaser | melwitt: i know you've worked some with the quota code. does nova have any locking on quota usages? neutron currently has a row-level lock on a resource/tenant_id which is causing me worlds of pain on highly parallelized port creation | |
| 23:38:24 | mnaser | it pretty much holds a lock the whole time (with a select for update) while it calculates the usage and updates... | |
| 23:46:10 | gmann | dansmith: seems oslo policy not able to format the target data for copy_image - https://zuul.opendev.org/t/openstack/build/aa1a5cd62aaa4210b19a5877e77b87db/log/controller/logs/screen-g-api.txt#600 | |
| 23:47:21 | gmann | trying to reproduce this locally to see if oslo trying to change 'hw_rng_model' for some reason. | |
| 23:52:06 | gmann | mnaser: not sure about lock (melwitt can confirm if we do have that) but nova does recheck quota again just before the resource creation to avoid(or to minimize as much as possible) any wrong calculation or over-quota in parallel request cases. | |
| 23:52:45 | mnaser | gmann: ok so nova doesnt do any row level locking for quota usage -- also yeah, i remember that, it's even an option to recheck quota in config if i remember right | |
| 23:53:28 | gmann | * after resource creation. this config - https://github.com/openstack/nova/blob/7385adb2f526166bb174dab055d060a428d4d4ff/nova/conf/quota.py#L189 | |
| 23:56:04 | gmann | mnaser: afaik, no locks but melwitt or dansmith can confirm it. | |
| #openstack-nova - 2020-07-15 | |||
| 01:09:23 | melwitt | mnaser: I implemented a re-architecting of quotas in pike, which began counting quota usage on-the-fly by counting resources (this only does a read). prior to pike, we did do the select-for-update locking thing | |
| 02:26:53 | openstackgerrit | Jeffrey Zhang proposed openstack/nova master: Support inject-nmi action in watchdog https://review.opendev.org/741072 | |
| 03:18:04 | openstackgerrit | Merged openstack/nova master: doc: Correct typo https://review.opendev.org/740670 | |
| 03:32:41 | openstackgerrit | Keigo Noha proposed openstack/nova master: Change default num_retries for glance to 3 https://review.opendev.org/740389 | |
| 05:38:27 | openstackgerrit | Jeffrey Zhang proposed openstack/nova master: Support inject-nmi action in watchdog https://review.opendev.org/741072 | |
| 06:22:49 | openstackgerrit | Jie Li proposed openstack/nova-specs master: add volume backed server rebuild spec https://review.opendev.org/739349 | |
| 08:50:12 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: hardware: Enable 'hw:cpu_dedicated_mask' for creating a mixed instance https://review.opendev.org/714706 | |
| 10:13:49 | dpaclt | Hi All I am unable to launch new vms .. http://paste.openstack.org/show/795937/ ..Anyone please suggest | |
| 10:15:50 | lyarwood | dpaclt: this isn't a support channel but grep for the req-4ce3ba68-34f8-453c-bb2a-9be9ef9adcbd id across the api/scheduler/conductor logs and you should see why all of your hosts are unable to schedule the instance | |
| 10:18:01 | lyarwood | elod: https://review.opendev.org/#/c/732717/ (and the two under it) is finally ready btw if you have time today | |
| 10:19:00 | dpaclt | Thanks lyarwood | |
| 10:44:39 | elod | lyarwood: sure, added to my TODOs :) | |
| 10:45:25 | lyarwood | thanks! | |
| 10:49:07 | openstackgerrit | Merged openstack/nova master: objects: Introduce 'pcpuset' field for InstanceNUMACell https://review.opendev.org/714658 | |
| 12:01:16 | openstack | bug 1550919 in OpenStack Compute (nova) queens "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Lee Yarwood (lyarwood) | |
| 12:01:16 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: Add functional test for bug 1550919 https://review.opendev.org/732715 | |
| 12:01:51 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: libvirt: Fix misleading debug msg "Instance is running" https://review.opendev.org/732716 | |
| 12:10:16 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/732717 | |
| 12:16:55 | jsuchome | gibi: hi, could you take another look at https://review.opendev.org/#/c/738738/ ? | |
| 12:27:08 | devfaz | Hi, is this a bug or known. If I try to disable an nova-compute service of a system terminated just seconds ago I get a trace with "An error occurred while updating the COMPUTE_STATUS_DISABLED trait on compute node resource providers managed by host XX. The trait will be synchronized automatically by the compute service when the update_available_resource periodic task runs" and rabbitmq-msgid timeout. | |
| 12:27:27 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.PY2 and six.PY3 https://review.opendev.org/726512 | |
| 12:27:41 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.add_metaclass https://review.opendev.org/726565 | |
| 12:28:01 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.reraise https://review.opendev.org/726898 | |
| 12:28:20 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224 | |
| 12:29:00 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757 | |
| 12:29:19 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777 | |
| 12:29:41 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094 | |
| 12:40:28 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109 | |
| 13:00:08 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117 | |
| 13:02:23 | openstack | Launchpad bug 1858667 in OpenStack Compute (nova) "TestMultiCellMigrate.test_poll_unconfirmed_resizes_with_(no_)upcall race failing with greenlet error: Error auto-confirming resize: Reader <greenlet.greenlet object at 0x7f55c19dd048> to writer privilege escalation not allowed. Will retry later." [Undecided,Confirmed] | |
| 13:02:23 | lyarwood | dansmith: https://bugs.launchpad.net/nova/+bug/1858667 - I'm hitting this odd locking issue in some func tests, this smells like some behaviour has changed in py38 tbh but I'm having a hard time pinning it down. Would you be able to take a look? | |
| 13:21:02 | lyarwood | melwitt: ^ also if you have any time, I'm not even sure why we are wrapping target_cell if I'm honest | |
| 13:21:35 | lyarwood | CELL_CACHE is the only global thing I can see? | |
| 13:30:55 | dansmith | lyarwood: the entire DB state is global, which is what the lock is protecting | |
| 13:31:29 | dansmith | lyarwood: your python REPL examples aren't representative I think because the lock grabbing is serial in the code, yet you're nesting them (I think, without indentation) | |
| 13:32:20 | dansmith | I'm not sure what the references to fasteners has to do with this, as we're using regular python.threading locks, IIRC | |
| 13:35:05 | dansmith | if I had to guess, I'd say maybe we're recursing on the lock somewhere, i.e. targeting the cell and then calling something that targets the cell again? | |
| 13:35:35 | dansmith | ah lockutils uses fasteners, I see | |
| 13:35:43 | lyarwood | dansmith: yeah sorry back | |
| 13:35:58 | lyarwood | dansmith: and yeah I did use indentation, launchpad-- just removed it for some reason | |
| 13:36:20 | lyarwood | dansmith: but yes my examples are bogus given what is actually being called | |
| 13:36:33 | dansmith | ack | |
| 13:36:49 | lyarwood | dansmith: I just hit the error and got carried away :) | |
| 13:37:40 | dansmith | is that in a periodic? poll unconfirmed resizes? | |
| 13:38:41 | dansmith | I may be making this up, but I think melwitt might have added something much later than this to make the periodics run targeted to the right cell, so maybe that's where the conflict is here... this is now getting lucky and running a periodic when we previously didn't need to change the target or something | |
| 13:38:44 | lyarwood | yeah | |
| 13:48:32 | masterpe | I try to live migrate a instance, but I get the message in the controller: "Unable to find record for source node compute40.domain.tld on compute48" | |
| 13:48:45 | masterpe | but compute40 is incorrect | |
| 13:49:03 | masterpe | What is wrong? | |
| 13:56:39 | sean-k-mooney | that looks like maybe the instace.host is wrong | |
| 13:56:59 | sean-k-mooney | which can happen if you have previous failed migrations in post live migrate | |
| 13:57:12 | sean-k-mooney | but im not sure | |
| 13:58:09 | sean-k-mooney | masterpe: i would check that the vm is actully on the host nova thinks its on | |
| 14:05:38 | masterpe | If I do a openstack server show instanceID: | |
| 14:05:41 | masterpe | OS-EXT-SRV-ATTR:host | compute48 | |
| 14:05:48 | masterpe | OS-EXT-SRV-ATTR:hypervisor_hostname | compute40.domain.tld | |
| 14:08:35 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933 | |
| 14:08:36 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime and shared CPUs https://review.opendev.org/728485 | |
| 14:10:52 | masterpe | sean-k-mooney: How to correct this? I have try to do a "update instances set node = 'compute48.domain.tld' where uuid = 'e848a42b-a7b2-4885-a290-34da7beb98ad';" | |
| 14:11:05 | masterpe | But that did not solve it | |
| 14:16:21 | masterpe | is is active on compute48 | |
| 14:17:24 | sean-k-mooney | you should not change that | |
| 14:17:46 | sean-k-mooney | actully that is strang | |
| 14:18:01 | sean-k-mooney | what hypervior are you using | |
| 14:19:25 | masterpe | qemu with kvm | |
| 14:23:41 | sean-k-mooney | ok in that cas esthe host and hypervior host should start the same | |
| 14:24:02 | sean-k-mooney | did you rename the host in the nova config? | |
| 14:24:31 | sean-k-mooney | or chage its hostname at any point | |
| 14:24:51 | masterpe | the hostname is not nova.conf | |
| 14:25:18 | masterpe | no | |
| 14:26:09 | sean-k-mooney | ok so you dont have host=compute48 https://github.com/openstack/nova/blob/master/nova/conf/netconf.py#L52-L56 | |
| 14:26:44 | sean-k-mooney | what is the actul host name currently | |
| 14:26:52 | sean-k-mooney | is it compute40 of compute 48 | |
| 14:33:25 | masterpe | command: hostnamectl status, gives Static hostname: compute48 | |
| 14:35:00 | masterpe | and /etc/hosts has 127.0.1.1 compute48.domain.tld compute48 & 10.8.1.48 compute48.domain.tld compute48 | |
| 14:38:58 | masterpe | nova-compute --version is on 19.1.1 | |
| 14:46:28 | masterpe | When I create a new instance on that compute all is fine | |
| 14:50:39 | lyarwood | dansmith: as you thought https://review.opendev.org/#/c/699291/7/nova/compute/api.py@3702 is the issue in that change, the failing tests are calling target_cell when running the _poll_unconfirmed_resizes periodic https://github.com/openstack/nova/blob/1fa6799e4171d7c93f4d6330f2437891733d7d27/nova/tests/functional/test_cross_cell_migrate.py#L988-L991 resulting in the reader to writer error. | |
| 14:51:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Handle Neutron errors in _post_live_migration() https://review.opendev.org/729763 | |
| 14:51:10 | openstack | bug 1879787 in OpenStack Compute (nova) "post_live_migration does not handle Neutron errors" [Medium,In progress] https://launchpad.net/bugs/1879787 - Assigned to Artom Lifshitz (notartom) | |
| 14:51:10 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Add regression test for bug 1879787 https://review.opendev.org/741230 | |