| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-15 | |||
| 12:27:08 | devfaz | Hi, is this a bug or known. If I try to disable an nova-compute service of a system terminated just seconds ago I get a trace with "An error occurred while updating the COMPUTE_STATUS_DISABLED trait on compute node resource providers managed by host XX. The trait will be synchronized automatically by the compute service when the update_available_resource periodic task runs" and rabbitmq-msgid timeout. | |
| 12:27:27 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.PY2 and six.PY3 https://review.opendev.org/726512 | |
| 12:27:41 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.add_metaclass https://review.opendev.org/726565 | |
| 12:28:01 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.reraise https://review.opendev.org/726898 | |
| 12:28:20 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224 | |
| 12:29:00 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757 | |
| 12:29:19 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777 | |
| 12:29:41 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094 | |
| 12:40:28 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109 | |
| 13:00:08 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117 | |
| 13:02:23 | openstack | Launchpad bug 1858667 in OpenStack Compute (nova) "TestMultiCellMigrate.test_poll_unconfirmed_resizes_with_(no_)upcall race failing with greenlet error: Error auto-confirming resize: Reader <greenlet.greenlet object at 0x7f55c19dd048> to writer privilege escalation not allowed. Will retry later." [Undecided,Confirmed] | |
| 13:02:23 | lyarwood | dansmith: https://bugs.launchpad.net/nova/+bug/1858667 - I'm hitting this odd locking issue in some func tests, this smells like some behaviour has changed in py38 tbh but I'm having a hard time pinning it down. Would you be able to take a look? | |
| 13:21:02 | lyarwood | melwitt: ^ also if you have any time, I'm not even sure why we are wrapping target_cell if I'm honest | |
| 13:21:35 | lyarwood | CELL_CACHE is the only global thing I can see? | |
| 13:30:55 | dansmith | lyarwood: the entire DB state is global, which is what the lock is protecting | |
| 13:31:29 | dansmith | lyarwood: your python REPL examples aren't representative I think because the lock grabbing is serial in the code, yet you're nesting them (I think, without indentation) | |
| 13:32:20 | dansmith | I'm not sure what the references to fasteners has to do with this, as we're using regular python.threading locks, IIRC | |
| 13:35:05 | dansmith | if I had to guess, I'd say maybe we're recursing on the lock somewhere, i.e. targeting the cell and then calling something that targets the cell again? | |
| 13:35:35 | dansmith | ah lockutils uses fasteners, I see | |
| 13:35:43 | lyarwood | dansmith: yeah sorry back | |
| 13:35:58 | lyarwood | dansmith: and yeah I did use indentation, launchpad-- just removed it for some reason | |
| 13:36:20 | lyarwood | dansmith: but yes my examples are bogus given what is actually being called | |
| 13:36:33 | dansmith | ack | |
| 13:36:49 | lyarwood | dansmith: I just hit the error and got carried away :) | |
| 13:37:40 | dansmith | is that in a periodic? poll unconfirmed resizes? | |
| 13:38:41 | dansmith | I may be making this up, but I think melwitt might have added something much later than this to make the periodics run targeted to the right cell, so maybe that's where the conflict is here... this is now getting lucky and running a periodic when we previously didn't need to change the target or something | |
| 13:38:44 | lyarwood | yeah | |
| 13:48:32 | masterpe | I try to live migrate a instance, but I get the message in the controller: "Unable to find record for source node compute40.domain.tld on compute48" | |
| 13:48:45 | masterpe | but compute40 is incorrect | |
| 13:49:03 | masterpe | What is wrong? | |
| 13:56:39 | sean-k-mooney | that looks like maybe the instace.host is wrong | |
| 13:56:59 | sean-k-mooney | which can happen if you have previous failed migrations in post live migrate | |
| 13:57:12 | sean-k-mooney | but im not sure | |
| 13:58:09 | sean-k-mooney | masterpe: i would check that the vm is actully on the host nova thinks its on | |
| 14:05:38 | masterpe | If I do a openstack server show instanceID: | |
| 14:05:41 | masterpe | OS-EXT-SRV-ATTR:host | compute48 | |
| 14:05:48 | masterpe | OS-EXT-SRV-ATTR:hypervisor_hostname | compute40.domain.tld | |
| 14:08:35 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933 | |
| 14:08:36 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime and shared CPUs https://review.opendev.org/728485 | |
| 14:10:52 | masterpe | sean-k-mooney: How to correct this? I have try to do a "update instances set node = 'compute48.domain.tld' where uuid = 'e848a42b-a7b2-4885-a290-34da7beb98ad';" | |
| 14:11:05 | masterpe | But that did not solve it | |
| 14:16:21 | masterpe | is is active on compute48 | |
| 14:17:24 | sean-k-mooney | you should not change that | |
| 14:17:46 | sean-k-mooney | actully that is strang | |
| 14:18:01 | sean-k-mooney | what hypervior are you using | |
| 14:19:25 | masterpe | qemu with kvm | |
| 14:23:41 | sean-k-mooney | ok in that cas esthe host and hypervior host should start the same | |
| 14:24:02 | sean-k-mooney | did you rename the host in the nova config? | |
| 14:24:31 | sean-k-mooney | or chage its hostname at any point | |
| 14:24:51 | masterpe | the hostname is not nova.conf | |
| 14:25:18 | masterpe | no | |
| 14:26:09 | sean-k-mooney | ok so you dont have host=compute48 https://github.com/openstack/nova/blob/master/nova/conf/netconf.py#L52-L56 | |
| 14:26:44 | sean-k-mooney | what is the actul host name currently | |
| 14:26:52 | sean-k-mooney | is it compute40 of compute 48 | |
| 14:33:25 | masterpe | command: hostnamectl status, gives Static hostname: compute48 | |
| 14:35:00 | masterpe | and /etc/hosts has 127.0.1.1 compute48.domain.tld compute48 & 10.8.1.48 compute48.domain.tld compute48 | |
| 14:38:58 | masterpe | nova-compute --version is on 19.1.1 | |
| 14:46:28 | masterpe | When I create a new instance on that compute all is fine | |
| 14:50:39 | lyarwood | dansmith: as you thought https://review.opendev.org/#/c/699291/7/nova/compute/api.py@3702 is the issue in that change, the failing tests are calling target_cell when running the _poll_unconfirmed_resizes periodic https://github.com/openstack/nova/blob/1fa6799e4171d7c93f4d6330f2437891733d7d27/nova/tests/functional/test_cross_cell_migrate.py#L988-L991 resulting in the reader to writer error. | |
| 14:51:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Handle Neutron errors in _post_live_migration() https://review.opendev.org/729763 | |
| 14:51:10 | openstack | bug 1879787 in OpenStack Compute (nova) "post_live_migration does not handle Neutron errors" [Medium,In progress] https://launchpad.net/bugs/1879787 - Assigned to Artom Lifshitz (notartom) | |
| 14:51:10 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Add regression test for bug 1879787 https://review.opendev.org/741230 | |
| 14:51:49 | dansmith | lyarwood: ah yep | |
| 14:57:03 | lyarwood | dansmith: is there anyway of checking if a context is already correctly targetted at a cell? | |
| 14:57:56 | lyarwood | actually nvm _get_source_compute_service doesn't have any other callers I can just mock it | |
| 14:58:34 | dansmith | yes, but it would be better to arrange for that not to happen I think, because if that gets called from a compute node it won't be able to target because it won't have api/connection | |
| 15:16:33 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Ensure source compute is up when confirming a resize https://review.opendev.org/699291 | |
| 15:20:42 | dansmith | lyarwood: you were only seeing the failure in your change there? | |
| 15:22:07 | lyarwood | dansmith: I am, however Matt had already created this bug a while ago. | |
| 15:22:48 | lyarwood | dansmith: ah it's the same change | |
| 15:22:51 | dansmith | yeah, but presumably that bug was just for the periodic not working, right? | |
| 15:22:57 | lyarwood | dansmith: so we can close this out | |
| 15:23:06 | lyarwood | dansmith: it's the same test failure | |
| 15:24:12 | dansmith | I'm foncused | |
| 15:24:25 | dansmith | you filed a bug for the target recursion | |
| 15:24:34 | melwitt | re: the earlier mention, I didn't add anything to target cell in periodics. I did briefly work on trying to get periodics to re-use a same RequestContext every time (so that request-id stays consistent in logs) but never finished it | |
| 15:25:34 | melwitt | so that might be what you were thinking of | |
| 15:25:39 | dansmith | melwitt: oh I thought you had done something specific to the targeting, although maybe that re-used context would have made them pre-targeted to the right cell? | |
| 15:26:05 | dansmith | lyarwood: did you see that target recursion anywhere other than on matt's change to the periodic? | |
| 15:26:07 | openstack | Launchpad bug 1858667 in OpenStack Compute (nova) "TestMultiCellMigrate.test_poll_unconfirmed_resizes_with_(no_)upcall race failing with greenlet error: Error auto-confirming resize: Reader <greenlet.greenlet object at 0x7f55c19dd048> to writer privilege escalation not allowed. Will retry later." [Undecided,Confirmed] | |
| 15:26:07 | lyarwood | dansmith: mriedem created https://bugs.launchpad.net/nova/+bug/1858667 a while ago that I referenced in my first ping, I missed that it only documents this failure in this unmerged change. | |
| 15:26:23 | dansmith | ah, heh okay gotcha | |
| 15:27:06 | dansmith | oh, I didn't realize that bug was old, I thought you had just filed it | |
| 15:27:09 | melwitt | I don't think so, I don't remember doing anything related to periodics and targeting. and the re-using context thing I never finished and it never landed | |
| 15:27:25 | dansmith | melwitt: okay | |
| 15:28:47 | melwitt | what's interesting is that you'd think this would always happen... because generally nova-api will set_target_cell and then the CheatingSerializer will pass it on to nova-compute already targeted, and generally we have used 'with target_cell' to run periodics | |
| 15:29:36 | dansmith | we can't use target_cell for periodics at least in reality because we don't (always) have the api db creds to be able to look up a mapping | |
| 15:29:47 | melwitt | right, I mean in func tests only | |
| 15:29:51 | dansmith | or do you mean in tests where we poke them.. okay | |
| 15:30:14 | dansmith | well, it only matters when the periodic makes a call to something that calls target a second time | |
| 15:30:22 | dansmith | which should never happen, | |
| 15:30:29 | melwitt | I'm just saying if it's a problem to 'with target_cell' on something that has already set_target_cell then lots of stuff should be failing all the time I'd think. surprising that it's only sometimes | |
| 15:30:31 | dansmith | because periodics need to be calling compute-only code that assume the local cell | |
| 15:31:08 | dansmith | it's not a problem with target_cell, it's a problem of calling code that calls target_cell from on the compute node I think | |
| 15:31:12 | melwitt | oh, I see, periodic is calling something that does a target_cell? I wonder why it's even doing that | |
| 15:31:19 | dansmith | well, | |
| 15:31:22 | dansmith | it's only that one change, | |
| 15:31:31 | dansmith | and it's for auto-confirming a cross-cell migration :) | |
| 15:32:12 | melwitt | I see, re-using (calling) a utility function that's normally only called by non nova-compute I guess? | |
| 15:32:31 | melwitt | I guess I could just look :P /me looks | |