Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-15
13:56:59 sean-k-mooney which can happen if you have previous failed migrations in post live migrate
13:57:12 sean-k-mooney but im not sure
13:58:09 sean-k-mooney masterpe: i would check that the vm is actully on the host nova thinks its on
14:05:38 masterpe If I do a openstack server show instanceID:
14:05:41 masterpe OS-EXT-SRV-ATTR:host | compute48
14:05:48 masterpe OS-EXT-SRV-ATTR:hypervisor_hostname | compute40.domain.tld
14:08:35 openstackgerrit Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933
14:08:36 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime and shared CPUs https://review.opendev.org/728485
14:10:52 masterpe sean-k-mooney: How to correct this? I have try to do a "update instances set node = 'compute48.domain.tld' where uuid = 'e848a42b-a7b2-4885-a290-34da7beb98ad';"
14:11:05 masterpe But that did not solve it
14:16:21 masterpe is is active on compute48
14:17:24 sean-k-mooney you should not change that
14:17:46 sean-k-mooney actully that is strang
14:18:01 sean-k-mooney what hypervior are you using
14:19:25 masterpe qemu with kvm
14:23:41 sean-k-mooney ok in that cas esthe host and hypervior host should start the same
14:24:02 sean-k-mooney did you rename the host in the nova config?
14:24:31 sean-k-mooney or chage its hostname at any point
14:24:51 masterpe the hostname is not nova.conf
14:25:18 masterpe no
14:26:09 sean-k-mooney ok so you dont have host=compute48 https://github.com/openstack/nova/blob/master/nova/conf/netconf.py#L52-L56
14:26:44 sean-k-mooney what is the actul host name currently
14:26:52 sean-k-mooney is it compute40 of compute 48
14:33:25 masterpe command: hostnamectl status, gives Static hostname: compute48
14:35:00 masterpe and /etc/hosts has 127.0.1.1 compute48.domain.tld compute48 & 10.8.1.48 compute48.domain.tld compute48
14:38:58 masterpe nova-compute --version is on 19.1.1
14:46:28 masterpe When I create a new instance on that compute all is fine
14:50:39 lyarwood dansmith: as you thought https://review.opendev.org/#/c/699291/7/nova/compute/api.py@3702 is the issue in that change, the failing tests are calling target_cell when running the _poll_unconfirmed_resizes periodic https://github.com/openstack/nova/blob/1fa6799e4171d7c93f4d6330f2437891733d7d27/nova/tests/functional/test_cross_cell_migrate.py#L988-L991 resulting in the reader to writer error.
14:51:08 openstackgerrit Artom Lifshitz proposed openstack/nova master: Handle Neutron errors in _post_live_migration() https://review.opendev.org/729763
14:51:10 openstack bug 1879787 in OpenStack Compute (nova) "post_live_migration does not handle Neutron errors" [Medium,In progress] https://launchpad.net/bugs/1879787 - Assigned to Artom Lifshitz (notartom)
14:51:10 openstackgerrit Artom Lifshitz proposed openstack/nova master: Add regression test for bug 1879787 https://review.opendev.org/741230
14:51:49 dansmith lyarwood: ah yep
14:57:03 lyarwood dansmith: is there anyway of checking if a context is already correctly targetted at a cell?
14:57:56 lyarwood actually nvm _get_source_compute_service doesn't have any other callers I can just mock it
14:58:34 dansmith yes, but it would be better to arrange for that not to happen I think, because if that gets called from a compute node it won't be able to target because it won't have api/connection
15:16:33 openstackgerrit Lee Yarwood proposed openstack/nova master: Ensure source compute is up when confirming a resize https://review.opendev.org/699291
15:20:42 dansmith lyarwood: you were only seeing the failure in your change there?
15:22:07 lyarwood dansmith: I am, however Matt had already created this bug a while ago.
15:22:48 lyarwood dansmith: ah it's the same change
15:22:51 dansmith yeah, but presumably that bug was just for the periodic not working, right?
15:22:57 lyarwood dansmith: so we can close this out
15:23:06 lyarwood dansmith: it's the same test failure
15:24:12 dansmith I'm foncused
15:24:25 dansmith you filed a bug for the target recursion
15:24:34 melwitt re: the earlier mention, I didn't add anything to target cell in periodics. I did briefly work on trying to get periodics to re-use a same RequestContext every time (so that request-id stays consistent in logs) but never finished it
15:25:34 melwitt so that might be what you were thinking of
15:25:39 dansmith melwitt: oh I thought you had done something specific to the targeting, although maybe that re-used context would have made them pre-targeted to the right cell?
15:26:05 dansmith lyarwood: did you see that target recursion anywhere other than on matt's change to the periodic?
15:26:07 openstack Launchpad bug 1858667 in OpenStack Compute (nova) "TestMultiCellMigrate.test_poll_unconfirmed_resizes_with_(no_)upcall race failing with greenlet error: Error auto-confirming resize: Reader <greenlet.greenlet object at 0x7f55c19dd048> to writer privilege escalation not allowed. Will retry later." [Undecided,Confirmed]
15:26:07 lyarwood dansmith: mriedem created https://bugs.launchpad.net/nova/+bug/1858667 a while ago that I referenced in my first ping, I missed that it only documents this failure in this unmerged change.
15:26:23 dansmith ah, heh okay gotcha
15:27:06 dansmith oh, I didn't realize that bug was old, I thought you had just filed it
15:27:09 melwitt I don't think so, I don't remember doing anything related to periodics and targeting. and the re-using context thing I never finished and it never landed
15:27:25 dansmith melwitt: okay
15:28:47 melwitt what's interesting is that you'd think this would always happen... because generally nova-api will set_target_cell and then the CheatingSerializer will pass it on to nova-compute already targeted, and generally we have used 'with target_cell' to run periodics
15:29:36 dansmith we can't use target_cell for periodics at least in reality because we don't (always) have the api db creds to be able to look up a mapping
15:29:47 melwitt right, I mean in func tests only
15:29:51 dansmith or do you mean in tests where we poke them.. okay
15:30:14 dansmith well, it only matters when the periodic makes a call to something that calls target a second time
15:30:22 dansmith which should never happen,
15:30:29 melwitt I'm just saying if it's a problem to 'with target_cell' on something that has already set_target_cell then lots of stuff should be failing all the time I'd think. surprising that it's only sometimes
15:30:31 dansmith because periodics need to be calling compute-only code that assume the local cell
15:31:08 dansmith it's not a problem with target_cell, it's a problem of calling code that calls target_cell from on the compute node I think
15:31:12 melwitt oh, I see, periodic is calling something that does a target_cell? I wonder why it's even doing that
15:31:19 dansmith well,
15:31:22 dansmith it's only that one change,
15:31:31 dansmith and it's for auto-confirming a cross-cell migration :)
15:32:12 melwitt I see, re-using (calling) a utility function that's normally only called by non nova-compute I guess?
15:32:31 melwitt I guess I could just look :P /me looks
15:33:00 dansmith exactly
15:33:56 melwitt huh, no it's not doing anything like that. mysterious ...
15:34:15 dansmith it's adding code that calls target
15:34:48 melwitt in nova/compute/api.py confirm_resize method
15:35:03 dansmith right, which is called by the periodic right?
15:35:52 melwitt must be... I guess I hadn't expected a nova-compute periodic calling something in nova.compute.API? but it must be I guess
15:36:05 dansmith we do that in places
15:36:13 dansmith like we call compute.api.stop_instance9)
15:36:20 dansmith and such
15:36:30 melwitt yeah, true
15:36:41 dansmith we probably shouldn't, but we've done that for a long time since before it mattered
15:37:10 dansmith ideally we'd have an inner method for "assume you're targeted or don't need to" that compute-specific code always calls, and then the compute "external" API just targets and then calls those things
15:37:27 melwitt yup there it is https://github.com/openstack/nova/blob/1fa6799e4171d7c93f4d6330f2437891733d7d27/nova/compute/manager.py#L9093
15:37:42 dansmith unmysterious :)
15:37:50 melwitt MYSTERY SOLVED
15:52:14 dansmith sean-k-mooney: IIRC, we have some local hook mechanism we use for checking the devstack deployment, maybe after tempest runs? we used to (or still do) run db archiving I think.. is that still a thing?
15:54:16 dansmith oh, is that post-run?
15:54:36 melwitt https://github.com/openstack/nova/blob/master/gate/post_test_hook.sh ?
15:56:48 melwitt only run by certain jobs https://github.com/openstack/nova/search?q=post_test_hook&unscoped_q=post_test_hook
15:57:20 melwitt nova-next, nova-live-migration, nova-grenade-multinode
15:57:32 dansmith ah yeah, but what runs that?
15:57:42 dansmith or is that baked into d-g?
15:58:03 melwitt actually maybe it's only nova-next. I don't understand what the other two are doing
15:58:49 melwitt it appears you run it by specifying this role in the post.yaml for the job https://github.com/openstack/nova/blob/89e65bd976be8e02fdb6932bb710b5165d7e00a3/playbooks/nova-next/post.yaml
15:59:04 melwitt and the role is defined here https://github.com/openstack/nova/blob/89e65bd976be8e02fdb6932bb710b5165d7e00a3/roles/run-post-test-hook/tasks/main.yaml
15:59:34 dansmith um,
15:59:48 dansmith but something has to tell the nova-next job to run that post.yaml right?
16:00:01 melwitt oh, yeah. that would be something in infra
16:00:06 melwitt I don't know which thing
16:00:33 melwitt maybe it is d-g
16:01:09 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Validate a BDMs disk_bus when provided https://review.opendev.org/724842

Earlier   Later