Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-15
14:58:34 dansmith yes, but it would be better to arrange for that not to happen I think, because if that gets called from a compute node it won't be able to target because it won't have api/connection
15:16:33 openstackgerrit Lee Yarwood proposed openstack/nova master: Ensure source compute is up when confirming a resize https://review.opendev.org/699291
15:20:42 dansmith lyarwood: you were only seeing the failure in your change there?
15:22:07 lyarwood dansmith: I am, however Matt had already created this bug a while ago.
15:22:48 lyarwood dansmith: ah it's the same change
15:22:51 dansmith yeah, but presumably that bug was just for the periodic not working, right?
15:22:57 lyarwood dansmith: so we can close this out
15:23:06 lyarwood dansmith: it's the same test failure
15:24:12 dansmith I'm foncused
15:24:25 dansmith you filed a bug for the target recursion
15:24:34 melwitt re: the earlier mention, I didn't add anything to target cell in periodics. I did briefly work on trying to get periodics to re-use a same RequestContext every time (so that request-id stays consistent in logs) but never finished it
15:25:34 melwitt so that might be what you were thinking of
15:25:39 dansmith melwitt: oh I thought you had done something specific to the targeting, although maybe that re-used context would have made them pre-targeted to the right cell?
15:26:05 dansmith lyarwood: did you see that target recursion anywhere other than on matt's change to the periodic?
15:26:07 openstack Launchpad bug 1858667 in OpenStack Compute (nova) "TestMultiCellMigrate.test_poll_unconfirmed_resizes_with_(no_)upcall race failing with greenlet error: Error auto-confirming resize: Reader <greenlet.greenlet object at 0x7f55c19dd048> to writer privilege escalation not allowed. Will retry later." [Undecided,Confirmed]
15:26:07 lyarwood dansmith: mriedem created https://bugs.launchpad.net/nova/+bug/1858667 a while ago that I referenced in my first ping, I missed that it only documents this failure in this unmerged change.
15:26:23 dansmith ah, heh okay gotcha
15:27:06 dansmith oh, I didn't realize that bug was old, I thought you had just filed it
15:27:09 melwitt I don't think so, I don't remember doing anything related to periodics and targeting. and the re-using context thing I never finished and it never landed
15:27:25 dansmith melwitt: okay
15:28:47 melwitt what's interesting is that you'd think this would always happen... because generally nova-api will set_target_cell and then the CheatingSerializer will pass it on to nova-compute already targeted, and generally we have used 'with target_cell' to run periodics
15:29:36 dansmith we can't use target_cell for periodics at least in reality because we don't (always) have the api db creds to be able to look up a mapping
15:29:47 melwitt right, I mean in func tests only
15:29:51 dansmith or do you mean in tests where we poke them.. okay
15:30:14 dansmith well, it only matters when the periodic makes a call to something that calls target a second time
15:30:22 dansmith which should never happen,
15:30:29 melwitt I'm just saying if it's a problem to 'with target_cell' on something that has already set_target_cell then lots of stuff should be failing all the time I'd think. surprising that it's only sometimes
15:30:31 dansmith because periodics need to be calling compute-only code that assume the local cell
15:31:08 dansmith it's not a problem with target_cell, it's a problem of calling code that calls target_cell from on the compute node I think
15:31:12 melwitt oh, I see, periodic is calling something that does a target_cell? I wonder why it's even doing that
15:31:19 dansmith well,
15:31:22 dansmith it's only that one change,
15:31:31 dansmith and it's for auto-confirming a cross-cell migration :)
15:32:12 melwitt I see, re-using (calling) a utility function that's normally only called by non nova-compute I guess?
15:32:31 melwitt I guess I could just look :P /me looks
15:33:00 dansmith exactly
15:33:56 melwitt huh, no it's not doing anything like that. mysterious ...
15:34:15 dansmith it's adding code that calls target
15:34:48 melwitt in nova/compute/api.py confirm_resize method
15:35:03 dansmith right, which is called by the periodic right?
15:35:52 melwitt must be... I guess I hadn't expected a nova-compute periodic calling something in nova.compute.API? but it must be I guess
15:36:05 dansmith we do that in places
15:36:13 dansmith like we call compute.api.stop_instance9)
15:36:20 dansmith and such
15:36:30 melwitt yeah, true
15:36:41 dansmith we probably shouldn't, but we've done that for a long time since before it mattered
15:37:10 dansmith ideally we'd have an inner method for "assume you're targeted or don't need to" that compute-specific code always calls, and then the compute "external" API just targets and then calls those things
15:37:27 melwitt yup there it is https://github.com/openstack/nova/blob/1fa6799e4171d7c93f4d6330f2437891733d7d27/nova/compute/manager.py#L9093
15:37:42 dansmith unmysterious :)
15:37:50 melwitt MYSTERY SOLVED
15:52:14 dansmith sean-k-mooney: IIRC, we have some local hook mechanism we use for checking the devstack deployment, maybe after tempest runs? we used to (or still do) run db archiving I think.. is that still a thing?
15:54:16 dansmith oh, is that post-run?
15:54:36 melwitt https://github.com/openstack/nova/blob/master/gate/post_test_hook.sh ?
15:56:48 melwitt only run by certain jobs https://github.com/openstack/nova/search?q=post_test_hook&unscoped_q=post_test_hook
15:57:20 melwitt nova-next, nova-live-migration, nova-grenade-multinode
15:57:32 dansmith ah yeah, but what runs that?
15:57:42 dansmith or is that baked into d-g?
15:58:03 melwitt actually maybe it's only nova-next. I don't understand what the other two are doing
15:58:49 melwitt it appears you run it by specifying this role in the post.yaml for the job https://github.com/openstack/nova/blob/89e65bd976be8e02fdb6932bb710b5165d7e00a3/playbooks/nova-next/post.yaml
15:59:04 melwitt and the role is defined here https://github.com/openstack/nova/blob/89e65bd976be8e02fdb6932bb710b5165d7e00a3/roles/run-post-test-hook/tasks/main.yaml
15:59:34 dansmith um,
15:59:48 dansmith but something has to tell the nova-next job to run that post.yaml right?
16:00:01 melwitt oh, yeah. that would be something in infra
16:00:06 melwitt I don't know which thing
16:00:33 melwitt maybe it is d-g
16:01:09 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Validate a BDMs disk_bus when provided https://review.opendev.org/724842
16:08:40 sean-k-mooney dansmith: we used to in devstack gate yes
16:08:55 sean-k-mooney a post job would be the zuuv3 way
16:09:01 sean-k-mooney *playbook
16:11:18 sean-k-mooney dansmith: this used to be defiend project-config
16:11:26 sean-k-mooney which hook to run
16:11:36 sean-k-mooney but i think it got moved in repo a long time ago
16:14:26 sean-k-mooney dansmith: melwitt we used to do something like this https://review.opendev.org/#/c/701051/1/playbooks/legacy/tempest-dsvm-networking-ovs-dpdk/run.yaml@a58
16:15:08 sean-k-mooney we defiend a pre_test_hook or post_test_hook function
16:15:18 sean-k-mooney which was invoked by devstack gate if it existed
16:16:38 sean-k-mooney we still do that for the grenate multi node https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-grenade-multinode/run.yaml#L57-L60
16:19:36 sean-k-mooney i may have missed the point of your question
16:19:50 sean-k-mooney dansmith: did any of ^ help
16:20:02 dansmith sean-k-mooney: on a call, biab
16:20:35 sean-k-mooney crap im ment to be on a call too i thnk
16:34:17 openstackgerrit Merged openstack/nova stable/ussuri: catch libvirt exception when nodedev not found. https://review.opendev.org/739593
17:24:31 openstackgerrit Merged openstack/nova stable/ussuri: Correct reported system memory https://review.opendev.org/740666
17:29:55 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934
17:29:55 openstackgerrit Stephen Finucane proposed openstack/nova master: Add emulated TPM support to Nova https://review.opendev.org/631363
17:29:56 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Move single use constants to their callers https://review.opendev.org/741280
17:29:56 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Add docs for vTPM support https://review.opendev.org/739213
17:29:57 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Remove 'test_servers.ServersTestBase' https://review.opendev.org/741282
17:29:57 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Define constants in '_IntegratedTestBase' https://review.opendev.org/741281
17:29:58 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Make '_IntegratedTestBase' subclass 'PlacementInstanceHelperMixin' https://review.opendev.org/741284
17:29:58 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Add 'PlacementHelperMixin', 'PlacementInstanceHelperMixin' https://review.opendev.org/741283
17:29:59 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Add helpers for suspend, resume and reboot of server https://review.opendev.org/741285
17:30:00 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Pass context, instance to '_create_domain' https://review.opendev.org/741286
17:30:23 stephenfin cleaning up after reverted resizes is hard work :(
17:31:25 gmann playbooks/nova-next/post.yaml which run that script (https://github.com/openstack/nova/blob/89e65bd976be8e02fdb6932bb710b5165d7e00a3/.zuul.yaml#L199)
17:31:25 gmann dansmith: melwitt sean-k-mooney nova-next is zuulv3 native jobs and d-g not in pic. d-g run post script for legacy jobs only. zuulv3 run three phase in order 1. 'pre-run' (in hierarchical inheritance order, parent >child) 2. 'run' (this is not run in inheritance order means only parent run if child define one). and last is 3 'post-run' (in hierarchical inheritance order, parent >child) here it trigger the
17:32:03 sean-k-mooney gmann: yep i know
17:32:12 dansmith gmann: ack, that's what I was suspecting,
17:32:21 dansmith that we had to call the post playbook, I didn't even look
17:32:53 dansmith I'm trying to get a glance job that will do some smoke testing of the devstack stuff, since baking devstack assumptions into tempest doesn't really matter
17:32:59 dansmith er, doesn't really make sense

Earlier   Later