Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-22
16:13:14 artom Since users can now set it?
16:13:40 artom Not sure I want to -1 over that this close to the deadline...
16:13:57 stephenfin Fair point. I can do a follow-up
16:14:49 lyarwood ooooff nice catch yeah
16:15:16 lyarwood that's a new microversion at that point right?
16:15:26 stephenfin this is already a new microversion
16:15:39 artom Yeah, it already is, just add the visibility thing to it
16:15:40 lyarwood stephenfin: ah true sorry
16:16:16 lyarwood brb kid just got home
16:16:35 artom By herself? o_O
16:23:38 gibi zigo, frickler: I think this is related https://review.opendev.org/c/openstack/requirements/+/761194
16:24:23 zigo gibi: I've bumped to 5.0.3, this fixes my issues, but now I'm getting:
16:24:23 zigo AttributeError: 'Semaphore' object has no attribute '_at_fork_reinit'
16:24:36 zigo which looks like a problem with Eventlet + Python 3.9... :(
16:24:36 gibi :/
16:24:56 zigo https://github.com/eventlet/eventlet/issues/646
16:25:41 gibi zigo: does eventlet officially states that it supports py3.9?
16:26:00 lyarwood artom: yeah we send them down the mines at ~6 months old around here /s
16:26:03 zigo gibi: I don't know, but Bullseye will be released with it (no py3.8 at all).
16:26:21 zigo This looks like fixing it, I'll try: https://github.com/eventlet/eventlet/pull/664/commits/087f2246ef5380cc5d519f614be7e928ab5ea025
16:27:06 zigo Debian has already 18 patches in the Eventlet package.
16:27:33 zigo Half a decade ago, there was some discussions to get rid of Eventlet in OpenStack. I'd love to have this effort started some day !
16:27:46 zigo Eventlet is a Bi... (pardon my language).
16:28:23 openstackgerrit Merged openstack/nova-specs master: Add spec for configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/770842
16:28:27 zigo Oh, we got that one already ... :/
16:29:47 gibi stephenfin: https://review.opendev.org/c/openstack/nova-specs/+/770842 will you push a followup fixing artom's comments?
16:29:59 stephenfin on it
16:30:44 gibi stephenfin: also I dont find the related bp in launchpad
16:30:45 openstackgerrit Lance Bragstad proposed openstack/placement master: Refactor usage tests to properly test project personas https://review.opendev.org/c/openstack/placement/+/772061
16:30:47 lbragstad stephenfin ^ that fixes the tests you modified to expose usage in placement to project users
16:33:06 stephenfin lbragstad: oh, the project_id was different /o\
16:33:38 stephenfin I even wrote a quick unit test that proved it worked, but I figured the unit test was wrong because the functional test was obviously correct /o\
16:33:55 lbragstad stephenfin right - so the tenancy check was doing what it should
16:34:07 lbragstad but i found something else that's concerning and i'm not sure how it's working now
16:34:33 lbragstad stephenfin https://review.opendev.org/c/openstack/placement/+/772061/1/placement/tests/functional/gabbits/usage-secure-rbac.yaml
16:35:04 lbragstad i added negative tests to ensure project users from anther project can't fetch usage information for projects they don't have authorization on
16:35:52 stephenfin Oh, I noticed that when I was debugging the rule
16:35:59 lbragstad and the project admin persona test fails consistently because the rule:admin_api rule is appended to the new default, regardless of what we're setting up in fixture
16:36:53 lbragstad i went splunking through the enforce_new_default configuration behavior and it appears to be working as expected
16:37:28 stephenfin Hmm, I'm guessing misconfiguration _somewhere_. We have a lot of examples of project reader policies in nova and I'm pretty sure we have unit tests for them all that prove that other project admins can't access $RESOURCE
16:37:34 lbragstad but that deprecated rule check get appended to the default rule magically
16:38:08 lbragstad yeah... i started looking at the placement fixture structure to see if it was doing something unexpected (not cleaning things up properly)?
16:38:16 stephenfin gibi: There's the BP https://blueprints.launchpad.net/nova/+spec/configurable-instance-hostnames
16:38:35 gibi stephenfin: on it
16:38:58 lbragstad but i couldn't find anything that stuck out - and i'm not that knowledgeable about gabbi/placement tests
16:39:31 lbragstad so - i pushed what i have based on your patch... but if you pull that down you should be able to recreate the issue, it'll fail the check gate
16:39:43 stephenfin Neither am I. They're tough to debug. I can't figure out how to even get logs from the placement server
16:40:08 lbragstad downgrade gabbi
16:40:08 stephenfin I'll have to poke cdent or efried when they're about, to see if they have any suggestions
16:41:05 efried Howdy. TLDR or should I read scrollback?
16:41:35 lbragstad if you're using gabbi > 2.0.0 the output logging is broken
16:41:37 lbragstad https://github.com/cdent/gabbi/issues/287
16:41:54 stephenfin lbragstad++ Well that makes my life much easier
16:42:02 zigo gibi: I just looked, Eventlet claims compat with ... python 3.7 ! :/
16:42:17 lbragstad stephenfin downgrading to 1.49.0 works for me
16:42:33 lbragstad in the sense that you can actually capture stdout in tests
16:42:48 gibi zigo: based on that we cannot even release OpenStack :/
16:43:01 zigo Yeah...
16:43:10 zigo Or I stay forever on Buster...
16:43:11 stephenfin efried: We're seeing some unusual policy behavior in placement as part of the RBAC work, and I was looking for a way to get more info from the placement server to debug. Sounds like it's just a downgrade of gabbi that's needed ^
16:43:30 efried cool
16:45:43 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Update spec for configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/772065
16:46:03 stephenfin gibi: And there's the spec amendment ^
16:46:13 stephenfin bauzas, lyarwood, artom also ^
16:47:52 gibi stephenfin: thanks
16:50:05 zigo Great, tests.patcher_test.test_fork_after_monkey_patch fails in Py 3.9 ... :/
16:53:48 melwitt lyarwood: I wanted to ask you about a failure in test_volume_swap I saw yesterday in the gate, have you seen a thing where it doesn't finish the copy and emits "COPY block job progress, current cursor: 1073741823 final cursor: 1073741824" a lot of times, showing that the cursor is only 1 from the end?
16:54:45 melwitt https://zuul.opendev.org/t/openstack/build/a078a17aa9924517b329cafc3f54fed4/log/controller/logs/screen-n-cpu.txt#11115
16:57:44 lyarwood melwitt: I've not seen it 1 block (!?) away and not finish no
16:58:16 lyarwood melwitt: it's typically much greater than that, did it get there pretty quickly and then stall?
16:58:17 melwitt ack
16:58:37 melwitt erm.. let me check.
16:59:33 melwitt lyarwood: yeah looks like it actually. got there and then stuck on the last block for 4 minutes
17:00:12 melwitt do you suppose this could be similar to problems with live migration where we've needed post copy/auto converge?
17:00:20 melwitt seems weird
17:00:48 lyarwood yeah it might be but I can't think that the cirros image would be writing that much to the volume if at all
17:00:58 lyarwood I think we write timestamps during the test and that's it
17:01:18 melwitt hm ok
17:06:12 lyarwood melwitt: we could write this up as a Ubuntu QEMU bug again and see if upstream can help debug this further
17:06:37 lyarwood melwitt: assuming there's something we can log that would help them
17:08:08 melwitt lyarwood: good idea, let me go through and collect more data (if there is more) and I'll open one if I can find more to go on
17:08:20 kashyap melwitt: So ... just reading the scrollback; that "current" and "final" cursors differing means: the copy (i.e. migration w/ storage) hasn't finished succesfully
17:08:46 kashyap In the past we've hit that, and debugged on list; and I recall filing a libvirt RFE to fix that ... let me check
17:09:18 melwitt kashyap: yeah, I think I understood that but the weird thing is it quickly gets to the last block and then stays stuck there for 4 minutes until the test wait in tempest times out and kills it
17:09:37 kashyap Hmm
17:09:44 kashyap melwitt: Yeah, this definitely looks ome something new
17:10:17 kashyap Because in that old behaviour, libvirt was just making "educated guess" when the syncing has finished
17:10:41 melwitt I see
17:10:42 kashyap Especially the "current cursor" terminology looks new to me. I haven't seen the word "cursor" in this error's context before
17:11:05 lyarwood this is blockCopy and not blockRebase btw kashyap
17:11:09 kashyap melwitt: I don't want to bore you with a long bug, but for the record, here it is: https://bugzilla.redhat.com/show_bug.cgi?id=1382165
17:11:11 openstack bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa
17:11:22 kashyap lyarwood: I see; nod
17:11:24 lyarwood I think we dropped that workaround a while ago
17:11:43 kashyap Yep
17:11:49 kashyap But just for clarity, blockCopy() is a _superset_ of blockRebase()
17:11:55 melwitt kashyap: any data/background helps :)
17:12:07 kashyap (So whatever worked, or failed w/ Rebase(), will also fails equal w/ Copy())
17:12:29 lyarwood yeah https://review.opendev.org/c/openstack/nova/+/729596 removed it
17:12:38 kashyap melwitt: But yeah; filing an upstream Ubuntu QEMU bug would be cool to start with

Earlier   Later