| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-22 | |||
| 17:22:47 | kashyap | melwitt: It looks like so; because I don't see the failure in this libvirtd log | |
| 17:22:58 | kashyap | And these copy failures are logged to libvirtd. So let me look in the other one | |
| 17:23:10 | sean-k-mooney | lyarwood: you should review gibis qos attach | |
| 17:23:16 | sean-k-mooney | series | |
| 17:23:20 | melwitt | kashyap: would there be a failure though? there was no error. it's just that the cursor never moved from the last block | |
| 17:23:41 | kashyap | melwitt: Not a failure per-se; but in Gate, we have libvirt log filters enabled | |
| 17:23:49 | melwitt | and the test timed out at the tempest level as a result of never seeing the 'available' status for the volume | |
| 17:23:50 | sean-k-mooney | lyarwood: im basically done with it and it looks ready to merge IMO | |
| 17:24:04 | kashyap | melwitt: ... so we should see more fine-grained info. And I was hoping to hurl a useful bit at one of the US-based libvirt developers :D | |
| 17:24:05 | sean-k-mooney | lyarwood: but it need a second core to review | |
| 17:24:17 | melwitt | kashyap: ack | |
| 17:25:10 | kashyap | melwitt: Oh, wait ... I first didn't see this error is _actually_ libvirt error. Looks like it's not | |
| 17:25:43 | sean-k-mooney | lyarwood: specifclly i was refering to https://review.opendev.org/q/topic:%22bp%252Fsupport-interface-attach-with-qos-ports%22+(status:open%20OR%20status:merged) if you are still lookign for somethign easy to review | |
| 17:26:15 | melwitt | kashyap: yeah it's not emitting an error afaik. it's just that we keep checking the job status and it shows it one block from the end and it stays that way for about 4 minutes and then we give up | |
| 17:28:38 | kashyap | melwitt: Have you got the instance name, please? | |
| 17:31:56 | kashyap | Found it | |
| 17:32:01 | melwitt | kashyap: uuid is 185d6573-58ee-47bc-8063-b447673db266, are you wanting the instanceNNNNNN id? | |
| 17:32:15 | melwitt | sorry, I was double checking whether I got the right one | |
| 17:32:29 | kashyap | melwitt: No-no; that's fine :-) | |
| 17:32:46 | melwitt | my eyes are like @_@ | |
| 17:33:07 | kashyap | melwitt: Heh; urgh; my FF browser is hung | |
| 17:33:54 | kashyap | Also there's an unpleasant easter egg in there (I knew it before too): ".txt" files are actually *gzip* files that will devour your RAM, if you try open them in your browser | |
| 17:35:03 | kashyap | Alright it came back. And it's this: instance-00000013 (185d6573-58ee-47bc-8063-b447673db266). Let's see if there's anything useful | |
| 17:37:04 | sean-k-mooney | kashyap: that depend on which of the ci provider the logs are uploaded too | |
| 17:37:41 | sean-k-mooney | the rackspace cloud for reason has there swift cluster configured in such a way that it always compresses them but does not send the correct headers | |
| 17:37:48 | sean-k-mooney | for things like curl to work | |
| 17:42:51 | kashyap | I see | |
| 17:42:59 | kashyap | melwitt: So ... something interesting (at least to me) :-) -- | |
| 17:43:21 | kashyap | I just extracted the exchange between libvirt and QEMU from the giant log here: https://kashyapc.fedorapeople.org/QMP_exchange_libvirtd_22Jan2021_NovaCI.txt | |
| 17:43:33 | kashyap | So ... the copy has actually *succeeded*. Here's the evidence: | |
| 17:43:46 | kashyap | - You `grep` for "blockdev-mirror" in there (at 2021-01-21 05:29:32.630+0000) -- that's the start of the copy job | |
| 17:44:29 | kashyap | - Then there's constant polling going on -- you'll see a lot of "io-status": "ok" going on there | |
| 17:44:58 | kashyap | - Finally, at 2021-01-21 05:34:24.571+0000 | |
| 17:45:40 | kashyap | We get the "ready" flag (which means copy has completed: notice the values of 'len' and 'offset' are equal: | |
| 17:45:43 | kashyap | {"return": [{"auto-finalize": true, "io-status": "ok", "device": "copy-vdb-libvirt-5-format", "auto-dismiss": false, "busy": true, "len": 1073741824, "offset": 1073741824, "status": "running", "paused": false, "speed": 0, "ready": false, "type": "mirror"}], "id": "libvirt-843"} | |
| 17:46:40 | kashyap | melwitt: So, all I can say for now is the copy has succeeded. So I'm not sure why it's saying the copy is not progressing | |
| 17:47:49 | kashyap | melwitt: I'm out of neurons, and need to make some food ... we can continue the chat in the (future) bug | |
| 17:48:48 | melwitt | thanks kashyap, that's helpful. have a nice weekend o/ | |
| 17:51:05 | kashyap | (I think it can't progress because it has reached the end of the copy -- the value in the error matches what I posted above: 1073741824. Hm, needs further digging.) | |
| 17:51:13 | kashyap | Have a good weekend, you too! | |
| 17:51:56 | melwitt | kashyap: yeah it seems the job_status api is returning the wrong thing then, or such | |
| 17:52:37 | melwitt | get_job_info https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L863-L873 | |
| 17:58:01 | lyarwood | is it just a race? | |
| 17:58:26 | lyarwood | but did libvirtd log the job as finished after n-cpu had already given up? | |
| 18:10:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246 | |
| 18:11:13 | melwitt | lyarwood: I will check | |
| 18:15:54 | gibi | have a nice weekend folks! o/ | |
| 18:16:01 | lyarwood | and you gibi \o | |
| 18:16:16 | lyarwood | o/ all | |
| 18:16:19 | sean-k-mooney | o/ | |
| 18:17:05 | melwitt | lyarwood: interesting, it looks like in libvirtd log it shows current == end but what we get in nova when we query is not, it's current == end - 1 | |
| 18:17:29 | melwitt | lyarwood: and also o/ | |
| 18:18:52 | sean-k-mooney | gibi: finished reviewing you qos series, +1 on most of it but left two -1s nothing major just look to confirm detach will work too and a minor issue with comments | |
| 18:19:22 | sean-k-mooney | gibi: but ya enjoy your weekend | |
| 18:26:15 | gmann | lbragstad: 1 comment on this https://review.opendev.org/c/openstack/placement/+/760240/15/placement/tests/functional/gabbits/resource-provider-legacy-rbac.yaml#20 | |
| 18:27:11 | gmann | lbragstad: we should check all persona for legacy policy also. | |
| 18:27:22 | gmann | basically checking all those tokens with enforce_scope false and true | |
| 18:27:42 | sean-k-mooney | ok im going to finish there too o/ | |
| 18:30:05 | lbragstad | gmann ok - i was thinking you only wanted testing for the default personas that were supported prior to secure RBAC? | |
| 18:30:34 | lbragstad | gmann does nova test system personas and reader personas with enforce_scope=False and enforce_new_defaults=False? | |
| 18:30:45 | openstackgerrit | Merged openstack/nova-specs master: Update spec for configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/772065 | |
| 18:31:05 | gmann | lbragstad: yeah we test all those with all those combination of config | |
| 18:31:38 | gmann | lbragstad: is there any admin-or-owner rule? | |
| 18:31:58 | lbragstad | for placement? | |
| 18:32:03 | gmann | yeah | |
| 18:32:14 | gmann | for nova I added all persona to check owner thing basically | |
| 18:32:20 | lbragstad | no - all of placement was protected with rule:admin_api which just checks role:admin essentially | |
| 18:32:56 | gmann | ok, in that case system-admin will also be able to access if enforce_scope is false | |
| 18:33:09 | gmann | as no project-id is checked there | |
| 18:33:36 | lbragstad | correct - system-admin will be able to access placement APIs | |
| 18:34:42 | gmann | admin is ok but reader we can make sure if new reader (project or system) cannot access in case of enforce_scope=false because we care changing the check_str to system_reader | |
| 18:35:02 | gmann | does it make sense? | |
| 18:35:31 | sean-k-mooney | we generally dont want project admins to be able to access placment by the way | |
| 18:35:50 | sean-k-mooney | and im not sure domain admins should eb able to either | |
| 18:36:30 | sean-k-mooney | plamcnet is one of those things that really only system admins should have access too | |
| 18:36:37 | gmann | sean-k-mooney: with new policy rigtht? | |
| 18:36:49 | sean-k-mooney | well system member would be ok too | |
| 18:36:53 | sean-k-mooney | gmann: yep | |
| 18:36:54 | gmann | yeah. that is lbragstad change does | |
| 18:37:10 | gmann | legacy Admin to SYSTEM_ADMIN | |
| 18:37:14 | sean-k-mooney | basically we need to enforce system scope | |
| 18:37:22 | sean-k-mooney | ya | |
| 18:37:35 | gmann | yeah | |
| 18:37:50 | lbragstad | well - there was a note in placement about exposing the usage API to project users | |
| 18:38:02 | sean-k-mooney | there may be uses for system reader or system member/user at somepoint | |
| 18:38:05 | lbragstad | from when mriedem refactored placement's policy enforcement and moved default policies into code | |
| 18:38:23 | lbragstad | stephenfin thought we should address that comment while we're in there mucking with the secure RBAC changes... | |
| 18:38:31 | sean-k-mooney | lbragstad: well currenlty plamcnet is not recommended to be exposed to endusers at all | |
| 18:39:06 | sean-k-mooney | at least in a public cloud case its provide a view into your infrastucre that you dont want normal tenants to have | |
| 18:39:22 | lbragstad | ok - in that case we should let stephenfin know? i'm assume i'm the person with the least amount of placement context | |
| 18:39:54 | lbragstad | https://review.opendev.org/c/openstack/placement/+/771964/1 | |
| 18:40:24 | sean-k-mooney | there is perhaps a usces with unified limits | |
| 18:40:42 | sean-k-mooney | where if we are counting quots with lmits some endpoint might be ok | |
| 18:40:58 | sean-k-mooney | the RP endpoint and allocation candiates endpoint no | |
| 18:41:12 | sean-k-mooney | but looking an allocations for your instance might be ok | |
| 18:41:55 | gmann | but does not it give usages for requested project | |
| 18:41:57 | sean-k-mooney | each resouce provider has the hostname as its name so we cant allow project user to do an RP show for example | |
| 18:42:57 | sean-k-mooney | i though the usage endpoint was teh usage of an invetory | |
| 18:43:42 | sean-k-mooney | ok https://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages | |
| 18:43:52 | sean-k-mooney | would be ok for project users | |