| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-22 | |||
| 17:31:56 | kashyap | Found it | |
| 17:32:01 | melwitt | kashyap: uuid is 185d6573-58ee-47bc-8063-b447673db266, are you wanting the instanceNNNNNN id? | |
| 17:32:15 | melwitt | sorry, I was double checking whether I got the right one | |
| 17:32:29 | kashyap | melwitt: No-no; that's fine :-) | |
| 17:32:46 | melwitt | my eyes are like @_@ | |
| 17:33:07 | kashyap | melwitt: Heh; urgh; my FF browser is hung | |
| 17:33:54 | kashyap | Also there's an unpleasant easter egg in there (I knew it before too): ".txt" files are actually *gzip* files that will devour your RAM, if you try open them in your browser | |
| 17:35:03 | kashyap | Alright it came back. And it's this: instance-00000013 (185d6573-58ee-47bc-8063-b447673db266). Let's see if there's anything useful | |
| 17:37:04 | sean-k-mooney | kashyap: that depend on which of the ci provider the logs are uploaded too | |
| 17:37:41 | sean-k-mooney | the rackspace cloud for reason has there swift cluster configured in such a way that it always compresses them but does not send the correct headers | |
| 17:37:48 | sean-k-mooney | for things like curl to work | |
| 17:42:51 | kashyap | I see | |
| 17:42:59 | kashyap | melwitt: So ... something interesting (at least to me) :-) -- | |
| 17:43:21 | kashyap | I just extracted the exchange between libvirt and QEMU from the giant log here: https://kashyapc.fedorapeople.org/QMP_exchange_libvirtd_22Jan2021_NovaCI.txt | |
| 17:43:33 | kashyap | So ... the copy has actually *succeeded*. Here's the evidence: | |
| 17:43:46 | kashyap | - You `grep` for "blockdev-mirror" in there (at 2021-01-21 05:29:32.630+0000) -- that's the start of the copy job | |
| 17:44:29 | kashyap | - Then there's constant polling going on -- you'll see a lot of "io-status": "ok" going on there | |
| 17:44:58 | kashyap | - Finally, at 2021-01-21 05:34:24.571+0000 | |
| 17:45:40 | kashyap | We get the "ready" flag (which means copy has completed: notice the values of 'len' and 'offset' are equal: | |
| 17:45:43 | kashyap | {"return": [{"auto-finalize": true, "io-status": "ok", "device": "copy-vdb-libvirt-5-format", "auto-dismiss": false, "busy": true, "len": 1073741824, "offset": 1073741824, "status": "running", "paused": false, "speed": 0, "ready": false, "type": "mirror"}], "id": "libvirt-843"} | |
| 17:46:40 | kashyap | melwitt: So, all I can say for now is the copy has succeeded. So I'm not sure why it's saying the copy is not progressing | |
| 17:47:49 | kashyap | melwitt: I'm out of neurons, and need to make some food ... we can continue the chat in the (future) bug | |
| 17:48:48 | melwitt | thanks kashyap, that's helpful. have a nice weekend o/ | |
| 17:51:05 | kashyap | (I think it can't progress because it has reached the end of the copy -- the value in the error matches what I posted above: 1073741824. Hm, needs further digging.) | |
| 17:51:13 | kashyap | Have a good weekend, you too! | |
| 17:51:56 | melwitt | kashyap: yeah it seems the job_status api is returning the wrong thing then, or such | |
| 17:52:37 | melwitt | get_job_info https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L863-L873 | |
| 17:58:01 | lyarwood | is it just a race? | |
| 17:58:26 | lyarwood | but did libvirtd log the job as finished after n-cpu had already given up? | |
| 18:10:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246 | |
| 18:11:13 | melwitt | lyarwood: I will check | |
| 18:15:54 | gibi | have a nice weekend folks! o/ | |
| 18:16:01 | lyarwood | and you gibi \o | |
| 18:16:16 | lyarwood | o/ all | |
| 18:16:19 | sean-k-mooney | o/ | |
| 18:17:05 | melwitt | lyarwood: interesting, it looks like in libvirtd log it shows current == end but what we get in nova when we query is not, it's current == end - 1 | |
| 18:17:29 | melwitt | lyarwood: and also o/ | |
| 18:18:52 | sean-k-mooney | gibi: finished reviewing you qos series, +1 on most of it but left two -1s nothing major just look to confirm detach will work too and a minor issue with comments | |
| 18:19:22 | sean-k-mooney | gibi: but ya enjoy your weekend | |
| 18:26:15 | gmann | lbragstad: 1 comment on this https://review.opendev.org/c/openstack/placement/+/760240/15/placement/tests/functional/gabbits/resource-provider-legacy-rbac.yaml#20 | |
| 18:27:11 | gmann | lbragstad: we should check all persona for legacy policy also. | |
| 18:27:22 | gmann | basically checking all those tokens with enforce_scope false and true | |
| 18:27:42 | sean-k-mooney | ok im going to finish there too o/ | |
| 18:30:05 | lbragstad | gmann ok - i was thinking you only wanted testing for the default personas that were supported prior to secure RBAC? | |
| 18:30:34 | lbragstad | gmann does nova test system personas and reader personas with enforce_scope=False and enforce_new_defaults=False? | |
| 18:30:45 | openstackgerrit | Merged openstack/nova-specs master: Update spec for configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/772065 | |
| 18:31:05 | gmann | lbragstad: yeah we test all those with all those combination of config | |
| 18:31:38 | gmann | lbragstad: is there any admin-or-owner rule? | |
| 18:31:58 | lbragstad | for placement? | |
| 18:32:03 | gmann | yeah | |
| 18:32:14 | gmann | for nova I added all persona to check owner thing basically | |
| 18:32:20 | lbragstad | no - all of placement was protected with rule:admin_api which just checks role:admin essentially | |
| 18:32:56 | gmann | ok, in that case system-admin will also be able to access if enforce_scope is false | |
| 18:33:09 | gmann | as no project-id is checked there | |
| 18:33:36 | lbragstad | correct - system-admin will be able to access placement APIs | |
| 18:34:42 | gmann | admin is ok but reader we can make sure if new reader (project or system) cannot access in case of enforce_scope=false because we care changing the check_str to system_reader | |
| 18:35:02 | gmann | does it make sense? | |
| 18:35:31 | sean-k-mooney | we generally dont want project admins to be able to access placment by the way | |
| 18:35:50 | sean-k-mooney | and im not sure domain admins should eb able to either | |
| 18:36:30 | sean-k-mooney | plamcnet is one of those things that really only system admins should have access too | |
| 18:36:37 | gmann | sean-k-mooney: with new policy rigtht? | |
| 18:36:49 | sean-k-mooney | well system member would be ok too | |
| 18:36:53 | sean-k-mooney | gmann: yep | |
| 18:36:54 | gmann | yeah. that is lbragstad change does | |
| 18:37:10 | gmann | legacy Admin to SYSTEM_ADMIN | |
| 18:37:14 | sean-k-mooney | basically we need to enforce system scope | |
| 18:37:22 | sean-k-mooney | ya | |
| 18:37:35 | gmann | yeah | |
| 18:37:50 | lbragstad | well - there was a note in placement about exposing the usage API to project users | |
| 18:38:02 | sean-k-mooney | there may be uses for system reader or system member/user at somepoint | |
| 18:38:05 | lbragstad | from when mriedem refactored placement's policy enforcement and moved default policies into code | |
| 18:38:23 | lbragstad | stephenfin thought we should address that comment while we're in there mucking with the secure RBAC changes... | |
| 18:38:31 | sean-k-mooney | lbragstad: well currenlty plamcnet is not recommended to be exposed to endusers at all | |
| 18:39:06 | sean-k-mooney | at least in a public cloud case its provide a view into your infrastucre that you dont want normal tenants to have | |
| 18:39:22 | lbragstad | ok - in that case we should let stephenfin know? i'm assume i'm the person with the least amount of placement context | |
| 18:39:54 | lbragstad | https://review.opendev.org/c/openstack/placement/+/771964/1 | |
| 18:40:24 | sean-k-mooney | there is perhaps a usces with unified limits | |
| 18:40:42 | sean-k-mooney | where if we are counting quots with lmits some endpoint might be ok | |
| 18:40:58 | sean-k-mooney | the RP endpoint and allocation candiates endpoint no | |
| 18:41:12 | sean-k-mooney | but looking an allocations for your instance might be ok | |
| 18:41:55 | gmann | but does not it give usages for requested project | |
| 18:41:57 | sean-k-mooney | each resouce provider has the hostname as its name so we cant allow project user to do an RP show for example | |
| 18:42:57 | sean-k-mooney | i though the usage endpoint was teh usage of an invetory | |
| 18:43:42 | sean-k-mooney | ok https://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages | |
| 18:43:52 | sean-k-mooney | would be ok for project users | |
| 18:44:05 | sean-k-mooney | that is the quota sutff | |
| 18:44:28 | sean-k-mooney | lbragstad: so https://review.opendev.org/c/openstack/placement/+/771964/1/placement/tests/functional/gabbits/usage-secure-rbac.yaml is fine | |
| 18:45:01 | gmann | yeah it make sense to have project scope there | |
| 18:45:52 | sean-k-mooney | i think /resource_providers /reshaper /allocation_candidates need to be system scope at lest by default | |
| 18:46:31 | sean-k-mooney | im not sure about /allocations/{consumer_uuid} | |
| 18:47:22 | lbragstad | yeah - stephenfin's change was specific to the usage API i believe | |
| 18:47:39 | sean-k-mooney | i think that would actully be ok as it is today for projects too https://docs.openstack.org/api-ref/placement/?expanded=list-allocations-detail#list-allocations | |
| 18:47:46 | lbragstad | because of https://review.opendev.org/c/openstack/placement/+/771964/1/placement/policies/usage.py#52 | |
| 18:49:27 | sean-k-mooney | ya ok i think stephenfin's patch is going in the right direction | |
| 18:49:52 | sean-k-mooney | i dont see anything that is definetly incorrect e.g. too borad | |
| 18:50:19 | sean-k-mooney | ill start it for monday | |
| 18:51:10 | lbragstad | ok - i'd appreciate it, i think there are some things in how placement sets up fixtures that need to be investigated, too https://review.opendev.org/c/openstack/placement/+/772061/1/placement/tests/functional/gabbits/usage-secure-rbac.yaml | |
| 18:56:32 | gmann | lbragstad: commented in this ^^. project_id is not passed for policy check | |
| 18:57:18 | gmann | if we pass req.project_id in context.can() then it can fail if it is different from context.project_id | |
| 18:57:52 | gmann | I have not tested that just checked it in code. I faced many such isuse in nova APIs | |