Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-22
17:33:54 kashyap Also there's an unpleasant easter egg in there (I knew it before too): ".txt" files are actually *gzip* files that will devour your RAM, if you try open them in your browser
17:35:03 kashyap Alright it came back. And it's this: instance-00000013 (185d6573-58ee-47bc-8063-b447673db266). Let's see if there's anything useful
17:37:04 sean-k-mooney kashyap: that depend on which of the ci provider the logs are uploaded too
17:37:41 sean-k-mooney the rackspace cloud for reason has there swift cluster configured in such a way that it always compresses them but does not send the correct headers
17:37:48 sean-k-mooney for things like curl to work
17:42:51 kashyap I see
17:42:59 kashyap melwitt: So ... something interesting (at least to me) :-) --
17:43:21 kashyap I just extracted the exchange between libvirt and QEMU from the giant log here: https://kashyapc.fedorapeople.org/QMP_exchange_libvirtd_22Jan2021_NovaCI.txt
17:43:33 kashyap So ... the copy has actually *succeeded*. Here's the evidence:
17:43:46 kashyap - You `grep` for "blockdev-mirror" in there (at 2021-01-21 05:29:32.630+0000) -- that's the start of the copy job
17:44:29 kashyap - Then there's constant polling going on -- you'll see a lot of "io-status": "ok" going on there
17:44:58 kashyap - Finally, at 2021-01-21 05:34:24.571+0000
17:45:40 kashyap We get the "ready" flag (which means copy has completed: notice the values of 'len' and 'offset' are equal:
17:45:43 kashyap {"return": [{"auto-finalize": true, "io-status": "ok", "device": "copy-vdb-libvirt-5-format", "auto-dismiss": false, "busy": true, "len": 1073741824, "offset": 1073741824, "status": "running", "paused": false, "speed": 0, "ready": false, "type": "mirror"}], "id": "libvirt-843"}
17:46:40 kashyap melwitt: So, all I can say for now is the copy has succeeded. So I'm not sure why it's saying the copy is not progressing
17:47:49 kashyap melwitt: I'm out of neurons, and need to make some food ... we can continue the chat in the (future) bug
17:48:48 melwitt thanks kashyap, that's helpful. have a nice weekend o/
17:51:05 kashyap (I think it can't progress because it has reached the end of the copy -- the value in the error matches what I posted above: 1073741824. Hm, needs further digging.)
17:51:13 kashyap Have a good weekend, you too!
17:51:56 melwitt kashyap: yeah it seems the job_status api is returning the wrong thing then, or such
17:52:37 melwitt get_job_info https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L863-L873
17:58:01 lyarwood is it just a race?
17:58:26 lyarwood but did libvirtd log the job as finished after n-cpu had already given up?
18:10:36 openstackgerrit Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246
18:11:13 melwitt lyarwood: I will check
18:15:54 gibi have a nice weekend folks! o/
18:16:01 lyarwood and you gibi \o
18:16:16 lyarwood o/ all
18:16:19 sean-k-mooney o/
18:17:05 melwitt lyarwood: interesting, it looks like in libvirtd log it shows current == end but what we get in nova when we query is not, it's current == end - 1
18:17:29 melwitt lyarwood: and also o/
18:18:52 sean-k-mooney gibi: finished reviewing you qos series, +1 on most of it but left two -1s nothing major just look to confirm detach will work too and a minor issue with comments
18:19:22 sean-k-mooney gibi: but ya enjoy your weekend
18:26:15 gmann lbragstad: 1 comment on this https://review.opendev.org/c/openstack/placement/+/760240/15/placement/tests/functional/gabbits/resource-provider-legacy-rbac.yaml#20
18:27:11 gmann lbragstad: we should check all persona for legacy policy also.
18:27:22 gmann basically checking all those tokens with enforce_scope false and true
18:27:42 sean-k-mooney ok im going to finish there too o/
18:30:05 lbragstad gmann ok - i was thinking you only wanted testing for the default personas that were supported prior to secure RBAC?
18:30:34 lbragstad gmann does nova test system personas and reader personas with enforce_scope=False and enforce_new_defaults=False?
18:30:45 openstackgerrit Merged openstack/nova-specs master: Update spec for configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/772065
18:31:05 gmann lbragstad: yeah we test all those with all those combination of config
18:31:38 gmann lbragstad: is there any admin-or-owner rule?
18:31:58 lbragstad for placement?
18:32:03 gmann yeah
18:32:14 gmann for nova I added all persona to check owner thing basically
18:32:20 lbragstad no - all of placement was protected with rule:admin_api which just checks role:admin essentially
18:32:56 gmann ok, in that case system-admin will also be able to access if enforce_scope is false
18:33:09 gmann as no project-id is checked there
18:33:36 lbragstad correct - system-admin will be able to access placement APIs
18:34:42 gmann admin is ok but reader we can make sure if new reader (project or system) cannot access in case of enforce_scope=false because we care changing the check_str to system_reader
18:35:02 gmann does it make sense?
18:35:31 sean-k-mooney we generally dont want project admins to be able to access placment by the way
18:35:50 sean-k-mooney and im not sure domain admins should eb able to either
18:36:30 sean-k-mooney plamcnet is one of those things that really only system admins should have access too
18:36:37 gmann sean-k-mooney: with new policy rigtht?
18:36:49 sean-k-mooney well system member would be ok too
18:36:53 sean-k-mooney gmann: yep
18:36:54 gmann yeah. that is lbragstad change does
18:37:10 gmann legacy Admin to SYSTEM_ADMIN
18:37:14 sean-k-mooney basically we need to enforce system scope
18:37:22 sean-k-mooney ya
18:37:35 gmann yeah
18:37:50 lbragstad well - there was a note in placement about exposing the usage API to project users
18:38:02 sean-k-mooney there may be uses for system reader or system member/user at somepoint
18:38:05 lbragstad from when mriedem refactored placement's policy enforcement and moved default policies into code
18:38:23 lbragstad stephenfin thought we should address that comment while we're in there mucking with the secure RBAC changes...
18:38:31 sean-k-mooney lbragstad: well currenlty plamcnet is not recommended to be exposed to endusers at all
18:39:06 sean-k-mooney at least in a public cloud case its provide a view into your infrastucre that you dont want normal tenants to have
18:39:22 lbragstad ok - in that case we should let stephenfin know? i'm assume i'm the person with the least amount of placement context
18:39:54 lbragstad https://review.opendev.org/c/openstack/placement/+/771964/1
18:40:24 sean-k-mooney there is perhaps a usces with unified limits
18:40:42 sean-k-mooney where if we are counting quots with lmits some endpoint might be ok
18:40:58 sean-k-mooney the RP endpoint and allocation candiates endpoint no
18:41:12 sean-k-mooney but looking an allocations for your instance might be ok
18:41:55 gmann but does not it give usages for requested project
18:41:57 sean-k-mooney each resouce provider has the hostname as its name so we cant allow project user to do an RP show for example
18:42:57 sean-k-mooney i though the usage endpoint was teh usage of an invetory
18:43:42 sean-k-mooney ok https://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages
18:43:52 sean-k-mooney would be ok for project users
18:44:05 sean-k-mooney that is the quota sutff
18:44:28 sean-k-mooney lbragstad: so https://review.opendev.org/c/openstack/placement/+/771964/1/placement/tests/functional/gabbits/usage-secure-rbac.yaml is fine
18:45:01 gmann yeah it make sense to have project scope there
18:45:52 sean-k-mooney i think /resource_providers /reshaper /allocation_candidates need to be system scope at lest by default
18:46:31 sean-k-mooney im not sure about /allocations/{consumer_uuid}
18:47:22 lbragstad yeah - stephenfin's change was specific to the usage API i believe
18:47:39 sean-k-mooney i think that would actully be ok as it is today for projects too https://docs.openstack.org/api-ref/placement/?expanded=list-allocations-detail#list-allocations
18:47:46 lbragstad because of https://review.opendev.org/c/openstack/placement/+/771964/1/placement/policies/usage.py#52
18:49:27 sean-k-mooney ya ok i think stephenfin's patch is going in the right direction
18:49:52 sean-k-mooney i dont see anything that is definetly incorrect e.g. too borad
18:50:19 sean-k-mooney ill start it for monday
18:51:10 lbragstad ok - i'd appreciate it, i think there are some things in how placement sets up fixtures that need to be investigated, too https://review.opendev.org/c/openstack/placement/+/772061/1/placement/tests/functional/gabbits/usage-secure-rbac.yaml
18:56:32 gmann lbragstad: commented in this ^^. project_id is not passed for policy check
18:57:18 gmann if we pass req.project_id in context.can() then it can fail if it is different from context.project_id
18:57:52 gmann I have not tested that just checked it in code. I faced many such isuse in nova APIs
18:58:36 lbragstad gmann i think we already fixed that in the previous patch
18:58:55 gmann oh, did not see that
21:01:03 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers https://review.opendev.org/c/openstack/placement/+/760240
21:05:15 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers https://review.opendev.org/c/openstack/placement/+/760240
22:47:39 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers https://review.opendev.org/c/openstack/placement/+/760240
22:47:40 openstackgerrit Lance Bragstad proposed openstack/placement master: Implement secure RBAC for aggregates https://review.opendev.org/c/openstack/placement/+/760235

Earlier   Later