Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-06-11
12:00:00 lyarwood gibi: so I'll work out how to capture that and then raise a QEMU bug depending on what we see
12:00:10 gibi ohh, OK, cool
12:00:18 lyarwood gibi: as I'm assuming live migration has had some kind of impact here
12:01:16 kashyap lyarwood: For the QEMU bug, do you think we have hope of reproducing this outside of the CI Gate?
12:01:38 kashyap lyarwood: E.g. in the same Ubuntu env; same versions, migrating a paused instance -- can that do it?
12:01:39 lyarwood really depends on what if anything we get back from the guestOS
12:01:49 kashyap Yeah; good point
12:01:56 lyarwood the instance isn't paused in this latest failure
12:12:10 masterpe[m] I have instances in the placement.allocations table but these instances does not exists anymore. "nova-manage placement audit" does not exists on Train. And I have tryed "nova-manage placement heal_allocations" but that does not work. Can I savely deleted those records in database?
12:17:23 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Skip block migration tests until bug #1931702 is resolved https://review.opendev.org/c/openstack/nova/+/795992
12:18:06 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to #1931702 https://review.opendev.org/c/openstack/nova/+/795992
12:18:07 lyarwood forgot to update the title sorry
12:22:16 frickler lyarwood: interesting issue, is that with cirros as guest or something else?
12:22:34 lyarwood frickler: yeah it's Cirros 0.5.2
12:32:41 lyarwood ^ should give us console logs at the time we send the detach to see how borked the guestOS is
12:33:33 opendevreview Lee Yarwood proposed openstack/nova master: DNM Revert "zuul: Skip block migration with attached volumes tests due to #1931702" https://review.opendev.org/c/openstack/nova/+/795997
12:33:33 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to bug #1931702 https://review.opendev.org/c/openstack/nova/+/795992
12:33:36 lyarwood whops, ^ there we go, git-review being slow again.
12:38:37 lyarwood gibi: https://review.opendev.org/c/openstack/nova/+/792415 simple docs review if you have time btw
12:39:05 lyarwood sean-k-mooney: https://review.opendev.org/c/openstack/nova-specs/+/794799 - would you mind taking a look at this?
13:37:23 opendevreview Lee Yarwood proposed openstack/nova master: libvirt: fup docs and typing for _hard_reboot flow https://review.opendev.org/c/openstack/nova/+/795081
13:38:10 lyarwood Wonderful now the nova-lvm job has failed randomly
14:00:20 gibi sorry I was pulled downstream, surfacing now...
14:05:36 opendevreview Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575
14:19:00 dansmith gmann: I was looking through nova's instance-get stuff to see if and how things have changed in preparation for system scope and RBAC stuff
14:19:22 dansmith and it looks to me like we're still enforcing instance.project_id==context.project_id all the way down at the db layer in model_query()
14:20:12 dansmith it only does that if we're a "user context", but I'm thinking that is going to be in the way of more flexible rbac stuff, if you can't even load an instance object to check a richer policy rule
14:20:31 dansmith am I right in thinking that will have to change?
14:26:37 gmann dansmith: currently we do not check get instance permission for other policy check which need instance object in target, are you saying to change that to check get instance permission first?
14:27:34 dansmith gmann: we don't check for "get instance permission" before doing something like an update -- that's what you mean right?
14:27:41 gmann yeah
14:27:56 gibi lyarwood: re: nova-lvm failure I see multiple hits of the same error in logstash but all fairly recent http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22WARNING%3A%20Failed%20to%20get%20udev%20device%20handler%20for%20device%20%2Fdev%2Fsda1%5C%22
14:28:32 gibi lyarwood: and apparently it is not nova-lvm specific
14:28:34 dansmith gmann: okay, but the problem is.. in order to do some update type thing that we might grant in policy, we have to db.get_instance(), which will fail to find it in model_query because it filters the owner very deep
14:28:49 gibi lyarwood: or did I have a wrong signature?
14:29:55 gmann dansmith: i see. and that is say system admin want to update server?
14:30:29 gmann or any other project member want to update some other project server?
14:31:08 dansmith well, right, I'm thinking the case where you want to do something more powerful, like say "project X is under project Y, so let project Y people manage project X servers too"
14:31:27 gmann humm
14:31:32 dansmith which is kinda the point of the RBAC work eventually, right? to make the policy actually let people do more complicated things like that.. no?
14:33:56 gmann dansmith: I think that is separate things if we want to allow than current secure RBAC which does not allow these kind of things instead restricting access
14:34:23 gmann project isolation is one things in new secure rbac but your use case is opposite to that
14:34:44 dansmith hrm
14:35:09 lyarwood <gibi "lyarwood: or did I have a wrong "> Yeah that's correct, iirc c-vol also hit something like this so maybe that's the duplicate?
14:35:54 dansmith gmann: my case is just my interpretation of what I thought the end goal was, so maybe I'm being too broad
14:36:16 gibi lyarwood: I see that we hit this during update_available_resource that simply skips the update and does not cause the job failure
14:36:20 lyarwood Sorry gibi, ^ was for you, somehow replied in element (matrix) and I'm not sure what that looks like in vanilla irc.
14:36:39 gibi lyarwood: it quotes part of the original message so it work for me :)
14:36:46 gmann dansmith: yeah because in most of the case we pass instance.project_id as target to oslo policy
14:36:57 gibi lyarwood: so most of the logstash hits are actual job SUCCESS as it only hit the during periodic
14:37:09 gmann dansmith: if we want to allow cross project operation we have to change that model and db things too what you mentioned
14:37:13 lyarwood Cool cool, the nova-lvm failure I saw was during instance deletion
14:37:31 gibi lyarwood: yeah, if we hit it during other operation then the test fails
14:37:36 lyarwood So maybe we just need to retry on failure here
14:37:52 lyarwood As it's just a basic lvs command
14:37:56 gibi lyarwood: but the hit in the periodic shows that the underlying issue it temprorary as the next periodic succeeds
14:38:10 gibi lyarwood: yeah, so a retry could help
14:38:14 lyarwood Yup cool, I'll work on that on Monday
14:38:35 gibi cool
14:39:01 gibi lyarwood: should I open a bug report on it?
14:39:09 gibi I have the context now
14:39:22 gibi and still have time today :)
14:40:03 gmann dansmith: yes, too broad :). I think in that case they should allocate that user a system token and ask to do other project things. but system to do project servers operation is another challenge what we discussded in PTG
14:40:29 lyarwood I already have
14:40:31 opendevreview Merged openstack/nova master: docs: Add releases to libvirt distro support matrix https://review.opendev.org/c/openstack/nova/+/792415
14:40:41 lyarwood Tagged under gate-failure
14:40:52 lyarwood Sorry I thought that's what you were looking at
14:41:16 gibi no problem, then I go and add more info under that
14:46:05 lyarwood Cool thanks
14:47:43 opendevreview Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575
14:50:31 gibi lyarwood: is this a new type of detach error https://zuul.opendev.org/t/openstack/build/02e6a99bf1574c978c663eb434705cbb/log/controller/logs/screen-n-cpu.txt?severity=0#34810 ?
14:51:11 gibi as far as I see it fails to detach from the live domain
14:51:20 gibi as the device is not there any more
14:51:44 gibi hmm, there is a DeviceRemovedEvent that was ignored :/
14:51:53 gibi sounds like a bug in the nova detach code
14:52:15 gibi I will file a bug and look into the root case
14:59:54 dansmith gmann: sorry I got pulled away
15:00:39 dansmith gmann: yeah, so I think I might be missing some keystone knowledge here.. and perhaps I'm trying to put too much capability in the projects for flexibility in terms of mapping people to abilities or somethin
15:03:41 gmann dansmith: afaik, one of the point to have system admin was this -to isolate project level stuff and remove project admin which was kind of global admin before new rbac
15:03:52 gmann as current project admin is admin to that project only
15:04:02 gmann current i mean after new rbac
15:05:08 dansmith yeah, I know that's one of the big tenets
15:40:45 dansmith gmann: okay I think I've straightened myself out on the cross-project thing
15:41:32 dansmith gmann: specifically related to enforcing policies in the api, I have another question
15:42:18 dansmith gmann: presumably if you have some role that lets you update a resource but not show it (possible if you configure policy that way), then you could not be able to show a resource, but make some trivial update to it and get a copy of it in the result of the PUT
15:42:57 dansmith I assume we would consider that acceptable because you gave that user update perms and the update call returns the resource...
15:43:19 dansmith just thinking of a case where you want some script to be able to reboot an instance but not see the metadata for it which contains a license key or something
15:44:00 gmann dansmith: yeah that's true. PUT has permission to show all info so they get.
15:44:39 dansmith okay, I figure that's the most straightforward thing to do -- not break the result of PUT just because you don't have get perms
15:44:40 gmann dansmith: in that case, we can introduce the new policy to restrict those metadata
15:45:15 dansmith yeah, sure, and that's legit, but I just wonder if someone would interpret restricting get to mean "they can never see this resource any way at all" which won't be the case
15:45:43 gmann dansmith: other way I think neutron does (need to check again) is check GET policy before PUT and deny if they do not have access to GET.
15:46:08 dansmith meaning require get and put perms to do a PUT, or just check get perms before returning the result?
15:46:38 gmann dansmith: I think yes, let me check. that is what i remember when i discussed it with amotoki in Toyko time.
15:46:47 dansmith the latter may make sense, but is probably not worth it, IMHO.. the former seems wrong
15:48:04 dansmith write-implies-read makes sense, write-requires-read does not, IMHO
15:56:57 gmann dansmith: cannot find that, will check later
15:57:13 dansmith gmann: no worries, not important at the moment, just curious
15:58:35 gmann dansmith: what we can do is restrict the PUT response if GET is not permit. GET-not-permit in this case (where PUT is allowed) means do not show server info instead of no access to server
15:59:23 dansmith gmann: yeah, that's the only thing that makes sense I think.. but I think you can argue that it's not worth that level of granularity

Earlier   Later