| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-11 | |||
| 11:58:31 | gibi | lyarwood: good stuff | |
| 11:59:15 | gibi | lyarwood: is there a way forward with the lockup other than skiping the test? | |
| 11:59:48 | lyarwood | gibi: yeah we don't have the complete console log as I guess it rotated | |
| 12:00:00 | lyarwood | gibi: so I'll work out how to capture that and then raise a QEMU bug depending on what we see | |
| 12:00:10 | gibi | ohh, OK, cool | |
| 12:00:18 | lyarwood | gibi: as I'm assuming live migration has had some kind of impact here | |
| 12:01:16 | kashyap | lyarwood: For the QEMU bug, do you think we have hope of reproducing this outside of the CI Gate? | |
| 12:01:38 | kashyap | lyarwood: E.g. in the same Ubuntu env; same versions, migrating a paused instance -- can that do it? | |
| 12:01:39 | lyarwood | really depends on what if anything we get back from the guestOS | |
| 12:01:49 | kashyap | Yeah; good point | |
| 12:01:56 | lyarwood | the instance isn't paused in this latest failure | |
| 12:12:10 | masterpe[m] | I have instances in the placement.allocations table but these instances does not exists anymore. "nova-manage placement audit" does not exists on Train. And I have tryed "nova-manage placement heal_allocations" but that does not work. Can I savely deleted those records in database? | |
| 12:17:23 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration tests until bug #1931702 is resolved https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:18:06 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to #1931702 https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:18:07 | lyarwood | forgot to update the title sorry | |
| 12:22:16 | frickler | lyarwood: interesting issue, is that with cirros as guest or something else? | |
| 12:22:34 | lyarwood | frickler: yeah it's Cirros 0.5.2 | |
| 12:32:41 | lyarwood | ^ should give us console logs at the time we send the detach to see how borked the guestOS is | |
| 12:33:33 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM Revert "zuul: Skip block migration with attached volumes tests due to #1931702" https://review.opendev.org/c/openstack/nova/+/795997 | |
| 12:33:33 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to bug #1931702 https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:33:36 | lyarwood | whops, ^ there we go, git-review being slow again. | |
| 12:38:37 | lyarwood | gibi: https://review.opendev.org/c/openstack/nova/+/792415 simple docs review if you have time btw | |
| 12:39:05 | lyarwood | sean-k-mooney: https://review.opendev.org/c/openstack/nova-specs/+/794799 - would you mind taking a look at this? | |
| 13:37:23 | opendevreview | Lee Yarwood proposed openstack/nova master: libvirt: fup docs and typing for _hard_reboot flow https://review.opendev.org/c/openstack/nova/+/795081 | |
| 13:38:10 | lyarwood | Wonderful now the nova-lvm job has failed randomly | |
| 14:00:20 | gibi | sorry I was pulled downstream, surfacing now... | |
| 14:05:36 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575 | |
| 14:19:00 | dansmith | gmann: I was looking through nova's instance-get stuff to see if and how things have changed in preparation for system scope and RBAC stuff | |
| 14:19:22 | dansmith | and it looks to me like we're still enforcing instance.project_id==context.project_id all the way down at the db layer in model_query() | |
| 14:20:12 | dansmith | it only does that if we're a "user context", but I'm thinking that is going to be in the way of more flexible rbac stuff, if you can't even load an instance object to check a richer policy rule | |
| 14:20:31 | dansmith | am I right in thinking that will have to change? | |
| 14:26:37 | gmann | dansmith: currently we do not check get instance permission for other policy check which need instance object in target, are you saying to change that to check get instance permission first? | |
| 14:27:34 | dansmith | gmann: we don't check for "get instance permission" before doing something like an update -- that's what you mean right? | |
| 14:27:41 | gmann | yeah | |
| 14:27:56 | gibi | lyarwood: re: nova-lvm failure I see multiple hits of the same error in logstash but all fairly recent http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22WARNING%3A%20Failed%20to%20get%20udev%20device%20handler%20for%20device%20%2Fdev%2Fsda1%5C%22 | |
| 14:28:32 | gibi | lyarwood: and apparently it is not nova-lvm specific | |
| 14:28:34 | dansmith | gmann: okay, but the problem is.. in order to do some update type thing that we might grant in policy, we have to db.get_instance(), which will fail to find it in model_query because it filters the owner very deep | |
| 14:28:49 | gibi | lyarwood: or did I have a wrong signature? | |
| 14:29:55 | gmann | dansmith: i see. and that is say system admin want to update server? | |
| 14:30:29 | gmann | or any other project member want to update some other project server? | |
| 14:31:08 | dansmith | well, right, I'm thinking the case where you want to do something more powerful, like say "project X is under project Y, so let project Y people manage project X servers too" | |
| 14:31:27 | gmann | humm | |
| 14:31:32 | dansmith | which is kinda the point of the RBAC work eventually, right? to make the policy actually let people do more complicated things like that.. no? | |
| 14:33:56 | gmann | dansmith: I think that is separate things if we want to allow than current secure RBAC which does not allow these kind of things instead restricting access | |
| 14:34:23 | gmann | project isolation is one things in new secure rbac but your use case is opposite to that | |
| 14:34:44 | dansmith | hrm | |
| 14:35:09 | lyarwood | <gibi "lyarwood: or did I have a wrong "> Yeah that's correct, iirc c-vol also hit something like this so maybe that's the duplicate? | |
| 14:35:54 | dansmith | gmann: my case is just my interpretation of what I thought the end goal was, so maybe I'm being too broad | |
| 14:36:16 | gibi | lyarwood: I see that we hit this during update_available_resource that simply skips the update and does not cause the job failure | |
| 14:36:20 | lyarwood | Sorry gibi, ^ was for you, somehow replied in element (matrix) and I'm not sure what that looks like in vanilla irc. | |
| 14:36:39 | gibi | lyarwood: it quotes part of the original message so it work for me :) | |
| 14:36:46 | gmann | dansmith: yeah because in most of the case we pass instance.project_id as target to oslo policy | |
| 14:36:57 | gibi | lyarwood: so most of the logstash hits are actual job SUCCESS as it only hit the during periodic | |
| 14:37:09 | gmann | dansmith: if we want to allow cross project operation we have to change that model and db things too what you mentioned | |
| 14:37:13 | lyarwood | Cool cool, the nova-lvm failure I saw was during instance deletion | |
| 14:37:31 | gibi | lyarwood: yeah, if we hit it during other operation then the test fails | |
| 14:37:36 | lyarwood | So maybe we just need to retry on failure here | |
| 14:37:52 | lyarwood | As it's just a basic lvs command | |
| 14:37:56 | gibi | lyarwood: but the hit in the periodic shows that the underlying issue it temprorary as the next periodic succeeds | |
| 14:38:10 | gibi | lyarwood: yeah, so a retry could help | |
| 14:38:14 | lyarwood | Yup cool, I'll work on that on Monday | |
| 14:38:35 | gibi | cool | |
| 14:39:01 | gibi | lyarwood: should I open a bug report on it? | |
| 14:39:09 | gibi | I have the context now | |
| 14:39:22 | gibi | and still have time today :) | |
| 14:40:03 | gmann | dansmith: yes, too broad :). I think in that case they should allocate that user a system token and ask to do other project things. but system to do project servers operation is another challenge what we discussded in PTG | |
| 14:40:29 | lyarwood | I already have | |
| 14:40:31 | opendevreview | Merged openstack/nova master: docs: Add releases to libvirt distro support matrix https://review.opendev.org/c/openstack/nova/+/792415 | |
| 14:40:41 | lyarwood | Tagged under gate-failure | |
| 14:40:52 | lyarwood | Sorry I thought that's what you were looking at | |
| 14:41:16 | gibi | no problem, then I go and add more info under that | |
| 14:46:05 | lyarwood | Cool thanks | |
| 14:47:43 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575 | |
| 14:50:31 | gibi | lyarwood: is this a new type of detach error https://zuul.opendev.org/t/openstack/build/02e6a99bf1574c978c663eb434705cbb/log/controller/logs/screen-n-cpu.txt?severity=0#34810 ? | |
| 14:51:11 | gibi | as far as I see it fails to detach from the live domain | |
| 14:51:20 | gibi | as the device is not there any more | |
| 14:51:44 | gibi | hmm, there is a DeviceRemovedEvent that was ignored :/ | |
| 14:51:53 | gibi | sounds like a bug in the nova detach code | |
| 14:52:15 | gibi | I will file a bug and look into the root case | |
| 14:59:54 | dansmith | gmann: sorry I got pulled away | |
| 15:00:39 | dansmith | gmann: yeah, so I think I might be missing some keystone knowledge here.. and perhaps I'm trying to put too much capability in the projects for flexibility in terms of mapping people to abilities or somethin | |
| 15:03:41 | gmann | dansmith: afaik, one of the point to have system admin was this -to isolate project level stuff and remove project admin which was kind of global admin before new rbac | |
| 15:03:52 | gmann | as current project admin is admin to that project only | |
| 15:04:02 | gmann | current i mean after new rbac | |
| 15:05:08 | dansmith | yeah, I know that's one of the big tenets | |
| 15:40:45 | dansmith | gmann: okay I think I've straightened myself out on the cross-project thing | |
| 15:41:32 | dansmith | gmann: specifically related to enforcing policies in the api, I have another question | |
| 15:42:18 | dansmith | gmann: presumably if you have some role that lets you update a resource but not show it (possible if you configure policy that way), then you could not be able to show a resource, but make some trivial update to it and get a copy of it in the result of the PUT | |
| 15:42:57 | dansmith | I assume we would consider that acceptable because you gave that user update perms and the update call returns the resource... | |
| 15:43:19 | dansmith | just thinking of a case where you want some script to be able to reboot an instance but not see the metadata for it which contains a license key or something | |
| 15:44:00 | gmann | dansmith: yeah that's true. PUT has permission to show all info so they get. | |
| 15:44:39 | dansmith | okay, I figure that's the most straightforward thing to do -- not break the result of PUT just because you don't have get perms | |
| 15:44:40 | gmann | dansmith: in that case, we can introduce the new policy to restrict those metadata | |
| 15:45:15 | dansmith | yeah, sure, and that's legit, but I just wonder if someone would interpret restricting get to mean "they can never see this resource any way at all" which won't be the case | |
| 15:45:43 | gmann | dansmith: other way I think neutron does (need to check again) is check GET policy before PUT and deny if they do not have access to GET. | |
| 15:46:08 | dansmith | meaning require get and put perms to do a PUT, or just check get perms before returning the result? | |
| 15:46:38 | gmann | dansmith: I think yes, let me check. that is what i remember when i discussed it with amotoki in Toyko time. | |
| 15:46:47 | dansmith | the latter may make sense, but is probably not worth it, IMHO.. the former seems wrong | |
| 15:48:04 | dansmith | write-implies-read makes sense, write-requires-read does not, IMHO | |
| 15:56:57 | gmann | dansmith: cannot find that, will check later | |