| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-11 | |||
| 08:29:59 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - reject server move and suspend https://review.opendev.org/c/openstack/nova/+/779913 | |
| 08:30:00 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - functional tests https://review.opendev.org/c/openstack/nova/+/780147 | |
| 08:59:48 | lyarwood | bauzas: ack I've +W'd it to chip away at the gate failures | |
| 09:00:46 | bauzas | lyarwood: ok, I'll also provide the revert change | |
| 09:47:53 | opendevreview | Sylvain Bauza proposed openstack/nova master: Revert "Removing mypy to fix the nova CI" https://review.opendev.org/c/openstack/nova/+/795973 | |
| 09:48:05 | bauzas | lyarwood: revert is up | |
| 11:39:31 | opendevreview | Merged openstack/nova master: Removing mypy to fix the nova CI https://review.opendev.org/c/openstack/nova/+/795744 | |
| 11:51:08 | gibi | lyarwood, stephenfin, kashyap: rechecked https://review.opendev.org/c/openstack/nova/+/795533 as it hit https://bugs.launchpad.net/nova/+bug/1912310 | |
| 11:51:23 | kashyap | gibi: Hi; /me clicks | |
| 11:51:27 | gibi | the bug being "libvirt.libvirtError: unable to connect to server at" | |
| 11:51:33 | kashyap | I guess it's the dreaded connection refused thing | |
| 11:51:38 | kashyap | Yep | |
| 11:52:02 | gibi | around the same time when the migration fails | |
| 11:52:23 | gibi | there is an error | |
| 11:52:24 | gibi | multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory\ | |
| 11:53:16 | gibi | on the dest compute | |
| 11:53:32 | gibi | I don't know if it is related but at least it correlates by time | |
| 11:53:46 | gibi | I've added the log links to the bug | |
| 11:55:44 | lyarwood | That shouldn't be related no | |
| 11:55:55 | lyarwood | it's likely n-cpu fetching the connector from os-brick | |
| 11:56:05 | lyarwood | and os-brick is checking if multipathd is present on the host | |
| 11:57:14 | lyarwood | https://bugs.launchpad.net/nova/+bug/1931702 - FWIW I've raised this and will push a skip of the live migration with attached volume tests now | |
| 11:58:31 | gibi | lyarwood: good stuff | |
| 11:59:15 | gibi | lyarwood: is there a way forward with the lockup other than skiping the test? | |
| 11:59:48 | lyarwood | gibi: yeah we don't have the complete console log as I guess it rotated | |
| 12:00:00 | lyarwood | gibi: so I'll work out how to capture that and then raise a QEMU bug depending on what we see | |
| 12:00:10 | gibi | ohh, OK, cool | |
| 12:00:18 | lyarwood | gibi: as I'm assuming live migration has had some kind of impact here | |
| 12:01:16 | kashyap | lyarwood: For the QEMU bug, do you think we have hope of reproducing this outside of the CI Gate? | |
| 12:01:38 | kashyap | lyarwood: E.g. in the same Ubuntu env; same versions, migrating a paused instance -- can that do it? | |
| 12:01:39 | lyarwood | really depends on what if anything we get back from the guestOS | |
| 12:01:49 | kashyap | Yeah; good point | |
| 12:01:56 | lyarwood | the instance isn't paused in this latest failure | |
| 12:12:10 | masterpe[m] | I have instances in the placement.allocations table but these instances does not exists anymore. "nova-manage placement audit" does not exists on Train. And I have tryed "nova-manage placement heal_allocations" but that does not work. Can I savely deleted those records in database? | |
| 12:17:23 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration tests until bug #1931702 is resolved https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:18:06 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to #1931702 https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:18:07 | lyarwood | forgot to update the title sorry | |
| 12:22:16 | frickler | lyarwood: interesting issue, is that with cirros as guest or something else? | |
| 12:22:34 | lyarwood | frickler: yeah it's Cirros 0.5.2 | |
| 12:32:41 | lyarwood | ^ should give us console logs at the time we send the detach to see how borked the guestOS is | |
| 12:33:33 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM Revert "zuul: Skip block migration with attached volumes tests due to #1931702" https://review.opendev.org/c/openstack/nova/+/795997 | |
| 12:33:33 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to bug #1931702 https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:33:36 | lyarwood | whops, ^ there we go, git-review being slow again. | |
| 12:38:37 | lyarwood | gibi: https://review.opendev.org/c/openstack/nova/+/792415 simple docs review if you have time btw | |
| 12:39:05 | lyarwood | sean-k-mooney: https://review.opendev.org/c/openstack/nova-specs/+/794799 - would you mind taking a look at this? | |
| 13:37:23 | opendevreview | Lee Yarwood proposed openstack/nova master: libvirt: fup docs and typing for _hard_reboot flow https://review.opendev.org/c/openstack/nova/+/795081 | |
| 13:38:10 | lyarwood | Wonderful now the nova-lvm job has failed randomly | |
| 14:00:20 | gibi | sorry I was pulled downstream, surfacing now... | |
| 14:05:36 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575 | |
| 14:19:00 | dansmith | gmann: I was looking through nova's instance-get stuff to see if and how things have changed in preparation for system scope and RBAC stuff | |
| 14:19:22 | dansmith | and it looks to me like we're still enforcing instance.project_id==context.project_id all the way down at the db layer in model_query() | |
| 14:20:12 | dansmith | it only does that if we're a "user context", but I'm thinking that is going to be in the way of more flexible rbac stuff, if you can't even load an instance object to check a richer policy rule | |
| 14:20:31 | dansmith | am I right in thinking that will have to change? | |
| 14:26:37 | gmann | dansmith: currently we do not check get instance permission for other policy check which need instance object in target, are you saying to change that to check get instance permission first? | |
| 14:27:34 | dansmith | gmann: we don't check for "get instance permission" before doing something like an update -- that's what you mean right? | |
| 14:27:41 | gmann | yeah | |
| 14:27:56 | gibi | lyarwood: re: nova-lvm failure I see multiple hits of the same error in logstash but all fairly recent http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22WARNING%3A%20Failed%20to%20get%20udev%20device%20handler%20for%20device%20%2Fdev%2Fsda1%5C%22 | |
| 14:28:32 | gibi | lyarwood: and apparently it is not nova-lvm specific | |
| 14:28:34 | dansmith | gmann: okay, but the problem is.. in order to do some update type thing that we might grant in policy, we have to db.get_instance(), which will fail to find it in model_query because it filters the owner very deep | |
| 14:28:49 | gibi | lyarwood: or did I have a wrong signature? | |
| 14:29:55 | gmann | dansmith: i see. and that is say system admin want to update server? | |
| 14:30:29 | gmann | or any other project member want to update some other project server? | |
| 14:31:08 | dansmith | well, right, I'm thinking the case where you want to do something more powerful, like say "project X is under project Y, so let project Y people manage project X servers too" | |
| 14:31:27 | gmann | humm | |
| 14:31:32 | dansmith | which is kinda the point of the RBAC work eventually, right? to make the policy actually let people do more complicated things like that.. no? | |
| 14:33:56 | gmann | dansmith: I think that is separate things if we want to allow than current secure RBAC which does not allow these kind of things instead restricting access | |
| 14:34:23 | gmann | project isolation is one things in new secure rbac but your use case is opposite to that | |
| 14:34:44 | dansmith | hrm | |
| 14:35:09 | lyarwood | <gibi "lyarwood: or did I have a wrong "> Yeah that's correct, iirc c-vol also hit something like this so maybe that's the duplicate? | |
| 14:35:54 | dansmith | gmann: my case is just my interpretation of what I thought the end goal was, so maybe I'm being too broad | |
| 14:36:16 | gibi | lyarwood: I see that we hit this during update_available_resource that simply skips the update and does not cause the job failure | |
| 14:36:20 | lyarwood | Sorry gibi, ^ was for you, somehow replied in element (matrix) and I'm not sure what that looks like in vanilla irc. | |
| 14:36:39 | gibi | lyarwood: it quotes part of the original message so it work for me :) | |
| 14:36:46 | gmann | dansmith: yeah because in most of the case we pass instance.project_id as target to oslo policy | |
| 14:36:57 | gibi | lyarwood: so most of the logstash hits are actual job SUCCESS as it only hit the during periodic | |
| 14:37:09 | gmann | dansmith: if we want to allow cross project operation we have to change that model and db things too what you mentioned | |
| 14:37:13 | lyarwood | Cool cool, the nova-lvm failure I saw was during instance deletion | |
| 14:37:31 | gibi | lyarwood: yeah, if we hit it during other operation then the test fails | |
| 14:37:36 | lyarwood | So maybe we just need to retry on failure here | |
| 14:37:52 | lyarwood | As it's just a basic lvs command | |
| 14:37:56 | gibi | lyarwood: but the hit in the periodic shows that the underlying issue it temprorary as the next periodic succeeds | |
| 14:38:10 | gibi | lyarwood: yeah, so a retry could help | |
| 14:38:14 | lyarwood | Yup cool, I'll work on that on Monday | |
| 14:38:35 | gibi | cool | |
| 14:39:01 | gibi | lyarwood: should I open a bug report on it? | |
| 14:39:09 | gibi | I have the context now | |
| 14:39:22 | gibi | and still have time today :) | |
| 14:40:03 | gmann | dansmith: yes, too broad :). I think in that case they should allocate that user a system token and ask to do other project things. but system to do project servers operation is another challenge what we discussded in PTG | |
| 14:40:29 | lyarwood | I already have | |
| 14:40:31 | opendevreview | Merged openstack/nova master: docs: Add releases to libvirt distro support matrix https://review.opendev.org/c/openstack/nova/+/792415 | |
| 14:40:41 | lyarwood | Tagged under gate-failure | |
| 14:40:52 | lyarwood | Sorry I thought that's what you were looking at | |
| 14:41:16 | gibi | no problem, then I go and add more info under that | |
| 14:46:05 | lyarwood | Cool thanks | |
| 14:47:43 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575 | |
| 14:50:31 | gibi | lyarwood: is this a new type of detach error https://zuul.opendev.org/t/openstack/build/02e6a99bf1574c978c663eb434705cbb/log/controller/logs/screen-n-cpu.txt?severity=0#34810 ? | |
| 14:51:11 | gibi | as far as I see it fails to detach from the live domain | |
| 14:51:20 | gibi | as the device is not there any more | |
| 14:51:44 | gibi | hmm, there is a DeviceRemovedEvent that was ignored :/ | |
| 14:51:53 | gibi | sounds like a bug in the nova detach code | |