| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-10 | |||
| 16:51:11 | melwitt | whyyyyyy /o\ | |
| 16:51:34 | gibi | he suspects that something in the guest OS makes the detach hanging | |
| 16:52:01 | gibi | hence https://review.opendev.org/c/openstack/tempest/+/794757 | |
| 16:52:17 | lyarwood | I'm going to fix that up quickly now | |
| 16:52:39 | gibi | (I looked at the failures in the tempest test but did not figured out what was wrong:/) | |
| 16:53:09 | melwitt | how does that help the guest OS? sorry I don't understand | |
| 16:53:51 | gibi | that tempest patch will make sure that the guest console log is dumped | |
| 16:54:01 | melwitt | ohhh ok | |
| 16:54:18 | lyarwood | yeah it's not going to resolve anything | |
| 16:54:39 | lyarwood | I had to rework the cleanup ordering to get wait_for_volume_attachment_remove first | |
| 16:54:41 | lyarwood | as that is able to dump the console | |
| 16:55:46 | melwitt | gotcha | |
| 18:02:53 | ganso | melwitt, gibi: hi! could you please take a look at this backport? It already has a +2. Thanks in advance! | |
| 18:04:30 | melwitt | ganso: link? | |
| 18:06:00 | ganso | melwitt: sorry! forgot the link! https://review.opendev.org/c/openstack/nova/+/794328 | |
| 18:06:16 | melwitt | thanks | |
| 18:35:32 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM testing tempest volume detach failure capture of console https://review.opendev.org/c/openstack/nova/+/794766 | |
| 19:15:39 | sean-k-mooney | gibi: looking at the ci result https://review.opendev.org/c/openstack/ovsdbapp/+/795789 actully does seam to if not fix greatly imporve the issue | |
| 19:15:55 | sean-k-mooney | gibi: there are no recoonection or stalls with that patch | |
| 19:16:40 | sean-k-mooney | im still going to look at addtional impormennt within os-vif but that looks to me like a viable path forward | |
| 22:16:29 | opendevreview | Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519 | |
| #openstack-nova - 2021-06-11 | |||
| 06:17:00 | lyarwood | melwitt: \o/ awesome thanks! | |
| 07:53:42 | opendevreview | liujiong proposed openstack/nova master: Do not create attachment for old root volume https://review.opendev.org/c/openstack/nova/+/795950 | |
| 08:17:48 | bauzas | gibi: stephenfin: so, https://review.opendev.org/c/openstack/nova/+/795533 got a gate failure again | |
| 08:18:05 | bauzas | can we please accept then https://review.opendev.org/c/openstack/nova/+/795744 ? | |
| 08:24:27 | bauzas | looking at project:openstack/nova label:Verified>=-2,Zuul -age:3d is:open we have a lot of changes getting -1 | |
| 08:29:52 | opendevreview | Yongli He proposed openstack/nova master: Smartnic support - cyborg drive https://review.opendev.org/c/openstack/nova/+/771362 | |
| 08:29:54 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - new vnic type https://review.opendev.org/c/openstack/nova/+/771363 | |
| 08:29:57 | opendevreview | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 08:29:59 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - reject server move and suspend https://review.opendev.org/c/openstack/nova/+/779913 | |
| 08:30:00 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - functional tests https://review.opendev.org/c/openstack/nova/+/780147 | |
| 08:59:48 | lyarwood | bauzas: ack I've +W'd it to chip away at the gate failures | |
| 09:00:46 | bauzas | lyarwood: ok, I'll also provide the revert change | |
| 09:47:53 | opendevreview | Sylvain Bauza proposed openstack/nova master: Revert "Removing mypy to fix the nova CI" https://review.opendev.org/c/openstack/nova/+/795973 | |
| 09:48:05 | bauzas | lyarwood: revert is up | |
| 11:39:31 | opendevreview | Merged openstack/nova master: Removing mypy to fix the nova CI https://review.opendev.org/c/openstack/nova/+/795744 | |
| 11:51:08 | gibi | lyarwood, stephenfin, kashyap: rechecked https://review.opendev.org/c/openstack/nova/+/795533 as it hit https://bugs.launchpad.net/nova/+bug/1912310 | |
| 11:51:23 | kashyap | gibi: Hi; /me clicks | |
| 11:51:27 | gibi | the bug being "libvirt.libvirtError: unable to connect to server at" | |
| 11:51:33 | kashyap | I guess it's the dreaded connection refused thing | |
| 11:51:38 | kashyap | Yep | |
| 11:52:02 | gibi | around the same time when the migration fails | |
| 11:52:23 | gibi | there is an error | |
| 11:52:24 | gibi | multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory\ | |
| 11:53:16 | gibi | on the dest compute | |
| 11:53:32 | gibi | I don't know if it is related but at least it correlates by time | |
| 11:53:46 | gibi | I've added the log links to the bug | |
| 11:55:44 | lyarwood | That shouldn't be related no | |
| 11:55:55 | lyarwood | it's likely n-cpu fetching the connector from os-brick | |
| 11:56:05 | lyarwood | and os-brick is checking if multipathd is present on the host | |
| 11:57:14 | lyarwood | https://bugs.launchpad.net/nova/+bug/1931702 - FWIW I've raised this and will push a skip of the live migration with attached volume tests now | |
| 11:58:31 | gibi | lyarwood: good stuff | |
| 11:59:15 | gibi | lyarwood: is there a way forward with the lockup other than skiping the test? | |
| 11:59:48 | lyarwood | gibi: yeah we don't have the complete console log as I guess it rotated | |
| 12:00:00 | lyarwood | gibi: so I'll work out how to capture that and then raise a QEMU bug depending on what we see | |
| 12:00:10 | gibi | ohh, OK, cool | |
| 12:00:18 | lyarwood | gibi: as I'm assuming live migration has had some kind of impact here | |
| 12:01:16 | kashyap | lyarwood: For the QEMU bug, do you think we have hope of reproducing this outside of the CI Gate? | |
| 12:01:38 | kashyap | lyarwood: E.g. in the same Ubuntu env; same versions, migrating a paused instance -- can that do it? | |
| 12:01:39 | lyarwood | really depends on what if anything we get back from the guestOS | |
| 12:01:49 | kashyap | Yeah; good point | |
| 12:01:56 | lyarwood | the instance isn't paused in this latest failure | |
| 12:12:10 | masterpe[m] | I have instances in the placement.allocations table but these instances does not exists anymore. "nova-manage placement audit" does not exists on Train. And I have tryed "nova-manage placement heal_allocations" but that does not work. Can I savely deleted those records in database? | |
| 12:17:23 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration tests until bug #1931702 is resolved https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:18:06 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to #1931702 https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:18:07 | lyarwood | forgot to update the title sorry | |
| 12:22:16 | frickler | lyarwood: interesting issue, is that with cirros as guest or something else? | |
| 12:22:34 | lyarwood | frickler: yeah it's Cirros 0.5.2 | |
| 12:32:41 | lyarwood | ^ should give us console logs at the time we send the detach to see how borked the guestOS is | |
| 12:33:33 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip block migration with attached volumes tests due to bug #1931702 https://review.opendev.org/c/openstack/nova/+/795992 | |
| 12:33:33 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM Revert "zuul: Skip block migration with attached volumes tests due to #1931702" https://review.opendev.org/c/openstack/nova/+/795997 | |
| 12:33:36 | lyarwood | whops, ^ there we go, git-review being slow again. | |
| 12:38:37 | lyarwood | gibi: https://review.opendev.org/c/openstack/nova/+/792415 simple docs review if you have time btw | |
| 12:39:05 | lyarwood | sean-k-mooney: https://review.opendev.org/c/openstack/nova-specs/+/794799 - would you mind taking a look at this? | |
| 13:37:23 | opendevreview | Lee Yarwood proposed openstack/nova master: libvirt: fup docs and typing for _hard_reboot flow https://review.opendev.org/c/openstack/nova/+/795081 | |
| 13:38:10 | lyarwood | Wonderful now the nova-lvm job has failed randomly | |
| 14:00:20 | gibi | sorry I was pulled downstream, surfacing now... | |
| 14:05:36 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575 | |
| 14:19:00 | dansmith | gmann: I was looking through nova's instance-get stuff to see if and how things have changed in preparation for system scope and RBAC stuff | |
| 14:19:22 | dansmith | and it looks to me like we're still enforcing instance.project_id==context.project_id all the way down at the db layer in model_query() | |
| 14:20:12 | dansmith | it only does that if we're a "user context", but I'm thinking that is going to be in the way of more flexible rbac stuff, if you can't even load an instance object to check a richer policy rule | |
| 14:20:31 | dansmith | am I right in thinking that will have to change? | |
| 14:26:37 | gmann | dansmith: currently we do not check get instance permission for other policy check which need instance object in target, are you saying to change that to check get instance permission first? | |
| 14:27:34 | dansmith | gmann: we don't check for "get instance permission" before doing something like an update -- that's what you mean right? | |
| 14:27:41 | gmann | yeah | |
| 14:27:56 | gibi | lyarwood: re: nova-lvm failure I see multiple hits of the same error in logstash but all fairly recent http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22WARNING%3A%20Failed%20to%20get%20udev%20device%20handler%20for%20device%20%2Fdev%2Fsda1%5C%22 | |
| 14:28:32 | gibi | lyarwood: and apparently it is not nova-lvm specific | |
| 14:28:34 | dansmith | gmann: okay, but the problem is.. in order to do some update type thing that we might grant in policy, we have to db.get_instance(), which will fail to find it in model_query because it filters the owner very deep | |
| 14:28:49 | gibi | lyarwood: or did I have a wrong signature? | |
| 14:29:55 | gmann | dansmith: i see. and that is say system admin want to update server? | |
| 14:30:29 | gmann | or any other project member want to update some other project server? | |
| 14:31:08 | dansmith | well, right, I'm thinking the case where you want to do something more powerful, like say "project X is under project Y, so let project Y people manage project X servers too" | |
| 14:31:27 | gmann | humm | |
| 14:31:32 | dansmith | which is kinda the point of the RBAC work eventually, right? to make the policy actually let people do more complicated things like that.. no? | |
| 14:33:56 | gmann | dansmith: I think that is separate things if we want to allow than current secure RBAC which does not allow these kind of things instead restricting access | |
| 14:34:23 | gmann | project isolation is one things in new secure rbac but your use case is opposite to that | |
| 14:34:44 | dansmith | hrm | |
| 14:35:09 | lyarwood | <gibi "lyarwood: or did I have a wrong "> Yeah that's correct, iirc c-vol also hit something like this so maybe that's the duplicate? | |
| 14:35:54 | dansmith | gmann: my case is just my interpretation of what I thought the end goal was, so maybe I'm being too broad | |
| 14:36:16 | gibi | lyarwood: I see that we hit this during update_available_resource that simply skips the update and does not cause the job failure | |