| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-12 | |||
| 07:11:06 | brinzhang | hi team, the series of remove tenant_id patches have been updated, please review them while you are free, thanks | |
| 07:19:56 | gibi | stephenfin: nice catch | |
| 08:14:49 | bauzas | good morning fokls | |
| 08:21:31 | lyarwood | Morning | |
| 08:22:09 | gibi | bauzas, lyarwood: happy Friday! | |
| 08:22:37 | bauzas | oh, the Friday dress code is back ? | |
| 08:23:21 | giblet | my neighbor decided to renovate his flat and the workers demolishing his bathroom at the moment which has a common wall with my flat. there is a constant noise so I feel like giblet already | |
| 08:23:39 | giblet | hence the name | |
| 08:24:54 | bauwser | hah, the likes of remote working | |
| 08:26:58 | lyarwood | urgh that's awful giblet | |
| 08:29:17 | giblet | I will survive | |
| 08:30:45 | bauwser | every last Thurday of the month, I appreciate my neighbor | |
| 08:31:41 | bauwser | just because then he gets 5 people for mowing his lawn | |
| 08:32:01 | giblet | a lot of lawn I assume | |
| 08:32:16 | bauwser | yeah, around ~2500sqm | |
| 08:32:31 | bauwser | but they are not having mowers | |
| 08:32:42 | bauwser | rather lawn tractors... | |
| 08:33:04 | bauwser | you can imagine the noise | |
| 08:39:20 | giblet | :/ | |
| 09:06:13 | opendevreview | Balazs Gibizer proposed openstack/nova stable/pike: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813437 | |
| 09:46:45 | opendevreview | Brin Zhang proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852 | |
| 09:46:46 | opendevreview | Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in os-quota-sets path https://review.opendev.org/c/openstack/nova/+/768851 | |
| 09:46:46 | opendevreview | Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in Limits API https://review.opendev.org/c/openstack/nova/+/768862 | |
| 09:52:35 | opendevreview | Merged openstack/nova master: tests: Restore - don't reset - warning filters https://review.opendev.org/c/openstack/nova/+/817641 | |
| 09:56:53 | bauwser | gibi: thanks for reviewing https://review.opendev.org/c/openstack/nova/+/816861 | |
| 09:57:06 | bauwser | ah, giblet ^ | |
| 09:57:26 | bauwser | giblet: your comment looks good to me, I'll provide a new revision | |
| 10:02:38 | giblet | bauwser: how do you feel about my top level comment there that I'm supportive to allows non-cores to state priority but not for patch owners to self +1? | |
| 10:03:09 | bauwser | giblet: honestly, I don't have any opinion | |
| 10:03:29 | bauwser | giblet: maybe we should not use this label for asking folks to look at our change | |
| 10:03:49 | opendevreview | Brin Zhang proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329 | |
| 10:04:02 | bauwser | giblet: but we should try to find some way to help contributors to ask for reviews without needing to go in IRC | |
| 10:04:30 | bauwser | so I'll provide 2 different points | |
| 10:04:41 | bauwser | and we'll discuss this in the change | |
| 10:17:34 | giblet | hm | |
| 10:17:50 | giblet | why people cannot come to IRC? | |
| 10:18:47 | giblet | is the problem that it is real time and therefore time zone dependent? | |
| 10:18:59 | giblet | then they can use the ML | |
| 10:19:34 | giblet | (but also I keep my client up and read scrollback so I can be reached from other timezones via IRC too) | |
| 10:34:05 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP: [doc] propose Review-Priority label for contribs https://review.opendev.org/c/openstack/nova/+/816861 | |
| 10:36:29 | bauwser | giblet: we provided a way to async ask for reviews previously with the etherpad | |
| 10:36:41 | bauwser | now, you need to ping people, ideally synchronously | |
| 10:37:05 | bauwser | and I don't want the ML to be used for begging reviews or other non-nova contributors will yell at us :) | |
| 10:38:42 | bauwser | but I need to go to gym | |
| 10:38:56 | bauwser | I'm starting to look at https://gerrit-review.googlesource.com/Documentation/user-attention-set.html | |
| 10:38:59 | bauwser | we could use it | |
| 11:18:48 | giblet | bauwser: attentions set is built on people added in CC and in Review field of a patch | |
| 11:19:05 | giblet | btw, why don't we say, add me in the review if you need me to look at the patch | |
| 11:19:17 | giblet | that is async and gerrit based | |
| 11:27:25 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Enable SQLAlchemy 2.0 deprecation warnings https://review.opendev.org/c/openstack/nova/+/804709 | |
| 11:27:25 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Replace use of Executable.scalar(), Executable.execute() https://review.opendev.org/c/openstack/nova/+/804878 | |
| 11:27:26 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Replace use of 'autoload' parameter https://review.opendev.org/c/openstack/nova/+/805734 | |
| 11:27:26 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Replace use of legacy select() calling style https://review.opendev.org/c/openstack/nova/+/805735 | |
| 11:27:27 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Replace 'insert.inline' parameter with 'Insert.inline()' method https://review.opendev.org/c/openstack/nova/+/805736 | |
| 11:27:27 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Don't pass strings to 'Connection.execute' https://review.opendev.org/c/openstack/nova/+/805737 | |
| 11:27:28 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Don't use legacy 'Row()' methods https://review.opendev.org/c/openstack/nova/+/817746 | |
| 11:27:29 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove use of 'bind' arguments https://review.opendev.org/c/openstack/nova/+/817747 | |
| 11:27:29 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove unnecessary warning filters https://review.opendev.org/c/openstack/nova/+/817748 | |
| 13:17:46 | giblet | lyarwood, sean-k-mooney: so the patch with the sleep before detach now reproduced the kernel panic https://review.opendev.org/c/openstack/nova/+/817564/3#message-8ef45ae21f54b427ae2c88bbc19a4c926ce00883 | |
| 13:18:15 | giblet | from the consol log it is clear that the kernel panic happened _before_ we trigger detach | |
| 13:18:28 | sean-k-mooney | is it possible the panic was tirggered by the attach | |
| 13:18:35 | giblet | or by the live migration | |
| 13:18:45 | giblet | or by the initial boot | |
| 13:18:50 | giblet | all possible | |
| 13:19:02 | giblet | so I will add more console printing step to the test case | |
| 13:19:10 | giblet | to see when the panic happens | |
| 13:20:19 | sean-k-mooney | looks liek it happend 19 seocnd after boot | |
| 13:20:53 | giblet | true, we can try to correlate that to the action in the test | |
| 13:21:22 | sean-k-mooney | the vm went active ~ 12:04:29 | |
| 13:21:59 | sean-k-mooney | we did the attach at 12:04:40 | |
| 13:22:34 | sean-k-mooney | actully no maybe i misread that | |
| 13:23:07 | giblet | 2021-11-12 12:04:30,036 99321 INFO [tempest.common.waiters] State transition "BUILD/spawning" ==> "ACTIVE/None" after 7 second wait | |
| 13:23:59 | giblet | 2021-11-12 12:04:38,573 99321 INFO [tempest.common.waiters] volume 0f307fda-dbef-408c-a029-97401db15945 reached in-use after waiting for 0.573394 seconds | |
| 13:24:16 | sean-k-mooney | looks liek we were just finsihing the mightation when the panic happend | |
| 13:24:47 | sean-k-mooney | it went active at 2021 12:04:52 | |
| 13:25:36 | giblet | 2021-11-12 12:04:40,615 <- trigger the live migration | |
| 13:25:48 | giblet | 2021-11-12 12:04:52,693 99321 INFO [tempest.common.waiters] State transition "MIGRATING/migrating" ==> "ACTIVE/None" after 12 second wait | |
| 13:26:04 | giblet | yeah so the panic is somehow related to the live migration | |
| 13:26:06 | sean-k-mooney | yep | |
| 13:26:23 | sean-k-mooney | it woudl have been i gues at :49 ish | |
| 13:27:54 | sean-k-mooney | giblet: just looking at the panic | |
| 13:28:07 | sean-k-mooney | its realted to ipv6 adress configuraiton | |
| 13:28:36 | giblet | I saw at least 3 different kernel panic | |
| 13:28:43 | giblet | that test can produce | |
| 13:28:47 | giblet | this is the 4th one :) | |
| 13:29:21 | sean-k-mooney | well its specifclay failing in _raw_spin_lock_bh | |
| 13:30:26 | lyarwood | sorry have various trades people in my house today trying to sort something out | |
| 13:30:34 | lyarwood | really hard to focus on anything | |
| 13:31:45 | sean-k-mooney | lyarwood: tl;dr the panic happen during the migration in an interupt handel form _raw_spin_lock_bh at least in this insntace | |
| 13:32:25 | lyarwood | giblet: remind me again, when did thi start failing? | |
| 13:32:25 | sean-k-mooney | so its before the detach in the cleanup phase if the time stames are correct | |
| 13:32:28 | lyarwood | this* | |
| 13:32:32 | giblet | tehre is lot of different RIP codes in the reproductions https://paste.ubuntu.com/p/QGrtv2nZWz/ | |
| 13:32:45 | giblet | lyarwood: on it ... | |
| 13:33:49 | giblet | lyarwood: the first panic was 2021-11-08T12:59:16 | |
| 13:33:52 | sean-k-mooney | giblet: yes altoh galoto of them look like they are related to lcoking/interupt handeling in general | |
| 13:34:15 | giblet | this one https://zuul.opendev.org/t/openstack/build/c445dc25cb2c4567b223f05d95134c47 | |
| 13:34:17 | lyarwood | giblet: kk it's not related to https://review.opendev.org/c/openstack/devstack/+/812928 then | |
| 13:36:41 | giblet | lyarwood: there is big gaps between nova-live-migration job run before 11.08. | |
| 13:36:44 | giblet | https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&project=openstack%2Fnova&branch=stable%2Fvictoria | |