| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-12 | |||
| 10:03:29 | bauwser | giblet: maybe we should not use this label for asking folks to look at our change | |
| 10:03:49 | opendevreview | Brin Zhang proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329 | |
| 10:04:02 | bauwser | giblet: but we should try to find some way to help contributors to ask for reviews without needing to go in IRC | |
| 10:04:30 | bauwser | so I'll provide 2 different points | |
| 10:04:41 | bauwser | and we'll discuss this in the change | |
| 10:17:34 | giblet | hm | |
| 10:17:50 | giblet | why people cannot come to IRC? | |
| 10:18:47 | giblet | is the problem that it is real time and therefore time zone dependent? | |
| 10:18:59 | giblet | then they can use the ML | |
| 10:19:34 | giblet | (but also I keep my client up and read scrollback so I can be reached from other timezones via IRC too) | |
| 10:34:05 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP: [doc] propose Review-Priority label for contribs https://review.opendev.org/c/openstack/nova/+/816861 | |
| 10:36:29 | bauwser | giblet: we provided a way to async ask for reviews previously with the etherpad | |
| 10:36:41 | bauwser | now, you need to ping people, ideally synchronously | |
| 10:37:05 | bauwser | and I don't want the ML to be used for begging reviews or other non-nova contributors will yell at us :) | |
| 10:38:42 | bauwser | but I need to go to gym | |
| 10:38:56 | bauwser | I'm starting to look at https://gerrit-review.googlesource.com/Documentation/user-attention-set.html | |
| 10:38:59 | bauwser | we could use it | |
| 11:18:48 | giblet | bauwser: attentions set is built on people added in CC and in Review field of a patch | |
| 11:19:05 | giblet | btw, why don't we say, add me in the review if you need me to look at the patch | |
| 11:19:17 | giblet | that is async and gerrit based | |
| 11:27:25 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Enable SQLAlchemy 2.0 deprecation warnings https://review.opendev.org/c/openstack/nova/+/804709 | |
| 11:27:25 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Replace use of Executable.scalar(), Executable.execute() https://review.opendev.org/c/openstack/nova/+/804878 | |
| 11:27:26 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Replace use of 'autoload' parameter https://review.opendev.org/c/openstack/nova/+/805734 | |
| 11:27:26 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Replace use of legacy select() calling style https://review.opendev.org/c/openstack/nova/+/805735 | |
| 11:27:27 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Replace 'insert.inline' parameter with 'Insert.inline()' method https://review.opendev.org/c/openstack/nova/+/805736 | |
| 11:27:27 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Don't pass strings to 'Connection.execute' https://review.opendev.org/c/openstack/nova/+/805737 | |
| 11:27:28 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Don't use legacy 'Row()' methods https://review.opendev.org/c/openstack/nova/+/817746 | |
| 11:27:29 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove use of 'bind' arguments https://review.opendev.org/c/openstack/nova/+/817747 | |
| 11:27:29 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove unnecessary warning filters https://review.opendev.org/c/openstack/nova/+/817748 | |
| 13:17:46 | giblet | lyarwood, sean-k-mooney: so the patch with the sleep before detach now reproduced the kernel panic https://review.opendev.org/c/openstack/nova/+/817564/3#message-8ef45ae21f54b427ae2c88bbc19a4c926ce00883 | |
| 13:18:15 | giblet | from the consol log it is clear that the kernel panic happened _before_ we trigger detach | |
| 13:18:28 | sean-k-mooney | is it possible the panic was tirggered by the attach | |
| 13:18:35 | giblet | or by the live migration | |
| 13:18:45 | giblet | or by the initial boot | |
| 13:18:50 | giblet | all possible | |
| 13:19:02 | giblet | so I will add more console printing step to the test case | |
| 13:19:10 | giblet | to see when the panic happens | |
| 13:20:19 | sean-k-mooney | looks liek it happend 19 seocnd after boot | |
| 13:20:53 | giblet | true, we can try to correlate that to the action in the test | |
| 13:21:22 | sean-k-mooney | the vm went active ~ 12:04:29 | |
| 13:21:59 | sean-k-mooney | we did the attach at 12:04:40 | |
| 13:22:34 | sean-k-mooney | actully no maybe i misread that | |
| 13:23:07 | giblet | 2021-11-12 12:04:30,036 99321 INFO [tempest.common.waiters] State transition "BUILD/spawning" ==> "ACTIVE/None" after 7 second wait | |
| 13:23:59 | giblet | 2021-11-12 12:04:38,573 99321 INFO [tempest.common.waiters] volume 0f307fda-dbef-408c-a029-97401db15945 reached in-use after waiting for 0.573394 seconds | |
| 13:24:16 | sean-k-mooney | looks liek we were just finsihing the mightation when the panic happend | |
| 13:24:47 | sean-k-mooney | it went active at 2021 12:04:52 | |
| 13:25:36 | giblet | 2021-11-12 12:04:40,615 <- trigger the live migration | |
| 13:25:48 | giblet | 2021-11-12 12:04:52,693 99321 INFO [tempest.common.waiters] State transition "MIGRATING/migrating" ==> "ACTIVE/None" after 12 second wait | |
| 13:26:04 | giblet | yeah so the panic is somehow related to the live migration | |
| 13:26:06 | sean-k-mooney | yep | |
| 13:26:23 | sean-k-mooney | it woudl have been i gues at :49 ish | |
| 13:27:54 | sean-k-mooney | giblet: just looking at the panic | |
| 13:28:07 | sean-k-mooney | its realted to ipv6 adress configuraiton | |
| 13:28:36 | giblet | I saw at least 3 different kernel panic | |
| 13:28:43 | giblet | that test can produce | |
| 13:28:47 | giblet | this is the 4th one :) | |
| 13:29:21 | sean-k-mooney | well its specifclay failing in _raw_spin_lock_bh | |
| 13:30:26 | lyarwood | sorry have various trades people in my house today trying to sort something out | |
| 13:30:34 | lyarwood | really hard to focus on anything | |
| 13:31:45 | sean-k-mooney | lyarwood: tl;dr the panic happen during the migration in an interupt handel form _raw_spin_lock_bh at least in this insntace | |
| 13:32:25 | lyarwood | giblet: remind me again, when did thi start failing? | |
| 13:32:25 | sean-k-mooney | so its before the detach in the cleanup phase if the time stames are correct | |
| 13:32:28 | lyarwood | this* | |
| 13:32:32 | giblet | tehre is lot of different RIP codes in the reproductions https://paste.ubuntu.com/p/QGrtv2nZWz/ | |
| 13:32:45 | giblet | lyarwood: on it ... | |
| 13:33:49 | giblet | lyarwood: the first panic was 2021-11-08T12:59:16 | |
| 13:33:52 | sean-k-mooney | giblet: yes altoh galoto of them look like they are related to lcoking/interupt handeling in general | |
| 13:34:15 | giblet | this one https://zuul.opendev.org/t/openstack/build/c445dc25cb2c4567b223f05d95134c47 | |
| 13:34:17 | lyarwood | giblet: kk it's not related to https://review.opendev.org/c/openstack/devstack/+/812928 then | |
| 13:36:41 | giblet | lyarwood: there is big gaps between nova-live-migration job run before 11.08. | |
| 13:36:44 | giblet | https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&project=openstack%2Fnova&branch=stable%2Fvictoria | |
| 13:36:58 | giblet | only 2 runs in october | |
| 13:38:14 | giblet | the newer october run still has the logs the older is already lost | |
| 13:39:26 | sean-k-mooney | looking at https://zuul.opendev.org/t/openstack/build/c445dc25cb2c4567b223f05d95134c47 that panic happend even eairlier in the test after 15 seconds | |
| 13:39:45 | sean-k-mooney | i have not check if that is during th migration or not but i suspect its also before the detach | |
| 13:40:54 | sean-k-mooney | 2021-11-08 12:41:10,502 99684 INFO [tempest.common.waiters] State transition "BUILD/spawning" ==> "ACTIVE/None" after 6 second wait | |
| 13:41:46 | sean-k-mooney | volume was attached at | |
| 13:41:48 | sean-k-mooney | 2021-11-08 12:41:18,780 99684 INFO [tempest.common.waiters] volume fb8d4e1c-5dae-41b4-a1d9-6ef20937997b reached in-use after waiting for 0.780033 seconds | |
| 13:42:09 | sean-k-mooney | and the migration finished at 2021-11-08 12:41:32,756 99684 INFO [tempest.common.waiters] State transition "MIGRATING/migrating" ==> "ACTIVE/None" after 12 second wait | |
| 13:42:31 | sean-k-mooney | so the panic was also during the live migration | |
| 13:43:05 | sean-k-mooney | so the detach is failing because the vm was dead after the migration | |
| 13:43:08 | giblet | I checked the oct 29 green run has the same qemu / libvirt version as the later failed runs | |
| 13:43:32 | giblet | qemu-system-x86 amd64 1:4.2-3ubuntu6.18 and libvirt0 amd64 6.0.0-0ubuntu8.14 | |
| 13:47:17 | sean-k-mooney | im just checking dowstram if we have an closed bugs with live migration or painc in ther enames | |
| 13:47:33 | sean-k-mooney | 11 | |
| 13:52:17 | sean-k-mooney | there are 3 nested virt issue that lok somewhat related but thos where this is happenign are not using that correct | |
| 13:55:04 | giblet | we have virt_type=qemu in the config | |
| 13:55:24 | giblet | so they are not nested guests | |
| 13:59:22 | sean-k-mooney | yes but im wonderign if nested virt is avaiabel and if the host are intel do they have kvm_intel.pml=1 | |
| 13:59:56 | sean-k-mooney | its not nested kvm but its still a nested qemu instnace | |
| 14:00:34 | sean-k-mooney | actully the pml issue seams to be more related to live migrating the l1 vm not the l2 vm | |
| 14:09:05 | sean-k-mooney | so ya i dont see anything downstram that imidiatly screams this is a fixed qemu/kernel bug | |
| 14:13:46 | giblet | I did a summary in the bug and linket do this chat | |
| 14:44:47 | belmoreira | hi, I remember that in the past there were some discussions to have live migrations between cells. | |
| 14:45:03 | belmoreira | is there any progress on this? | |
| 14:45:55 | giblet | belmoreira: I think the cross cell migration work was stopped when mriedem left | |
| 14:45:57 | sean-k-mooney | we have not implemeted it | |
| 14:46:06 | sean-k-mooney | we have cross cell cold migratiohn/resize | |
| 14:46:23 | sean-k-mooney | belmoreira: there is no assumtion that hyperviors can talk to each other directly between cells | |
| 14:46:38 | sean-k-mooney | belmoreira: they might be abel too but we decied we could not assuem that | |