| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-12 | |||
| 11:27:28 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Don't use legacy 'Row()' methods https://review.opendev.org/c/openstack/nova/+/817746 | |
| 11:27:29 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove unnecessary warning filters https://review.opendev.org/c/openstack/nova/+/817748 | |
| 11:27:29 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove use of 'bind' arguments https://review.opendev.org/c/openstack/nova/+/817747 | |
| 13:17:46 | giblet | lyarwood, sean-k-mooney: so the patch with the sleep before detach now reproduced the kernel panic https://review.opendev.org/c/openstack/nova/+/817564/3#message-8ef45ae21f54b427ae2c88bbc19a4c926ce00883 | |
| 13:18:15 | giblet | from the consol log it is clear that the kernel panic happened _before_ we trigger detach | |
| 13:18:28 | sean-k-mooney | is it possible the panic was tirggered by the attach | |
| 13:18:35 | giblet | or by the live migration | |
| 13:18:45 | giblet | or by the initial boot | |
| 13:18:50 | giblet | all possible | |
| 13:19:02 | giblet | so I will add more console printing step to the test case | |
| 13:19:10 | giblet | to see when the panic happens | |
| 13:20:19 | sean-k-mooney | looks liek it happend 19 seocnd after boot | |
| 13:20:53 | giblet | true, we can try to correlate that to the action in the test | |
| 13:21:22 | sean-k-mooney | the vm went active ~ 12:04:29 | |
| 13:21:59 | sean-k-mooney | we did the attach at 12:04:40 | |
| 13:22:34 | sean-k-mooney | actully no maybe i misread that | |
| 13:23:07 | giblet | 2021-11-12 12:04:30,036 99321 INFO [tempest.common.waiters] State transition "BUILD/spawning" ==> "ACTIVE/None" after 7 second wait | |
| 13:23:59 | giblet | 2021-11-12 12:04:38,573 99321 INFO [tempest.common.waiters] volume 0f307fda-dbef-408c-a029-97401db15945 reached in-use after waiting for 0.573394 seconds | |
| 13:24:16 | sean-k-mooney | looks liek we were just finsihing the mightation when the panic happend | |
| 13:24:47 | sean-k-mooney | it went active at 2021 12:04:52 | |
| 13:25:36 | giblet | 2021-11-12 12:04:40,615 <- trigger the live migration | |
| 13:25:48 | giblet | 2021-11-12 12:04:52,693 99321 INFO [tempest.common.waiters] State transition "MIGRATING/migrating" ==> "ACTIVE/None" after 12 second wait | |
| 13:26:04 | giblet | yeah so the panic is somehow related to the live migration | |
| 13:26:06 | sean-k-mooney | yep | |
| 13:26:23 | sean-k-mooney | it woudl have been i gues at :49 ish | |
| 13:27:54 | sean-k-mooney | giblet: just looking at the panic | |
| 13:28:07 | sean-k-mooney | its realted to ipv6 adress configuraiton | |
| 13:28:36 | giblet | I saw at least 3 different kernel panic | |
| 13:28:43 | giblet | that test can produce | |
| 13:28:47 | giblet | this is the 4th one :) | |
| 13:29:21 | sean-k-mooney | well its specifclay failing in _raw_spin_lock_bh | |
| 13:30:26 | lyarwood | sorry have various trades people in my house today trying to sort something out | |
| 13:30:34 | lyarwood | really hard to focus on anything | |
| 13:31:45 | sean-k-mooney | lyarwood: tl;dr the panic happen during the migration in an interupt handel form _raw_spin_lock_bh at least in this insntace | |
| 13:32:25 | sean-k-mooney | so its before the detach in the cleanup phase if the time stames are correct | |
| 13:32:25 | lyarwood | giblet: remind me again, when did thi start failing? | |
| 13:32:28 | lyarwood | this* | |
| 13:32:32 | giblet | tehre is lot of different RIP codes in the reproductions https://paste.ubuntu.com/p/QGrtv2nZWz/ | |
| 13:32:45 | giblet | lyarwood: on it ... | |
| 13:33:49 | giblet | lyarwood: the first panic was 2021-11-08T12:59:16 | |
| 13:33:52 | sean-k-mooney | giblet: yes altoh galoto of them look like they are related to lcoking/interupt handeling in general | |
| 13:34:15 | giblet | this one https://zuul.opendev.org/t/openstack/build/c445dc25cb2c4567b223f05d95134c47 | |
| 13:34:17 | lyarwood | giblet: kk it's not related to https://review.opendev.org/c/openstack/devstack/+/812928 then | |
| 13:36:41 | giblet | lyarwood: there is big gaps between nova-live-migration job run before 11.08. | |
| 13:36:44 | giblet | https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&project=openstack%2Fnova&branch=stable%2Fvictoria | |
| 13:36:58 | giblet | only 2 runs in october | |
| 13:38:14 | giblet | the newer october run still has the logs the older is already lost | |
| 13:39:26 | sean-k-mooney | looking at https://zuul.opendev.org/t/openstack/build/c445dc25cb2c4567b223f05d95134c47 that panic happend even eairlier in the test after 15 seconds | |
| 13:39:45 | sean-k-mooney | i have not check if that is during th migration or not but i suspect its also before the detach | |
| 13:40:54 | sean-k-mooney | 2021-11-08 12:41:10,502 99684 INFO [tempest.common.waiters] State transition "BUILD/spawning" ==> "ACTIVE/None" after 6 second wait | |
| 13:41:46 | sean-k-mooney | volume was attached at | |
| 13:41:48 | sean-k-mooney | 2021-11-08 12:41:18,780 99684 INFO [tempest.common.waiters] volume fb8d4e1c-5dae-41b4-a1d9-6ef20937997b reached in-use after waiting for 0.780033 seconds | |
| 13:42:09 | sean-k-mooney | and the migration finished at 2021-11-08 12:41:32,756 99684 INFO [tempest.common.waiters] State transition "MIGRATING/migrating" ==> "ACTIVE/None" after 12 second wait | |
| 13:42:31 | sean-k-mooney | so the panic was also during the live migration | |
| 13:43:05 | sean-k-mooney | so the detach is failing because the vm was dead after the migration | |
| 13:43:08 | giblet | I checked the oct 29 green run has the same qemu / libvirt version as the later failed runs | |
| 13:43:32 | giblet | qemu-system-x86 amd64 1:4.2-3ubuntu6.18 and libvirt0 amd64 6.0.0-0ubuntu8.14 | |
| 13:47:17 | sean-k-mooney | im just checking dowstram if we have an closed bugs with live migration or painc in ther enames | |
| 13:47:33 | sean-k-mooney | 11 | |
| 13:52:17 | sean-k-mooney | there are 3 nested virt issue that lok somewhat related but thos where this is happenign are not using that correct | |
| 13:55:04 | giblet | we have virt_type=qemu in the config | |
| 13:55:24 | giblet | so they are not nested guests | |
| 13:59:22 | sean-k-mooney | yes but im wonderign if nested virt is avaiabel and if the host are intel do they have kvm_intel.pml=1 | |
| 13:59:56 | sean-k-mooney | its not nested kvm but its still a nested qemu instnace | |
| 14:00:34 | sean-k-mooney | actully the pml issue seams to be more related to live migrating the l1 vm not the l2 vm | |
| 14:09:05 | sean-k-mooney | so ya i dont see anything downstram that imidiatly screams this is a fixed qemu/kernel bug | |
| 14:13:46 | giblet | I did a summary in the bug and linket do this chat | |
| 14:44:47 | belmoreira | hi, I remember that in the past there were some discussions to have live migrations between cells. | |
| 14:45:03 | belmoreira | is there any progress on this? | |
| 14:45:55 | giblet | belmoreira: I think the cross cell migration work was stopped when mriedem left | |
| 14:45:57 | sean-k-mooney | we have not implemeted it | |
| 14:46:06 | sean-k-mooney | we have cross cell cold migratiohn/resize | |
| 14:46:23 | sean-k-mooney | belmoreira: there is no assumtion that hyperviors can talk to each other directly between cells | |
| 14:46:38 | belmoreira | yeah... is what I recall. And I think he was only working in cold migration as a starting point | |
| 14:46:38 | sean-k-mooney | belmoreira: they might be abel too but we decied we could not assuem that | |
| 14:47:06 | sean-k-mooney | giblet: the cross cell migration work is completed | |
| 14:47:14 | sean-k-mooney | giblet: that works but only cold migration | |
| 14:47:22 | sean-k-mooney | via effectivly a shelve | |
| 14:47:29 | giblet | yepp | |
| 14:47:30 | sean-k-mooney | and then unshelve to the new cell | |
| 14:48:35 | belmoreira | humm... I missed that in the release notes. | |
| 14:48:52 | dansmith | belmoreira: it's been like a couple years at this point.. :P | |
| 14:49:16 | dansmith | I dunno how reliable it is because I think it's very infrequently used, if that :/ | |
| 14:49:55 | sean-k-mooney | i think its in train | |
| 14:50:11 | sean-k-mooney | but ya i don tknow if any of our downstream custoemr use it | |
| 14:50:24 | sean-k-mooney | most of them are not multi cell | |
| 14:50:26 | belmoreira | honestly, I was searching for it just few minutes ago... in all releases after Rocky because I recall that mriedem was working on it | |
| 14:51:08 | belmoreira | but anyway... what I'm looking at is a way to live migrate not cold migrate | |
| 14:51:11 | sean-k-mooney | oh its ussuri https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/cross-cell-resize.html | |
| 14:51:27 | sean-k-mooney | belmoreira: right we basically agree to not implemnte live migration | |
| 14:51:34 | sean-k-mooney | effectivly ever | |
| 14:51:40 | dansmith | sean-k-mooney: did we? | |
| 14:51:44 | belmoreira | :) | |
| 14:51:59 | sean-k-mooney | we could but it would reuiqre assumeing conenctivty betwenn the hyperviors in diffenet cells | |
| 14:52:01 | dansmith | IIRC it was going to be more complicated, very little audience, and so had no plans | |
| 14:52:19 | sean-k-mooney | dansmith: well i thinke the connectivty was the main blocker | |
| 14:52:21 | dansmith | sure, but live migration already assumes connectivity between hosts that not everyone allows, even in one cel | |
| 14:52:36 | sean-k-mooney | i gues that is true | |
| 14:52:52 | belmoreira | my use case is a one off... I'm trying to get rid of nova-network | |
| 14:53:06 | dansmith | my recollection was "if there's large uptake of cells and this becomes a thing lots of people want, we can consider, but no plans until then" | |