| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-12 | |||
| 13:26:23 | sean-k-mooney | it woudl have been i gues at :49 ish | |
| 13:27:54 | sean-k-mooney | giblet: just looking at the panic | |
| 13:28:07 | sean-k-mooney | its realted to ipv6 adress configuraiton | |
| 13:28:36 | giblet | I saw at least 3 different kernel panic | |
| 13:28:43 | giblet | that test can produce | |
| 13:28:47 | giblet | this is the 4th one :) | |
| 13:29:21 | sean-k-mooney | well its specifclay failing in _raw_spin_lock_bh | |
| 13:30:26 | lyarwood | sorry have various trades people in my house today trying to sort something out | |
| 13:30:34 | lyarwood | really hard to focus on anything | |
| 13:31:45 | sean-k-mooney | lyarwood: tl;dr the panic happen during the migration in an interupt handel form _raw_spin_lock_bh at least in this insntace | |
| 13:32:25 | lyarwood | giblet: remind me again, when did thi start failing? | |
| 13:32:25 | sean-k-mooney | so its before the detach in the cleanup phase if the time stames are correct | |
| 13:32:28 | lyarwood | this* | |
| 13:32:32 | giblet | tehre is lot of different RIP codes in the reproductions https://paste.ubuntu.com/p/QGrtv2nZWz/ | |
| 13:32:45 | giblet | lyarwood: on it ... | |
| 13:33:49 | giblet | lyarwood: the first panic was 2021-11-08T12:59:16 | |
| 13:33:52 | sean-k-mooney | giblet: yes altoh galoto of them look like they are related to lcoking/interupt handeling in general | |
| 13:34:15 | giblet | this one https://zuul.opendev.org/t/openstack/build/c445dc25cb2c4567b223f05d95134c47 | |
| 13:34:17 | lyarwood | giblet: kk it's not related to https://review.opendev.org/c/openstack/devstack/+/812928 then | |
| 13:36:41 | giblet | lyarwood: there is big gaps between nova-live-migration job run before 11.08. | |
| 13:36:44 | giblet | https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&project=openstack%2Fnova&branch=stable%2Fvictoria | |
| 13:36:58 | giblet | only 2 runs in october | |
| 13:38:14 | giblet | the newer october run still has the logs the older is already lost | |
| 13:39:26 | sean-k-mooney | looking at https://zuul.opendev.org/t/openstack/build/c445dc25cb2c4567b223f05d95134c47 that panic happend even eairlier in the test after 15 seconds | |
| 13:39:45 | sean-k-mooney | i have not check if that is during th migration or not but i suspect its also before the detach | |
| 13:40:54 | sean-k-mooney | 2021-11-08 12:41:10,502 99684 INFO [tempest.common.waiters] State transition "BUILD/spawning" ==> "ACTIVE/None" after 6 second wait | |
| 13:41:46 | sean-k-mooney | volume was attached at | |
| 13:41:48 | sean-k-mooney | 2021-11-08 12:41:18,780 99684 INFO [tempest.common.waiters] volume fb8d4e1c-5dae-41b4-a1d9-6ef20937997b reached in-use after waiting for 0.780033 seconds | |
| 13:42:09 | sean-k-mooney | and the migration finished at 2021-11-08 12:41:32,756 99684 INFO [tempest.common.waiters] State transition "MIGRATING/migrating" ==> "ACTIVE/None" after 12 second wait | |
| 13:42:31 | sean-k-mooney | so the panic was also during the live migration | |
| 13:43:05 | sean-k-mooney | so the detach is failing because the vm was dead after the migration | |
| 13:43:08 | giblet | I checked the oct 29 green run has the same qemu / libvirt version as the later failed runs | |
| 13:43:32 | giblet | qemu-system-x86 amd64 1:4.2-3ubuntu6.18 and libvirt0 amd64 6.0.0-0ubuntu8.14 | |
| 13:47:17 | sean-k-mooney | im just checking dowstram if we have an closed bugs with live migration or painc in ther enames | |
| 13:47:33 | sean-k-mooney | 11 | |
| 13:52:17 | sean-k-mooney | there are 3 nested virt issue that lok somewhat related but thos where this is happenign are not using that correct | |
| 13:55:04 | giblet | we have virt_type=qemu in the config | |
| 13:55:24 | giblet | so they are not nested guests | |
| 13:59:22 | sean-k-mooney | yes but im wonderign if nested virt is avaiabel and if the host are intel do they have kvm_intel.pml=1 | |
| 13:59:56 | sean-k-mooney | its not nested kvm but its still a nested qemu instnace | |
| 14:00:34 | sean-k-mooney | actully the pml issue seams to be more related to live migrating the l1 vm not the l2 vm | |
| 14:09:05 | sean-k-mooney | so ya i dont see anything downstram that imidiatly screams this is a fixed qemu/kernel bug | |
| 14:13:46 | giblet | I did a summary in the bug and linket do this chat | |
| 14:44:47 | belmoreira | hi, I remember that in the past there were some discussions to have live migrations between cells. | |
| 14:45:03 | belmoreira | is there any progress on this? | |
| 14:45:55 | giblet | belmoreira: I think the cross cell migration work was stopped when mriedem left | |
| 14:45:57 | sean-k-mooney | we have not implemeted it | |
| 14:46:06 | sean-k-mooney | we have cross cell cold migratiohn/resize | |
| 14:46:23 | sean-k-mooney | belmoreira: there is no assumtion that hyperviors can talk to each other directly between cells | |
| 14:46:38 | sean-k-mooney | belmoreira: they might be abel too but we decied we could not assuem that | |
| 14:46:38 | belmoreira | yeah... is what I recall. And I think he was only working in cold migration as a starting point | |
| 14:47:06 | sean-k-mooney | giblet: the cross cell migration work is completed | |
| 14:47:14 | sean-k-mooney | giblet: that works but only cold migration | |
| 14:47:22 | sean-k-mooney | via effectivly a shelve | |
| 14:47:29 | giblet | yepp | |
| 14:47:30 | sean-k-mooney | and then unshelve to the new cell | |
| 14:48:35 | belmoreira | humm... I missed that in the release notes. | |
| 14:48:52 | dansmith | belmoreira: it's been like a couple years at this point.. :P | |
| 14:49:16 | dansmith | I dunno how reliable it is because I think it's very infrequently used, if that :/ | |
| 14:49:55 | sean-k-mooney | i think its in train | |
| 14:50:11 | sean-k-mooney | but ya i don tknow if any of our downstream custoemr use it | |
| 14:50:24 | sean-k-mooney | most of them are not multi cell | |
| 14:50:26 | belmoreira | honestly, I was searching for it just few minutes ago... in all releases after Rocky because I recall that mriedem was working on it | |
| 14:51:08 | belmoreira | but anyway... what I'm looking at is a way to live migrate not cold migrate | |
| 14:51:11 | sean-k-mooney | oh its ussuri https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/cross-cell-resize.html | |
| 14:51:27 | sean-k-mooney | belmoreira: right we basically agree to not implemnte live migration | |
| 14:51:34 | sean-k-mooney | effectivly ever | |
| 14:51:40 | dansmith | sean-k-mooney: did we? | |
| 14:51:44 | belmoreira | :) | |
| 14:51:59 | sean-k-mooney | we could but it would reuiqre assumeing conenctivty betwenn the hyperviors in diffenet cells | |
| 14:52:01 | dansmith | IIRC it was going to be more complicated, very little audience, and so had no plans | |
| 14:52:19 | sean-k-mooney | dansmith: well i thinke the connectivty was the main blocker | |
| 14:52:21 | dansmith | sure, but live migration already assumes connectivity between hosts that not everyone allows, even in one cel | |
| 14:52:36 | sean-k-mooney | i gues that is true | |
| 14:52:52 | belmoreira | my use case is a one off... I'm trying to get rid of nova-network | |
| 14:53:06 | dansmith | my recollection was "if there's large uptake of cells and this becomes a thing lots of people want, we can consider, but no plans until then" | |
| 14:53:08 | sean-k-mooney | belmoreira: if you have a need for it and or people to work on it i would be happy to review a spec | |
| 14:53:16 | dansmith | and so far, no such demand that I know of | |
| 14:53:29 | sean-k-mooney | dansmith: yep i think that is a fiar summary | |
| 14:53:43 | belmoreira | the hypothetical plan would be to live migrate the instances from a nova-network cell to a neutron cell | |
| 14:54:06 | dansmith | belmoreira: well, unlikely that would work anyway I think, without very special planning on our part, | |
| 14:54:15 | dansmith | which we obviously can't and won't do without the code being in master :) | |
| 14:55:08 | belmoreira | yeah... honestly I'm not asking for it... but it would be great if we can discuss the blockers... in case I need to deal with them | |
| 14:55:57 | belmoreira | the main problem in my view would be to make both source and dest talk (different conductors) and populate the DB | |
| 14:56:19 | dansmith | belmoreira: it would need to work the same way the cross-cell cold migration works, | |
| 14:56:28 | dansmith | which is the superconductor does that bridging from the nova perspective | |
| 14:56:52 | dansmith | i.e. it's the thing that creates the instance in the target cell and deletes it from the old cell when the time comes | |
| 14:57:27 | sean-k-mooney | dansmith: we might also need to proxy the conntion between the hypervirous via the the super condctor host or some other host if we coudl not asusme direct conenctivity | |
| 14:57:46 | belmoreira | in my case I can have direct connectivity | |
| 14:57:48 | sean-k-mooney | there are also complcaiton when volumes or ceph gets invovled | |
| 14:58:05 | sean-k-mooney | belmoreira: is your cinder/ceph storage shared accross cells | |
| 14:58:19 | belmoreira | yes, it is | |
| 14:58:46 | dansmith | sean-k-mooney: well, or we have to redesign the live migration flow to go through (super)conductor in a more push-pull sort of way | |
| 15:00:05 | sean-k-mooney | well i ment makign it so qemu on one host could talke to qemu on another via proxign the coneection. | |
| 15:01:23 | dansmith | oh, I dunno, I think you'd just make it a requirement that they can talk direct, | |
| 15:01:29 | dansmith | even if for a short time | |
| 15:01:55 | dansmith | implementing a well-performing proxy at the superconductor (which is likely just a container on a host) seems like more work than necessary | |
| 15:02:24 | sean-k-mooney | ya it does | |
| 15:02:35 | belmoreira | those are not limitations for me... | |
| 15:02:49 | sean-k-mooney | belmoreira: yep the were just limiation in general | |