| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-12 | |||
| 13:36:58 | giblet | only 2 runs in october | |
| 13:38:14 | giblet | the newer october run still has the logs the older is already lost | |
| 13:39:26 | sean-k-mooney | looking at https://zuul.opendev.org/t/openstack/build/c445dc25cb2c4567b223f05d95134c47 that panic happend even eairlier in the test after 15 seconds | |
| 13:39:45 | sean-k-mooney | i have not check if that is during th migration or not but i suspect its also before the detach | |
| 13:40:54 | sean-k-mooney | 2021-11-08 12:41:10,502 99684 INFO [tempest.common.waiters] State transition "BUILD/spawning" ==> "ACTIVE/None" after 6 second wait | |
| 13:41:46 | sean-k-mooney | volume was attached at | |
| 13:41:48 | sean-k-mooney | 2021-11-08 12:41:18,780 99684 INFO [tempest.common.waiters] volume fb8d4e1c-5dae-41b4-a1d9-6ef20937997b reached in-use after waiting for 0.780033 seconds | |
| 13:42:09 | sean-k-mooney | and the migration finished at 2021-11-08 12:41:32,756 99684 INFO [tempest.common.waiters] State transition "MIGRATING/migrating" ==> "ACTIVE/None" after 12 second wait | |
| 13:42:31 | sean-k-mooney | so the panic was also during the live migration | |
| 13:43:05 | sean-k-mooney | so the detach is failing because the vm was dead after the migration | |
| 13:43:08 | giblet | I checked the oct 29 green run has the same qemu / libvirt version as the later failed runs | |
| 13:43:32 | giblet | qemu-system-x86 amd64 1:4.2-3ubuntu6.18 and libvirt0 amd64 6.0.0-0ubuntu8.14 | |
| 13:47:17 | sean-k-mooney | im just checking dowstram if we have an closed bugs with live migration or painc in ther enames | |
| 13:47:33 | sean-k-mooney | 11 | |
| 13:52:17 | sean-k-mooney | there are 3 nested virt issue that lok somewhat related but thos where this is happenign are not using that correct | |
| 13:55:04 | giblet | we have virt_type=qemu in the config | |
| 13:55:24 | giblet | so they are not nested guests | |
| 13:59:22 | sean-k-mooney | yes but im wonderign if nested virt is avaiabel and if the host are intel do they have kvm_intel.pml=1 | |
| 13:59:56 | sean-k-mooney | its not nested kvm but its still a nested qemu instnace | |
| 14:00:34 | sean-k-mooney | actully the pml issue seams to be more related to live migrating the l1 vm not the l2 vm | |
| 14:09:05 | sean-k-mooney | so ya i dont see anything downstram that imidiatly screams this is a fixed qemu/kernel bug | |
| 14:13:46 | giblet | I did a summary in the bug and linket do this chat | |
| 14:44:47 | belmoreira | hi, I remember that in the past there were some discussions to have live migrations between cells. | |
| 14:45:03 | belmoreira | is there any progress on this? | |
| 14:45:55 | giblet | belmoreira: I think the cross cell migration work was stopped when mriedem left | |
| 14:45:57 | sean-k-mooney | we have not implemeted it | |
| 14:46:06 | sean-k-mooney | we have cross cell cold migratiohn/resize | |
| 14:46:23 | sean-k-mooney | belmoreira: there is no assumtion that hyperviors can talk to each other directly between cells | |
| 14:46:38 | belmoreira | yeah... is what I recall. And I think he was only working in cold migration as a starting point | |
| 14:46:38 | sean-k-mooney | belmoreira: they might be abel too but we decied we could not assuem that | |
| 14:47:06 | sean-k-mooney | giblet: the cross cell migration work is completed | |
| 14:47:14 | sean-k-mooney | giblet: that works but only cold migration | |
| 14:47:22 | sean-k-mooney | via effectivly a shelve | |
| 14:47:29 | giblet | yepp | |
| 14:47:30 | sean-k-mooney | and then unshelve to the new cell | |
| 14:48:35 | belmoreira | humm... I missed that in the release notes. | |
| 14:48:52 | dansmith | belmoreira: it's been like a couple years at this point.. :P | |
| 14:49:16 | dansmith | I dunno how reliable it is because I think it's very infrequently used, if that :/ | |
| 14:49:55 | sean-k-mooney | i think its in train | |
| 14:50:11 | sean-k-mooney | but ya i don tknow if any of our downstream custoemr use it | |
| 14:50:24 | sean-k-mooney | most of them are not multi cell | |
| 14:50:26 | belmoreira | honestly, I was searching for it just few minutes ago... in all releases after Rocky because I recall that mriedem was working on it | |
| 14:51:08 | belmoreira | but anyway... what I'm looking at is a way to live migrate not cold migrate | |
| 14:51:11 | sean-k-mooney | oh its ussuri https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/cross-cell-resize.html | |
| 14:51:27 | sean-k-mooney | belmoreira: right we basically agree to not implemnte live migration | |
| 14:51:34 | sean-k-mooney | effectivly ever | |
| 14:51:40 | dansmith | sean-k-mooney: did we? | |
| 14:51:44 | belmoreira | :) | |
| 14:51:59 | sean-k-mooney | we could but it would reuiqre assumeing conenctivty betwenn the hyperviors in diffenet cells | |
| 14:52:01 | dansmith | IIRC it was going to be more complicated, very little audience, and so had no plans | |
| 14:52:19 | sean-k-mooney | dansmith: well i thinke the connectivty was the main blocker | |
| 14:52:21 | dansmith | sure, but live migration already assumes connectivity between hosts that not everyone allows, even in one cel | |
| 14:52:36 | sean-k-mooney | i gues that is true | |
| 14:52:52 | belmoreira | my use case is a one off... I'm trying to get rid of nova-network | |
| 14:53:06 | dansmith | my recollection was "if there's large uptake of cells and this becomes a thing lots of people want, we can consider, but no plans until then" | |
| 14:53:08 | sean-k-mooney | belmoreira: if you have a need for it and or people to work on it i would be happy to review a spec | |
| 14:53:16 | dansmith | and so far, no such demand that I know of | |
| 14:53:29 | sean-k-mooney | dansmith: yep i think that is a fiar summary | |
| 14:53:43 | belmoreira | the hypothetical plan would be to live migrate the instances from a nova-network cell to a neutron cell | |
| 14:54:06 | dansmith | belmoreira: well, unlikely that would work anyway I think, without very special planning on our part, | |
| 14:54:15 | dansmith | which we obviously can't and won't do without the code being in master :) | |
| 14:55:08 | belmoreira | yeah... honestly I'm not asking for it... but it would be great if we can discuss the blockers... in case I need to deal with them | |
| 14:55:57 | belmoreira | the main problem in my view would be to make both source and dest talk (different conductors) and populate the DB | |
| 14:56:19 | dansmith | belmoreira: it would need to work the same way the cross-cell cold migration works, | |
| 14:56:28 | dansmith | which is the superconductor does that bridging from the nova perspective | |
| 14:56:52 | dansmith | i.e. it's the thing that creates the instance in the target cell and deletes it from the old cell when the time comes | |
| 14:57:27 | sean-k-mooney | dansmith: we might also need to proxy the conntion between the hypervirous via the the super condctor host or some other host if we coudl not asusme direct conenctivity | |
| 14:57:46 | belmoreira | in my case I can have direct connectivity | |
| 14:57:48 | sean-k-mooney | there are also complcaiton when volumes or ceph gets invovled | |
| 14:58:05 | sean-k-mooney | belmoreira: is your cinder/ceph storage shared accross cells | |
| 14:58:19 | belmoreira | yes, it is | |
| 14:58:46 | dansmith | sean-k-mooney: well, or we have to redesign the live migration flow to go through (super)conductor in a more push-pull sort of way | |
| 15:00:05 | sean-k-mooney | well i ment makign it so qemu on one host could talke to qemu on another via proxign the coneection. | |
| 15:01:23 | dansmith | oh, I dunno, I think you'd just make it a requirement that they can talk direct, | |
| 15:01:29 | dansmith | even if for a short time | |
| 15:01:55 | dansmith | implementing a well-performing proxy at the superconductor (which is likely just a container on a host) seems like more work than necessary | |
| 15:02:24 | sean-k-mooney | ya it does | |
| 15:02:35 | belmoreira | those are not limitations for me... | |
| 15:02:49 | sean-k-mooney | belmoreira: yep the were just limiation in general | |
| 15:02:59 | sean-k-mooney | e.g. if you map cells to edge sites | |
| 15:03:12 | sean-k-mooney | cross cell migration get a lot harder if its over the wan | |
| 15:04:38 | belmoreira | it looks like it's a lot of work... because its a live migration the instance entries need to be created in the dest cell DB, domain XML needs to be created, take care of the network logic, and possible ceph volumes attached... qemu starts the migration and it needs to be removed from the source DB, placement updated | |
| 15:04:55 | sean-k-mooney | i dont link config driven api behavior but i kind of feel like we woudl need a way to enabel/disabel this perhapes via the sschuler config in soem way | |
| 15:05:33 | sean-k-mooney | belmoreira: we would have to also update the instnace cell mapping in the api db | |
| 15:05:45 | belmoreira | true | |
| 15:05:50 | sean-k-mooney | which is parly why this need to be driven by the super conductor | |
| 15:05:57 | dansmith | sean-k-mooney: we disabled it via policy now | |
| 15:06:09 | sean-k-mooney | dansmith: cross cell live migration | |
| 15:06:29 | sean-k-mooney | how does the policy rule know its corsscell | |
| 15:07:06 | sean-k-mooney | are we actully doign a lookup of the host if we force the destionaion? | |
| 15:07:39 | belmoreira | Cross cell resize was added in Ussuri (i was searching for migration, that's why I missed it) | |
| 15:08:18 | dansmith | sean-k-mooney: we have a flag to the api that says "can cross cell boundaries" I think, and only allow that if policy passes | |
| 15:09:00 | sean-k-mooney | belmoreira: ah ya same code path internelly | |
| 15:12:13 | belmoreira | great... looks like I will be the oldest nova-network user around :) | |
| 15:12:38 | belmoreira | dansmith sean-k-mooney thanks a lot for the comments and insights | |
| 15:15:01 | dansmith | sean-k-mooney: well, I thought we expressed intent to the api that we wanted to cross, but apparently it's only the config flag, but yeah we check the policy deeper in compute/api to decide if the user is allowed | |
| 15:18:19 | sean-k-mooney | ya that would work. i dont think we modify the request body to request cross cell either, so ya it woudl have to be checked later in the compute api | |
| 15:35:18 | giblet | sean-k-mooney: Am I correct that nova uses the physnet of the neutron network in two cases 1) matching it with the physical_network property of the PciDevice (coming from the whitelist) during scheduling if there is InstancePciRequest 2) if numa aware vswitches are configured then matching against novas configuration | |
| 15:35:23 | giblet | https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/numa-aware-vswitches.html#id33 ? | |
| 15:36:26 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Test token expiration during live migration https://review.opendev.org/c/openstack/nova/+/817778 | |