Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-15
15:22:07 dansmith the two ends of a migration are storing the machine-local context in their own libvirts
15:22:22 dansmith maybe for the revert case?
15:22:23 efried alex_xu: why would we need it in migration context?
15:22:24 alex_xu I need copy the source pmem data to the dest pmem. I need dest pmem device path
15:22:46 dansmith wait
15:22:58 dansmith so for cold migration you're going to migrate the data as well?
15:23:05 alex_xu yes
15:23:11 efried there's a conf option
15:23:25 alex_xu yes, it is configurable by the extra spec
15:23:31 dansmith how is that going to work? write it to disk, consider it a disk image to move and then write it back?
15:23:41 sean-k-mooney dansmith: not by defualt
15:23:44 alex_xu https://review.opendev.org/#/c/634556/12/nova/privsep/libvirt.py
15:23:54 dansmith I *love* this snowflake feature you guys have come up with
15:23:56 sean-k-mooney we are default to not copy the data but we optionally can copy it
15:23:58 alex_xu ^ here is, ssh...
15:24:17 dansmith alex_xu: so you're going to add another scp operation?
15:24:27 alex_xu yes...
15:24:34 alex_xu it is something we hate...
15:24:34 dansmith gross
15:24:59 efried What's the alternative? Block migrations etc. Which is worse?
15:25:04 dansmith so if/when we get to the point of being able to use libvirt to move the images over the TLS tunnel, we're stuck scp'ing this thing?
15:25:23 dansmith efried: not promising that this data is persistent across moves is one option
15:25:37 dansmith efried: isn't this being added as a snowflake "you can never live migrate" feature?
15:25:54 openstackgerrit Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.opendev.org/637955
15:25:57 efried I thought live migrate was supported
15:25:58 sean-k-mooney dansmith: in the livemigration case qemu/libvirt can copy the data
15:26:01 alex_xu it support lm
15:26:23 sean-k-mooney for cold migrate we have to do it but maybe we can make a libvirt feature request?
15:26:35 dansmith okay, I thought it wasn't going to be able to move that data
15:26:44 alex_xu i'm ok without copy for resize
15:27:10 efried http://specs.openstack.org/openstack/nova-specs/specs/train/approved/virtual-persistent-memory.html#live-migration
15:27:11 sean-k-mooney alex_xu: the issue with scp/rsync for resize is it wont work for cross cell resize
15:27:22 efried apparently libvirt moves it along with RAM
15:27:33 sean-k-mooney alex_xu: e.g. for cross cell resize we cant assume teh compute nodes have network connectivtiy
15:27:41 alex_xu sean-k-mooney: yes, we are going to stop cross cell resize in the initial proposal
15:27:45 sean-k-mooney at least not in the edge deployment case
15:28:05 dansmith live migration is transparent to the user, so that's good, and resize is not, so I think it's reasonable to say that there's no data copy when you resize
15:28:14 dansmith you could resize to a flavor with/without pmem too,
15:28:24 dansmith so not copying would cover that case in both directions
15:28:38 efried per the spec, "reduction" in vpmem would not be allowed
15:28:50 dansmith because of this
15:29:08 dansmith I have to jump on a call now,
15:29:22 dansmith but I'm really unhappy with us adding another "just scp it across" thing
15:29:35 efried alex_xu: Sounds like the spec update https://review.opendev.org/669970 needs to be rethought :)
15:29:52 alex_xu yea, will update again
15:30:05 alex_xu I think the copy for resize can be removed
15:30:30 efried & cold migration
15:30:36 efried (which is the same thing?)
15:30:38 dansmith they are the same thing
15:31:22 alex_xu remove the copy, remove the vpmem field
15:31:23 efried doesn't the VM see the vpmem as persistent storage, though?
15:31:35 efried won't it freak out if it boots and the data is gone?
15:31:35 sean-k-mooney efried: not really
15:31:45 efried okay, then yeah, that vastly simplifies things.
15:31:56 sean-k-mooney it sees it as ram/dimms
15:32:06 efried unfortunate that we already merged https://review.opendev.org/#/c/662697/ -- seems as though we won't be using that?
15:32:07 sean-k-mooney the data is ment to be persitent
15:32:11 alex_xu and the most of usecase is for cache, so it is ok
15:32:37 sean-k-mooney but you should really store your data soewhere else and keep the working set in teh pmem
15:33:47 sean-k-mooney ya most workload use it as high capasity scratch space for operating on a subset of the data but the long term storage of the data should be in a cinder volume
15:36:38 efried I've put a hold on https://review.opendev.org/#/c/634548/ for now
15:51:01 gibi efried: thanks for catching bug in https://review.opendev.org/#/c/637955/ I've fixed it.
15:51:21 efried gibi: cool
15:51:33 efried got a nice backlog today, but hopefully I can get back around to it.
15:51:55 efried gibi: since you're around, would you mind pushing https://review.opendev.org/#/c/657464/ ?
15:52:11 gibi efried: on it
15:52:40 openstackgerrit Stephen Finucane proposed openstack/nova master: Update supported transports for iscsi connector https://review.opendev.org/524443
15:52:51 efried thanks
16:00:41 efried sean-k-mooney: is rebuild admin-only?
16:00:46 sean-k-mooney no
16:00:57 sean-k-mooney rebuild and resize can both be done by tenants
16:01:34 efried sean-k-mooney: okay, so back to that thing we were looking at earlier, it looks like setup_networks_on_host is caled in rebuild and resize flows as well as migrate
16:03:17 efried hm, unless teardown=True only happens on migrations...
16:03:36 openstackgerrit Merged openstack/os-resource-classes master: Propose FPGA and PGPU resource classes https://review.opendev.org/657464
16:03:42 efried does _confirm_resize only happen on migration-y resizes?
16:04:53 sean-k-mooney efried: it happesn on resize or cold migration
16:05:22 sean-k-mooney in both cases we go into resize_verify and you chave to do reseize --confim
16:05:25 efried rats, nother meeting, will come back to this...
16:06:01 sean-k-mooney efried: stephenfin is adding an openstack server migrate --confirm as synatactic sugar but its the same call underneath
16:07:14 stephenfin that threw me too, fwiw
16:07:32 stephenfin (whether cold migrations needed to be confirmed or not)
16:07:51 sean-k-mooney stephenfin: well that depends on your config settings
16:07:58 sean-k-mooney but ya
16:08:40 sean-k-mooney i hit that oddity back in hevana so at this point i dont even think about them differently anymore.
16:08:48 sean-k-mooney (cold migraton vs resize)
17:17:36 bbobrov sean-k-mooney: hey
17:17:46 bbobrov sean-k-mooney: regarding your comment to https://review.opendev.org/#/c/638680/24/nova/virt/libvirt/driver.py
17:18:09 bbobrov sean-k-mooney: do you have a traceback how it blows up?
17:18:34 sean-k-mooney i have fixed it in https://review.opendev.org/#/c/670189/
17:19:15 sean-k-mooney bbobrov: basicaly the old code would use q35 if there was not default for an archatecutre
17:20:04 sean-k-mooney which is invalid if you have sparc or many other arch specific qemu-* packages installed
17:20:28 sean-k-mooney so libvirt would raise an exception saying q35 is not supported by emulator X
17:21:25 bbobrov sean-k-mooney: understood, thanks. Don't we have a ci job to catch this kind of stuff?
17:21:59 sean-k-mooney this was not caught as this is currently not used outside of the tests and the test mock out all calls to libvirt
17:22:08 sean-k-mooney so it was passing because of invalid test data
17:22:43 sean-k-mooney if you tried to call that finciton in the agent then you got tracebacks
17:23:13 sean-k-mooney on ubunutu 18.04 they package all the emultor by defualt and install them when you install qemu and qemu-kvm
17:23:30 bbobrov ok, thanks. I'll review 670189 then
17:23:31 sean-k-mooney so i would have expect this to break but only when we added code to call this
17:25:02 sean-k-mooney i think the signiture of get_domain_capabilities is proably not what we want but i have fixed up the function without changing it for now to not break the sev code an allow me to contue with my own work

Earlier   Later