Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-26
14:48:58 shilpasd mriedem: means nova live-migration <instance_id>, it triggers live migration, but any another way to live migrate, any periodic call or something
14:49:33 mriedem no nova doesn't auto-live migrate things for you
14:49:51 shilpasd mriedem: i am in process of verifying all move operations on NFS changes done against https://review.opendev.org/#/c/650188/
14:50:05 shilpasd so wnat to take care of all move operations
14:50:37 shilpasd so just want to know @ it
14:51:27 mriedem all move operations are user-initiated
14:51:56 mriedem as far as i know anyway
14:52:00 shilpasd ok, as of now verifying SHELVE + SHELVE with offload + UNSHELVE + REBUILD + RESIZE + RESIZE REVERT + EVACUATION + COLD MIGRATION + COLD MIGRATION REVERT + LIVE MIGRATION
14:52:10 shilpasd just list if i missed anything
14:52:17 mriedem by rebuild i assume you mean evacuate
14:52:27 mriedem rebuild (the server action in the api) isn't a move,
14:52:29 mriedem but evacuate is
14:52:32 efried brinzhang: I'm here now, what's up?
14:52:43 mriedem evacuate = rebuild on another host
14:52:51 shilpasd rebuild using another image
14:53:00 mriedem rebuild + a new image is not a move
14:53:10 mriedem it's rebuilding the server's root disk image on the same host
14:53:11 bauzas mriedem: not sure I understood your point in https://bugs.launchpad.net/nova/+bug/1793569/comments/5
14:53:12 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza)
14:53:37 mriedem also, shelve w/o offload and then unshelve is also not a move operation,
14:53:45 bauzas mriedem: do you want heal_allocations to support this or the "placement audit' rather ?
14:53:48 mriedem if the instance is shelved but not offloaded, and then the user unshelves, it's just unshelved on the same host
14:53:52 shilpasd mriedem: ok, noted
14:54:31 shilpasd mriedem: what @ resize
14:55:14 shilpasd its move operation, right, since resizing on another host also
14:55:26 mriedem shilpasd: maybe :)
14:55:43 mriedem unless nova is configured with allow_resize_to_same_host and the scheduler picks the same host the instance is already one,
14:55:55 mriedem which is possible in a small edge site or if the server is in a strict affinity group and can't be moved
14:56:17 mriedem *already on
14:56:51 shilpasd got it
14:56:52 mriedem https://bugs.launchpad.net/nova/+bug/1790204 is all about that problem
14:56:53 openstack Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged]
14:58:00 mriedem bauzas: i think i meant to say "nova-manage placement audit" there,
14:58:18 mriedem since heal_allocations doesn't report on things really, nor does it delete allocations, it only adds allocations for instances (not migrations) that are missing
14:58:51 bauzas mriedem: ack, will add this there then
15:00:27 mriedem i went on to continue talking about heal_allocations but idk, it's a blur
15:00:47 shilpasd mriedem: one more query, i have NFS configuration, and performing resize on another host, and it goes for creating a instance data file on the dest system via SSH
15:00:59 shilpasd refer code at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8861
15:01:37 shilpasd mriedem: during shared resource provider check, why this check is necessary?
15:02:05 shilpasd _is_storage_shared_with()
15:03:26 mriedem shilpasd: it may be ssh or rsync, it depends on config https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.remote_filesystem_transport
15:03:34 mriedem the default is ssh
15:04:23 mriedem i'm less familiar with this code, but for one we don't have shared storage provider support in the libvirt driver anyway,
15:04:58 mriedem but this is presumably one of the things we could replace if we had compute nodes modeled in a shared storage aggregate and we could avoid the "temp file create" tests and such for shared storage
15:05:57 mriedem d get ssh'ed/rsync'ed around
15:05:57 mriedem as i'm sure lyarwood and mdbooth could probably attest, shared storage support in the libvirt driver can be very confusing because there are the instance files like console logs and such, and there is the image backend, and that can all be different and be a mix of shared storage and non-shared storage, e.g. the root disk images might be in the rbd image backend but the instance files, like console logs, could be on local dis
15:06:26 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.ensure_libvirt_rbd_instance_dir_cleanup
15:06:26 mriedem e.g.
15:07:28 sean-k-mooney mriedem: yes it is but if we keep getting bug with it i might have to raise it. but ya not before m2 likely not before m3 if in train at all.
15:07:44 sean-k-mooney ^ network info cache rework
15:07:45 mriedem bauzas: i think what i was thinking of was an audit command could detect that you have orphaned allocations tied to a not-in-progress migration, e.g. a migration that failed but we failed to cleanup the allocations,
15:08:19 mriedem bauzas: and then that information could be provided to the admin to then determine what to do, e.g. delete the allocations for the migration record consumer and potentially the related instance,
15:08:30 bauzas mriedem: yeah ok
15:08:33 mriedem and if they delete the allocations for the instance, then they could run heal_allocations on the instance to fix things up
15:08:47 mriedem we could also eventually build on that to make it automatic with options
15:08:56 mriedem e.g. nova-manage placement audit --heal
15:08:58 mriedem something like that
15:09:01 shilpasd mriedem: thanks for discussing doubts, will go through the sharings and get back to you for any further
15:09:57 mriedem sean-k-mooney: redoing nova's nw info cache at this point in the game is going to be a big undertaking, and i would not be surprised if trying to use a global cache like memcache or etcd or something just generates more or different kinds of bugs than what we've already been patching lo these many years, as i'm sure dansmith can agree
15:10:31 dansmith oh mahgod
15:10:44 efried yonglihe: I'm going to fix your pep8 error on https://review.opendev.org/#/c/627765/ real quick, k?
15:10:57 dansmith why do we need a memcache? it's in the database
15:10:59 efried it's due to a new rule that recently merged.
15:11:31 sean-k-mooney dansmith: i was suggesting not keeping in in the database and only having a dict cache or maybe use memcache
15:11:47 dansmith sean-k-mooney: ...why?
15:11:50 sean-k-mooney mriedem: and ya it would be a blueprint or spec not a bug fix
15:11:51 mriedem sean-k-mooney: we can just as easily f that up
15:12:24 sean-k-mooney well if its in process as a dict cache then if we f it up it fixed by restarting the compute agent
15:12:42 sean-k-mooney memcahce is proably not going to help with anything
15:12:45 dansmith we store some stuff in nwinfo that isn't anywhere else, IIRC, like which ports we created vs. the user, so that has to be persisted somewhere if we were going to use memcache
15:12:49 dansmith ...yeah ;)
15:13:02 dansmith what problem is being solved here?
15:13:22 mriedem i don't think that overhauling to use an external cache service and restarting the compute is the giant hammer we really need for what we're trying to solve
15:13:29 sean-k-mooney nothing at the momemnt reworking it is unrelated to what we are trying to fix
15:13:40 openstackgerrit Eric Fried proposed openstack/nova master: clean up orphan instances https://review.opendev.org/627765
15:13:40 openstackgerrit Eric Fried proposed openstack/nova master: Clean up orphan instances virt driver https://review.opendev.org/648912
15:13:44 mriedem so this is a....thought exercise?
15:13:49 efried sean-k-mooney, gibi: Would y'all please have another look at these --^
15:13:50 sean-k-mooney yes
15:14:11 sean-k-mooney its on my todo list to figure ot if it makes sense to even do
15:14:13 gibi efried: I have it open
15:15:17 efried thanks gibi
15:15:25 efried thanks sean-k-mooney
15:15:34 efried sean-k-mooney: fyi it's apparently a thing stx cares about
15:15:51 efried thus presumably it "makes sense" in some capacity :)
15:17:05 mriedem efried: hyperv ci is happy with the update_provider_tree patch https://review.opendev.org/#/c/667417/
15:17:17 efried mriedem: thanks for the reminder
15:17:49 mriedem efried: fwiw that cleanup orphan instances thing is also something that the public cloud SIG (and huawei public cloud ops) care about as well, which i was initially reviewing it awhile back
15:18:01 mriedem *why i was
15:18:34 mriedem the concern at the last ptg was how much duplication there was with the existing periodic to cleanup running deleted (but not orphaned) instances
15:20:24 efried okay, thanks for that background.
15:21:13 mriedem something something live migration fails and you've got untracked guests on the host consuming resources (which aren't tracked obviously) so then trying to schedule things to those hosts fails b/c you're out of resources
15:21:39 efried sounds like we need a patch to clean up those orphaned instances
15:22:31 mriedem i'm sure lots of operators have already just written scripts to detect and clean those types of thing sup
15:22:33 mriedem *up
15:22:38 mriedem but yeah it's better to have it native probably
15:42:22 efried mriedem: We don't have a way to prove the xen one is being hit, do we? (update_provider_tree)
15:42:25 efried since their CI is dead?
15:43:54 efried mriedem: also, if you haven't already, there should be a note to the ML warning of this (and another before we remove the code path, obvsly)
15:44:06 efried ...for oot folk

Earlier   Later