| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-26 | |||
| 14:32:26 | sean-k-mooney | well my next question was going to be "i assume this is because of nova networks legacy choices" | |
| 14:32:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove no longer required "inner" methods. https://review.opendev.org/655282 | |
| 14:32:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove unused FP device creation and deletion methods. https://review.opendev.org/635433 | |
| 14:32:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.opendev.org/635431 | |
| 14:32:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.opendev.org/635436 | |
| 14:32:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Privsep the ebtables modification code. https://review.opendev.org/635435 | |
| 14:32:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move dnsmasq restarts to privsep. https://review.opendev.org/639280 | |
| 14:32:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move iptables rule fetching and setting to privsep. https://review.opendev.org/636508 | |
| 14:32:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move calls to ovs-vsctl to privsep. https://review.opendev.org/639282 | |
| 14:32:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move router advertisement daemon restarts to privsep. https://review.opendev.org/639281 | |
| 14:32:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move setting of device trust to privsep. https://review.opendev.org/639283 | |
| 14:32:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: We no longer need rootwrap. https://review.opendev.org/554438 | |
| 14:32:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Cleanup the _execute shim in nova/network. https://review.opendev.org/639581 | |
| 14:32:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move final bridge commands to privsep. https://review.opendev.org/639580 | |
| 14:32:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Cleanup no longer required filters and add a release note. https://review.opendev.org/639826 | |
| 14:33:00 | mriedem | sean-k-mooney: idk, you'd have to do some digging to find out when the network info cache was introduced, i don't know if it was before quantum or not | |
| 14:33:20 | mriedem | but we also store bdms in the db which are essentially the same thing - a cache of volume information for the server | |
| 14:33:28 | mriedem | which was probably before cinder existed | |
| 14:33:41 | sean-k-mooney | im seeing a pattern there | |
| 14:34:17 | sean-k-mooney | ok well lets fix the current issue first but i think i might look into if we could remove storing it to the db | |
| 14:34:31 | amorin | I would love that | |
| 14:34:35 | amorin | :p | |
| 14:34:46 | sean-k-mooney | cacheing in memory in the compute agent would likely be enough | |
| 14:35:18 | sean-k-mooney | we would have to rebuild it every time the compute agent restarts but i think that is fine | |
| 14:37:22 | sean-k-mooney | actully we could use memcache to cache it too which would mean all the services would have acess to it anway its now on my todo list | |
| 14:38:02 | sean-k-mooney | messing up the netron policy and currpting the network info cache is what cause our ci cloud production outage at the weekend | |
| 14:45:46 | mriedem | TheJulia: is this a known busted job? http://logs.openstack.org/17/667417/1/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/db33ba3/controller/logs/devstacklog.txt.gz#_2019-06-26_05_47_14_168 | |
| 14:46:53 | mriedem | sean-k-mooney: redoing how the nw info cache works is hopefully wayyyyyyy down on your todo lits | |
| 14:46:55 | mriedem | *list | |
| 14:47:30 | shilpasd | efried: mriedem: can you tell me how to trigger live migration sync and async way, any CLI commands? | |
| 14:48:00 | mriedem | shilpasd: i don't know what you mean, sync and async way | |
| 14:48:58 | shilpasd | mriedem: means nova live-migration <instance_id>, it triggers live migration, but any another way to live migrate, any periodic call or something | |
| 14:49:33 | mriedem | no nova doesn't auto-live migrate things for you | |
| 14:49:51 | shilpasd | mriedem: i am in process of verifying all move operations on NFS changes done against https://review.opendev.org/#/c/650188/ | |
| 14:50:05 | shilpasd | so wnat to take care of all move operations | |
| 14:50:37 | shilpasd | so just want to know @ it | |
| 14:51:27 | mriedem | all move operations are user-initiated | |
| 14:51:56 | mriedem | as far as i know anyway | |
| 14:52:00 | shilpasd | ok, as of now verifying SHELVE + SHELVE with offload + UNSHELVE + REBUILD + RESIZE + RESIZE REVERT + EVACUATION + COLD MIGRATION + COLD MIGRATION REVERT + LIVE MIGRATION | |
| 14:52:10 | shilpasd | just list if i missed anything | |
| 14:52:17 | mriedem | by rebuild i assume you mean evacuate | |
| 14:52:27 | mriedem | rebuild (the server action in the api) isn't a move, | |
| 14:52:29 | mriedem | but evacuate is | |
| 14:52:32 | efried | brinzhang: I'm here now, what's up? | |
| 14:52:43 | mriedem | evacuate = rebuild on another host | |
| 14:52:51 | shilpasd | rebuild using another image | |
| 14:53:00 | mriedem | rebuild + a new image is not a move | |
| 14:53:10 | mriedem | it's rebuilding the server's root disk image on the same host | |
| 14:53:11 | bauzas | mriedem: not sure I understood your point in https://bugs.launchpad.net/nova/+bug/1793569/comments/5 | |
| 14:53:12 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 14:53:37 | mriedem | also, shelve w/o offload and then unshelve is also not a move operation, | |
| 14:53:45 | bauzas | mriedem: do you want heal_allocations to support this or the "placement audit' rather ? | |
| 14:53:48 | mriedem | if the instance is shelved but not offloaded, and then the user unshelves, it's just unshelved on the same host | |
| 14:53:52 | shilpasd | mriedem: ok, noted | |
| 14:54:31 | shilpasd | mriedem: what @ resize | |
| 14:55:14 | shilpasd | its move operation, right, since resizing on another host also | |
| 14:55:26 | mriedem | shilpasd: maybe :) | |
| 14:55:43 | mriedem | unless nova is configured with allow_resize_to_same_host and the scheduler picks the same host the instance is already one, | |
| 14:55:55 | mriedem | which is possible in a small edge site or if the server is in a strict affinity group and can't be moved | |
| 14:56:17 | mriedem | *already on | |
| 14:56:51 | shilpasd | got it | |
| 14:56:52 | mriedem | https://bugs.launchpad.net/nova/+bug/1790204 is all about that problem | |
| 14:56:53 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged] | |
| 14:58:00 | mriedem | bauzas: i think i meant to say "nova-manage placement audit" there, | |
| 14:58:18 | mriedem | since heal_allocations doesn't report on things really, nor does it delete allocations, it only adds allocations for instances (not migrations) that are missing | |
| 14:58:51 | bauzas | mriedem: ack, will add this there then | |
| 15:00:27 | mriedem | i went on to continue talking about heal_allocations but idk, it's a blur | |
| 15:00:47 | shilpasd | mriedem: one more query, i have NFS configuration, and performing resize on another host, and it goes for creating a instance data file on the dest system via SSH | |
| 15:00:59 | shilpasd | refer code at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8861 | |
| 15:01:37 | shilpasd | mriedem: during shared resource provider check, why this check is necessary? | |
| 15:02:05 | shilpasd | _is_storage_shared_with() | |
| 15:03:26 | mriedem | shilpasd: it may be ssh or rsync, it depends on config https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.remote_filesystem_transport | |
| 15:03:34 | mriedem | the default is ssh | |
| 15:04:23 | mriedem | i'm less familiar with this code, but for one we don't have shared storage provider support in the libvirt driver anyway, | |
| 15:04:58 | mriedem | but this is presumably one of the things we could replace if we had compute nodes modeled in a shared storage aggregate and we could avoid the "temp file create" tests and such for shared storage | |
| 15:05:57 | mriedem | d get ssh'ed/rsync'ed around | |
| 15:05:57 | mriedem | as i'm sure lyarwood and mdbooth could probably attest, shared storage support in the libvirt driver can be very confusing because there are the instance files like console logs and such, and there is the image backend, and that can all be different and be a mix of shared storage and non-shared storage, e.g. the root disk images might be in the rbd image backend but the instance files, like console logs, could be on local dis | |
| 15:06:26 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.ensure_libvirt_rbd_instance_dir_cleanup | |
| 15:06:26 | mriedem | e.g. | |
| 15:07:28 | sean-k-mooney | mriedem: yes it is but if we keep getting bug with it i might have to raise it. but ya not before m2 likely not before m3 if in train at all. | |
| 15:07:44 | sean-k-mooney | ^ network info cache rework | |
| 15:07:45 | mriedem | bauzas: i think what i was thinking of was an audit command could detect that you have orphaned allocations tied to a not-in-progress migration, e.g. a migration that failed but we failed to cleanup the allocations, | |
| 15:08:19 | mriedem | bauzas: and then that information could be provided to the admin to then determine what to do, e.g. delete the allocations for the migration record consumer and potentially the related instance, | |
| 15:08:30 | bauzas | mriedem: yeah ok | |
| 15:08:33 | mriedem | and if they delete the allocations for the instance, then they could run heal_allocations on the instance to fix things up | |
| 15:08:47 | mriedem | we could also eventually build on that to make it automatic with options | |
| 15:08:56 | mriedem | e.g. nova-manage placement audit --heal | |
| 15:08:58 | mriedem | something like that | |
| 15:09:01 | shilpasd | mriedem: thanks for discussing doubts, will go through the sharings and get back to you for any further | |
| 15:09:57 | mriedem | sean-k-mooney: redoing nova's nw info cache at this point in the game is going to be a big undertaking, and i would not be surprised if trying to use a global cache like memcache or etcd or something just generates more or different kinds of bugs than what we've already been patching lo these many years, as i'm sure dansmith can agree | |
| 15:10:31 | dansmith | oh mahgod | |
| 15:10:44 | efried | yonglihe: I'm going to fix your pep8 error on https://review.opendev.org/#/c/627765/ real quick, k? | |
| 15:10:57 | dansmith | why do we need a memcache? it's in the database | |
| 15:10:59 | efried | it's due to a new rule that recently merged. | |
| 15:11:31 | sean-k-mooney | dansmith: i was suggesting not keeping in in the database and only having a dict cache or maybe use memcache | |
| 15:11:47 | dansmith | sean-k-mooney: ...why? | |
| 15:11:50 | sean-k-mooney | mriedem: and ya it would be a blueprint or spec not a bug fix | |
| 15:11:51 | mriedem | sean-k-mooney: we can just as easily f that up | |
| 15:12:24 | sean-k-mooney | well if its in process as a dict cache then if we f it up it fixed by restarting the compute agent | |
| 15:12:42 | sean-k-mooney | memcahce is proably not going to help with anything | |