Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-06
21:01:55 sean-k-mooney at the moment numa stuff wont work with qemu via nova so its fine that this applies to both
21:02:34 sean-k-mooney if the instance has a numa toplogy and we are migrating it it means it was spawned on a kvm host
21:07:17 sean-k-mooney artom: fyi this is the most imporant comment out of the one i just left on the review https://review.openstack.org/#/c/634606/9/nova/compute/manager.py@6011
21:07:37 artom sean-k-mooney, that explains so much!
21:08:06 artom I hadn't rebased locally, line numbers in CI output didn't match, and _get_resource_tracker() was clearly present
21:08:07 sean-k-mooney it broke the sriov migrtaion series too
21:08:08 artom Thanks!
21:08:50 sean-k-mooney i fixed it in that yesterday so when i saw the fuction and the filing ci it clicked :)
21:08:54 openstackgerrit Merged openstack/nova stable/queens: Fix port dns_name reset https://review.openstack.org/633807
21:09:04 openstackgerrit Merged openstack/nova master: update flavor admin docs https://review.openstack.org/635198
21:09:22 artom Well, CI was going to fail regardless, but I was hoping it'd fail less
21:15:04 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: warn about changing/unsetting AZ name with instances https://review.openstack.org/635315
21:18:39 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: mention policy defaults for aggregates https://review.openstack.org/635317
21:48:40 mriedem cfriesen: jackding: https://review.openstack.org/#/c/603844/
21:48:53 mriedem i like this approach better, but the patch needs work
21:49:02 mriedem and i think we probably don't want to mess with instances whose task_state is not None
21:50:11 mriedem also funny that i was looking at a very similar thing in something i was writing yesterday for cross-cell resize https://review.openstack.org/#/c/635080/1/nova/compute/manager.py@5029
21:55:03 jackding mriedem: thanks. Yeh rollback can be messy.
22:15:44 openstackgerrit Matt Riedemann proposed openstack/nova master: neutron: make migrate_instance_finish call setup_instance_network_on_host https://review.openstack.org/635343
22:18:58 openstackgerrit Michal Arbet proposed openstack/nova master: Fix python3 compatibility https://review.openstack.org/635220
22:36:05 openstackgerrit Michal Arbet proposed openstack/nova master: Fix python3 compatibility https://review.openstack.org/635220
22:39:55 openstackgerrit Matt Riedemann proposed openstack/nova master: De-cruftify the finish_resize methods https://review.openstack.org/635349
22:42:00 openstackgerrit Eric Fried proposed openstack/nova master: Use :oslo-config: role in hypervisor-kvm doc https://review.openstack.org/634798
22:50:19 hogepodge Hi, someone just asked if there's a way to explicitly disable live migrations, and after looking at the configuration docs I can see why they're stumped. Anyone have some info on how to do that?
22:50:55 sean-k-mooney there is a way to do it yes ill have to look it up
22:51:26 sean-k-mooney hogepodge: it is admin only thing so nomally you dont have to expcltly disable it
22:53:38 sean-k-mooney hogepodge: you might have to do it via the polcy.json
22:54:14 hogepodge ok, thanks. I thought that might be the case. I asked for more context
22:54:32 openstackgerrit Chris Friesen proposed openstack/nova master: fix up numa-topology live migration hypervisor check https://review.openstack.org/635350
22:54:34 sean-k-mooney i think the way this is normally done is to block the api request using the policy.json
22:54:58 sean-k-mooney dansmith: do you recall another way to disable live migrations?
22:55:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Delete the obj_as_admin context manager https://review.openstack.org/635351
22:55:48 mriedem https://docs.openstack.org/nova/latest/configuration/policy.html
22:55:52 mriedem os_compute_api:os-migrate-server:migrate_live
22:56:45 mriedem change that to !
22:56:48 mriedem to disable it for everyone
22:56:50 mriedem hogepodge: ^
22:58:01 mriedem oh hey look legacy-grenade-dsvm-neutron-multinode-live-migration is fixed
22:58:04 mriedem https://review.openstack.org/#/c/634962/
22:58:17 mriedem http://logs.openstack.org/62/634962/5/check/legacy-grenade-dsvm-neutron-multinode-live-migration/982e90b/
23:01:40 hogepodge mriedem: sean-k-mooney: thanks
23:03:26 sean-k-mooney mriedem: so is that related to the block migration issue where it was complaing it was not on shared shared storage
23:03:39 sean-k-mooney ah it is
23:03:48 sean-k-mooney i was looking for https://bugs.launchpad.net/nova/+bug/1813216 yesterday
23:03:49 openstack Launchpad bug 1813216 in OpenStack Compute (nova) "legacy-grenade-dsvm-neutron-multinode-live-migration failing with "is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since Jan 21" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
23:05:21 mriedem yeah nova-compute during the ceph run didn't have the rbd config in it
23:05:59 sean-k-mooney mriedem: its not really a block migration if you are using the RBD iamge backend
23:07:27 sean-k-mooney its not a bfv instance either but when using RBD it really is a shared filesystem.
23:08:09 sean-k-mooney do we run the block migration tests on any multinode job without ceph/nfs
23:08:33 mriedem yes
23:08:37 mriedem well, i think so,
23:08:42 mriedem neutron multinode or whatever
23:09:30 mriedem http://logs.openstack.org/44/603844/19/check/tempest-multinode-full/698a95c/job-output.txt.gz#_2019-02-05_22_28_06_654011
23:10:22 sean-k-mooney ah this one https://github.com/openstack/nova/blob/master/.zuul.yaml#L252-L254
23:10:32 mriedem we don't have anything that tests the crazy environment lyarwood keeps trying to fix which is rbd shared storage for the root disk but local storage for the instance files
23:10:36 sean-k-mooney ok i was wondering if that came in form one of the templates
23:10:48 mriedem nova, for the most part, assumes that if you're using rbd you're shared everywhere
23:10:55 mriedem which apparently isn't true for some people
23:11:27 sean-k-mooney i dont understand the "local storage for the instance files" bit
23:11:39 sean-k-mooney i can ask lyarwood
23:11:41 mriedem the stuff under $instances_path
23:11:51 mriedem console logs, the image, config drive (i think)
23:11:52 mriedem etc
23:11:59 sean-k-mooney oh the libivt logs and stuff
23:12:28 mriedem yes this crazy thing https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.ensure_libvirt_rbd_instance_dir_cleanup
23:12:57 sean-k-mooney i assumed if you set then image backend to ceph those where on ceph too well the config drive and swap at least
23:13:19 mriedem that's what the libvirt driver assumes as well in some places
23:13:55 mriedem but the rbd image backend just controls the disks, not the other files associated with the instance on the host
23:14:05 mriedem it's very fun
23:14:05 sean-k-mooney that explain some of my issue on my old newton cloud...
23:14:26 sean-k-mooney well the more you know
23:14:30 mriedem could probably add that to the pile-o-problems for cdent to work on shared storage providers...
23:15:12 sean-k-mooney i know the file that is created when you suspend an instace was written to the compute node but ya
23:15:27 sean-k-mooney i wonder how many people over look that as i did
23:16:31 sean-k-mooney oh i was talking to lyarwood about a multi attach live migration bug earlier in the week
23:16:37 mriedem well, we don't really help anyone with no mention of this in the docs really, or config options for that matter
23:17:18 mriedem we should probably have a "things to consider if you plan to use shared storage" section in the admin docs
23:17:28 sean-k-mooney assuming we dont already have a tempest senario test i think im going to write one and add it to the tempest-full-multinode job
23:17:57 mriedem there are no tempest tests for live migration + volume multiattach that i'm aware of
23:18:05 mriedem there are for resize though
23:19:17 sean-k-mooney mriedem: lyarwood was looking at a posibly race. i dont have the context to had but i was thinking a simple test that boots 2 vms with the same multi attach volume and migrate one of the vms back and forth
23:21:22 sean-k-mooney i think it was related to concurent update of the attachnets
23:21:40 sean-k-mooney mriedem: oh lyarwood filled an upstream bug https://bugs.launchpad.net/nova/+bug/1814245
23:21:41 openstack Launchpad bug 1814245 in OpenStack Compute (nova) "_disconnect_volume incorrectly called for multiattach volumes during post_live_migration" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood)
23:24:26 mriedem so it disconnects the volume from the other server that wasn't live migrated
23:24:32 mriedem if both servers start on the same source host
23:24:33 mriedem yeah?
23:24:42 sean-k-mooney ya
23:25:27 mriedem i guess that bug says https://review.openstack.org/#/c/551302/ fixes it,
23:25:31 mriedem but ^ needs to be updated
23:27:23 mriedem sean-k-mooney: i don't know that a tempest test would catch this unless that test also ssh's into the guest and tries to verify the volume is still connected on the source host
23:27:40 mriedem and and connected to the live migrated server on the dest host
23:27:59 sean-k-mooney mriedem: yes i planned to ssh in touch a file migrate and check i could read the file on both vms
23:28:02 mriedem this is similar to https://review.openstack.org/#/c/548356/
23:28:17 mriedem maybe you want to work on cleaning up ^ first
23:28:22 mriedem i'm sure steve is long gone by now
23:28:56 mriedem he was only on loan from oracle until they got multiattach support in queens
23:29:13 sean-k-mooney am i can take a look. i have not worked with tempest in a while so i kind of wanted to use this to get familar with it again
23:30:55 sean-k-mooney if that is isshing into the vm it really should be a senario test not an api test... but that is a different issue
23:31:56 sean-k-mooney the api tests "should" be able to run with the nova fake or libvirt fake driver

Earlier   Later