| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-06 | |||
| 20:53:40 | cfriesen | sean-k-mooney: agreed, that's what I was thinking. QEMU rather than KVM | |
| 20:53:40 | sean-k-mooney | cfriesen: so ya when virt tipe is set to kvm it is still qemu i just check on a local devstack. | |
| 20:54:52 | cfriesen | the tricky bit is that I think qemu with TCG doesn't support numa stuff | |
| 20:55:07 | sean-k-mooney | it does | |
| 20:55:18 | sean-k-mooney | it just does not support per cpu pinning | |
| 20:55:33 | cfriesen | ah, okay. | |
| 20:55:33 | sean-k-mooney | you can create numa nodes and use hugepages | |
| 20:55:38 | cfriesen | should be all right then | |
| 20:55:53 | sean-k-mooney | but nova's implematnion does not work due to the per cpu pinning it adds on top | |
| 20:56:34 | sean-k-mooney | the mttcg backend (multi thread TCG) remvoes the cpu pinning limitation too | |
| 20:56:49 | sean-k-mooney | libvirt will have support for that soon | |
| 21:00:23 | cfriesen | I'll propose a fix, see what stephen has to say tomorrow. | |
| 21:00:56 | sean-k-mooney | cfriesen: ya looking at the compute node object in the db it does not have the virt type | |
| 21:01:04 | sean-k-mooney | so i guess just update it to qemu | |
| 21:01:55 | sean-k-mooney | at the moment numa stuff wont work with qemu via nova so its fine that this applies to both | |
| 21:02:34 | sean-k-mooney | if the instance has a numa toplogy and we are migrating it it means it was spawned on a kvm host | |
| 21:07:17 | sean-k-mooney | artom: fyi this is the most imporant comment out of the one i just left on the review https://review.openstack.org/#/c/634606/9/nova/compute/manager.py@6011 | |
| 21:07:37 | artom | sean-k-mooney, that explains so much! | |
| 21:08:06 | artom | I hadn't rebased locally, line numbers in CI output didn't match, and _get_resource_tracker() was clearly present | |
| 21:08:07 | sean-k-mooney | it broke the sriov migrtaion series too | |
| 21:08:08 | artom | Thanks! | |
| 21:08:50 | sean-k-mooney | i fixed it in that yesterday so when i saw the fuction and the filing ci it clicked :) | |
| 21:08:54 | openstackgerrit | Merged openstack/nova stable/queens: Fix port dns_name reset https://review.openstack.org/633807 | |
| 21:09:04 | openstackgerrit | Merged openstack/nova master: update flavor admin docs https://review.openstack.org/635198 | |
| 21:09:22 | artom | Well, CI was going to fail regardless, but I was hoping it'd fail less | |
| 21:15:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: warn about changing/unsetting AZ name with instances https://review.openstack.org/635315 | |
| 21:18:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: mention policy defaults for aggregates https://review.openstack.org/635317 | |
| 21:48:40 | mriedem | cfriesen: jackding: https://review.openstack.org/#/c/603844/ | |
| 21:48:53 | mriedem | i like this approach better, but the patch needs work | |
| 21:49:02 | mriedem | and i think we probably don't want to mess with instances whose task_state is not None | |
| 21:50:11 | mriedem | also funny that i was looking at a very similar thing in something i was writing yesterday for cross-cell resize https://review.openstack.org/#/c/635080/1/nova/compute/manager.py@5029 | |
| 21:55:03 | jackding | mriedem: thanks. Yeh rollback can be messy. | |
| 22:15:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: neutron: make migrate_instance_finish call setup_instance_network_on_host https://review.openstack.org/635343 | |
| 22:18:58 | openstackgerrit | Michal Arbet proposed openstack/nova master: Fix python3 compatibility https://review.openstack.org/635220 | |
| 22:36:05 | openstackgerrit | Michal Arbet proposed openstack/nova master: Fix python3 compatibility https://review.openstack.org/635220 | |
| 22:39:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: De-cruftify the finish_resize methods https://review.openstack.org/635349 | |
| 22:42:00 | openstackgerrit | Eric Fried proposed openstack/nova master: Use :oslo-config: role in hypervisor-kvm doc https://review.openstack.org/634798 | |
| 22:50:19 | hogepodge | Hi, someone just asked if there's a way to explicitly disable live migrations, and after looking at the configuration docs I can see why they're stumped. Anyone have some info on how to do that? | |
| 22:50:55 | sean-k-mooney | there is a way to do it yes ill have to look it up | |
| 22:51:26 | sean-k-mooney | hogepodge: it is admin only thing so nomally you dont have to expcltly disable it | |
| 22:53:38 | sean-k-mooney | hogepodge: you might have to do it via the polcy.json | |
| 22:54:14 | hogepodge | ok, thanks. I thought that might be the case. I asked for more context | |
| 22:54:32 | openstackgerrit | Chris Friesen proposed openstack/nova master: fix up numa-topology live migration hypervisor check https://review.openstack.org/635350 | |
| 22:54:34 | sean-k-mooney | i think the way this is normally done is to block the api request using the policy.json | |
| 22:54:58 | sean-k-mooney | dansmith: do you recall another way to disable live migrations? | |
| 22:55:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete the obj_as_admin context manager https://review.openstack.org/635351 | |
| 22:55:48 | mriedem | https://docs.openstack.org/nova/latest/configuration/policy.html | |
| 22:55:52 | mriedem | os_compute_api:os-migrate-server:migrate_live | |
| 22:56:45 | mriedem | change that to ! | |
| 22:56:48 | mriedem | to disable it for everyone | |
| 22:56:50 | mriedem | hogepodge: ^ | |
| 22:58:01 | mriedem | oh hey look legacy-grenade-dsvm-neutron-multinode-live-migration is fixed | |
| 22:58:04 | mriedem | https://review.openstack.org/#/c/634962/ | |
| 22:58:17 | mriedem | http://logs.openstack.org/62/634962/5/check/legacy-grenade-dsvm-neutron-multinode-live-migration/982e90b/ | |
| 23:01:40 | hogepodge | mriedem: sean-k-mooney: thanks | |
| 23:03:26 | sean-k-mooney | mriedem: so is that related to the block migration issue where it was complaing it was not on shared shared storage | |
| 23:03:39 | sean-k-mooney | ah it is | |
| 23:03:48 | sean-k-mooney | i was looking for https://bugs.launchpad.net/nova/+bug/1813216 yesterday | |
| 23:03:49 | openstack | Launchpad bug 1813216 in OpenStack Compute (nova) "legacy-grenade-dsvm-neutron-multinode-live-migration failing with "is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since Jan 21" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 23:05:21 | mriedem | yeah nova-compute during the ceph run didn't have the rbd config in it | |
| 23:05:59 | sean-k-mooney | mriedem: its not really a block migration if you are using the RBD iamge backend | |
| 23:07:27 | sean-k-mooney | its not a bfv instance either but when using RBD it really is a shared filesystem. | |
| 23:08:09 | sean-k-mooney | do we run the block migration tests on any multinode job without ceph/nfs | |
| 23:08:33 | mriedem | yes | |
| 23:08:37 | mriedem | well, i think so, | |
| 23:08:42 | mriedem | neutron multinode or whatever | |
| 23:09:30 | mriedem | http://logs.openstack.org/44/603844/19/check/tempest-multinode-full/698a95c/job-output.txt.gz#_2019-02-05_22_28_06_654011 | |
| 23:10:22 | sean-k-mooney | ah this one https://github.com/openstack/nova/blob/master/.zuul.yaml#L252-L254 | |
| 23:10:32 | mriedem | we don't have anything that tests the crazy environment lyarwood keeps trying to fix which is rbd shared storage for the root disk but local storage for the instance files | |
| 23:10:36 | sean-k-mooney | ok i was wondering if that came in form one of the templates | |
| 23:10:48 | mriedem | nova, for the most part, assumes that if you're using rbd you're shared everywhere | |
| 23:10:55 | mriedem | which apparently isn't true for some people | |
| 23:11:27 | sean-k-mooney | i dont understand the "local storage for the instance files" bit | |
| 23:11:39 | sean-k-mooney | i can ask lyarwood | |
| 23:11:41 | mriedem | the stuff under $instances_path | |
| 23:11:51 | mriedem | console logs, the image, config drive (i think) | |
| 23:11:52 | mriedem | etc | |
| 23:11:59 | sean-k-mooney | oh the libivt logs and stuff | |
| 23:12:28 | mriedem | yes this crazy thing https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.ensure_libvirt_rbd_instance_dir_cleanup | |
| 23:12:57 | sean-k-mooney | i assumed if you set then image backend to ceph those where on ceph too well the config drive and swap at least | |
| 23:13:19 | mriedem | that's what the libvirt driver assumes as well in some places | |
| 23:13:55 | mriedem | but the rbd image backend just controls the disks, not the other files associated with the instance on the host | |
| 23:14:05 | mriedem | it's very fun | |
| 23:14:05 | sean-k-mooney | that explain some of my issue on my old newton cloud... | |
| 23:14:26 | sean-k-mooney | well the more you know | |
| 23:14:30 | mriedem | could probably add that to the pile-o-problems for cdent to work on shared storage providers... | |
| 23:15:12 | sean-k-mooney | i know the file that is created when you suspend an instace was written to the compute node but ya | |
| 23:15:27 | sean-k-mooney | i wonder how many people over look that as i did | |
| 23:16:31 | sean-k-mooney | oh i was talking to lyarwood about a multi attach live migration bug earlier in the week | |
| 23:16:37 | mriedem | well, we don't really help anyone with no mention of this in the docs really, or config options for that matter | |
| 23:17:18 | mriedem | we should probably have a "things to consider if you plan to use shared storage" section in the admin docs | |
| 23:17:28 | sean-k-mooney | assuming we dont already have a tempest senario test i think im going to write one and add it to the tempest-full-multinode job | |
| 23:17:57 | mriedem | there are no tempest tests for live migration + volume multiattach that i'm aware of | |
| 23:18:05 | mriedem | there are for resize though | |
| 23:19:17 | sean-k-mooney | mriedem: lyarwood was looking at a posibly race. i dont have the context to had but i was thinking a simple test that boots 2 vms with the same multi attach volume and migrate one of the vms back and forth | |
| 23:21:22 | sean-k-mooney | i think it was related to concurent update of the attachnets | |
| 23:21:40 | sean-k-mooney | mriedem: oh lyarwood filled an upstream bug https://bugs.launchpad.net/nova/+bug/1814245 | |
| 23:21:41 | openstack | Launchpad bug 1814245 in OpenStack Compute (nova) "_disconnect_volume incorrectly called for multiattach volumes during post_live_migration" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 23:24:26 | mriedem | so it disconnects the volume from the other server that wasn't live migrated | |
| 23:24:32 | mriedem | if both servers start on the same source host | |