| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-30 | |||
| 09:18:13 | viks____ | However, in vm which is created in the same host, i get | |
| 09:18:19 | viks____ | https://www.irccloud.com/pastebin/6LLVRG5F/ | |
| 09:18:43 | viks____ | Why read speed in the vm is very less and why the write speed is high on vm compared to host? I use below commands to test read and write using sysbench: | |
| 09:18:50 | viks____ | https://www.irccloud.com/pastebin/bzcImC3H/ | |
| 09:19:00 | viks____ | Can someone tell me why i see the above behaviour. Also How do i get better performance for disk I/o? | |
| 09:32:12 | elod | lyarwood: thanks for working on the live-migration job issue! ... and sorry, I have some questions in the live-migration patch. o:) | |
| 09:33:41 | lyarwood | elod: np I'll look now | |
| 09:33:55 | lyarwood | viks____: can you provide any more context, which virt driver are you using, storage backend etc | |
| 09:35:14 | viks____ | KVM/libvert | |
| 09:35:55 | viks____ | also i use qemu images | |
| 09:36:04 | viks____ | sorry qcow2* | |
| 09:44:16 | kashyap | viks____: When benchmarking I/O patterns, this is a good post to refer to: http://blog.vmsplice.net/2017/11/common-disk-benchmarking-mistakes.html | |
| 09:46:05 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/victoria: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/c/openstack/nova/+/764432 | |
| 09:46:06 | lyarwood | elod: ^ updated, hopefully that's cleaner and easier to understand now. | |
| 09:47:10 | elod | lyarwood: looking | |
| 09:58:11 | elod | lyarwood: one more quetsion: so now nova-evacuate is merged into the live-migration job by calling 'run-evacuate-hook' role. so the 'nova-evacuate' playbook is not needed anymore. Did you leave that playbook (playbook/nova-evacuate/run.yaml) there in case someone wants to run it explicitly? | |
| 09:59:45 | lyarwood | elod: nope that's a mistake, I'll remove it from master now | |
| 10:00:23 | elod | lyarwood: oh, thanks! | |
| 10:01:56 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-evacuate: Remove leftover playbook from standalone job https://review.opendev.org/c/openstack/nova/+/764618 | |
| 10:08:12 | viks____ | kashyap: Thanks.. in that post, fio is used. I'm using sysbench. also i'm trying to compare the results b/n host and vm using the same set of commands.. i think that should be fine.. or ?? | |
| 10:10:29 | kashyap | viks____: Same set as in with "sysbench"? | |
| 10:14:18 | viks____ | kashyap: i meant i am running the above commands in vm as well as the host running the vm and then comparing the results | |
| 10:14:43 | kashyap | viks____: As another case: also run on the host _without_ running the VM | |
| 10:14:55 | kashyap | viks____: Then you can compare and contrast further | |
| 10:16:45 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Move query param generation to RequestGroup o.vo https://review.opendev.org/c/openstack/nova/+/756894 | |
| 10:21:17 | viks____ | kashyap: .. The above is the result of that only.. first on the host without any vm.. then after creating the vm, running test within the vm.. and i was expecting some kind of similar results b/n them.. if u see above, read speed on host is too high compared to that in vm... and write speed is more in vm than that of host itself.. | |
| 10:21:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor _claim_pci_device_for_interface_attach to prepare for qos https://review.opendev.org/c/openstack/nova/+/756895 | |
| 10:23:25 | openstackgerrit | Merged openstack/nova master: Revert "zuul: Skip test_attach_scsi_disk_with_config_drive in nova-ceph-multistore" https://review.opendev.org/c/openstack/nova/+/764311 | |
| 10:23:47 | kashyap | viks____: Hmm, I don't have bright ideas; can you come over to #qemu on OFTC network? There are more Block Layer devs who have better ideas on these kind of perf issues | |
| 10:27:28 | kashyap | viks____: If you can't; I can ask for you in a bit. (I"m in the middle of something) | |
| 10:28:36 | viks____ | kashyap: is that related to both kvm and qemu? or just qemu? Because i'm using KVM... | |
| 10:29:07 | openstackgerrit | Adit Sarfaty proposed openstack/nova stable/train: Retry on vmware create_vm when it fails https://review.opendev.org/c/openstack/nova/+/764515 | |
| 10:30:08 | kashyap | viks____: It's for QEMU. But even if you're using KVM, QEMU is doing most of the heavy-lifting. All the I/O stuff is handled by QEMU's block layer. | |
| 10:30:38 | kashyap | viks____: QEMU is implicit in "KVM/libvirt" | |
| 10:30:52 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: Add regression test for bug #1899649 https://review.opendev.org/c/openstack/nova/+/764625 | |
| 10:30:52 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: compute: Don't detach volumes when RescheduledException raised without retry https://review.opendev.org/c/openstack/nova/+/764626 | |
| 10:30:53 | openstack | bug 1899649 in OpenStack Compute (nova) "Volume marked as available after a failure to build" [Undecided,In progress] https://launchpad.net/bugs/1899649 - Assigned to Lee Yarwood (lyarwood) | |
| 10:40:55 | kashyap | viks____: I have just asked your question to a QEMU dev on your "write perf better in guest than on host" question; I'll let you know if there's a response | |
| 10:41:30 | kashyap | (I asked the author of the earlier-mentioned benchmarking blog post.) | |
| 10:44:36 | viks____ | Ok.. thanks | |
| 10:48:59 | kashyap | viks____: So, here we go: | |
| 10:50:53 | kashyap | viks____: It's likely that the [host] page cache is being used, so host vs guest numbers could be wildly different to due different amounts of RAM. | |
| 10:51:09 | kashyap | viks____: So the first step is to check whether the benchmark is going through the page cache. If it does, then it's possible that the amount of physical disk I/O being done is quite different between guest and host, and therefore performance is also very different. | |
| 10:52:23 | kashyap | viks____: It's point-2 in the blog post abov ("Bypass the page cache"). You might want to re-test with 'fio' to compare and contrast. | |
| 10:54:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Remove KeyPair model from main DB https://review.opendev.org/c/openstack/nova/+/763989 | |
| 10:57:13 | stephenfin | lyarwood: Could you hit https://review.opendev.org/c/openstack/nova/+/749315/ https://review.opendev.org/c/openstack/nova/+/749312/ and https://review.opendev.org/c/openstack/nova/+/759114/ (remove XenAPI final bits)? | |
| 11:03:30 | viks____ | kashyap: ok.. thanks a lot.. will check with fio once | |
| 11:04:55 | lyarwood | stephenfin: ack sure | |
| 11:29:46 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/759114/ - As it isn't passing CI anyway can you remove `Remove` from the commit subject ;) | |
| 11:30:10 | stephenfin | um, yeah, fair /o\ | |
| 11:31:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Neuter 'BandwidthUsage', 'BandwidthUsageList' https://review.opendev.org/c/openstack/nova/+/759114 | |
| 11:31:59 | stephenfin | lyarwood: ^ | |
| 11:32:43 | lyarwood | ta | |
| 11:34:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Drop models for removed services, features https://review.opendev.org/c/openstack/nova/+/698706 | |
| 11:41:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Remove KeyPair model from main DB https://review.opendev.org/c/openstack/nova/+/763989 | |
| 11:55:44 | stephenfin | gibi: I corrected the docstring as requested at https://review.opendev.org/c/openstack/nova/+/764376/ fwiw | |
| 12:00:18 | priteau | Hello. I have an affinity server group, with a single instance in it. The instance is shut off. I can't cold migrate this instance, ServerGroupAffinityFilter is removing all hosts. Is this a known issue? | |
| 12:01:11 | priteau | I found https://bugs.launchpad.net/nova/+bug/1890065 but I am not sure it's the same problem, since it is about 2 instances being live migrated | |
| 12:01:11 | openstack | Launchpad bug 1890065 in OpenStack Compute (nova) "Impossible to migrate affinity instances" [Wishlist,Opinion] | |
| 12:58:07 | gibi | stephenfin: ack, looks good | |
| 12:59:20 | gibi | priteau: I think it is the general behavior is known, if you have instances in an affinity group then you cannot migrate them. (the power state does not matter either). The edge case when you have only one instance in the affinity group is not handled spcially. Do you have a use case where you need a single instance in an affinity group? | |
| 13:06:21 | priteau | gibi: I've encountered this while migrating every instance on an OpenStack cloud during a c7 to c8 migration, I don't know why the user set it up that way. I suppose one use case would be to a server group for all instances of a cluster, which could potentially scale down to 1. | |
| 13:09:09 | gibi | priteau: solving the migration problem generally is hard, as that needs a way to say ignore-the-affinity-rule-for-this-migration which is most probably means an API impact. The single instance case could be solved by always ignoring the affinity rule, so that is easier but I don't know that solving only the single instance case would really help much | |
| 13:09:52 | priteau | Agreed. | |
| 13:10:26 | priteau | On the advice of johnthetubaguy I temporarily changed the policy in the instance_group_policy table and reverted it after the migration. | |
| 13:42:33 | openstackgerrit | Merged openstack/nova master: doc: Fix rendering in the PTL guide https://review.opendev.org/c/openstack/nova/+/764175 | |
| 13:56:38 | gibi | did anybody see pip error like "ERROR: Links are not allowed as constraints" I got hit by it recently in grenade https://39757d668f2e13d2ecb5-94a4d63114c603d5be2910eb980b9a2e.ssl.cf1.rackcdn.com/756894/3/check/grenade/fe26d2e/job-output.txt | |
| 14:02:31 | gibi | both grenade and grenade-multinode hits it | |
| 14:02:42 | gibi | this is from the same patch running the grenade-multinode https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_57e/756894/3/check/nova-grenade-multinode/57ea016/logs/grenade.sh.txt | |
| 14:12:27 | gibi | it seems it is started happening a lot in the last hour or so http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Links%20are%20not%20allowed%20as%20constraints%5C%22 | |
| 14:14:08 | jangutter | gibi: I was just about to start sending up signal flares! | |
| 14:14:21 | gibi | jangutter: thanks | |
| 14:14:45 | jangutter | gibi: happened to me too, I was working on stable/victoria and at first I thought I did something wrong. | |
| 14:17:31 | jangutter | best I can figure out: https://github.com/openstack/devstack/blob/master/inc/python#L386 <--- devstack has been using this method for ages now. I wonder if a pip update converted the warning to an error. | |
| 14:20:45 | gibi | yeah, that sound like a possibility | |
| 14:24:06 | jangutter | Removing that line (and the preceding backslash) seems to unblock it for me, but figuring out the consequences of that needs someone with a lot more experience on requirements, pip and the projects needs. | |
| 14:37:13 | gibi | hm, I cannot push to opendev gerrit now | |
| 14:37:24 | gibi | fatal: Unpack error, check server log | |
| 14:37:24 | gibi | error: remote unpack failed: error Missing tree 466cd024745bbeffebf3f8e598759fcc9b75df4d | |
| 14:47:19 | kashyap | stephenfin: gibi: I'm still to comb through some review comments on Secure Boot spec before the spec-review day tomm. Excuse my delay. | |
| 14:48:18 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: DNM https://review.opendev.org/c/openstack/nova/+/764766 | |
| 14:52:16 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: Add regression test for bug #1899649 https://review.opendev.org/c/openstack/nova/+/764625 | |
| 14:52:16 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: compute: Don't detach volumes when RescheduledException raised without retry https://review.opendev.org/c/openstack/nova/+/764626 | |
| 14:52:17 | openstack | bug 1899649 in OpenStack Compute (nova) "Volume marked as available after a failure to build" [Undecided,In progress] https://launchpad.net/bugs/1899649 - Assigned to Lee Yarwood (lyarwood) | |
| 14:52:40 | gibi | kashyap: no worries, Milestone 1 is this week but we only freeze specs at M2 | |
| 14:53:44 | kashyap | gibi: Ah, thank you. I didn't realize that spec-freeze is at M2. | |
| 14:54:40 | gibi | kashyap: M2 is spec freze M3 is feature freeze | |
| 14:55:06 | gibi | I guess in the past there was a time when we freezed specs at M1 but we don't do that in the last couple of cycles | |
| 14:55:20 | kashyap | I see | |
| 15:00:00 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow extending an existing instance allocation https://review.opendev.org/c/openstack/nova/+/757109 | |
| 15:00:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make remove allocation symmetric with add allocation https://review.opendev.org/c/openstack/nova/+/757110 | |
| 15:12:42 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: DNM https://review.opendev.org/c/openstack/nova/+/764771 | |
| 15:14:41 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: DNM https://review.opendev.org/c/openstack/nova/+/764772 | |
| 15:16:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: DNM https://review.opendev.org/c/openstack/nova/+/764773 | |
| 15:17:30 | gibi | (sorry for the noise) | |
| 15:18:18 | jangutter | gibi: still checking the pip thing? | |
| 15:18:32 | gibi | jangutter: nope, I got hit by a gerrit issue | |
| 15:36:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Factor out port resource deallocation https://review.opendev.org/c/openstack/nova/+/757111 | |
| 15:42:39 | stephenfin | lyarwood: Don't know if I linked these already, but those changes to improve logging for PCI requests are here https://review.opendev.org/q/topic:%2522bug/1852727%2522+status:open | |
| 15:42:50 | stephenfin | If you care to cast an eye over them :) | |