| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-31 | |||
| 15:00:57 | mdbooth | kaisers: Hmm, kinda find the reason I thought that now. Could be I was imagining it. Have you tested this manually, btw? | |
| 15:01:13 | mdbooth | kinda? can't | |
| 15:01:57 | openstackgerrit | Sujitha proposed openstack/nova master: Adding tags field to InstancePayload https://review.openstack.org/407228 | |
| 15:02:18 | leakypipes | sfinucan: quick review on this please? https://review.openstack.org/#/c/451635/2 | |
| 15:10:05 | openstackgerrit | Sujitha proposed openstack/nova master: Change tags to default field in Instance object. https://review.openstack.org/415298 | |
| 15:10:18 | openstackgerrit | Sujitha proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185 | |
| 15:10:32 | openstackgerrit | Sujitha proposed openstack/nova master: Reduce calls to load_tags() to 0 https://review.openstack.org/435146 | |
| 15:12:42 | openstackgerrit | Sujitha proposed openstack/nova master: Add keypairs field to InstancePayload https://review.openstack.org/419730 | |
| 15:14:39 | cfriesen | are there any qemu-knowledgable folks around? When using qcow2 instance storage I'm wondering why we convert backing file images to raw rather than just leaving them in whatever format they were in (raw, qcow2). | |
| 15:18:15 | kaisers_ | mdbooth: Hi! Yep, i tested this manually. --scope works for me. I actually hacked it into the driver in a test instance and killed the n-cpu process hard, mount point remained active | |
| 15:18:36 | mdbooth | kashyap: That's not quite the same thing | |
| 15:18:37 | kashyap | cfriesen: Good question, I once recall digging it, and now lost it from memory | |
| 15:18:39 | mdbooth | Eurgh | |
| 15:18:45 | mdbooth | kaisers: ^^^ | |
| 15:18:55 | kaisers_ | mdbooth: :) | |
| 15:19:08 | mdbooth | kaisers_: Did you try stopping the nova-compute service? | |
| 15:19:16 | kaisers_ | mdbooth: systemd produces a slice unit who becomes parent of the process | |
| 15:19:27 | kaisers_ | hmm | |
| 15:19:31 | kaisers_ | mdbooth: i can revisit that | |
| 15:19:56 | kaisers_ | mdbooth: but that should behave similarily | |
| 15:20:00 | mdbooth | kaisers_: Having read it again, I suspect it should work | |
| 15:20:14 | kaisers_ | mdbooth: as the bug comes up as soon as the cgroup is closed | |
| 15:20:27 | mdbooth | And frankly this is in your driver, so I'd personally be inclined to take your word for it as a functional test would be logistically difficult | |
| 15:20:33 | kashyap | cfriesen: Are you hitting this? -- "Converting image from qcow2 to raw is slow for images with large virtual size" -- https://bugs.launchpad.net/nova/+bug/1484992 | |
| 15:20:33 | openstack | Launchpad bug 1484992 in OpenStack Compute (nova) "Converting image from qcow2 to raw is slow for images with large virtual size" [Wishlist,Confirmed] | |
| 15:20:54 | kaisers_ | mdbooth: ack | |
| 15:20:54 | mdbooth | kaisers_: However, given the bug you're trying to fix, if it were me I'd want to test this | |
| 15:21:11 | kaisers_ | yep, i'll do that | |
| 15:24:14 | sfinucan | leakypipes: Done | |
| 15:24:23 | leakypipes | sfinucan: thanks! | |
| 15:24:37 | leakypipes | sfinucan: or however you say thanks in Irish :P | |
| 15:25:03 | sfinucan | leakypipes: Go raibh maith agat ;) | |
| 15:25:11 | leakypipes | ah, well, that too! :) | |
| 15:25:42 | sfinucan | leakypipes: This is one to stick in your review queue for today :) https://review.openstack.org/#/c/361140/ | |
| 15:26:21 | sfinucan | ...or for next weeks spec review drive | |
| 15:26:28 | cfriesen | kashyap: yep. and in our case it's made worse by the fact that we do the conversion with ionice and no output caching (necessary to minimize impact of the conversion IO load on the instances using the same disk) | |
| 15:26:39 | kashyap | cfriesen: I'm looking at this function _disk_qcow2_to_raw() | |
| 15:26:48 | kashyap | And it's not being used anywhere from my current `grep`ing | |
| 15:27:23 | kashyap | It was introduced as a helper to be used in nova/virt/libvirt -> finish_migration() | |
| 15:27:24 | cfriesen | kashyap: in our case we're hitting it when downloading the backing image file during a block live migration for a qcow2-based instance | |
| 15:28:35 | kashyap | cfriesen: Do you have a full test case? | |
| 15:29:32 | cfriesen | kashyap: boot instance with qcow2 storage, live-migrate it to a compute node that doesn't have the image cached yet. | |
| 15:29:56 | kashyap | cfriesen: I _think_ I know why we do the conversion: | |
| 15:30:06 | mriedem | dansmith: weird the logs are not found for https://review.openstack.org/#/c/452006/ | |
| 15:30:09 | mriedem | on the failed jobs | |
| 15:30:12 | mriedem | anyway i left some comments | |
| 15:30:15 | kashyap | There's an epic comment from mdbooth in the function finish_migration() | |
| 15:30:27 | kashyap | cfriesen: It's to avoid CVEs from format probing | |
| 15:30:28 | mriedem | dansmith: i'm be wary of using the IntegratedHelperMixin | |
| 15:30:44 | dansmith | mriedem: known issue apparently | |
| 15:30:45 | leakypipes | sfinucan: yeah, will try to get to that once I complete a review of jianghuaw's VGPU stuff. | |
| 15:30:48 | mriedem | and I'd use USES_DB_SELF = True with the NoDbTestCase i think so you can setup the db yourself | |
| 15:30:53 | dansmith | mriedem: I was copying the server groups test | |
| 15:30:56 | mriedem | avoiding default cells or something | |
| 15:31:24 | kashyap | cfriesen: Hmm, but wait -- format probing is not the only rationale, because to avoid that, the solution is to explicitly specify the format of the backing file | |
| 15:31:38 | leakypipes | sfinucan: plus, I'm technically snikitin_'s boss so if I don't get to that review today, he can bitch at me all he wants to over the weekend or Monday ;P | |
| 15:31:41 | mriedem | dansmith: fwiw this is how i've been doing them lately https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py | |
| 15:31:44 | mriedem | completely self-contained | |
| 15:31:51 | mdbooth | kashyap: IIRC that code is only applicable when the source and destination hosts have different storage configurations | |
| 15:31:53 | kashyap | cfriesen: Can you reproduce it with Git Master? | |
| 15:32:01 | sfinucan | leakypipes: :D | |
| 15:32:02 | mdbooth | As you say, it stops a security bug in that case | |
| 15:32:13 | mdbooth | We really shouldn't be doing it, thoug | |
| 15:32:14 | dansmith | mriedem: ack | |
| 15:32:14 | mdbooth | h | |
| 15:32:20 | cfriesen | kashyap: I think that's different. in our case the conversion is done under pre_live_migration(), wheich eventually calls down to libvirt.utils.fetch_image(), which calls fetch_to_raw() | |
| 15:32:32 | cfriesen | kashyap: let me check master | |
| 15:32:36 | kashyap | mdbooth: But cfriesen seems to imply that we unconditionally convert any raw to qcow2 -- do we do that at all? | |
| 15:32:38 | mdbooth | cfriesen: What's the issue? | |
| 15:32:53 | kashyap | cfriesen: Oh right, fetch_to_raw(), it's from utils.py | |
| 15:32:54 | mdbooth | cfriesen: Are you getting different backing file formats on source and destination? | |
| 15:33:01 | kashyap | mdbooth: See the bug in the scrollback | |
| 15:33:08 | kashyap | mdbooth: https://bugs.launchpad.net/nova/+bug/1484992 | |
| 15:33:08 | openstack | Launchpad bug 1484992 in OpenStack Compute (nova) "Converting image from qcow2 to raw is slow for images with large virtual size" [Wishlist,Confirmed] | |
| 15:33:16 | kashyap | Chris is hitting the above | |
| 15:33:41 | cfriesen | mdbooth: no, we're hitting a performance issue when converting a downloaded qcow2 image to raw for use as a backing file, as part of pre_live_migration() | |
| 15:33:49 | mdbooth | cfriesen: Ah | |
| 15:33:56 | mdbooth | So, you don't actually have to convert those | |
| 15:34:01 | mdbooth | That's a config option | |
| 15:34:19 | mdbooth | However, it's an ongoing issue that changing storage config after deployment is a minefield | |
| 15:34:21 | kashyap | Ah, right, I thought I saw one | |
| 15:35:10 | cfriesen | mdbooth: force_raw_images ? | |
| 15:35:13 | kashyap | 'force_raw_images = True' | |
| 15:35:19 | mdbooth | cfriesen: Sounds about right, yeah | |
| 15:35:28 | kashyap | Yep, that seems to be it - (BoolOpt) Force backing images to raw format | |
| 15:35:38 | mdbooth | cfriesen: You're using LVM? | |
| 15:35:49 | cfriesen | mdbooth: no, qcow2 in this case. | |
| 15:35:56 | cfriesen | mdbooth: lvm in other cases though | |
| 15:36:12 | mdbooth | Ok, because I didn't think live migration worked on lvm... does it? | |
| 15:36:26 | mdbooth | Or is that only cold migration? | |
| 15:36:44 | cfriesen | mdbooth: cold works in our environment, we patched it in. :) | |
| 15:36:50 | mdbooth | Hehe, ok | |
| 15:38:16 | mdbooth | cfriesen: Yeah, if you want to avoid the conversion step, force_raw_images = False is what you want | |
| 15:38:28 | mdbooth | It should probably be the default, tbh | |
| 15:38:43 | openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova master: Return 400 when compute host is not found https://review.openstack.org/447355 | |
| 15:38:43 | openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova master: Reset instance status on invalid host https://review.openstack.org/401009 | |
| 15:39:06 | mriedem | lyarwood: ah i realize now we don't have any unit test coverage on detach or driver_detach in nova.virt.block_device now | |
| 15:39:17 | mriedem | you had the TODOs in test_compute_mgr | |
| 15:39:18 | kashyap | mdbooth: There should surely be some rationale documented as to why it was chosen to be 'True' by default? | |
| 15:39:27 | mdbooth | surely | |