| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-08 | |||
| 15:23:03 | kashyap | mriedem: Isn't it 16GB? | |
| 15:23:20 | kashyap | It's the size in bytes | |
| 15:23:38 | kashyap | It is indeed 16GB. | |
| 15:24:06 | mriedem | the nova code says kb | |
| 15:24:07 | mriedem | dev.resize(resize_to * units.Gi / units.Ki) | |
| 15:24:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Final cleanups in conf/network https://review.openstack.org/431023 | |
| 15:24:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Fix formatting of network options https://review.openstack.org/443086 | |
| 15:25:24 | kashyap | mriedem: I just checked the QEMU QAPI schema docs; the size there definitely means bytes. | |
| 15:25:49 | kashyap | Note, the 'size' you see there is the QEMU argument to the QMP command 'block_resize' | |
| 15:25:59 | kashyap | mriedem: What type of storage is this, BTW? | |
| 15:26:04 | kashyap | QCOW2? | |
| 15:27:39 | lyarwood | kashyap: are you sure that resize only takes Bytes? | |
| 15:27:54 | mriedem | kashyap: i'm not sure what the volume is that it's resizing to, but this is what it's resizing from (i think) | |
| 15:28:07 | mriedem | http://paste.openstack.org/show/601953/ | |
| 15:28:24 | kashyap | lyarwood: If you see the pastebin from mriedem, on line-4, the quoted size is the argument to the QMP command 'block_resize' | |
| 15:28:40 | kashyap | mriedem: The QEMU folks are asking about the details of the kind of storage | |
| 15:29:18 | mriedem | this is where the size is calculated https://review.openstack.org/#/c/433793/3/nova/virt/libvirt/driver.py@1396 | |
| 15:29:21 | gibi | tojuvone: thanks for the info. I still feel it would be nice to have a non nova impacting alternative mentioned in the spec. but I have no hard feelings is we go forward with the nova based solution | |
| 15:29:41 | kashyap | lyarwood: See line 996 there -- http://git.qemu-project.org/?p=qemu.git;a=blob;f=qapi/block-core.json;h=9bb7f4a17ba9765245616212542725c60ec361a1;hb=HEAD#l996 | |
| 15:29:54 | mriedem | ibm a9000 is what they are resizing to | |
| 15:30:07 | kashyap | And, from what? | |
| 15:30:58 | mriedem | HP 3PAR | |
| 15:31:02 | lyarwood | kashyap: thanks | |
| 15:31:51 | tojuvone | gibi, Thanks, I will work on that | |
| 15:32:23 | kashyap | mriedem: I've conveyed this detail to the QEMU folks on IRC, let you know once one of the block layer guys in EST wakes up | |
| 15:34:40 | mriedem | kashyap: thanks | |
| 15:34:55 | kashyap | mriedem: One more | |
| 15:35:08 | mriedem | even if it's 16GB, https://bugzilla.redhat.com/show_bug.cgi?id=1070531 seems to indicate it's still too large | |
| 15:35:08 | openstack | bugzilla.redhat.com bug 1070531 in qemu-kvm "qmp can not give a reasonable hint when block_resize a scsi_debug disk with 10G" [Low,Assigned] - Assigned to armbru | |
| 15:35:39 | kashyap | mriedem: What is the storage backend - is it a plain raw file, a qcow2 file, a SCSI block device, or something else? | |
| 15:36:29 | openstackgerrit | Maciej Szankin proposed openstack/nova master: conf: fix default values reporting infra worker https://review.openstack.org/441227 | |
| 15:37:00 | mriedem | kashyap: i'll have to dig for that | |
| 15:37:15 | mriedem | kashyap: fo the thing being swapped to or from? | |
| 15:37:25 | dansmith | jaypipes: did you see my comments here? https://review.openstack.org/#/c/441544/4 | |
| 15:37:35 | dansmith | jaypipes: I can fix them up if you don't have time | |
| 15:37:36 | kashyap | mriedem: The context for that question is, from danpb: | |
| 15:37:39 | kashyap | mriedem: To figure out where EINVAL comes from we need to understand what qemu block driver is getting used | |
| 15:38:03 | dansmith | jaypipes: specifically this: https://review.openstack.org/#/c/441544/4/nova/virt/ironic/driver.py | |
| 15:38:42 | kashyap | mriedem: Both. | |
| 15:40:51 | kashyap | mriedem: One more (credit for the guess: Eric Blake), about potentially what's going on: | |
| 15:40:52 | mriedem | maybe i can find some details from the compute logs | |
| 15:41:23 | jaypipes | dansmith: yeah, will get to that after meetings. | |
| 15:41:40 | kashyap | mriedem: | |
| 15:41:40 | kashyap | 16:39 < eblake> I know qcow2 prevents resizes that shrink the volume | |
| 15:41:40 | kashyap | 16:39 < eblake> could it be that the resize failed because of that? | |
| 15:41:40 | kashyap | 16:39 < kashyap> eblake: Could you see that from the pastebin, or are you guessing that there's a shrink going on? | |
| 15:41:43 | kashyap | 16:40 < eblake> I'm just guessing, as that would be one reason for calling the resize requested size invalid | |
| 15:41:49 | dansmith | jaypipes: okay, and then kinda next for this effort is the flavor override patch and the allocation side of this right? | |
| 15:42:00 | jaypipes | dansmith: yup. | |
| 15:42:17 | dansmith | jaypipes: okay cool | |
| 15:43:04 | mriedem | kashyap: we only resize if the new volume that we're resizing to is bigger https://review.openstack.org/#/c/433793/3/nova/compute/manager.py@5006 | |
| 15:43:29 | mriedem | looking at the logs, the name on the A9000 volume implies it's an iscsi block device | |
| 15:43:39 | mriedem | however, it also says the size is 10GB, | |
| 15:43:49 | mriedem | but that's not what i'm seeing when we actually run the block_rebase command | |
| 15:43:54 | kashyap | mriedem: On the 16GB size comment, I'm being told it is "any size is valid as long as its below the max possible size; and 16G is trivially in range" | |
| 15:43:59 | mriedem | *block_resize | |
| 15:44:26 | kashyap | And the max possible size is at least a a TB of even EB, apparently | |
| 15:44:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174 | |
| 15:44:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Add functools.wraps to rapper https://review.openstack.org/443175 | |
| 15:45:57 | kashyap | mriedem: Apart from implication, can you confirm that it's indeed an iSCSI block dev? | |
| 15:47:43 | mriedem | yeah it's iscsi | |
| 15:47:57 | efried | sean-k-mooney mriedem https://review.openstack.org/#/c/435575/ didn't break our CI; it just makes the jenkins pep8 job fail on our code where it didn't before. Still figuring out how to pull in that plugin so I can validate locally, but without having to fix the whole nova project (which presumably got done by somebody else before this change was merged). | |
| 15:48:16 | mriedem | another thing is they swap from the A9000 to the 3PAR successfully and then they try to swap from the 3PAR to the A9000 and that fails | |
| 15:48:22 | mriedem | and i think that's actually a thing that was recently fixed | |
| 15:48:39 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: PowerVM Driver: SSP emphemeral disk support https://review.openstack.org/443189 | |
| 15:49:25 | sean-k-mooney | mriedem: efried it broke our ci by what i consider to be a bug in devstack | |
| 15:49:25 | mriedem | i wonder if they are hitting https://review.openstack.org/#/c/431540/ | |
| 15:49:30 | mriedem | kashyap: lyarwood ^ | |
| 15:50:22 | sean-k-mooney | mriedem: when devstack does a pip install of the nova source it installs both the requirements.txt and test-requiremnts.txt gloablly | |
| 15:50:26 | lyarwood | mriedem: yeah that allows you to swap multiple times now | |
| 15:51:08 | kashyap | mriedem: Okay, danpb seems to have nailed it, with his QEMU POV: | |
| 15:52:13 | kashyap | Ok, so QEMU will be using a block device and you can't resize those. More specifically the 'block_resize' command when used against a block device, will just refresh qemu's view of the block device size, and if the requested resize size is larger than the block device size, it'l give EINVAL. IOW, you can use 'block_resize' on a SCSI device *if* you arrange for a resize to be done in the remote | |
| 15:52:19 | kashyap | iSCSI server first, and if you then tell the client host to rescan LUNs, and only then tell QEMU to apply the new size. | |
| 15:52:29 | sean-k-mooney | mriedem: that means that anyting else that uses pep8 is now runnning the addtional ordering tests | |
| 15:52:52 | sean-k-mooney | mriedem: as a result we cant build ovs-dpdk from sources as they dont enforcre the import ordering. | |
| 15:53:23 | sean-k-mooney | mriedem: devstack should not do https://github.com/openstack-dev/devstack/blob/6523d6e09789076d0a430bf7b608511fcb1c406a/inc/python#L336-L348 by default | |
| 15:55:43 | dansmith | johnthetubaguy: see the questions pending on this quotas thing? https://review.openstack.org/#/c/416521 | |
| 15:55:52 | dansmith | johnthetubaguy: it doesn't call you out specifically, but it might as well :P | |
| 15:56:19 | johnthetubaguy | dansmith: on the TC call, those should keep me amused while I am listening | |
| 15:56:21 | mriedem | kashyap: so i got the GET volume response info from the compute logs which has the size for each volume, and they both say 10 | |
| 15:56:27 | efried | sean-k-mooney Sounds reasonable to me. Launchpad bug? Propose a fix? | |
| 15:56:27 | mriedem | so i don't see why we're even attempting a resize | |
| 15:56:29 | sean-k-mooney | mriedem: the connect to ovs may not be apparent but there makefile enabled flake8 if its installed which it is tanks to devstack and the test defined by flake8-import-order seam to be enabled by default | |
| 15:56:39 | dansmith | johnthetubaguy: even quotas are better than TC stuff :) | |
| 15:56:55 | kashyap | mriedem: Heh, that's a good place to step back and see, if it's even needed. | |
| 15:56:58 | mriedem | kashyap: because resize_to should be 0 since this would be false https://review.openstack.org/#/c/433793/3/nova/compute/manager.py@5006 | |
| 15:57:01 | sean-k-mooney | efried: am yes i have asked our ci team to open one and submit a patch to make it optional | |
| 15:58:21 | mriedem | kashyap: multipath is also enabled on these things, which i'm just going to wildly make a guess could ruin everything | |
| 15:58:29 | sean-k-mooney | efried: if we are gong to run the unit tests in the ci we will just use tox and it will install the test-requriments in the virtual env itself so i dont know why devstack is installing them by default for every thing that is installed via pip | |
| 15:58:41 | mriedem | b/c we sure as hell don't test swap volume with multipath backends | |
| 15:58:57 | openstackgerrit | Gábor Antal proposed openstack/nova-specs master: Show persisted scheduler hints in server details https://review.openstack.org/440580 | |
| 15:59:28 | kashyap | mriedem: Yeah, 'multipath' -- never in my life I could grok that goddamend thing. Whenever I set out to focus on it to learn, I get distracted. | |
| 16:01:00 | sean-k-mooney | efried: mriedem looks like they have already opened the bug https://bugs.launchpad.net/devstack/+bug/1671144 | |
| 16:01:00 | openstack | Launchpad bug 1671144 in devstack "test-requirements are wrongly installed system-wide" [Undecided,New] | |
| 16:01:21 | openstackgerrit | John Garbutt proposed openstack/nova master: POC: add context.check_scope https://review.openstack.org/435485 | |
| 16:03:05 | lyarwood | mriedem: we don't appear to test this resize path during our swap_volume tempest tests btw, I'll follow up with that. | |
| 16:07:19 | mriedem | kashyap: ah there are 2 A9K volumes | |
| 16:07:22 | mriedem | 2 swap operations | |
| 16:07:34 | mriedem | 10GB A9K -> 10GB 3PAR, | |
| 16:07:42 | kashyap | And... | |