Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-08
15:12:36 kashyap Yep; maybe he confused it because I was learning Dutch (and last night was my class, so left early)
15:12:58 kashyap mriedem: Okay, we've got at least the complete command libvirt is sending QEMU
15:13:07 kashyap From line-3, qemuMonitorJSONCommandWithFd:296
15:13:10 tojuvone gibi, Surely also working the get alarm to user trough Vitrage
15:13:17 kashyap Now to decipher this error message, let me look a bit more
15:14:11 gibi tojuvone: I don't know. Today the I don't get network status or block device backend status through the GET /servers API but I can accept that planned maintenance is different and that might have place in the nova API
15:14:15 tojuvone gibi, as it could receive notification
15:14:26 sfinucan mriedem, johnthetubaguy: Could I ask you to take a look at the PCI weigher spec again? Think I've resolved all your comments https://review.openstack.org/#/c/364468/
15:15:24 gibi tojuvone: I agree that we need to notify the end user I just wondering if this notification should come from nova our from carlton
15:15:44 kashyap mriedem: Yes, I remember the failure, let me check with QEMU folks on OFTC
15:16:38 mriedem kashyap: i found https://bugzilla.redhat.com/show_bug.cgi?id=1070531
15:16:38 openstack bugzilla.redhat.com bug 1070531 in qemu-kvm "qmp can not give a reasonable hint when block_resize a scsi_debug disk with 10G" [Low,Assigned] - Assigned to armbru
15:16:45 mriedem i wonder if the size arg is too big?
15:16:55 tojuvone gibi, Craton would be building notifications soon. That is also something if not use Craton, then one might like to have the Nova notif
15:17:26 kashyap mriedem: Potentially; don't want to say something meaningless without digging a bit more :-)
15:17:54 mriedem if i'm looking at this correctly, it's a 15GB block resize
15:18:45 gibi tojuvone: does it mean there will be two alternative implementation of the same feature one in Nova and one in Craton?
15:18:49 mriedem umm, 15360GB?
15:19:00 mriedem self._guest._domain.blockResize(self._disk, size_kb)
15:19:14 mriedem "size":16106127360
15:20:19 kashyap Oh dear
15:21:45 tojuvone gibi, For the notification? If you would use Craton, yes there would be 2 different then.
15:22:52 openstackgerrit Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174
15:22:52 openstackgerrit Dan Smith proposed openstack/nova master: Add functools.wraps to rapper https://review.openstack.org/443175
15:23:03 kashyap mriedem: Isn't it 16GB?
15:23:20 kashyap It's the size in bytes
15:23:38 kashyap It is indeed 16GB.
15:24:06 mriedem the nova code says kb
15:24:07 mriedem dev.resize(resize_to * units.Gi / units.Ki)
15:24:42 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Final cleanups in conf/network https://review.openstack.org/431023
15:24:42 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Fix formatting of network options https://review.openstack.org/443086
15:25:24 kashyap mriedem: I just checked the QEMU QAPI schema docs; the size there definitely means bytes.
15:25:49 kashyap Note, the 'size' you see there is the QEMU argument to the QMP command 'block_resize'
15:25:59 kashyap mriedem: What type of storage is this, BTW?
15:26:04 kashyap QCOW2?
15:27:39 lyarwood kashyap: are you sure that resize only takes Bytes?
15:27:54 mriedem kashyap: i'm not sure what the volume is that it's resizing to, but this is what it's resizing from (i think)
15:28:07 mriedem http://paste.openstack.org/show/601953/
15:28:24 kashyap lyarwood: If you see the pastebin from mriedem, on line-4, the quoted size is the argument to the QMP command 'block_resize'
15:28:40 kashyap mriedem: The QEMU folks are asking about the details of the kind of storage
15:29:18 mriedem this is where the size is calculated https://review.openstack.org/#/c/433793/3/nova/virt/libvirt/driver.py@1396
15:29:21 gibi tojuvone: thanks for the info. I still feel it would be nice to have a non nova impacting alternative mentioned in the spec. but I have no hard feelings is we go forward with the nova based solution
15:29:41 kashyap lyarwood: See line 996 there -- http://git.qemu-project.org/?p=qemu.git;a=blob;f=qapi/block-core.json;h=9bb7f4a17ba9765245616212542725c60ec361a1;hb=HEAD#l996
15:29:54 mriedem ibm a9000 is what they are resizing to
15:30:07 kashyap And, from what?
15:30:58 mriedem HP 3PAR
15:31:02 lyarwood kashyap: thanks
15:31:51 tojuvone gibi, Thanks, I will work on that
15:32:23 kashyap mriedem: I've conveyed this detail to the QEMU folks on IRC, let you know once one of the block layer guys in EST wakes up
15:34:40 mriedem kashyap: thanks
15:34:55 kashyap mriedem: One more
15:35:08 mriedem even if it's 16GB, https://bugzilla.redhat.com/show_bug.cgi?id=1070531 seems to indicate it's still too large
15:35:08 openstack bugzilla.redhat.com bug 1070531 in qemu-kvm "qmp can not give a reasonable hint when block_resize a scsi_debug disk with 10G" [Low,Assigned] - Assigned to armbru
15:35:39 kashyap mriedem: What is the storage backend - is it a plain raw file, a qcow2 file, a SCSI block device, or something else?
15:36:29 openstackgerrit Maciej Szankin proposed openstack/nova master: conf: fix default values reporting infra worker https://review.openstack.org/441227
15:37:00 mriedem kashyap: i'll have to dig for that
15:37:15 mriedem kashyap: fo the thing being swapped to or from?
15:37:25 dansmith jaypipes: did you see my comments here? https://review.openstack.org/#/c/441544/4
15:37:35 dansmith jaypipes: I can fix them up if you don't have time
15:37:36 kashyap mriedem: The context for that question is, from danpb:
15:37:39 kashyap mriedem: To figure out where EINVAL comes from we need to understand what qemu block driver is getting used
15:38:03 dansmith jaypipes: specifically this: https://review.openstack.org/#/c/441544/4/nova/virt/ironic/driver.py
15:38:42 kashyap mriedem: Both.
15:40:51 kashyap mriedem: One more (credit for the guess: Eric Blake), about potentially what's going on:
15:40:52 mriedem maybe i can find some details from the compute logs
15:41:23 jaypipes dansmith: yeah, will get to that after meetings.
15:41:40 kashyap mriedem:
15:41:40 kashyap 16:39 < eblake> I know qcow2 prevents resizes that shrink the volume
15:41:40 kashyap 16:39 < eblake> could it be that the resize failed because of that?
15:41:40 kashyap 16:39 < kashyap> eblake: Could you see that from the pastebin, or are you guessing that there's a shrink going on?
15:41:43 kashyap 16:40 < eblake> I'm just guessing, as that would be one reason for calling the resize requested size invalid
15:41:49 dansmith jaypipes: okay, and then kinda next for this effort is the flavor override patch and the allocation side of this right?
15:42:00 jaypipes dansmith: yup.
15:42:17 dansmith jaypipes: okay cool
15:43:04 mriedem kashyap: we only resize if the new volume that we're resizing to is bigger https://review.openstack.org/#/c/433793/3/nova/compute/manager.py@5006
15:43:29 mriedem looking at the logs, the name on the A9000 volume implies it's an iscsi block device
15:43:39 mriedem however, it also says the size is 10GB,
15:43:49 mriedem but that's not what i'm seeing when we actually run the block_rebase command
15:43:54 kashyap mriedem: On the 16GB size comment, I'm being told it is "any size is valid as long as its below the max possible size; and 16G is trivially in range"
15:43:59 mriedem *block_resize
15:44:26 kashyap And the max possible size is at least a a TB of even EB, apparently
15:44:56 openstackgerrit Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174
15:44:56 openstackgerrit Dan Smith proposed openstack/nova master: Add functools.wraps to rapper https://review.openstack.org/443175
15:45:57 kashyap mriedem: Apart from implication, can you confirm that it's indeed an iSCSI block dev?
15:47:43 mriedem yeah it's iscsi
15:47:57 efried sean-k-mooney mriedem https://review.openstack.org/#/c/435575/ didn't break our CI; it just makes the jenkins pep8 job fail on our code where it didn't before. Still figuring out how to pull in that plugin so I can validate locally, but without having to fix the whole nova project (which presumably got done by somebody else before this change was merged).
15:48:16 mriedem another thing is they swap from the A9000 to the 3PAR successfully and then they try to swap from the 3PAR to the A9000 and that fails
15:48:22 mriedem and i think that's actually a thing that was recently fixed
15:48:39 openstackgerrit Eric Fried proposed openstack/nova master: WIP: PowerVM Driver: SSP emphemeral disk support https://review.openstack.org/443189
15:49:25 sean-k-mooney mriedem: efried it broke our ci by what i consider to be a bug in devstack
15:49:25 mriedem i wonder if they are hitting https://review.openstack.org/#/c/431540/
15:49:30 mriedem kashyap: lyarwood ^
15:50:22 sean-k-mooney mriedem: when devstack does a pip install of the nova source it installs both the requirements.txt and test-requiremnts.txt gloablly
15:50:26 lyarwood mriedem: yeah that allows you to swap multiple times now
15:51:08 kashyap mriedem: Okay, danpb seems to have nailed it, with his QEMU POV:
15:52:13 kashyap Ok, so QEMU will be using a block device and you can't resize those. More specifically the 'block_resize' command when used against a block device, will just refresh qemu's view of the block device size, and if the requested resize size is larger than the block device size, it'l give EINVAL. IOW, you can use 'block_resize' on a SCSI device *if* you arrange for a resize to be done in the remote
15:52:19 kashyap iSCSI server first, and if you then tell the client host to rescan LUNs, and only then tell QEMU to apply the new size.
15:52:29 sean-k-mooney mriedem: that means that anyting else that uses pep8 is now runnning the addtional ordering tests
15:52:52 sean-k-mooney mriedem: as a result we cant build ovs-dpdk from sources as they dont enforcre the import ordering.
15:53:23 sean-k-mooney mriedem: devstack should not do https://github.com/openstack-dev/devstack/blob/6523d6e09789076d0a430bf7b608511fcb1c406a/inc/python#L336-L348 by default

Earlier   Later