Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-12-08
14:04:05 gibi it fails in nova-ceph-multistore too so it is not grenade specific
14:04:12 sean-k-mooney not sure if that actully what is happening as i tough we used master tempetst regradels of the branch but i would guess its soemthign like that
14:04:26 sean-k-mooney oh ok
14:04:46 sean-k-mooney so may somethign related to our jobs
14:05:38 frickler yes, that allowlist needs to include tempest
14:06:02 sean-k-mooney frickler: im not sure it should
14:06:09 sean-k-mooney tempest shoudl be installed in teh venv
14:06:21 sean-k-mooney we shoudl not be using tempest form teh host right?
14:06:41 sean-k-mooney we install tempest in a venv because it is often not the same version of openstack
14:06:54 gibi this is where we fail https://github.com/openstack/devstack/blob/master/lib/tempest#L731
14:06:56 sean-k-mooney and we do it that way to avoid the possibel depency conflcit that woudl create
14:07:16 sean-k-mooney so tempest should __not__ be in the allowlist
14:07:41 sean-k-mooney ya so that shoudl be using tempst form the venv
14:08:09 sean-k-mooney shoudl it be tox -evenv-tempest https://github.com/openstack/devstack/blob/master/lib/tempest#L709
14:08:14 sean-k-mooney instead of venv
14:08:47 sean-k-mooney tox -evenv-tempest seams to be what we use on the other lines
14:08:57 gibi hm, good point
14:09:02 gibi that seems like the wrong env
14:09:40 gibi I will push a patch
14:09:41 frickler oh, cool, so this found an actual bug, nice
14:11:33 ralonsoh sean-k-mooney, thanks! I saw this issue in out CI too
14:11:38 ralonsoh our*
14:14:41 gibi devstack fix is up https://review.opendev.org/c/openstack/devstack/+/866997
14:28:38 opendevreview Konrad Gube proposed openstack/nova-specs master: Use extend volume completion action https://review.opendev.org/c/openstack/nova-specs/+/855490
14:44:19 opendevreview Bence Romsics proposed openstack/nova master: doc: soft delete and shadow tables https://review.opendev.org/c/openstack/nova/+/867001
15:06:12 opendevreview Bence Romsics proposed openstack/nova master: doc: soft delete and shadow tables https://review.opendev.org/c/openstack/nova/+/867001
15:19:39 elendrys Hello here, is this channel open to nova related questions or is it dedicated to developers ?
15:45:00 frickler elendrys: this is a bit of a grey area, #openstack is what is meant for general questions, but the chance to get an answer there is low, so you might as well try here directly
15:47:14 fungi also the openstack-discuss@lists.openstack.org mailing list has a much higher chance of getting you an answer eventually
15:54:07 elendrys Ok thank you
15:54:35 sean-k-mooney elendrys: did you have a question in general
15:55:00 sean-k-mooney while the channel is not for supprot we will try an point you in the right direction in genreal
15:55:44 elendrys Yes
15:56:23 elendrys I have servers with cinder volumes, backed in PureStorage array over iscsi
15:57:21 sean-k-mooney ok so the vm is using host mount iscsi block devices passthroug to the vms
15:57:28 elendrys When i resize volumes attached to running servers, I can see in nova's log that it rescans the array to get the new size when cinder notifies
15:57:55 elendrys But in the vm, the size always have 1 notification late size
15:58:23 elendrys I mean, a 10Gb vol expended to 20 stills show as 10, but if I extend to 30, it shows 20
15:58:49 elendrys I can't get why (probably libvirt) it is missing something
15:58:54 sean-k-mooney hum interesting.
15:59:13 sean-k-mooney am im not sure this is libvirt/qemu related
15:59:22 elendrys If I stop then start the server it shows 30 as intended
15:59:28 sean-k-mooney i woudl suppect this is more likely to be related to iscsid or simialr
15:59:59 sean-k-mooney elendrys: did you check on the host if the host kernel sees the correct volume size
16:00:30 sean-k-mooney you should be able to find the dev path form the vm xml and check with lsblk or a similar too to see if it sees the correct current size
16:01:06 sean-k-mooney if the hosts sees the correct size then the issue is likely at the qemu level
16:01:45 sean-k-mooney elendrys: i dont think libvirt is invovled for voulme resize in this case
16:02:22 elendrys It does
16:05:33 sean-k-mooney im wondering is there anything you can do at the gues os level
16:05:38 elendrys It is queued somewhere by nova
16:06:25 elendrys I tried to force scsci bus scan but it doesn't change
16:06:52 elendrys but when I extend an extra time, the previous size is seen instantaneously by the guest
16:07:10 elendrys it's just 1 extend late
16:17:20 sean-k-mooney well nova as far as i am aware will not need to notify qemu but perhaps we do
16:20:15 clarkb and those are sizes reported by lsblk?
16:20:35 sean-k-mooney unfortunetly i dont currently have the low level details loaded in my brain
16:20:39 sean-k-mooney clarkb: ya
16:20:55 sean-k-mooney the size of the block device repoted to the guest
16:21:12 sean-k-mooney which should eventualy be show in lsblk on the host and guest
16:21:44 clarkb ya I'm wondering if the guest does an explicit check via lsblk if it sees correct info and the only problem is in propogating the not on demand info
16:25:09 elendrys If it is better I can open the discussion on the mailing list
16:26:29 kgube_ the size seems to come from os-brick LinuxSCSI which calls "blockdev --getsize64" after a rescan
16:31:39 kgube_ so maybe the rescan in LinuxSCSI.extend_volume is not working
16:32:28 kgube_ elendrys, have you tried it with debug logging?
16:33:16 elendrys nope, but I can do that quickly
16:35:29 kgube_ there should be "Starting size: " and "volume size after scsi device rescan " in the log
16:36:37 elendrys Starting size: 76235669504 extend_volume /usr/lib/python3/dist-packages/os_brick/initiator/linuxscsi.py:555
16:36:56 elendrys volume size after scsi device rescan 80530636800 extend_volume
16:39:03 elendrys hem
16:39:12 elendrys - /dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de has shown up. wait_for_path
16:39:21 elendrys mpath(/dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de) current size 76235669504
16:39:29 elendrys mpath(/dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de) new size 76235669504
16:40:04 elendrys Extend iSCSI Volume /dev/dm-28; new_size=76235669504 extend_volume
16:40:11 elendrys Resizing target device /dev/dm-28 to 76235669504 _resize_attached_volume
16:40:35 elendrys I think that it rely on multipath but it is not refreshed quickly enough
16:43:01 kgube_ yeah that seems to come from the multipath code
16:43:03 kgube_ https://github.com/openstack/os-brick/blob/master/os_brick/initiator/linuxscsi.py#L639
16:45:00 elendrys it explains why I see the previous size in the guest
16:47:18 kgube_ I don't know enough about scsi to have any idea what to do about this, though
16:47:47 elendrys Should I fill an issue on the tracker or just ask the mailing list ?
16:48:05 elendrys Thank you for your time :)
16:53:14 kgube_ hm, I'm not sure. I'm very new to Nova dev
16:55:14 elodilles sean-k-mooney: sorry for pinging, but if you have some free minutes to review this old stable backport, that would be awesome: https://review.opendev.org/c/openstack/nova/+/764504
16:55:27 elodilles (it has a lot of +1 on it)
16:57:49 kgube_ elendrys, maybe try the mailing list first there is also a lot of openstack users reading it
16:58:12 kgube_ elendrys, maybe someone has seen this before and found a workaround
17:08:58 sean-k-mooney oh ya that sure happy to see that merge
17:09:41 sean-k-mooney elodilles: give me 5 mins to finsih somehting and ill review it then
17:12:20 elodilles thanks in advance o/
17:27:35 sean-k-mooney elodilles: once that merges im happy to continue moving this back to train so ill keep an eye out for that over then few days
17:28:12 sean-k-mooney elodilles: how are the stabel gates at the moment have they been affected by the tempest tox issue on master or the tox 4.0 issues
17:32:33 elodilles sean-k-mooney: thanks, i'll try to keep an eye on them, too :) good question about the gate, I haven't checked that yet :S but i guess we will see :/
#openstack-nova - 2022-12-09
06:03:04 opendevreview Hiroki Narukawa proposed openstack/nova master: libvirt: retry libvirt connection on live_migration_monitor https://review.opendev.org/c/openstack/nova/+/867077
08:16:31 opendevreview Bence Romsics proposed openstack/nova master: doc: soft delete and shadow tables https://review.opendev.org/c/openstack/nova/+/867001
08:52:03 kgube sean-k-mooney, sorry for pinging, but have you had a chance to look at this yet? https://review.opendev.org/c/openstack/nova-specs/+/855490
08:52:16 kgube sean-k-mooney, whoami-rajat said he would like to avoid investing time in reviewing the cinder spec if it isn't clear that that the new direction of the change is acceptable for Nova
08:54:50 kgube sean-k-mooney, but cinder has spec freeze in a week so there is not alot of time left
09:32:19 whoami-rajat kgube, sean-k-mooney yeah, we already had reviewed the original spec several times and it was close to approval until the direction changed from nova side, it would be good to know if the new spec is the correct way forward
09:43:39 elendrys kgube: After several tests the enhanced code is present only in Zed (Ussuri here), and backporting it does not quite work either because the map/resize functions awaits either for the multipath name or the device mapper location, and the wwn is passed as argument
09:44:49 elendrys my conclusion may be that I should use an older version of multipathd, i'll see if someone reply on the list

Earlier   Later