| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-12-08 | |||
| 08:00:22 | opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. Signed-off-by: yangzhipeng |
|
| 08:03:18 | opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. https://review.opendev.org/c/openstack/nova/+/865362 | |
| 08:04:56 | opendevreview | norman shen proposed openstack/nova master: Skip deleting instance info for same host migration https://review.opendev.org/c/openstack/nova/+/866521 | |
| 09:48:55 | opendevreview | yzp proposed openstack/nova master: Refresh instance when init instance in rebuilding https://review.opendev.org/c/openstack/nova/+/866960 | |
| 10:27:12 | opendevreview | yzp proposed openstack/nova master: Refresh instance when init instance. https://review.opendev.org/c/openstack/nova/+/866960 | |
| 13:24:38 | gibi | do we have a blocked nova gate? https://zuul.opendev.org/t/openstack/build/32c3c4a56eec4d7eac956a629abdee362 | |
| 13:24:54 | gibi | it looks we have constant failure in https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack/nova | |
| 13:25:04 | gibi | since yesterday | |
| 13:25:20 | gibi | also in https://zuul.opendev.org/t/openstack/builds?job_name=nova-grenade-multinode&project=openstack/nova | |
| 13:26:05 | gibi | "venv: failed with tempest is not allowed, use allowlist_externals to allow it" | |
| 13:47:36 | sean-k-mooney | that sound like we are not installing tempet in the venv properly | |
| 13:51:22 | gibi | yeah | |
| 13:51:35 | gibi | grenade fails a bit differently but maybe related too | |
| 13:52:54 | gibi | this seems related https://review.opendev.org/c/openstack/tempest/+/865314/1/tox.ini | |
| 13:53:16 | gibi | but merged couple weeks ago | |
| 13:53:23 | sean-k-mooney | ya so there was a mail about that | |
| 13:53:38 | sean-k-mooney | the allowlist_external=* is problematic | |
| 13:57:53 | sean-k-mooney | gibi: https://lists.openstack.org/pipermail/openstack-discuss/2022-November/031343.html | |
| 13:59:16 | sean-k-mooney | clarkb: ^ so that tempest fix may have broken grenade? | |
| 14:02:35 | sean-k-mooney | gibi: my guess is the tempest venv is not being recreated | |
| 14:03:00 | sean-k-mooney | so on zed it would have had allowlist_external=* and used global pip | |
| 14:03:31 | sean-k-mooney | as a result tempest was likely insalled outside of the venv? | |
| 14:03:45 | sean-k-mooney | and then with the master tox file it would have failed? | |
| 14:04:05 | gibi | it fails in nova-ceph-multistore too so it is not grenade specific | |
| 14:04:12 | sean-k-mooney | not sure if that actully what is happening as i tough we used master tempetst regradels of the branch but i would guess its soemthign like that | |
| 14:04:26 | sean-k-mooney | oh ok | |
| 14:04:46 | sean-k-mooney | so may somethign related to our jobs | |
| 14:05:38 | frickler | yes, that allowlist needs to include tempest | |
| 14:06:02 | sean-k-mooney | frickler: im not sure it should | |
| 14:06:09 | sean-k-mooney | tempest shoudl be installed in teh venv | |
| 14:06:21 | sean-k-mooney | we shoudl not be using tempest form teh host right? | |
| 14:06:41 | sean-k-mooney | we install tempest in a venv because it is often not the same version of openstack | |
| 14:06:54 | gibi | this is where we fail https://github.com/openstack/devstack/blob/master/lib/tempest#L731 | |
| 14:06:56 | sean-k-mooney | and we do it that way to avoid the possibel depency conflcit that woudl create | |
| 14:07:16 | sean-k-mooney | so tempest should __not__ be in the allowlist | |
| 14:07:41 | sean-k-mooney | ya so that shoudl be using tempst form the venv | |
| 14:08:09 | sean-k-mooney | shoudl it be tox -evenv-tempest https://github.com/openstack/devstack/blob/master/lib/tempest#L709 | |
| 14:08:14 | sean-k-mooney | instead of venv | |
| 14:08:47 | sean-k-mooney | tox -evenv-tempest seams to be what we use on the other lines | |
| 14:08:57 | gibi | hm, good point | |
| 14:09:02 | gibi | that seems like the wrong env | |
| 14:09:40 | gibi | I will push a patch | |
| 14:09:41 | frickler | oh, cool, so this found an actual bug, nice | |
| 14:11:33 | ralonsoh | sean-k-mooney, thanks! I saw this issue in out CI too | |
| 14:11:38 | ralonsoh | our* | |
| 14:14:41 | gibi | devstack fix is up https://review.opendev.org/c/openstack/devstack/+/866997 | |
| 14:28:38 | opendevreview | Konrad Gube proposed openstack/nova-specs master: Use extend volume completion action https://review.opendev.org/c/openstack/nova-specs/+/855490 | |
| 14:44:19 | opendevreview | Bence Romsics proposed openstack/nova master: doc: soft delete and shadow tables https://review.opendev.org/c/openstack/nova/+/867001 | |
| 15:06:12 | opendevreview | Bence Romsics proposed openstack/nova master: doc: soft delete and shadow tables https://review.opendev.org/c/openstack/nova/+/867001 | |
| 15:19:39 | elendrys | Hello here, is this channel open to nova related questions or is it dedicated to developers ? | |
| 15:45:00 | frickler | elendrys: this is a bit of a grey area, #openstack is what is meant for general questions, but the chance to get an answer there is low, so you might as well try here directly | |
| 15:47:14 | fungi | also the openstack-discuss@lists.openstack.org mailing list has a much higher chance of getting you an answer eventually | |
| 15:54:07 | elendrys | Ok thank you | |
| 15:54:35 | sean-k-mooney | elendrys: did you have a question in general | |
| 15:55:00 | sean-k-mooney | while the channel is not for supprot we will try an point you in the right direction in genreal | |
| 15:55:44 | elendrys | Yes | |
| 15:56:23 | elendrys | I have servers with cinder volumes, backed in PureStorage array over iscsi | |
| 15:57:21 | sean-k-mooney | ok so the vm is using host mount iscsi block devices passthroug to the vms | |
| 15:57:28 | elendrys | When i resize volumes attached to running servers, I can see in nova's log that it rescans the array to get the new size when cinder notifies | |
| 15:57:55 | elendrys | But in the vm, the size always have 1 notification late size | |
| 15:58:23 | elendrys | I mean, a 10Gb vol expended to 20 stills show as 10, but if I extend to 30, it shows 20 | |
| 15:58:49 | elendrys | I can't get why (probably libvirt) it is missing something | |
| 15:58:54 | sean-k-mooney | hum interesting. | |
| 15:59:13 | sean-k-mooney | am im not sure this is libvirt/qemu related | |
| 15:59:22 | elendrys | If I stop then start the server it shows 30 as intended | |
| 15:59:28 | sean-k-mooney | i woudl suppect this is more likely to be related to iscsid or simialr | |
| 15:59:59 | sean-k-mooney | elendrys: did you check on the host if the host kernel sees the correct volume size | |
| 16:00:30 | sean-k-mooney | you should be able to find the dev path form the vm xml and check with lsblk or a similar too to see if it sees the correct current size | |
| 16:01:06 | sean-k-mooney | if the hosts sees the correct size then the issue is likely at the qemu level | |
| 16:01:45 | sean-k-mooney | elendrys: i dont think libvirt is invovled for voulme resize in this case | |
| 16:02:22 | elendrys | It does | |
| 16:05:33 | sean-k-mooney | im wondering is there anything you can do at the gues os level | |
| 16:05:38 | elendrys | It is queued somewhere by nova | |
| 16:06:25 | elendrys | I tried to force scsci bus scan but it doesn't change | |
| 16:06:52 | elendrys | but when I extend an extra time, the previous size is seen instantaneously by the guest | |
| 16:07:10 | elendrys | it's just 1 extend late | |
| 16:17:20 | sean-k-mooney | well nova as far as i am aware will not need to notify qemu but perhaps we do | |
| 16:20:15 | clarkb | and those are sizes reported by lsblk? | |
| 16:20:35 | sean-k-mooney | unfortunetly i dont currently have the low level details loaded in my brain | |
| 16:20:39 | sean-k-mooney | clarkb: ya | |
| 16:20:55 | sean-k-mooney | the size of the block device repoted to the guest | |
| 16:21:12 | sean-k-mooney | which should eventualy be show in lsblk on the host and guest | |
| 16:21:44 | clarkb | ya I'm wondering if the guest does an explicit check via lsblk if it sees correct info and the only problem is in propogating the not on demand info | |
| 16:25:09 | elendrys | If it is better I can open the discussion on the mailing list | |
| 16:26:29 | kgube_ | the size seems to come from os-brick LinuxSCSI which calls "blockdev --getsize64" after a rescan | |
| 16:31:39 | kgube_ | so maybe the rescan in LinuxSCSI.extend_volume is not working | |
| 16:32:28 | kgube_ | elendrys, have you tried it with debug logging? | |
| 16:33:16 | elendrys | nope, but I can do that quickly | |
| 16:35:29 | kgube_ | there should be "Starting size: " and "volume size after scsi device rescan " in the log | |
| 16:36:37 | elendrys | Starting size: 76235669504 extend_volume /usr/lib/python3/dist-packages/os_brick/initiator/linuxscsi.py:555 | |
| 16:36:56 | elendrys | volume size after scsi device rescan 80530636800 extend_volume | |
| 16:39:03 | elendrys | hem | |
| 16:39:12 | elendrys | - /dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de has shown up. wait_for_path | |
| 16:39:21 | elendrys | mpath(/dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de) current size 76235669504 | |
| 16:39:29 | elendrys | mpath(/dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de) new size 76235669504 | |
| 16:40:04 | elendrys | Extend iSCSI Volume /dev/dm-28; new_size=76235669504 extend_volume | |
| 16:40:11 | elendrys | Resizing target device /dev/dm-28 to 76235669504 _resize_attached_volume | |
| 16:40:35 | elendrys | I think that it rely on multipath but it is not refreshed quickly enough | |
| 16:43:01 | kgube_ | yeah that seems to come from the multipath code | |
| 16:43:03 | kgube_ | https://github.com/openstack/os-brick/blob/master/os_brick/initiator/linuxscsi.py#L639 | |