| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-12-07 | |||
| 18:59:26 | melwitt | I ran it locally a bunch and it was very consistent | |
| 18:59:44 | sean-k-mooney | weird | |
| 18:59:47 | melwitt | I didn't run it in a long running loop but just while I was messing with it I ran it several times | |
| 19:00:41 | melwitt | if I left it running in a loop overnight, maybe it would pass at some point 😂 | |
| 19:06:42 | sean-k-mooney | thats a lower pass rate then is desireable in ci | |
| 19:06:55 | sean-k-mooney | so i think we are good with your change | |
| 19:06:56 | melwitt | a little | |
| #openstack-nova - 2022-12-08 | |||
| 06:46:21 | opendevreview | yangzhipeng proposed openstack/nova master: when evacuate is performing, and restart compute node, if get instance info early, the instance state is not latest. this will reset instance task to error incorrectly, so refresh instance when modify instance state. https://review.opendev.org/c/openstack/nova/+/866960 | |
| 06:53:50 | opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. https://review.opendev.org/c/openstack/nova/+/865362 | |
| 07:54:21 | opendevreview | yzp proposed openstack/nova master: Refresh instance when init instance in rebuilding https://review.opendev.org/c/openstack/nova/+/866960 | |
| 07:58:33 | opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. Signed-off-by: yangzhipeng |
|
| 07:58:33 | opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. Signed-off-by: yangzhipeng |
|
| 08:00:22 | opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. Signed-off-by: yangzhipeng |
|
| 08:03:18 | opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. https://review.opendev.org/c/openstack/nova/+/865362 | |
| 08:04:56 | opendevreview | norman shen proposed openstack/nova master: Skip deleting instance info for same host migration https://review.opendev.org/c/openstack/nova/+/866521 | |
| 09:48:55 | opendevreview | yzp proposed openstack/nova master: Refresh instance when init instance in rebuilding https://review.opendev.org/c/openstack/nova/+/866960 | |
| 10:27:12 | opendevreview | yzp proposed openstack/nova master: Refresh instance when init instance. https://review.opendev.org/c/openstack/nova/+/866960 | |
| 13:24:38 | gibi | do we have a blocked nova gate? https://zuul.opendev.org/t/openstack/build/32c3c4a56eec4d7eac956a629abdee362 | |
| 13:24:54 | gibi | it looks we have constant failure in https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack/nova | |
| 13:25:04 | gibi | since yesterday | |
| 13:25:20 | gibi | also in https://zuul.opendev.org/t/openstack/builds?job_name=nova-grenade-multinode&project=openstack/nova | |
| 13:26:05 | gibi | "venv: failed with tempest is not allowed, use allowlist_externals to allow it" | |
| 13:47:36 | sean-k-mooney | that sound like we are not installing tempet in the venv properly | |
| 13:51:22 | gibi | yeah | |
| 13:51:35 | gibi | grenade fails a bit differently but maybe related too | |
| 13:52:54 | gibi | this seems related https://review.opendev.org/c/openstack/tempest/+/865314/1/tox.ini | |
| 13:53:16 | gibi | but merged couple weeks ago | |
| 13:53:23 | sean-k-mooney | ya so there was a mail about that | |
| 13:53:38 | sean-k-mooney | the allowlist_external=* is problematic | |
| 13:57:53 | sean-k-mooney | gibi: https://lists.openstack.org/pipermail/openstack-discuss/2022-November/031343.html | |
| 13:59:16 | sean-k-mooney | clarkb: ^ so that tempest fix may have broken grenade? | |
| 14:02:35 | sean-k-mooney | gibi: my guess is the tempest venv is not being recreated | |
| 14:03:00 | sean-k-mooney | so on zed it would have had allowlist_external=* and used global pip | |
| 14:03:31 | sean-k-mooney | as a result tempest was likely insalled outside of the venv? | |
| 14:03:45 | sean-k-mooney | and then with the master tox file it would have failed? | |
| 14:04:05 | gibi | it fails in nova-ceph-multistore too so it is not grenade specific | |
| 14:04:12 | sean-k-mooney | not sure if that actully what is happening as i tough we used master tempetst regradels of the branch but i would guess its soemthign like that | |
| 14:04:26 | sean-k-mooney | oh ok | |
| 14:04:46 | sean-k-mooney | so may somethign related to our jobs | |
| 14:05:38 | frickler | yes, that allowlist needs to include tempest | |
| 14:06:02 | sean-k-mooney | frickler: im not sure it should | |
| 14:06:09 | sean-k-mooney | tempest shoudl be installed in teh venv | |
| 14:06:21 | sean-k-mooney | we shoudl not be using tempest form teh host right? | |
| 14:06:41 | sean-k-mooney | we install tempest in a venv because it is often not the same version of openstack | |
| 14:06:54 | gibi | this is where we fail https://github.com/openstack/devstack/blob/master/lib/tempest#L731 | |
| 14:06:56 | sean-k-mooney | and we do it that way to avoid the possibel depency conflcit that woudl create | |
| 14:07:16 | sean-k-mooney | so tempest should __not__ be in the allowlist | |
| 14:07:41 | sean-k-mooney | ya so that shoudl be using tempst form the venv | |
| 14:08:09 | sean-k-mooney | shoudl it be tox -evenv-tempest https://github.com/openstack/devstack/blob/master/lib/tempest#L709 | |
| 14:08:14 | sean-k-mooney | instead of venv | |
| 14:08:47 | sean-k-mooney | tox -evenv-tempest seams to be what we use on the other lines | |
| 14:08:57 | gibi | hm, good point | |
| 14:09:02 | gibi | that seems like the wrong env | |
| 14:09:40 | gibi | I will push a patch | |
| 14:09:41 | frickler | oh, cool, so this found an actual bug, nice | |
| 14:11:33 | ralonsoh | sean-k-mooney, thanks! I saw this issue in out CI too | |
| 14:11:38 | ralonsoh | our* | |
| 14:14:41 | gibi | devstack fix is up https://review.opendev.org/c/openstack/devstack/+/866997 | |
| 14:28:38 | opendevreview | Konrad Gube proposed openstack/nova-specs master: Use extend volume completion action https://review.opendev.org/c/openstack/nova-specs/+/855490 | |
| 14:44:19 | opendevreview | Bence Romsics proposed openstack/nova master: doc: soft delete and shadow tables https://review.opendev.org/c/openstack/nova/+/867001 | |
| 15:06:12 | opendevreview | Bence Romsics proposed openstack/nova master: doc: soft delete and shadow tables https://review.opendev.org/c/openstack/nova/+/867001 | |
| 15:19:39 | elendrys | Hello here, is this channel open to nova related questions or is it dedicated to developers ? | |
| 15:45:00 | frickler | elendrys: this is a bit of a grey area, #openstack is what is meant for general questions, but the chance to get an answer there is low, so you might as well try here directly | |
| 15:47:14 | fungi | also the openstack-discuss@lists.openstack.org mailing list has a much higher chance of getting you an answer eventually | |
| 15:54:07 | elendrys | Ok thank you | |
| 15:54:35 | sean-k-mooney | elendrys: did you have a question in general | |
| 15:55:00 | sean-k-mooney | while the channel is not for supprot we will try an point you in the right direction in genreal | |
| 15:55:44 | elendrys | Yes | |
| 15:56:23 | elendrys | I have servers with cinder volumes, backed in PureStorage array over iscsi | |
| 15:57:21 | sean-k-mooney | ok so the vm is using host mount iscsi block devices passthroug to the vms | |
| 15:57:28 | elendrys | When i resize volumes attached to running servers, I can see in nova's log that it rescans the array to get the new size when cinder notifies | |
| 15:57:55 | elendrys | But in the vm, the size always have 1 notification late size | |
| 15:58:23 | elendrys | I mean, a 10Gb vol expended to 20 stills show as 10, but if I extend to 30, it shows 20 | |
| 15:58:49 | elendrys | I can't get why (probably libvirt) it is missing something | |
| 15:58:54 | sean-k-mooney | hum interesting. | |
| 15:59:13 | sean-k-mooney | am im not sure this is libvirt/qemu related | |
| 15:59:22 | elendrys | If I stop then start the server it shows 30 as intended | |
| 15:59:28 | sean-k-mooney | i woudl suppect this is more likely to be related to iscsid or simialr | |
| 15:59:59 | sean-k-mooney | elendrys: did you check on the host if the host kernel sees the correct volume size | |
| 16:00:30 | sean-k-mooney | you should be able to find the dev path form the vm xml and check with lsblk or a similar too to see if it sees the correct current size | |
| 16:01:06 | sean-k-mooney | if the hosts sees the correct size then the issue is likely at the qemu level | |
| 16:01:45 | sean-k-mooney | elendrys: i dont think libvirt is invovled for voulme resize in this case | |
| 16:02:22 | elendrys | It does | |
| 16:05:33 | sean-k-mooney | im wondering is there anything you can do at the gues os level | |
| 16:05:38 | elendrys | It is queued somewhere by nova | |
| 16:06:25 | elendrys | I tried to force scsci bus scan but it doesn't change | |
| 16:06:52 | elendrys | but when I extend an extra time, the previous size is seen instantaneously by the guest | |
| 16:07:10 | elendrys | it's just 1 extend late | |
| 16:17:20 | sean-k-mooney | well nova as far as i am aware will not need to notify qemu but perhaps we do | |
| 16:20:15 | clarkb | and those are sizes reported by lsblk? | |
| 16:20:35 | sean-k-mooney | unfortunetly i dont currently have the low level details loaded in my brain | |
| 16:20:39 | sean-k-mooney | clarkb: ya | |
| 16:20:55 | sean-k-mooney | the size of the block device repoted to the guest | |
| 16:21:12 | sean-k-mooney | which should eventualy be show in lsblk on the host and guest | |
| 16:21:44 | clarkb | ya I'm wondering if the guest does an explicit check via lsblk if it sees correct info and the only problem is in propogating the not on demand info | |
| 16:25:09 | elendrys | If it is better I can open the discussion on the mailing list | |
| 16:26:29 | kgube_ | the size seems to come from os-brick LinuxSCSI which calls "blockdev --getsize64" after a rescan | |
| 16:31:39 | kgube_ | so maybe the rescan in LinuxSCSI.extend_volume is not working | |
| 16:32:28 | kgube_ | elendrys, have you tried it with debug logging? | |
| 16:33:16 | elendrys | nope, but I can do that quickly | |