| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-17 | |||
| 15:30:06 | openstack | Launchpad bug 1918945 in OpenStack Compute (nova) "Nova API fails with 500s when called with non-project-scoped keystone tokens" [Undecided,New] | |
| 15:31:24 | gmann | gibi: yes this is not regression, it is as per keystone current design for system scope. we will discuss about that more before and then in PTG. i have added this in etherpad L220 https://etherpad.opendev.org/p/nova-xena-ptg | |
| 15:32:27 | gmann | gibi: may be we should add this as cross project topic with keystone. lbragstad dansmith ? | |
| 15:32:57 | gmann | gibi: lyarwood ack 780333, I will check after my breakfast | |
| 15:33:08 | dansmith | it's fodder for the conversation about system scope for sure | |
| 15:33:23 | dansmith | figured that one that lbragstad has the etherpad for is probably cross-project already | |
| 15:33:28 | elod | melwitt: sorry, i did miss it somehow :S ...but now it's +2+W'd | |
| 15:33:45 | lbragstad | i haven't created any etherpads, yet | |
| 15:34:02 | lbragstad | just the one for summarizing the issue | |
| 15:34:16 | lbragstad | but i didn't target it to be a topic for the PTG officially or anything | |
| 15:35:11 | melwitt | elod: thanks! question for you, how did you figure out that stable/pike was failing for lack of https://review.opendev.org/c/openstack/nova/+/780852 ? just curious bc I can't figure it out. the error I see in nova-live-migration there is "Error EINVAL: bad entity name" and I was curious how you knew what was happening | |
| 15:35:20 | dansmith | lbragstad: https://etherpad.opendev.org/p/nova-xena-ptg L220, I thought that was you but I guess it was sean | |
| 15:35:54 | lbragstad | oh - cool | |
| 15:36:34 | lbragstad | fwiw - i tried summarizing everything in a ml post and i sent it to the operator list | |
| 15:36:54 | lbragstad | i'd like to know if they have any opinions on how this should work | |
| 15:36:55 | lbragstad | http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021131.html | |
| 15:38:03 | dansmith | yeah, I have that marked, but haven't seen any replies yet | |
| 15:38:12 | elod | melwitt: well, I just saw, that it is ceph related, and since we just pinned ceph in newer stable branches I gave it a try | |
| 15:38:33 | melwitt | elod: ahh :) ok cool | |
| 15:44:58 | gmann | dansmith: lbragstad I added it in nova L220 but we can move to cross project section | |
| 15:45:10 | dansmith | ack | |
| 15:49:16 | gmann | dansmith: gibi lbragstad done, moved to L47 - https://etherpad.opendev.org/p/nova-xena-ptg | |
| 15:49:17 | elod | melwitt: and this failure (without the pin) also suggests that we are using ceph octopus in stable/pike (which is now pinned to nautilus in queens and newer): https://zuul.opendev.org/t/openstack/build/ea688ac7f8044e70b1503a517fc36db3/log/job-output.txt#4278 | |
| 15:50:04 | melwitt | elod: ohh good spot, thank you | |
| 15:55:36 | elod | melwitt: thank you too for the +2 :) | |
| 15:57:42 | melwitt | elod: np, thank you for fixing it! | |
| 15:57:52 | gibi | gmann, dansmith, lbragstad: ack, thanks. I will check with the keystone team about a timeslot for that cross project discussion | |
| 15:59:01 | melwitt | elod: btw, I opened the result on the pinned ceph version patch for nova-grenade-live-migration and see it's failing with "ERROR: This script does not work on Python 2.7 The minimum supported Python version is 3.6. Please use https://bootstrap.pypa.io/pip/2.7/get-pip.py instead." https://zuul.opendev.org/t/openstack/build/6583ef90f2954a57934781e31d17c68d/log/logs/grenade.sh.txt#2656 do you recall a fix for that going around in the | |
| 15:59:01 | melwitt | past? I can't find | |
| 15:59:50 | gibi | gmann: thanks for confirming that it is not a regression | |
| 16:01:04 | ignaziocassano | Hello All, please anyone can help me with live migrations on queens? Only for some vm live migration does not work. When the vm tries to start on destination host, in instance log I read: 2021-03-15T14:16:12.593707Z qemu-kvm: Failed to load virtio/extra_state:extra_state | |
| 16:02:44 | ignaziocassano | If I hard reset the vm it starts and then I do not got any live migration errors anymore | |
| 16:03:57 | elod | melwitt: hmmm, that failure is related to this: https://review.opendev.org/q/topic:%22get-pip-py27%22 | |
| 16:04:07 | elod | melwitt: i'll try to fix it | |
| 16:04:59 | melwitt | elod: oh, derp, how did I miss that in my gerrit search 😆 yeah, I thought it sounded familiar like something that got fixed recently. I'll lookout for your upcoming patch :) | |
| 16:05:45 | ignaziocassano | I wrote an email to openstack-discuss two days ago but I have not received any suggestion yet. Not found any bug related to to this issue | |
| 16:06:18 | elod | melwitt: correction: I won't be able to :/ as that is because it's ocata, and ocata cannot be fixed if I remember correctly... some catch 22... | |
| 16:07:28 | lyarwood | time to nuke the grenade job on pike then I guess? | |
| 16:07:53 | melwitt | elod: you mean pike? it's on pike but it's non-voting, perhaps for similar reasons | |
| 16:08:11 | elod | melwitt: stestr and other contradicting requirements :S (this was my trial: https://review.opendev.org/777614 ) | |
| 16:08:29 | elod | melwitt: no, ocata, as grenade on pike starts with ocata install | |
| 16:09:34 | melwitt | elod: oh sorry 😓 I didn't know that. ok, good to know | |
| 16:09:42 | elod | lyarwood: yes, actually, I was thinking on that, too (I hoped I could fix it somehow, but 1. seems not really possible; 2. had another priority all the time... o:)) | |
| 16:10:30 | melwitt | yeah, agreed if it can't be fixed then we can save CI resources | |
| 16:11:40 | elod | melwitt: sure. actually not so many patches arrive to pike, but still, it would save some resource :) | |
| 16:12:00 | melwitt | elod: oh, heh true | |
| 16:45:02 | kukacz | hi, is instance rebuild with volume backed instance and selecting same image supposed to actually rebuild the volume to state of the image? | |
| 16:47:09 | lyarwood | kukacz: https://docs.openstack.org/api-ref/compute/?expanded=rebuild-server-rebuild-action-detail & https://bugs.launchpad.net/nova/+bug/1482040 | |
| 16:47:11 | openstack | Launchpad bug 1482040 in OpenStack Compute (nova) pike "rebuild volume-backed instance with image does not replace image in root disk" [Medium,Confirmed] | |
| 16:47:26 | lyarwood | kukacz: it's a known limitation that the volume isn't rebuilt using the new image | |
| 16:47:42 | lyarwood | kukacz: iirc we had a spec agreed to look at correcting this but the changes never landed | |
| 16:48:27 | lyarwood | https://specs.openstack.org/openstack/nova-specs/specs/train/approved/volume-backed-server-rebuild.html | |
| 16:53:49 | lyarwood | melwitt: https://review.opendev.org/c/openstack/nova/+/765769 https://review.opendev.org/c/openstack/nova/+/773443 https://review.opendev.org/c/openstack/nova/+/757306 - would you mind also hitting these? Apologies but my backlog is getting way to big now so I need to start pinging people for reviews. | |
| 16:53:50 | kukacz | lyarwood: clear, thanks! | |
| 16:54:22 | kukacz | somehow I thought that this was working when using identical image | |
| 16:54:28 | kukacz | my bad | |
| 17:23:13 | openstackgerrit | Elod Illes proposed openstack/nova stable/pike: [stable-only] Move grenade jobs to experimental https://review.opendev.org/c/openstack/nova/+/781134 | |
| 17:33:54 | gibi | did we fixed some pure PCI passthrough evacuate issue since victoria? I cannot reproduce this on master https://bugs.launchpad.net/nova/+bug/1910663 | |
| 17:33:55 | openstack | Launchpad bug 1910663 in OpenStack Compute (nova) "pci device duplicate attach after intance evacuated" [Undecided,New] | |
| 17:34:38 | tosky | time to abandon stable/ocata for real? | |
| 17:34:51 | tosky | (looking at that change above ^^) | |
| 18:00:58 | openstackgerrit | Lee Yarwood proposed openstack/nova master: block_device: Use initialize APIs to refresh when reported as idempotent https://review.opendev.org/c/openstack/nova/+/720769 | |
| 18:01:00 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Reject requests to commit active snapshot of an inactive instance https://review.opendev.org/c/openstack/nova/+/781138 | |
| 18:01:01 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Add devstack-plugin-nfs-tempest-full to the check queue https://review.opendev.org/c/openstack/nova/+/781139 | |
| 18:06:31 | openstackgerrit | Merged openstack/nova stable/stein: [stable-only] Specify IPv6 CIDR in tempest-slow-py3 https://review.opendev.org/c/openstack/nova/+/780991 | |
| 18:06:50 | openstackgerrit | Merged openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | |
| 18:23:33 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova stable/stein: Use subqueryload() instead of joinedload() for (system_)metadata https://review.opendev.org/c/openstack/nova/+/761812 | |
| 18:24:04 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova stable/stein: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/757085 | |
| 18:24:06 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova stable/stein: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780784 | |
| 18:56:09 | openstackgerrit | Merged openstack/nova stable/pike: [stable-only] gate: Pin CEPH_RELEASE to nautilus in LM hook https://review.opendev.org/c/openstack/nova/+/780852 | |
| 19:33:30 | lbragstad | gmann ping - i have a fresh devstack up with nova configured to enforce scope and enforce the new default policies | |
| 19:33:40 | lbragstad | but i'm still able to list compute hypervisors as a project-admin | |
| 19:33:49 | lbragstad | did i miss a configuration switch somewhere? | |
| 19:34:51 | gmann | lbragstad: no that should be enough. | |
| 19:35:18 | gmann | gibi: lbragstad stephenfin i think we should go with 400 return code which is most close and mentioned in api-wg guidelines too https://review.opendev.org/c/openstack/nova/+/780333 | |
| 19:35:25 | gmann | lyarwood: ^^ | |
| 19:35:32 | gmann | lbragstad: sorry, wrong mention. | |
| 19:36:44 | lbragstad | gmann no worries - http://paste.openstack.org/show/803682/ is what i'm seeing | |
| 19:36:56 | lbragstad | line 49 shows the project-scoped token response | |
| 19:38:22 | lbragstad | mm - nevermind... i used test-config to setup nova.conf, and n-api must not have been bounced after the config update | |
| 19:38:48 | gmann | lbragstad: it should not allowed | |
| 19:39:12 | lbragstad | yep - you're right, i restarted the nova-api and it looks like it picked up the oslo_policy configs | |
| 19:39:23 | gmann | ohk, for policy change n-api does not need to restart but yes for config change it need | |
| 19:39:46 | gmann | lbragstad: nice. | |
| 22:29:34 | melwitt | lyarwood: no apology needed, will hit those. unrelated, curious if you've seen this before for gate failure in swap volume "Failure rebasing volume /dev/sdb on vdb.: libvirt.libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/sdb': Permission denied" https://zuul.opendev.org/t/openstack/build/c9515f1edc774e74a84e91a7dee50074/log/compute1/logs/screen-n-cpu.txt#2138 seeing it only on stein | |
| 22:29:35 | melwitt | and train | |
| 23:26:23 | eandersson | ops melwitt missed that you had already re-checked the rocky patch :D | |
| 23:26:36 | eandersson | *stein | |
| 23:26:56 | melwitt | eandersson: np, maybe it will give us better luck :) | |
| 23:27:20 | eandersson | fingerscrossed! | |
| #openstack-nova - 2021-03-18 | |||
| 00:37:37 | melwitt | same root failure during create of cinder resources on stable/stein "libvirt.libvirtError: operation failed: 'serial' is deprecated, please use the corresponding option of '-device' instead Could not open '/dev/sdb': Permission denied" https://zuul.opendev.org/t/openstack/build/e1050a661fec47b49d6d7babc89d71d3/log/logs/screen-n-cpu.txt#3267 | |
| 00:37:57 | melwitt | why can't it open /dev/sdb -_- | |
| 00:39:07 | melwitt | rather, why is it Permission Denied. argh. | |
| 00:42:38 | melwitt | opened https://bugs.launchpad.net/openstack-gate/+bug/1919920 fwiw | |
| 00:42:40 | openstack | Launchpad bug 1919920 in OpenStack-Gate "volume related tests sometimes fail with " Could not open '/dev/sdb': Permission denied" on stable/train and stable/stein" [Undecided,New] | |
| 05:40:35 | openstackgerrit | Merged openstack/nova stable/ussuri: libvirt: Increase incremental and max sleep time during device detach https://review.opendev.org/c/openstack/nova/+/757306 | |
| 06:48:41 | openstackgerrit | Merged openstack/nova stable/pike: [stable-only] Move grenade jobs to experimental https://review.opendev.org/c/openstack/nova/+/781134 | |
| 07:02:36 | eandersson | The whole settings in the database is one of the hardest things for new operators to troubleshoot | |
| 07:03:04 | eandersson | When changing the names of rmq server there seems to be no errors in the logs for some reasons | |
| 07:03:20 | eandersson | And if operators are not aware of that "feature" it is probably the last place they will oook | |