| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-17 | |||
| 15:38:03 | dansmith | yeah, I have that marked, but haven't seen any replies yet | |
| 15:38:12 | elod | melwitt: well, I just saw, that it is ceph related, and since we just pinned ceph in newer stable branches I gave it a try | |
| 15:38:33 | melwitt | elod: ahh :) ok cool | |
| 15:44:58 | gmann | dansmith: lbragstad I added it in nova L220 but we can move to cross project section | |
| 15:45:10 | dansmith | ack | |
| 15:49:16 | gmann | dansmith: gibi lbragstad done, moved to L47 - https://etherpad.opendev.org/p/nova-xena-ptg | |
| 15:49:17 | elod | melwitt: and this failure (without the pin) also suggests that we are using ceph octopus in stable/pike (which is now pinned to nautilus in queens and newer): https://zuul.opendev.org/t/openstack/build/ea688ac7f8044e70b1503a517fc36db3/log/job-output.txt#4278 | |
| 15:50:04 | melwitt | elod: ohh good spot, thank you | |
| 15:55:36 | elod | melwitt: thank you too for the +2 :) | |
| 15:57:42 | melwitt | elod: np, thank you for fixing it! | |
| 15:57:52 | gibi | gmann, dansmith, lbragstad: ack, thanks. I will check with the keystone team about a timeslot for that cross project discussion | |
| 15:59:01 | melwitt | elod: btw, I opened the result on the pinned ceph version patch for nova-grenade-live-migration and see it's failing with "ERROR: This script does not work on Python 2.7 The minimum supported Python version is 3.6. Please use https://bootstrap.pypa.io/pip/2.7/get-pip.py instead." https://zuul.opendev.org/t/openstack/build/6583ef90f2954a57934781e31d17c68d/log/logs/grenade.sh.txt#2656 do you recall a fix for that going around in the | |
| 15:59:01 | melwitt | past? I can't find | |
| 15:59:50 | gibi | gmann: thanks for confirming that it is not a regression | |
| 16:01:04 | ignaziocassano | Hello All, please anyone can help me with live migrations on queens? Only for some vm live migration does not work. When the vm tries to start on destination host, in instance log I read: 2021-03-15T14:16:12.593707Z qemu-kvm: Failed to load virtio/extra_state:extra_state | |
| 16:02:44 | ignaziocassano | If I hard reset the vm it starts and then I do not got any live migration errors anymore | |
| 16:03:57 | elod | melwitt: hmmm, that failure is related to this: https://review.opendev.org/q/topic:%22get-pip-py27%22 | |
| 16:04:07 | elod | melwitt: i'll try to fix it | |
| 16:04:59 | melwitt | elod: oh, derp, how did I miss that in my gerrit search 😆 yeah, I thought it sounded familiar like something that got fixed recently. I'll lookout for your upcoming patch :) | |
| 16:05:45 | ignaziocassano | I wrote an email to openstack-discuss two days ago but I have not received any suggestion yet. Not found any bug related to to this issue | |
| 16:06:18 | elod | melwitt: correction: I won't be able to :/ as that is because it's ocata, and ocata cannot be fixed if I remember correctly... some catch 22... | |
| 16:07:28 | lyarwood | time to nuke the grenade job on pike then I guess? | |
| 16:07:53 | melwitt | elod: you mean pike? it's on pike but it's non-voting, perhaps for similar reasons | |
| 16:08:11 | elod | melwitt: stestr and other contradicting requirements :S (this was my trial: https://review.opendev.org/777614 ) | |
| 16:08:29 | elod | melwitt: no, ocata, as grenade on pike starts with ocata install | |
| 16:09:34 | melwitt | elod: oh sorry 😓 I didn't know that. ok, good to know | |
| 16:09:42 | elod | lyarwood: yes, actually, I was thinking on that, too (I hoped I could fix it somehow, but 1. seems not really possible; 2. had another priority all the time... o:)) | |
| 16:10:30 | melwitt | yeah, agreed if it can't be fixed then we can save CI resources | |
| 16:11:40 | elod | melwitt: sure. actually not so many patches arrive to pike, but still, it would save some resource :) | |
| 16:12:00 | melwitt | elod: oh, heh true | |
| 16:45:02 | kukacz | hi, is instance rebuild with volume backed instance and selecting same image supposed to actually rebuild the volume to state of the image? | |
| 16:47:09 | lyarwood | kukacz: https://docs.openstack.org/api-ref/compute/?expanded=rebuild-server-rebuild-action-detail & https://bugs.launchpad.net/nova/+bug/1482040 | |
| 16:47:11 | openstack | Launchpad bug 1482040 in OpenStack Compute (nova) pike "rebuild volume-backed instance with image does not replace image in root disk" [Medium,Confirmed] | |
| 16:47:26 | lyarwood | kukacz: it's a known limitation that the volume isn't rebuilt using the new image | |
| 16:47:42 | lyarwood | kukacz: iirc we had a spec agreed to look at correcting this but the changes never landed | |
| 16:48:27 | lyarwood | https://specs.openstack.org/openstack/nova-specs/specs/train/approved/volume-backed-server-rebuild.html | |
| 16:53:49 | lyarwood | melwitt: https://review.opendev.org/c/openstack/nova/+/765769 https://review.opendev.org/c/openstack/nova/+/773443 https://review.opendev.org/c/openstack/nova/+/757306 - would you mind also hitting these? Apologies but my backlog is getting way to big now so I need to start pinging people for reviews. | |
| 16:53:50 | kukacz | lyarwood: clear, thanks! | |
| 16:54:22 | kukacz | somehow I thought that this was working when using identical image | |
| 16:54:28 | kukacz | my bad | |
| 17:23:13 | openstackgerrit | Elod Illes proposed openstack/nova stable/pike: [stable-only] Move grenade jobs to experimental https://review.opendev.org/c/openstack/nova/+/781134 | |
| 17:33:54 | gibi | did we fixed some pure PCI passthrough evacuate issue since victoria? I cannot reproduce this on master https://bugs.launchpad.net/nova/+bug/1910663 | |
| 17:33:55 | openstack | Launchpad bug 1910663 in OpenStack Compute (nova) "pci device duplicate attach after intance evacuated" [Undecided,New] | |
| 17:34:38 | tosky | time to abandon stable/ocata for real? | |
| 17:34:51 | tosky | (looking at that change above ^^) | |
| 18:00:58 | openstackgerrit | Lee Yarwood proposed openstack/nova master: block_device: Use initialize APIs to refresh when reported as idempotent https://review.opendev.org/c/openstack/nova/+/720769 | |
| 18:01:00 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Reject requests to commit active snapshot of an inactive instance https://review.opendev.org/c/openstack/nova/+/781138 | |
| 18:01:01 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Add devstack-plugin-nfs-tempest-full to the check queue https://review.opendev.org/c/openstack/nova/+/781139 | |
| 18:06:31 | openstackgerrit | Merged openstack/nova stable/stein: [stable-only] Specify IPv6 CIDR in tempest-slow-py3 https://review.opendev.org/c/openstack/nova/+/780991 | |
| 18:06:50 | openstackgerrit | Merged openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | |
| 18:23:33 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova stable/stein: Use subqueryload() instead of joinedload() for (system_)metadata https://review.opendev.org/c/openstack/nova/+/761812 | |
| 18:24:04 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova stable/stein: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/757085 | |
| 18:24:06 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova stable/stein: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780784 | |
| 18:56:09 | openstackgerrit | Merged openstack/nova stable/pike: [stable-only] gate: Pin CEPH_RELEASE to nautilus in LM hook https://review.opendev.org/c/openstack/nova/+/780852 | |
| 19:33:30 | lbragstad | gmann ping - i have a fresh devstack up with nova configured to enforce scope and enforce the new default policies | |
| 19:33:40 | lbragstad | but i'm still able to list compute hypervisors as a project-admin | |
| 19:33:49 | lbragstad | did i miss a configuration switch somewhere? | |
| 19:34:51 | gmann | lbragstad: no that should be enough. | |
| 19:35:18 | gmann | gibi: lbragstad stephenfin i think we should go with 400 return code which is most close and mentioned in api-wg guidelines too https://review.opendev.org/c/openstack/nova/+/780333 | |
| 19:35:25 | gmann | lyarwood: ^^ | |
| 19:35:32 | gmann | lbragstad: sorry, wrong mention. | |
| 19:36:44 | lbragstad | gmann no worries - http://paste.openstack.org/show/803682/ is what i'm seeing | |
| 19:36:56 | lbragstad | line 49 shows the project-scoped token response | |
| 19:38:22 | lbragstad | mm - nevermind... i used test-config to setup nova.conf, and n-api must not have been bounced after the config update | |
| 19:38:48 | gmann | lbragstad: it should not allowed | |
| 19:39:12 | lbragstad | yep - you're right, i restarted the nova-api and it looks like it picked up the oslo_policy configs | |
| 19:39:23 | gmann | ohk, for policy change n-api does not need to restart but yes for config change it need | |
| 19:39:46 | gmann | lbragstad: nice. | |
| 22:29:34 | melwitt | lyarwood: no apology needed, will hit those. unrelated, curious if you've seen this before for gate failure in swap volume "Failure rebasing volume /dev/sdb on vdb.: libvirt.libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/sdb': Permission denied" https://zuul.opendev.org/t/openstack/build/c9515f1edc774e74a84e91a7dee50074/log/compute1/logs/screen-n-cpu.txt#2138 seeing it only on stein | |
| 22:29:35 | melwitt | and train | |
| 23:26:23 | eandersson | ops melwitt missed that you had already re-checked the rocky patch :D | |
| 23:26:36 | eandersson | *stein | |
| 23:26:56 | melwitt | eandersson: np, maybe it will give us better luck :) | |
| 23:27:20 | eandersson | fingerscrossed! | |
| #openstack-nova - 2021-03-18 | |||
| 00:37:37 | melwitt | same root failure during create of cinder resources on stable/stein "libvirt.libvirtError: operation failed: 'serial' is deprecated, please use the corresponding option of '-device' instead Could not open '/dev/sdb': Permission denied" https://zuul.opendev.org/t/openstack/build/e1050a661fec47b49d6d7babc89d71d3/log/logs/screen-n-cpu.txt#3267 | |
| 00:37:57 | melwitt | why can't it open /dev/sdb -_- | |
| 00:39:07 | melwitt | rather, why is it Permission Denied. argh. | |
| 00:42:38 | melwitt | opened https://bugs.launchpad.net/openstack-gate/+bug/1919920 fwiw | |
| 00:42:40 | openstack | Launchpad bug 1919920 in OpenStack-Gate "volume related tests sometimes fail with " Could not open '/dev/sdb': Permission denied" on stable/train and stable/stein" [Undecided,New] | |
| 05:40:35 | openstackgerrit | Merged openstack/nova stable/ussuri: libvirt: Increase incremental and max sleep time during device detach https://review.opendev.org/c/openstack/nova/+/757306 | |
| 06:48:41 | openstackgerrit | Merged openstack/nova stable/pike: [stable-only] Move grenade jobs to experimental https://review.opendev.org/c/openstack/nova/+/781134 | |
| 07:02:36 | eandersson | The whole settings in the database is one of the hardest things for new operators to troubleshoot | |
| 07:03:04 | eandersson | When changing the names of rmq server there seems to be no errors in the logs for some reasons | |
| 07:03:20 | eandersson | And if operators are not aware of that "feature" it is probably the last place they will oook | |
| 07:03:23 | eandersson | *look | |
| 07:04:30 | eandersson | I guess maybe because the old rabbitmq servers are still online, so it's just a really bad "split brain". | |
| 07:05:31 | eandersson | At least for us a common scenario has been to have RabbitMQ and OpenStack services on the same node, but when scaling out having dedicated rmq servers is a thing | |
| 07:06:05 | eandersson | but it's not clear to anyone that the rmq config is actually in the database | |
| 07:06:17 | eandersson | (and again super difficult to troubleshoot) | |
| 09:35:23 | stephenfin | lyarwood, gibi: So in opting for HTTP 409, I was actually following two existing examples: we do this already for both SEV and vTPM issues | |
| 09:37:48 | stephenfin | Everything I've read about HTTP 403 suggests it's to do with authorization issues. That's why we use it for policy. There's are no issues with authorization here. The issue is that the server understood the request but can't support it. HTTP 409 seemed the best of a bad bunch | |
| 09:38:44 | stephenfin | HTTP 400 doesn't seem right either. That would suggest that the request was incorrectly formed. Again, it's correct but we simply can't support it | |
| 09:39:23 | stephenfin | lyarwood, gibi: https://github.com/openstack/nova/blob/master/nova/exception.py#L528-L537 | |
| 09:44:07 | lyarwood | stephenfin: right, I saw the examples and I was initially okay with this | |
| 09:44:46 | lyarwood | stephenfin: I honestly don't care either way tbh, both are a compromise AFAICT | |
| 09:45:18 | lyarwood | stephenfin: lets chat with gmann later and agree which way to land on this before the deadline tomorrow that gibi raised yesterday | |
| 09:45:26 | stephenfin | I think so too. I thought there was an explicit "the server understands this request but can't service it" code, but I can't find one | |
| 09:45:47 | stephenfin | Sounds good. I'll propose a spec update on the assumption that we're going with HTTP 409. I can update if gmann talks us around | |
| 09:47:53 | lyarwood | sounds good | |
| 09:47:57 | lyarwood | ops snap | |