| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-17 | |||
| 16:07:53 | melwitt | elod: you mean pike? it's on pike but it's non-voting, perhaps for similar reasons | |
| 16:08:11 | elod | melwitt: stestr and other contradicting requirements :S (this was my trial: https://review.opendev.org/777614 ) | |
| 16:08:29 | elod | melwitt: no, ocata, as grenade on pike starts with ocata install | |
| 16:09:34 | melwitt | elod: oh sorry 😓 I didn't know that. ok, good to know | |
| 16:09:42 | elod | lyarwood: yes, actually, I was thinking on that, too (I hoped I could fix it somehow, but 1. seems not really possible; 2. had another priority all the time... o:)) | |
| 16:10:30 | melwitt | yeah, agreed if it can't be fixed then we can save CI resources | |
| 16:11:40 | elod | melwitt: sure. actually not so many patches arrive to pike, but still, it would save some resource :) | |
| 16:12:00 | melwitt | elod: oh, heh true | |
| 16:45:02 | kukacz | hi, is instance rebuild with volume backed instance and selecting same image supposed to actually rebuild the volume to state of the image? | |
| 16:47:09 | lyarwood | kukacz: https://docs.openstack.org/api-ref/compute/?expanded=rebuild-server-rebuild-action-detail & https://bugs.launchpad.net/nova/+bug/1482040 | |
| 16:47:11 | openstack | Launchpad bug 1482040 in OpenStack Compute (nova) pike "rebuild volume-backed instance with image does not replace image in root disk" [Medium,Confirmed] | |
| 16:47:26 | lyarwood | kukacz: it's a known limitation that the volume isn't rebuilt using the new image | |
| 16:47:42 | lyarwood | kukacz: iirc we had a spec agreed to look at correcting this but the changes never landed | |
| 16:48:27 | lyarwood | https://specs.openstack.org/openstack/nova-specs/specs/train/approved/volume-backed-server-rebuild.html | |
| 16:53:49 | lyarwood | melwitt: https://review.opendev.org/c/openstack/nova/+/765769 https://review.opendev.org/c/openstack/nova/+/773443 https://review.opendev.org/c/openstack/nova/+/757306 - would you mind also hitting these? Apologies but my backlog is getting way to big now so I need to start pinging people for reviews. | |
| 16:53:50 | kukacz | lyarwood: clear, thanks! | |
| 16:54:22 | kukacz | somehow I thought that this was working when using identical image | |
| 16:54:28 | kukacz | my bad | |
| 17:23:13 | openstackgerrit | Elod Illes proposed openstack/nova stable/pike: [stable-only] Move grenade jobs to experimental https://review.opendev.org/c/openstack/nova/+/781134 | |
| 17:33:54 | gibi | did we fixed some pure PCI passthrough evacuate issue since victoria? I cannot reproduce this on master https://bugs.launchpad.net/nova/+bug/1910663 | |
| 17:33:55 | openstack | Launchpad bug 1910663 in OpenStack Compute (nova) "pci device duplicate attach after intance evacuated" [Undecided,New] | |
| 17:34:38 | tosky | time to abandon stable/ocata for real? | |
| 17:34:51 | tosky | (looking at that change above ^^) | |
| 18:00:58 | openstackgerrit | Lee Yarwood proposed openstack/nova master: block_device: Use initialize APIs to refresh when reported as idempotent https://review.opendev.org/c/openstack/nova/+/720769 | |
| 18:01:00 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Reject requests to commit active snapshot of an inactive instance https://review.opendev.org/c/openstack/nova/+/781138 | |
| 18:01:01 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Add devstack-plugin-nfs-tempest-full to the check queue https://review.opendev.org/c/openstack/nova/+/781139 | |
| 18:06:31 | openstackgerrit | Merged openstack/nova stable/stein: [stable-only] Specify IPv6 CIDR in tempest-slow-py3 https://review.opendev.org/c/openstack/nova/+/780991 | |
| 18:06:50 | openstackgerrit | Merged openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | |
| 18:23:33 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova stable/stein: Use subqueryload() instead of joinedload() for (system_)metadata https://review.opendev.org/c/openstack/nova/+/761812 | |
| 18:24:04 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova stable/stein: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/757085 | |
| 18:24:06 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova stable/stein: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780784 | |
| 18:56:09 | openstackgerrit | Merged openstack/nova stable/pike: [stable-only] gate: Pin CEPH_RELEASE to nautilus in LM hook https://review.opendev.org/c/openstack/nova/+/780852 | |
| 19:33:30 | lbragstad | gmann ping - i have a fresh devstack up with nova configured to enforce scope and enforce the new default policies | |
| 19:33:40 | lbragstad | but i'm still able to list compute hypervisors as a project-admin | |
| 19:33:49 | lbragstad | did i miss a configuration switch somewhere? | |
| 19:34:51 | gmann | lbragstad: no that should be enough. | |
| 19:35:18 | gmann | gibi: lbragstad stephenfin i think we should go with 400 return code which is most close and mentioned in api-wg guidelines too https://review.opendev.org/c/openstack/nova/+/780333 | |
| 19:35:25 | gmann | lyarwood: ^^ | |
| 19:35:32 | gmann | lbragstad: sorry, wrong mention. | |
| 19:36:44 | lbragstad | gmann no worries - http://paste.openstack.org/show/803682/ is what i'm seeing | |
| 19:36:56 | lbragstad | line 49 shows the project-scoped token response | |
| 19:38:22 | lbragstad | mm - nevermind... i used test-config to setup nova.conf, and n-api must not have been bounced after the config update | |
| 19:38:48 | gmann | lbragstad: it should not allowed | |
| 19:39:12 | lbragstad | yep - you're right, i restarted the nova-api and it looks like it picked up the oslo_policy configs | |
| 19:39:23 | gmann | ohk, for policy change n-api does not need to restart but yes for config change it need | |
| 19:39:46 | gmann | lbragstad: nice. | |
| 22:29:34 | melwitt | lyarwood: no apology needed, will hit those. unrelated, curious if you've seen this before for gate failure in swap volume "Failure rebasing volume /dev/sdb on vdb.: libvirt.libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/sdb': Permission denied" https://zuul.opendev.org/t/openstack/build/c9515f1edc774e74a84e91a7dee50074/log/compute1/logs/screen-n-cpu.txt#2138 seeing it only on stein | |
| 22:29:35 | melwitt | and train | |
| 23:26:23 | eandersson | ops melwitt missed that you had already re-checked the rocky patch :D | |
| 23:26:36 | eandersson | *stein | |
| 23:26:56 | melwitt | eandersson: np, maybe it will give us better luck :) | |
| 23:27:20 | eandersson | fingerscrossed! | |
| #openstack-nova - 2021-03-18 | |||
| 00:37:37 | melwitt | same root failure during create of cinder resources on stable/stein "libvirt.libvirtError: operation failed: 'serial' is deprecated, please use the corresponding option of '-device' instead Could not open '/dev/sdb': Permission denied" https://zuul.opendev.org/t/openstack/build/e1050a661fec47b49d6d7babc89d71d3/log/logs/screen-n-cpu.txt#3267 | |
| 00:37:57 | melwitt | why can't it open /dev/sdb -_- | |
| 00:39:07 | melwitt | rather, why is it Permission Denied. argh. | |
| 00:42:38 | melwitt | opened https://bugs.launchpad.net/openstack-gate/+bug/1919920 fwiw | |
| 00:42:40 | openstack | Launchpad bug 1919920 in OpenStack-Gate "volume related tests sometimes fail with " Could not open '/dev/sdb': Permission denied" on stable/train and stable/stein" [Undecided,New] | |
| 05:40:35 | openstackgerrit | Merged openstack/nova stable/ussuri: libvirt: Increase incremental and max sleep time during device detach https://review.opendev.org/c/openstack/nova/+/757306 | |
| 06:48:41 | openstackgerrit | Merged openstack/nova stable/pike: [stable-only] Move grenade jobs to experimental https://review.opendev.org/c/openstack/nova/+/781134 | |
| 07:02:36 | eandersson | The whole settings in the database is one of the hardest things for new operators to troubleshoot | |
| 07:03:04 | eandersson | When changing the names of rmq server there seems to be no errors in the logs for some reasons | |
| 07:03:20 | eandersson | And if operators are not aware of that "feature" it is probably the last place they will oook | |
| 07:03:23 | eandersson | *look | |
| 07:04:30 | eandersson | I guess maybe because the old rabbitmq servers are still online, so it's just a really bad "split brain". | |
| 07:05:31 | eandersson | At least for us a common scenario has been to have RabbitMQ and OpenStack services on the same node, but when scaling out having dedicated rmq servers is a thing | |
| 07:06:05 | eandersson | but it's not clear to anyone that the rmq config is actually in the database | |
| 07:06:17 | eandersson | (and again super difficult to troubleshoot) | |
| 09:35:23 | stephenfin | lyarwood, gibi: So in opting for HTTP 409, I was actually following two existing examples: we do this already for both SEV and vTPM issues | |
| 09:37:48 | stephenfin | Everything I've read about HTTP 403 suggests it's to do with authorization issues. That's why we use it for policy. There's are no issues with authorization here. The issue is that the server understood the request but can't support it. HTTP 409 seemed the best of a bad bunch | |
| 09:38:44 | stephenfin | HTTP 400 doesn't seem right either. That would suggest that the request was incorrectly formed. Again, it's correct but we simply can't support it | |
| 09:39:23 | stephenfin | lyarwood, gibi: https://github.com/openstack/nova/blob/master/nova/exception.py#L528-L537 | |
| 09:44:07 | lyarwood | stephenfin: right, I saw the examples and I was initially okay with this | |
| 09:44:46 | lyarwood | stephenfin: I honestly don't care either way tbh, both are a compromise AFAICT | |
| 09:45:18 | lyarwood | stephenfin: lets chat with gmann later and agree which way to land on this before the deadline tomorrow that gibi raised yesterday | |
| 09:45:26 | stephenfin | I think so too. I thought there was an explicit "the server understands this request but can't service it" code, but I can't find one | |
| 09:45:47 | stephenfin | Sounds good. I'll propose a spec update on the assumption that we're going with HTTP 409. I can update if gmann talks us around | |
| 09:47:53 | lyarwood | sounds good | |
| 09:47:57 | lyarwood | ops snap | |
| 09:48:09 | lyarwood | stephenfin: how's the hangover btw? | |
| 09:48:39 | stephenfin | from all the pints in the pubs I wasn't allowed into? | |
| 09:48:40 | stephenfin | :) | |
| 09:49:26 | lyarwood | stephenfin: fair point, I did stumble across this yesterday and suddenly wanted a pint https://soundsofthepub.com/ | |
| 09:50:39 | lyarwood | 24 days until beer gardens open | |
| 09:50:43 | lyarwood | and I bet it rains, all day. | |
| 09:50:45 | stephenfin | hahaha | |
| 09:50:49 | stephenfin | nice one | |
| 09:51:20 | gibi | stephenfin, lyarwood: yeah, as I said before I have no hard opininon. And also nova is not consistent about such error code so nothing to force ourselves into. | |
| 09:51:40 | kashyap | lyarwood: What do you mean 24 days? Is it some UK deadline? | |
| 09:51:48 | kashyap | Morning, BTW | |
| 09:51:49 | lyarwood | kashyap: yes | |
| 09:52:09 | kashyap | lyarwood: Sweet. On rain ... paradise is a well-watered garden, ya know :D | |
| 09:52:12 | lyarwood | kashyap: beer gardens (pubs) and outside dining allowed in ~24 days. | |
| 09:52:18 | gibi | stephenfin: I think the understood, but cannot support is a 500 code like 501 | |
| 09:52:27 | kashyap | lyarwood: Nice. I thought I'll slowly go senile and die before seeing any of these | |
| 09:52:53 | kashyap | (I mean, they're making similar murmurs here too of opening by 01May) | |
| 09:54:49 | gibi | stephenfin: but the https://specs.openstack.org/openstack/api-wg/guidelines/http/response-codes.html#use-of-501-not-implemented disagrees with me about 501 | |
| 09:56:18 | gibi | if we follow that guide ^^ then we should use 400 | |
| 09:56:42 | gibi | but I agree to wait for gmann to get up and make a decision | |
| 10:00:46 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/victoria: DNM - Testing glance backports to fix bug #1919993 https://review.opendev.org/c/openstack/nova/+/781237 | |
| 10:00:48 | openstack | bug 1919993 in Glance "test_image_web_download_import_with_bad_url failing on stable/victoria" [Undecided,New] https://launchpad.net/bugs/1919993 | |