| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-14 | |||
| 00:54:39 | opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | |
| 01:50:16 | gmann | seems like tempest-integrated-compute-centos-8-stream is failing consistently https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-centos-8-stream | |
| 02:04:04 | opendevreview | Ghanshyam proposed openstack/nova master: Server actions APIs scoped to project scope https://review.opendev.org/c/openstack/nova/+/824358 | |
| 02:04:07 | melwitt | gmann: :( thanks, I wondered about it | |
| 02:08:45 | gmann | melwitt: I am seeing this traceback in logs, not sure about root cause https://zuul.opendev.org/t/openstack/build/e0db6a9a7ba04e66b0781ba7d259357d/log/controller/logs/screen-q-svc.txt#32875 | |
| 02:10:36 | gmann | pinged neutron folks about it in case they are aware or can find the rootcause | |
| 02:12:12 | melwitt | ack, that's not something I've seen before | |
| 02:14:50 | gmann | it seems it started failing since ~1.5 hrs https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-centos-8-stream | |
| 02:16:07 | melwitt | that's consistent with what I observed. I had been following that change and rechecking it and it had been working until not too long ago | |
| 08:57:15 | bauzas | good morning Nova | |
| 08:57:31 | gibi | good morning Nova | |
| 08:57:51 | gibi | I have a full day workshop downstream so I my presence will be spotty | |
| 08:59:39 | bauzas | gibi: spotty ? fine | |
| 08:59:49 | bauzas | use a headset to listen to it | |
| 08:59:54 | gibi | yeah | |
| 09:00:04 | bauzas | ok, my pun is terrible | |
| 09:00:09 | gibi | video is mandatoryt | |
| 09:01:47 | bauzas | http://www.reactiongifs.com/r/weeds.gif | |
| 09:02:32 | gibi | sorry I'm distracted | |
| 09:05:49 | gibi | kashyap: o/ | |
| 09:06:34 | kashyap | [Off-topic] If any of you use fancy ergonomic mechanical keyboards | |
| 09:12:29 | bauzas | hmmm, I forgot about this spec https://review.opendev.org/c/openstack/nova-specs/+/819510 | |
| 09:12:36 | bauzas | sad it got a -2 | |
| 11:50:01 | opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| 16:51:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reject AZ changes during aggregate add / remove host https://review.opendev.org/c/openstack/nova/+/821423 | |
| 16:58:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: trigger nova-next with new tempest test https://review.opendev.org/c/openstack/nova/+/824607 | |
| 17:12:10 | gmann | melwitt: bauzas gibi for centos8 stream job failing, I have filed the bug https://bugs.launchpad.net/neutron/+bug/1957941 | |
| 17:12:26 | gmann | and making job as non voting to unblock the nova and tempest gate https://review.opendev.org/c/openstack/tempest/+/824740 | |
| 17:30:45 | sean-k-mooney | gmann: oh i see the binding detail are failing a scope check | |
| 17:31:07 | sean-k-mooney | and other fileds | |
| 17:31:41 | sean-k-mooney | with the new deftionsing some of those shoudl be project_admin and the rest project member | |
| 17:31:56 | sean-k-mooney | i.e. host_id woudl be project_admin | |
| 17:32:29 | sean-k-mooney | as would the physical_network and segmentation_id i belive | |
| 17:33:10 | sean-k-mooney | have neutron acidentlly started enforcing scope by default? | |
| 18:20:30 | noonedeadpunk | hey! I was wondering - does keystone_authtoken/neutron sections should be defined for nova-compute? | |
| 18:20:53 | noonedeadpunk | eventually does it needs interaction with keystone? I bet no as it only talk to conductor? | |
| 18:22:45 | noonedeadpunk | likely only key_manager is required though | |
| 18:25:09 | sean-k-mooney | nova-compute directly calls neutron | |
| 18:25:30 | noonedeadpunk | aha | |
| 18:25:31 | sean-k-mooney | nova-compute directly talks to most other services rest apis | |
| 18:25:57 | sean-k-mooney | it only uses the condutor to talk to the scheduler, db and conductor itslef | |
| 18:26:38 | noonedeadpunk | We in #openstack-ansible were just asked what can be done http://seclab.cs.sunysb.edu/seclab/pubs/asiaccs16.pdf and it feels what they write about RPC is a bit stupid since you have keystone admin credentials stored on each compute... | |
| 18:27:11 | sean-k-mooney | you dont nessisarly have keystone admin creds | |
| 18:27:31 | sean-k-mooney | in fact you shoudl not have keystone admin creds | |
| 18:27:32 | noonedeadpunk | well, ok, service creds | |
| 18:27:45 | sean-k-mooney | ok yes | |
| 18:27:45 | noonedeadpunk | but it would still have kind of admin privileges? | |
| 18:27:57 | sean-k-mooney | iddeally you woudl use application credentils | |
| 18:28:22 | sean-k-mooney | am yes nova need to all some other services as admin | |
| 18:28:24 | noonedeadpunk | so the idea there was - how much you fucked if one compute node is hacked :) | |
| 18:28:35 | sean-k-mooney | but those service creditely shoudl only give admin on those spercices | |
| 18:29:09 | sean-k-mooney | it depend on what you put there | |
| 18:29:30 | sean-k-mooney | if you use app creds you can avoid using a password in your config | |
| 18:29:37 | sean-k-mooney | which makes revocation simpler | |
| 18:29:37 | noonedeadpunk | I don't think it's still smth that keystone has out of the box? I mean even with all these scoped tokens you would still need to mess up with policies to get yourself covered? | |
| 18:30:02 | noonedeadpunk | like credential per compute... hm... | |
| 18:30:10 | noonedeadpunk | that's interesting idea | |
| 18:30:23 | sean-k-mooney | yep you can do cred per service per compute today | |
| 18:30:29 | sean-k-mooney | if you really want too | |
| 18:31:02 | sean-k-mooney | also if you are deploying in contiaenr you dont need the nova.conf to be visable to the nova_libvirt container | |
| 18:31:33 | sean-k-mooney | so even if you break out of the vm you then have to also escap the container and selix and the file permisions to read the config file | |
| 18:31:49 | sean-k-mooney | at least with ooo that is | |
| 18:32:24 | sean-k-mooney | so for ooo the vms are runing itn the filesystem namesapce of nova_libvirt as the qemu user | |
| 18:32:33 | sean-k-mooney | that contaienr does not have the nova.conf visable to it | |
| 18:32:55 | sean-k-mooney | and selinux + file system permission woudl prevent the vm process form reading it | |
| 18:32:58 | clarkb | wouldn't that be true for any setup running libvirt as not the nova user and basic file permissions? | |
| 18:33:00 | clarkb | ya that | |
| 18:33:23 | clarkb | however libvirt is privileged | |
| 18:33:25 | noonedeadpunk | yeah, I guess in ubuntu apparmour does kind of same anyway | |
| 18:33:31 | clarkb | so might be defeatable? | |
| 18:33:34 | sean-k-mooney | yep even in a normal rpm/deb install selinux/apparmor + file system permisions can help | |
| 18:33:41 | melwitt | thanks gmann++ | |
| 18:33:59 | sean-k-mooney | clarkb: libvirt is but qemu does not run with the same user or group as libvirt | |
| 18:34:05 | clarkb | aha got it | |
| 18:34:44 | noonedeadpunk | but yeah, I got the idea:) At least I thought that we do smth stupid by placing access to keystone in nova-compute.conf but no, it's required) | |
| 18:34:45 | opendevreview | Merged openstack/nova master: Add regression test for bug #1937084 https://review.opendev.org/c/openstack/nova/+/812126 | |
| 18:34:51 | sean-k-mooney | so in general qemu shoudl not be part of the nova group so should not be able to read nova config files | |
| 18:35:51 | sean-k-mooney | nova is often part of the qemu/libvirt group but that direction makes sense | |
| 18:36:26 | sean-k-mooney | since nova need to be able to talk to libvirt at least and create files that qemu can read | |
| 18:37:32 | melwitt | lyarwood: in case you didn't see, I addressed your comments on https://review.opendev.org/c/openstack/python-openstackclient/+/818306 | |
| 18:38:18 | sean-k-mooney | noonedeadpunk: so ya i have not read that paper but im not sure how valid it is and if they have correctly deployed thigns | |
| 18:39:31 | sean-k-mooney | noonedeadpunk: with devstack for example most thigns are owned by the stack user so the filesystem doesnt do much for you but opensack ansible, kolla or ooo should provide much more protection | |
| 18:40:25 | noonedeadpunk | wasnt it reading in details, but yeah, it weird, as they read nova.conf, get rabbitmq creds, then were snifffing tokens (why when they could jsut take that from keystone_authotoken?) | |
| 18:40:57 | noonedeadpunk | but yeah, I got idea, thanks sean-k-mooney a lot! | |
| 18:41:19 | noonedeadpunk | as always, I got bunch of good ideas how to improve things) | |
| 18:43:30 | sean-k-mooney | noonedeadpunk: looking at https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_pre_install.yml#L85-L121 by the way it looks like osa is already creating /etc/nova in the nova user and group | |
| 18:43:47 | noonedeadpunk | yep, sure we do that! | |
| 18:43:50 | sean-k-mooney | you likely dont want to use 755 for the mode | |
| 18:43:52 | sean-k-mooney | https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_pre_install.yml#L92 | |
| 18:44:04 | sean-k-mooney | since that is world readable | |
| 18:44:14 | noonedeadpunk | good point | |
| 18:44:15 | sean-k-mooney | say 750 | |
| 18:44:38 | sean-k-mooney | or 650 for files | |
| 18:45:05 | noonedeadpunk | we should totally review that asap... | |
| 18:45:34 | noonedeadpunk | nova.conf is stored 0640 though | |
| 18:45:46 | sean-k-mooney | ack then its likely fin already | |
| 18:45:54 | noonedeadpunk | https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_post_install.yml#L58-L81 | |
| 18:45:54 | sean-k-mooney | nova is part fo the libvirt group https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/drivers/kvm/nova_compute_kvm.yml#L56-L63 | |
| 18:45:57 | sean-k-mooney | which is corrct | |
| 18:46:08 | sean-k-mooney | and i dont see qemu or libvirt beign added to nova | |