| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-14 | |||
| 16:58:40 | noonedeadpunk | we changed neither of that and yeah - code is quite old indeed | |
| 16:58:46 | sean-k-mooney | but that does not mean other code around it had not changed the data its reciving | |
| 16:59:05 | noonedeadpunk | I wonder if that could boil down to smth like sdk version :D | |
| 16:59:29 | sean-k-mooney | not in this case since we are not using the sdk altough maybe neutron clinet | |
| 16:59:31 | noonedeadpunk | that changes output of `subnet.get_meta('dhcp_server')` | |
| 16:59:54 | noonedeadpunk | but that should be neutron client object? | |
| 17:00:43 | noonedeadpunk | sorry really need to run as otherwise my wife will feed my cold corps to pigs | |
| 17:00:44 | sean-k-mooney | not nessisarly | |
| 17:00:53 | sean-k-mooney | cool | |
| 17:00:59 | sean-k-mooney | ping us on monday | |
| 17:01:13 | noonedeadpunk | sure, sorry for that :) | |
| 17:01:19 | noonedeadpunk | have a good weekends again | |
| 17:01:56 | sean-k-mooney | no its an iteresting issue | |
| 17:02:15 | sean-k-mooney | i was jut checkign if it was perhaps oru Subnet model object we use form the network info case but no | |
| 17:02:17 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/model.py#L299-L339 | |
| 17:02:28 | sean-k-mooney | that does not have .get_meta | |
| 17:06:38 | sean-k-mooney | https://github.com/openstack/nova/commit/c7f572d65b57e034c1391ed49d84f6e5f1d672ad seams related | |
| 17:07:15 | sean-k-mooney | johnsom: noonedeadpunk when ye are aroudn on monday check if there is a dhcp port on the network in question | |
| 17:07:55 | sean-k-mooney | this might be caused by useing ovn with or without the dhcp agent | |
| 17:08:03 | sean-k-mooney | if deployed without it we might not have the info | |
| 17:08:10 | sean-k-mooney | btu with it we would? | |
| 17:08:15 | sean-k-mooney | that is my best guess right now | |
| 17:08:25 | dansmith | bauzas: you would not believe how many unit tests depend on this broken behavior (incorrectly) | |
| 17:09:42 | sean-k-mooney | johnsom: looking at https://github.com/openstack/nova/blob/master/nova/virt/netutils.py#L292-L297 | |
| 17:09:55 | sean-k-mooney | we dont have the same workaroudn for that edgecase | |
| 17:10:36 | johnsom | I don't have the dhcp agent running, mine is pure ovn | |
| 17:10:55 | sean-k-mooney | and you have the info | |
| 17:11:02 | johnsom | correct | |
| 17:11:09 | sean-k-mooney | ok lets see if the its the same for noonedeadpunk on monday | |
| 17:11:43 | sean-k-mooney | the way we are doing the check might still be realted but im not sure | |
| 17:15:46 | sean-k-mooney | johnsom: this si workinf form info in the network infocache by the way | |
| 17:16:23 | sean-k-mooney | https://github.com/openstack/nova/blob/ae42400b7663bc58d5562de99e976c95131b77a9/nova/virt/netutils.py#L168-L224 | |
| 17:17:35 | sean-k-mooney | ovn is racy with how its sends network_vif_plugged events | |
| 17:17:48 | sean-k-mooney | so its posible there is a race here | |
| 17:19:02 | dansmith | ugh, the whole point of network_vif_plugged was to close a race :/ | |
| 17:20:26 | sean-k-mooney | ya i know but ovn did not have a way orgian to knwo if the flwo were programed | |
| 17:20:39 | sean-k-mooney | so it said it was pluged when it was bound | |
| 17:20:50 | dansmith | yeah, that's just unfortunate | |
| 17:20:53 | sean-k-mooney | now it has a websocket callback meachnium | |
| 17:21:08 | sean-k-mooney | but it still does not wait for the ovn metadta agaent to compltes | |
| 17:22:04 | sean-k-mooney | which makes metadata and i think dncp racy on boot which as you know is why network vif plugged was added | |
| 17:22:27 | sean-k-mooney | we dont have the dhcp race if using ovn native dhcp | |
| 17:22:49 | sean-k-mooney | but if you use the dhcp agent for ironic support i think it has a race but not sure | |
| 19:09:47 | noonedeadpunk | I'd say my attempts fail quite reliably many times in a row without passing any single time | |
| 19:10:15 | noonedeadpunk | so doesn't look like race in my case at least | |
| 19:26:10 | noonedeadpunk | thankfully it's super easy to fix now. But this doesn't really solve the mystery of why we're having different results with same input | |
| 20:38:24 | opendevreview | melanie witt proposed openstack/nova master: Call volume detach rollback API if detach fails https://review.opendev.org/c/openstack/nova/+/880399 | |
| 21:30:19 | opendevreview | Merged openstack/nova master: Update to the PTL guide https://review.opendev.org/c/openstack/nova/+/875730 | |
| 22:36:52 | opendevreview | Robert Breker proposed openstack/nova master: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/880557 | |
| #openstack-nova - 2023-04-17 | |||
| 01:08:27 | opendevreview | Merged openstack/nova master: db: Remove legacy migrations https://review.opendev.org/c/openstack/nova/+/872428 | |
| 09:05:30 | Uggla | sean-k-mooney, gibi, bauzas I'm facing an issue. I can mount a share on the compute if I'am an admin, but not a regular user. I think it is something around privsep but I did not figure what it could be. Any idea what could be the issue ? | |
| 09:06:57 | gibi | Uggla: how this fails? | |
| 09:07:06 | Uggla | I receive this error as a regular user : mount.nfs: mounting 192.168.122.76:/opt/stack/data/manila/mnt/share-7643c9cb-bf52-4263-b048-2b3589451cd9 failed, reason given by server: No such file or directory\n' | |
| 09:07:24 | Uggla | however the directory exists. | |
| 09:08:54 | Uggla | as an example using the mount -t nfs 192.168.122.76:/opt/stack/data/manila/mnt/share-7643c9cb-bf52-4263-b048-2b3589451cd9 /opt/stack/data/nova/mnt/57c1c5f49444e0ccdcd1d46b5d640827 command as root is ok. | |
| 09:09:14 | opendevreview | Stephen Finucane proposed openstack/nova master: doc: Update version info https://review.opendev.org/c/openstack/nova/+/880614 | |
| 09:09:26 | Uggla | as root | |
| 09:15:23 | Uggla | gibi, https://paste.opendev.org/show/bypnmFjxs469LqyF1nsB/ | |
| 09:16:58 | bauzas | Uggla: I'm unclear, is the mount failing ? | |
| 09:18:18 | Uggla | bauzas, yes the mount is failing if I am an openstack "regular" user (demo). Not if I am an admin. | |
| 09:18:46 | bauzas | I see | |
| 09:19:10 | bauzas | which service is doing the mount ? | |
| 09:19:15 | bauzas | nova-compute right? | |
| 09:19:18 | Uggla | bauzas, compute | |
| 09:19:19 | Uggla | yes | |
| 09:19:39 | bauzas | sec | |
| 09:20:50 | gibi | it is strange the mount call does not take the ctx so I'm not sure how this code can depend on the user token | |
| 09:23:35 | bauzas | I'm confused, I'm trying to see where we internally syscall mount | |
| 09:23:44 | Uggla | gibi, hopefully I have a witness (artom) prooving I'm not totally mad. ;) | |
| 09:24:28 | bauzas | ah, found it | |
| 09:24:37 | bauzas | in https://review.opendev.org/c/openstack/nova/+/833090/29/nova/virt/libvirt/driver.py#4115 | |
| 09:25:49 | bauzas | you're calling nfs.LibvirtNFSVolumeDriver with connect_volume() | |
| 09:25:54 | bauzas | Uggla: amirite ? | |
| 09:26:15 | Uggla | yes | |
| 09:26:46 | Uggla | then it is calling mount --> privsep --> fs | |
| 09:27:52 | bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/fs.py#L113 | |
| 09:27:55 | bauzas | found it | |
| 09:28:51 | bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/mount.py#L262 this is what's eventually called | |
| 09:29:20 | bauzas | which indeed calls a prevsep helper https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/mount.py#L308 | |
| 09:29:53 | bauzas | and yeah look https://github.com/openstack/nova/blob/master/nova/privsep/fs.py#L30 | |
| 09:33:01 | bauzas | oh wait | |
| 09:33:25 | Uggla | bauzas, yes and as gibi said there is no link with the context. That's the reason why I don't understand the issue. | |
| 09:33:37 | bauzas | Uggla: as a reminder, the privsep helper doesn't run as root | |
| 09:34:49 | bauzas | it runs as the nova user with sudo rights, which is different | |
| 09:36:43 | bauzas | iirc, rootwrap only runs privsep with https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters | |
| 09:37:14 | bauzas | and then the privsep client connects to the privsep socket | |
| 09:37:58 | bauzas | that's why I'm suspecting you miss some privsep capability | |
| 09:39:37 | Uggla | bauzas, hum you lost me. As privsep is a replacement for rootwrap in my mind. | |
| 09:41:31 | bauzas | Uggla: privsep is a deamon https://github.com/openstack/oslo.privsep/blob/master/oslo_privsep/daemon.py | |
| 09:41:54 | bauzas | and iirc, we start it thru rootwrap https://github.com/openstack/oslo.privsep/blob/master/oslo_privsep/daemon.py#L32 | |
| 09:42:23 | bauzas | then, the privsep client connects to the deamon which has escalated rights | |
| 09:42:24 | gibi | bauzas: if the same sequence works with and admin user token but not with the demo user token then I don't think this is a privsep capability issue but I have no better idea :/ | |
| 09:42:45 | bauzas | gibi: I haven't understand the same | |
| 09:43:10 | bauzas | I thought Uggla was saying that the mount wasn't working with a nova user, but a mount command was working fine with sudo righrs | |
| 09:43:35 | bauzas | hence me thinking this was about a missing cap | |
| 09:44:12 | bauzas | but agreed, if an admin API call can mount thru nova, then yeah, this is a keystone context problem and not a privsep one | |
| 09:46:37 | Uggla | bauzas, the sudo mount command was just to prove that it is not an issue with missing directory and that mount.nfs is possible on the host. And it works fine as an os admin and not as a os user. | |
| 09:46:50 | bauzas | ohok | |
| 09:46:54 | bauzas | my bad then | |
| 09:47:03 | bauzas | soooo, pdb it | |