| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-14 | |||
| 16:51:05 | johnsom | That is odd given I have DHCP on the subnet, but the metadata still has the fixed IP | |
| 16:51:19 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.injected_network_template | |
| 16:51:40 | noonedeadpunk | I for sure have not changed that | |
| 16:52:02 | sean-k-mooney | thats good becasue that is not tested anywhwere | |
| 16:52:05 | noonedeadpunk | I was debugging OSA CI job that start failing month ago | |
| 16:52:06 | sean-k-mooney | and has not been for years | |
| 16:52:12 | sean-k-mooney | so it proably does not work | |
| 16:52:21 | noonedeadpunk | Or well. I t was passing last time month ago | |
| 16:52:58 | noonedeadpunk | and in the meanwhile what we changed was nova/octavia/neutron versions we're isntalling and ansible collection that creates networks | |
| 16:53:12 | sean-k-mooney | so thats this https://github.com/openstack/nova/blob/master/nova/virt/interfaces.template but thats not actuly for the data your looking at | |
| 16:53:25 | noonedeadpunk | but it's interesting why me and johnsomhave different result given we both have ddhcp enabled | |
| 16:53:54 | sean-k-mooney | johnsom: noonedeadpunk what ml2 drivers are you using | |
| 16:54:02 | sean-k-mooney | ml2/ovs ml2/ovn | |
| 16:54:21 | sean-k-mooney | this code should not care but just wondering | |
| 16:55:13 | noonedeadpunk | ovn | |
| 16:55:18 | johnsom | ovn | |
| 16:55:37 | noonedeadpunk | btw, upgrade job that is passing is using lxb | |
| 16:55:56 | noonedeadpunk | (upgrade from Y to 2023.1) | |
| 16:56:00 | sean-k-mooney | ok so it started breaking in ovn but only somethimes | |
| 16:56:59 | noonedeadpunk | I have a sandbox vm where it's reproducible | |
| 16:57:04 | johnsom | We haven't seen any changes in the Octavia jobs, they are all still passing as expected. | |
| 16:57:15 | sean-k-mooney | johnsom: noonedeadpunk are ye seeign this in yoru downstream installs or is this also seen in upstram devstack tempest jobs | |
| 16:57:42 | noonedeadpunk | but it's in OSA jobs | |
| 16:57:42 | sean-k-mooney | noonedeadpunk: your using OSA right | |
| 16:57:49 | noonedeadpunk | yup | |
| 16:57:56 | noonedeadpunk | No idea about downstream yet | |
| 16:58:05 | sean-k-mooney | ya im wondering if there is some neturon/nova config setting at play that changes this behvior | |
| 16:58:12 | sean-k-mooney | or an enabled neutron api extention | |
| 16:58:24 | noonedeadpunk | well, it could be... | |
| 16:58:29 | sean-k-mooney | most of that code has not changed in 4-6 years | |
| 16:58:40 | noonedeadpunk | we changed neither of that and yeah - code is quite old indeed | |
| 16:58:46 | sean-k-mooney | but that does not mean other code around it had not changed the data its reciving | |
| 16:59:05 | noonedeadpunk | I wonder if that could boil down to smth like sdk version :D | |
| 16:59:29 | sean-k-mooney | not in this case since we are not using the sdk altough maybe neutron clinet | |
| 16:59:31 | noonedeadpunk | that changes output of `subnet.get_meta('dhcp_server')` | |
| 16:59:54 | noonedeadpunk | but that should be neutron client object? | |
| 17:00:43 | noonedeadpunk | sorry really need to run as otherwise my wife will feed my cold corps to pigs | |
| 17:00:44 | sean-k-mooney | not nessisarly | |
| 17:00:53 | sean-k-mooney | cool | |
| 17:00:59 | sean-k-mooney | ping us on monday | |
| 17:01:13 | noonedeadpunk | sure, sorry for that :) | |
| 17:01:19 | noonedeadpunk | have a good weekends again | |
| 17:01:56 | sean-k-mooney | no its an iteresting issue | |
| 17:02:15 | sean-k-mooney | i was jut checkign if it was perhaps oru Subnet model object we use form the network info case but no | |
| 17:02:17 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/model.py#L299-L339 | |
| 17:02:28 | sean-k-mooney | that does not have .get_meta | |
| 17:06:38 | sean-k-mooney | https://github.com/openstack/nova/commit/c7f572d65b57e034c1391ed49d84f6e5f1d672ad seams related | |
| 17:07:15 | sean-k-mooney | johnsom: noonedeadpunk when ye are aroudn on monday check if there is a dhcp port on the network in question | |
| 17:07:55 | sean-k-mooney | this might be caused by useing ovn with or without the dhcp agent | |
| 17:08:03 | sean-k-mooney | if deployed without it we might not have the info | |
| 17:08:10 | sean-k-mooney | btu with it we would? | |
| 17:08:15 | sean-k-mooney | that is my best guess right now | |
| 17:08:25 | dansmith | bauzas: you would not believe how many unit tests depend on this broken behavior (incorrectly) | |
| 17:09:42 | sean-k-mooney | johnsom: looking at https://github.com/openstack/nova/blob/master/nova/virt/netutils.py#L292-L297 | |
| 17:09:55 | sean-k-mooney | we dont have the same workaroudn for that edgecase | |
| 17:10:36 | johnsom | I don't have the dhcp agent running, mine is pure ovn | |
| 17:10:55 | sean-k-mooney | and you have the info | |
| 17:11:02 | johnsom | correct | |
| 17:11:09 | sean-k-mooney | ok lets see if the its the same for noonedeadpunk on monday | |
| 17:11:43 | sean-k-mooney | the way we are doing the check might still be realted but im not sure | |
| 17:15:46 | sean-k-mooney | johnsom: this si workinf form info in the network infocache by the way | |
| 17:16:23 | sean-k-mooney | https://github.com/openstack/nova/blob/ae42400b7663bc58d5562de99e976c95131b77a9/nova/virt/netutils.py#L168-L224 | |
| 17:17:35 | sean-k-mooney | ovn is racy with how its sends network_vif_plugged events | |
| 17:17:48 | sean-k-mooney | so its posible there is a race here | |
| 17:19:02 | dansmith | ugh, the whole point of network_vif_plugged was to close a race :/ | |
| 17:20:26 | sean-k-mooney | ya i know but ovn did not have a way orgian to knwo if the flwo were programed | |
| 17:20:39 | sean-k-mooney | so it said it was pluged when it was bound | |
| 17:20:50 | dansmith | yeah, that's just unfortunate | |
| 17:20:53 | sean-k-mooney | now it has a websocket callback meachnium | |
| 17:21:08 | sean-k-mooney | but it still does not wait for the ovn metadta agaent to compltes | |
| 17:22:04 | sean-k-mooney | which makes metadata and i think dncp racy on boot which as you know is why network vif plugged was added | |
| 17:22:27 | sean-k-mooney | we dont have the dhcp race if using ovn native dhcp | |
| 17:22:49 | sean-k-mooney | but if you use the dhcp agent for ironic support i think it has a race but not sure | |
| 19:09:47 | noonedeadpunk | I'd say my attempts fail quite reliably many times in a row without passing any single time | |
| 19:10:15 | noonedeadpunk | so doesn't look like race in my case at least | |
| 19:26:10 | noonedeadpunk | thankfully it's super easy to fix now. But this doesn't really solve the mystery of why we're having different results with same input | |
| 20:38:24 | opendevreview | melanie witt proposed openstack/nova master: Call volume detach rollback API if detach fails https://review.opendev.org/c/openstack/nova/+/880399 | |
| 21:30:19 | opendevreview | Merged openstack/nova master: Update to the PTL guide https://review.opendev.org/c/openstack/nova/+/875730 | |
| 22:36:52 | opendevreview | Robert Breker proposed openstack/nova master: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/880557 | |
| #openstack-nova - 2023-04-17 | |||
| 01:08:27 | opendevreview | Merged openstack/nova master: db: Remove legacy migrations https://review.opendev.org/c/openstack/nova/+/872428 | |
| 09:05:30 | Uggla | sean-k-mooney, gibi, bauzas I'm facing an issue. I can mount a share on the compute if I'am an admin, but not a regular user. I think it is something around privsep but I did not figure what it could be. Any idea what could be the issue ? | |
| 09:06:57 | gibi | Uggla: how this fails? | |
| 09:07:06 | Uggla | I receive this error as a regular user : mount.nfs: mounting 192.168.122.76:/opt/stack/data/manila/mnt/share-7643c9cb-bf52-4263-b048-2b3589451cd9 failed, reason given by server: No such file or directory\n' | |
| 09:07:24 | Uggla | however the directory exists. | |
| 09:08:54 | Uggla | as an example using the mount -t nfs 192.168.122.76:/opt/stack/data/manila/mnt/share-7643c9cb-bf52-4263-b048-2b3589451cd9 /opt/stack/data/nova/mnt/57c1c5f49444e0ccdcd1d46b5d640827 command as root is ok. | |
| 09:09:14 | opendevreview | Stephen Finucane proposed openstack/nova master: doc: Update version info https://review.opendev.org/c/openstack/nova/+/880614 | |
| 09:09:26 | Uggla | as root | |
| 09:15:23 | Uggla | gibi, https://paste.opendev.org/show/bypnmFjxs469LqyF1nsB/ | |
| 09:16:58 | bauzas | Uggla: I'm unclear, is the mount failing ? | |
| 09:18:18 | Uggla | bauzas, yes the mount is failing if I am an openstack "regular" user (demo). Not if I am an admin. | |
| 09:18:46 | bauzas | I see | |
| 09:19:10 | bauzas | which service is doing the mount ? | |
| 09:19:15 | bauzas | nova-compute right? | |
| 09:19:18 | Uggla | bauzas, compute | |
| 09:19:19 | Uggla | yes | |
| 09:19:39 | bauzas | sec | |
| 09:20:50 | gibi | it is strange the mount call does not take the ctx so I'm not sure how this code can depend on the user token | |
| 09:23:35 | bauzas | I'm confused, I'm trying to see where we internally syscall mount | |
| 09:23:44 | Uggla | gibi, hopefully I have a witness (artom) prooving I'm not totally mad. ;) | |
| 09:24:28 | bauzas | ah, found it | |