Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-08-04
14:38:33 gibi ralonsoh: placement log stars at 13:43 and the failed query starts at 2020-08-04 13:42:39.140
14:39:13 gibi sorry, never mind
14:39:32 gibi the placement log ends at https://www.sciencealert.com/images/How-Much-Are-We-Worth-V2-large.jpg
14:39:40 gibi bah
14:39:58 gibi so the placement log ends 13:43 so it should have some clue in it
14:40:14 ralonsoh gibi, ok I took a look at the placement logs
14:40:14 ralonsoh https://026716e2072b41f3360b-f5b932bfaec6245f8b3ddc87df48d7a4.ssl.cf2.rackcdn.com/640258/23/check/neutron-grenade-multinode/bcc8e1e/controller/logs/screen-placement-api.txt
14:40:33 ralonsoh I don't know why is failing
14:40:51 ralonsoh keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for identity service not found
14:44:24 gibi the config looks similar before and after the placement restart based on the log
14:52:54 gibi keystone log seems normal around that time
14:53:11 ralonsoh yes, I don't see any error there
14:55:02 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP zuul: Migrate to zuulv3 and break up the nova-live-migration job https://review.opendev.org/711604
14:56:51 gibi nova compute also fails as gets back the same HTTP 500 from placement than the grenade script
14:59:49 gibi ralonsoh: neutron server also gets eystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for identity service not found from keystone
14:59:58 gibi around the same time Aug 04 13:42:19.
15:00:14 ralonsoh gibi, I'm trying to find why placement stops replying
15:00:25 gibi so this is not a placement issue but something is probably wrong with keystone
15:00:36 ralonsoh ok
15:08:24 ralonsoh gibi, just one question
15:08:31 gibi sure
15:08:33 ralonsoh who sets this RP?
15:08:50 ralonsoh because I see n-conductor starting just at this moment
15:08:50 ralonsoh https://026716e2072b41f3360b-f5b932bfaec6245f8b3ddc87df48d7a4.ssl.cf2.rackcdn.com/640258/23/check/neutron-grenade-multinode/bcc8e1e/controller/logs/screen-n-cond.txt
15:09:03 ralonsoh 13:41:14 ~ 15
15:10:08 gibi the RP is created by n-cpi
15:10:13 gibi n-cpu
15:10:18 gibi which fails to create it
15:10:23 gibi as placement returns http 500
15:11:27 gibi but even if the RP would exists the grenade script cannot read it as it also gets http 500 from placement
15:11:41 ralonsoh exactly
15:12:35 gibi based on the exception placement cannot find the internal endpoints in keystone
15:13:08 gibi also around the same time neutron also tries and fails to find the internal endpoints in keystone (independendly from placement)
15:14:17 ralonsoh I think, according to the timestamps that this problem could be due to https://review.opendev.org/#/c/744507/
15:14:40 ralonsoh this is just a guess, I'll try that in a patch
15:15:03 gibi culd be a change in the middleware yes
15:55:28 prometheanfire do we still need the nova client to evacuate (do the docs need updating) https://docs.openstack.org/nova/ussuri/admin/evacuate.html
15:55:42 sean-k-mooney prometheanfire: yes
15:56:00 sean-k-mooney i started a patch a year ago but never got aroudn to finishing it
15:56:31 sean-k-mooney https://review.opendev.org/#/c/643578/
15:56:57 sean-k-mooney i have local change to change it back to evacuate but i never finished renaming it back
15:57:37 prometheanfire ah, kk thanks :D
15:59:39 sean-k-mooney if you feel like updating it feel free. i guess i can try and finish it again
15:59:52 sean-k-mooney more urgent things keep coming up
16:00:10 sean-k-mooney but it should be done sooner rather then later
16:02:12 prometheanfire eh, was more a curiousity, trying to wean off project clients and all
16:02:37 sean-k-mooney ya that is why i created the patch in the first place
16:02:54 sean-k-mooney i wrote it after we had the denver fourm session on closing the gaps
16:03:26 sean-k-mooney then it didnt really get reviewed for a few months and i kind of lost interest
16:04:05 prometheanfire yarp
16:49:42 openstackgerrit Elancheran S proposed openstack/nova master: Removes the delta file once image is extracted https://review.opendev.org/734421
17:07:08 dansmith lyarwood: in case you didn't notice, that failure you're seeing appears to be a race on the glance side that I opened a bug for and have a fix up for,
17:07:32 dansmith but it's a pretty fundamentally-broken thing on the glance side so the fix is non-trivial and will likely take a while to get agreement on and land
17:08:05 dansmith lyarwood: if it's happening a lot, then I guess we could move that to non-voting and add back the base ceph job for the time being, but... it's nice to keep the pressure up for a fix too :/
17:08:57 lyarwood dansmith: yeah no issues, that's only the second hit tha tI had seen against the job
17:09:03 dansmith ack
17:09:05 lyarwood dansmith: so I'm happy to leave it voting for now
17:09:07 dansmith it's actually cool,
17:09:34 dansmith because I filed that bug as "I only saw this when something else was broken, but clearly there's a real race here" and the fact that we're hitting it in the happy path is sweet sweet vindication :P
17:10:12 lyarwood ^_^
17:10:23 lyarwood have you closed my bug out as a dup already?
17:10:36 lyarwood if not I can do it now
17:11:18 dansmith no, is that what you want?
17:11:40 dansmith you've got links to logs which is good, but I guess maybe the dup linkage will keep it handy enough
17:11:57 lyarwood true but I can just copy that across
17:12:04 lyarwood feel free to close it as a dup
17:12:46 dansmith ack
18:45:04 sean-k-mooney gibi: finally got around to testing the WIP attach/detach patch. i commented no the review but http://paste.openstack.org/show/796572/ are the results it looks like thre are still issues with macvtap and direct phsyical.
19:54:13 openstackgerrit Ghanshyam Mann proposed openstack/nova master: migrate testing to ubuntu focal https://review.opendev.org/738126
22:24:54 openstackgerrit sean mooney proposed openstack/nova master: Lookup nic feature by PCI address https://review.opendev.org/739131
22:27:12 openstackgerrit sean mooney proposed openstack/nova master: Lookup nic feature by PCI address https://review.opendev.org/739131
22:30:05 sean-k-mooney stephenfin: gibi ^ adressed your comments then did a rebase
23:11:13 openstackgerrit sean mooney proposed openstack/nova master: remove deprecated schduler filters https://review.opendev.org/744800
23:35:16 sean-k-mooney gmann: that os-vif failure is real
23:35:39 sean-k-mooney gmann: ip route change on of there funciton to return an empty list instead of raise an excption
23:35:59 sean-k-mooney gmann: im going to fix it
23:36:18 gmann sean-k-mooney: ohk, I thought it is something constraint i made incompatible for bionic and focal.
23:37:09 sean-k-mooney am well im testing on focal
23:37:18 sean-k-mooney so it might be related to focal
23:38:21 sean-k-mooney in any case ill fix it
23:38:42 gmann but the current failure is on bionic as current PS is without depends-on
23:38:57 sean-k-mooney oh ok then its both
23:39:31 gmann ok
23:39:44 sean-k-mooney iproute basically used to raise ipexc.NetlinkError
23:39:53 sean-k-mooney if you looked up an interface that did not exist
23:39:59 sean-k-mooney not it returns an empty list
23:40:16 sean-k-mooney so i guess we just dont cap that in upper-constiratins or a new release has come out
23:40:19 sean-k-mooney let me check
23:40:54 sean-k-mooney ya july 27th
23:40:57 sean-k-mooney https://pypi.org/project/pyroute2/#history
23:41:05 sean-k-mooney 0.5.13
23:42:15 gmann yeah that is what installed on failing jobs
23:42:29 sean-k-mooney ya 0.5.12 proably will work.
23:42:41 sean-k-mooney let me stash my change and ill down grade it locally
23:44:59 sean-k-mooney ya 0.5.12 works
23:45:36 gmann https://review.opendev.org/#/c/743277/
23:45:50 gmann sean-k-mooney: it is better to run os-vif unit test on requirement gate
23:46:47 sean-k-mooney we could they are really fast
23:46:48 sean-k-mooney Sum of execute time for each test: 0.3669 sec.
23:46:57 sean-k-mooney less then a second

Earlier   Later