| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-04 | |||
| 13:18:51 | artom | lyarwood, tbh, I don't think anyone will look, unless they understand it's a WIP and want to check in/help | |
| 13:18:55 | artom | Aka no shame | |
| 13:19:33 | sean-k-mooney | artom: i think lyarwood just did not want to waste gate resouces | |
| 13:20:02 | artom | sean-k-mooney, ah, yeah. Well, what are you gonna do | |
| 13:20:59 | sean-k-mooney | disable all jobs you are not updatinging and ask stephenfin to run unit test locally :P | |
| 13:21:36 | sean-k-mooney | you can replicate the gate as i said but its a lot of setup if you dont do this regularly | |
| 13:22:22 | sean-k-mooney | that is true | |
| 13:22:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Update references to image properties https://review.opendev.org/744198 | |
| 13:22:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Drop support for UML https://review.opendev.org/743230 | |
| 13:22:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Drop support for Xen https://review.opendev.org/743231 | |
| 13:22:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove 'hypervisor_version' from 'libvirt_info' https://review.opendev.org/744199 | |
| 13:23:53 | sean-k-mooney | stephenfin: https://www.sciencealert.com/images/How-Much-Are-We-Worth-V2-large.jpg | |
| 13:24:22 | sean-k-mooney | thats even before you take into account your value as a software engineer or a good person | |
| 13:24:31 | sean-k-mooney | machines are cheap in comparison :P | |
| 13:24:38 | kashyap | sean-k-mooney: Heh; I never realized the 0.2mg of gold even | |
| 13:29:47 | artom | "It would take 40,000 people to get enough gold for one uk sovereign coin" | |
| 13:29:55 | artom | Challenge accepted! | |
| 13:49:23 | stephenfin | artom the terrible | |
| 13:49:41 | artom | Great Uncle Ivan would be proud | |
| 14:29:46 | ralonsoh | hello folks | |
| 14:29:55 | ralonsoh | we are hitting in the Neutron CI | |
| 14:30:02 | ralonsoh | a problem with grenade jobs | |
| 14:30:07 | ralonsoh | https://026716e2072b41f3360b-f5b932bfaec6245f8b3ddc87df48d7a4.ssl.cf2.rackcdn.com/640258/23/check/neutron-grenade-multinode/bcc8e1e/controller/logs/grenade.sh_log.txt | |
| 14:30:14 | ralonsoh | just an example | |
| 14:31:02 | ralonsoh | when retrieving the RP for VCPU | |
| 14:31:03 | ralonsoh | openstack resource provider list -f value | head -n1 | cut -d ' ' -f 1 | |
| 14:31:08 | ralonsoh | that returns nothing | |
| 14:37:22 | ralonsoh | stephenfin, hi ^^ | |
| 14:38:33 | gibi | ralonsoh: placement log stars at 13:43 and the failed query starts at 2020-08-04 13:42:39.140 | |
| 14:39:13 | gibi | sorry, never mind | |
| 14:39:32 | gibi | the placement log ends at https://www.sciencealert.com/images/How-Much-Are-We-Worth-V2-large.jpg | |
| 14:39:40 | gibi | bah | |
| 14:39:58 | gibi | so the placement log ends 13:43 so it should have some clue in it | |
| 14:40:14 | ralonsoh | gibi, ok I took a look at the placement logs | |
| 14:40:14 | ralonsoh | https://026716e2072b41f3360b-f5b932bfaec6245f8b3ddc87df48d7a4.ssl.cf2.rackcdn.com/640258/23/check/neutron-grenade-multinode/bcc8e1e/controller/logs/screen-placement-api.txt | |
| 14:40:33 | ralonsoh | I don't know why is failing | |
| 14:40:51 | ralonsoh | keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for identity service not found | |
| 14:44:24 | gibi | the config looks similar before and after the placement restart based on the log | |
| 14:52:54 | gibi | keystone log seems normal around that time | |
| 14:53:11 | ralonsoh | yes, I don't see any error there | |
| 14:55:02 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Migrate to zuulv3 and break up the nova-live-migration job https://review.opendev.org/711604 | |
| 14:56:51 | gibi | nova compute also fails as gets back the same HTTP 500 from placement than the grenade script | |
| 14:59:49 | gibi | ralonsoh: neutron server also gets eystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for identity service not found from keystone | |
| 14:59:58 | gibi | around the same time Aug 04 13:42:19. | |
| 15:00:14 | ralonsoh | gibi, I'm trying to find why placement stops replying | |
| 15:00:25 | gibi | so this is not a placement issue but something is probably wrong with keystone | |
| 15:00:36 | ralonsoh | ok | |
| 15:08:24 | ralonsoh | gibi, just one question | |
| 15:08:31 | gibi | sure | |
| 15:08:33 | ralonsoh | who sets this RP? | |
| 15:08:50 | ralonsoh | because I see n-conductor starting just at this moment | |
| 15:08:50 | ralonsoh | https://026716e2072b41f3360b-f5b932bfaec6245f8b3ddc87df48d7a4.ssl.cf2.rackcdn.com/640258/23/check/neutron-grenade-multinode/bcc8e1e/controller/logs/screen-n-cond.txt | |
| 15:09:03 | ralonsoh | 13:41:14 ~ 15 | |
| 15:10:08 | gibi | the RP is created by n-cpi | |
| 15:10:13 | gibi | n-cpu | |
| 15:10:18 | gibi | which fails to create it | |
| 15:10:23 | gibi | as placement returns http 500 | |
| 15:11:27 | gibi | but even if the RP would exists the grenade script cannot read it as it also gets http 500 from placement | |
| 15:11:41 | ralonsoh | exactly | |
| 15:12:35 | gibi | based on the exception placement cannot find the internal endpoints in keystone | |
| 15:13:08 | gibi | also around the same time neutron also tries and fails to find the internal endpoints in keystone (independendly from placement) | |
| 15:14:17 | ralonsoh | I think, according to the timestamps that this problem could be due to https://review.opendev.org/#/c/744507/ | |
| 15:14:40 | ralonsoh | this is just a guess, I'll try that in a patch | |
| 15:15:03 | gibi | culd be a change in the middleware yes | |
| 15:55:28 | prometheanfire | do we still need the nova client to evacuate (do the docs need updating) https://docs.openstack.org/nova/ussuri/admin/evacuate.html | |
| 15:55:42 | sean-k-mooney | prometheanfire: yes | |
| 15:56:00 | sean-k-mooney | i started a patch a year ago but never got aroudn to finishing it | |
| 15:56:31 | sean-k-mooney | https://review.opendev.org/#/c/643578/ | |
| 15:56:57 | sean-k-mooney | i have local change to change it back to evacuate but i never finished renaming it back | |
| 15:57:37 | prometheanfire | ah, kk thanks :D | |
| 15:59:39 | sean-k-mooney | if you feel like updating it feel free. i guess i can try and finish it again | |
| 15:59:52 | sean-k-mooney | more urgent things keep coming up | |
| 16:00:10 | sean-k-mooney | but it should be done sooner rather then later | |
| 16:02:12 | prometheanfire | eh, was more a curiousity, trying to wean off project clients and all | |
| 16:02:37 | sean-k-mooney | ya that is why i created the patch in the first place | |
| 16:02:54 | sean-k-mooney | i wrote it after we had the denver fourm session on closing the gaps | |
| 16:03:26 | sean-k-mooney | then it didnt really get reviewed for a few months and i kind of lost interest | |
| 16:04:05 | prometheanfire | yarp | |
| 16:49:42 | openstackgerrit | Elancheran S proposed openstack/nova master: Removes the delta file once image is extracted https://review.opendev.org/734421 | |
| 17:07:08 | dansmith | lyarwood: in case you didn't notice, that failure you're seeing appears to be a race on the glance side that I opened a bug for and have a fix up for, | |
| 17:07:32 | dansmith | but it's a pretty fundamentally-broken thing on the glance side so the fix is non-trivial and will likely take a while to get agreement on and land | |
| 17:08:05 | dansmith | lyarwood: if it's happening a lot, then I guess we could move that to non-voting and add back the base ceph job for the time being, but... it's nice to keep the pressure up for a fix too :/ | |
| 17:08:57 | lyarwood | dansmith: yeah no issues, that's only the second hit tha tI had seen against the job | |
| 17:09:03 | dansmith | ack | |
| 17:09:05 | lyarwood | dansmith: so I'm happy to leave it voting for now | |
| 17:09:07 | dansmith | it's actually cool, | |
| 17:09:34 | dansmith | because I filed that bug as "I only saw this when something else was broken, but clearly there's a real race here" and the fact that we're hitting it in the happy path is sweet sweet vindication :P | |
| 17:10:12 | lyarwood | ^_^ | |
| 17:10:23 | lyarwood | have you closed my bug out as a dup already? | |
| 17:10:36 | lyarwood | if not I can do it now | |
| 17:11:18 | dansmith | no, is that what you want? | |
| 17:11:40 | dansmith | you've got links to logs which is good, but I guess maybe the dup linkage will keep it handy enough | |
| 17:11:57 | lyarwood | true but I can just copy that across | |
| 17:12:04 | lyarwood | feel free to close it as a dup | |
| 17:12:46 | dansmith | ack | |
| 18:45:04 | sean-k-mooney | gibi: finally got around to testing the WIP attach/detach patch. i commented no the review but http://paste.openstack.org/show/796572/ are the results it looks like thre are still issues with macvtap and direct phsyical. | |
| 19:54:13 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: migrate testing to ubuntu focal https://review.opendev.org/738126 | |
| 22:24:54 | openstackgerrit | sean mooney proposed openstack/nova master: Lookup nic feature by PCI address https://review.opendev.org/739131 | |
| 22:27:12 | openstackgerrit | sean mooney proposed openstack/nova master: Lookup nic feature by PCI address https://review.opendev.org/739131 | |
| 22:30:05 | sean-k-mooney | stephenfin: gibi ^ adressed your comments then did a rebase | |