Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-27
21:34:29 melwitt "/placemen//" well, that's really weird :/
21:57:03 clarkb ya I can't see any updates to keystoneauth or nova taht would explain that
22:03:19 gmann nor on placement sie
22:03:20 gmann side
22:04:16 melwitt same 😑 still looking for where that's coming from. no updates to keystoneauth, nova, placement, keystone. config files look normal
22:09:13 gmann its all started failing since last hr
22:13:52 clarkb let me know if we want to hold a node then we can debug directly
22:14:03 clarkb though in theory a local devstack would exhibit the same behavior if you ahve one
22:16:10 melwitt true
22:16:38 melwitt guess I'll go ahead and try that since I've not found anything else
22:30:17 gmann I do not see any installed pkgs version difference in passing vs failing build
22:45:36 melwitt my local devstack doesn't even get that far:
22:45:38 melwitt Installing collected packages: soupsieve, waitress, sortedcontainers, pyroute2.core, beautifulsoup4, WebTest, simplejson, [...]
22:45:40 melwitt Attempting uninstall: simplejson
22:45:42 melwitt Found existing installation: simplejson 3.16.0
22:45:44 melwitt ERROR: Cannot uninstall 'simplejson'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
22:47:56 clarkb melwitt: the issue there is whatever system you ran devstack on already has simplejson installed via system packages and pip can't uninstall it because the system package is deficient
22:48:32 clarkb I'ev brought it up with zigo as a good thing to help drive in the debuntu world by converting those packages to setuptools instead of distutils. Until then you can manually uninstall python-simplejson/python3-simplejson if on ubuntu. I assume similar package names on fedora/centos
22:48:42 melwitt hm, ok. I just created a new ubuntu focal VM. haven't had this happene before
22:49:34 melwitt ok thanks
22:54:02 gmann i am also creating new 20.04 machine
22:58:01 melwitt I noticed someone mentioned in #openstack-qa earlier today that they were able to hit the same error locally https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2021-09-27.log.html#t2021-09-27T19:54:38
22:58:12 johnsom I am hitting it local
22:58:24 melwitt so hopefully one of us will figure out what's wrong
#openstack-nova - 2021-09-28
00:00:49 gmann I am checking with sync mode also in case async tasks are creating issue here
00:01:31 gmann and some logging for list all endpoints though endpoints seems fine in this case but will be helpful for other issues https://review.opendev.org/c/openstack/devstack/+/811246
00:15:24 sean-k-mooney[m] melwitt if i remember correctly its pulled in by cloud-init
00:15:49 sean-k-mooney[m] just uninstall python-simplejson
00:16:13 sean-k-mooney[m] you have had to do that on fedora in the past and python-yaml too form time to time
01:01:28 gmann seeing same error with async disable too
06:35:56 gibi good morning
06:36:09 gibi I see you had a busy day while I was asleep
07:32:20 bauzas good morning Nova
07:32:55 gibi bauzas: o/
07:33:11 bauzas yeah I saw this
07:33:58 bauzas I'm glad Ian found the root cause
07:35:45 gibi yepp
09:27:33 opendevreview Merged openstack/nova master: tools: Ignore bot-generated branch creation patches https://review.opendev.org/c/openstack/nova/+/810285
10:09:16 gibi bauzas: I broke the old qos min bw support in Xena when I extended it with qos min pps support: https://bugs.launchpad.net/nova/+bug/1945310 So this bug is probably a Xena regression, but I will only be certain that it wasn't broken after I backported the repro tests to wallaby to see if the fault is not there
10:09:35 bauzas ah zut
10:11:48 gibi I have the fixs locally so I think we can still fit this in RC2
10:12:10 gibi but we have a broken gate so maybe not
10:12:35 bauzas yep, I dunno what to say
10:13:02 bauzas gibi: maybe we should accept the RC2 proposal for the moment and see whether the fix could be merged tomorrow for providing a RC3
10:13:17 gibi bauzas: good point
10:13:22 gibi that is the safest bet I think
10:13:47 bauzas gibi: add a comment for our meeting
10:13:48 gibi so lets talk about this on the meeting today, accept the RC2 if nobody against it and I can push for RC3
10:13:53 bauzas yep
10:14:12 gibi adding it to the meeting..
10:15:26 gibi done
10:17:53 bauzas gibi: adding the xena-rc-potential tag to your bug report
10:18:29 bauzas can you triage it ?
10:18:30 gibi ack, there is a slight chance that it was similarly broken before xena too hence my hesitation in that
10:19:04 gibi I will triage it
10:24:46 lyarwood do we not have a bug for the Apache/Placement issue at the moment?
10:25:43 gibi lyarwood: I don't see any
10:25:47 gibi lyarwood: I only see the fix https://review.opendev.org/c/openstack/devstack/+/811303
10:26:01 gibi partial fix, as grenade still needs love
10:26:01 lyarwood yeah same was just checking I hadn't missed something
10:26:17 lyarwood yeah I can't test that until the devstack fix lands
10:26:18 gibi lyarwood: I guess this is not even a placement bug but a devstack one
10:26:25 lyarwood actually I'll just pull both in
10:26:42 gibi lyarwood: wait, there are two grenade issues
10:26:48 lyarwood oh wait sorry I see what you mean re grenade, we need it fixed on both branches
10:26:56 gibi lyarwood: in the https://review.opendev.org/c/openstack/devstack/+/811303 grenade still fails with the placement issue
10:26:58 lyarwood the devstack issue
10:27:03 lyarwood right
10:27:20 lyarwood yeah so we need to land on stable/xena first
10:27:24 lyarwood then master
10:27:31 lyarwood good old grenade
10:27:38 gibi lyarwood: that was my guess but I wasn't sure
10:27:54 gibi lyarwood: so grenade used the devstack from the stable branch?
10:28:09 lyarwood gibi: yup I believe so
10:28:16 gibi nice
10:30:07 elodilles yes, grenade issues usually have to be fixed first at the oldest branch (where the problem exists) and merge "backward"
10:30:27 elodilles is this such an issue?
10:31:10 lyarwood yeah I think so if it's blocking nova-status checks during the initial install
10:31:39 lyarwood I guess it needs to go back as far as that apache update landed
10:32:19 elodilles :S
10:32:36 gibi and the apache update is in focal
10:32:46 gibi so it effect all our focal grenade jobs?
10:33:03 lyarwood it could, I'm just watching the stable/xena job install stable/wallaby now
10:33:10 lyarwood https://zuul.opendev.org/t/openstack/stream/540dea35d7a84633bb00d706fe02136d?logfile=console.log
10:33:30 elodilles if it is only related to focal then we need to fix it first on victoria afaik
10:34:39 elodilles (victoria is the first branch based on focal)
10:37:38 lyarwood yeah it failed
10:37:42 lyarwood fun times
10:38:25 gibi lyarwood: could you drop a mail to the ML thread about this fun finding?
10:38:56 lyarwood sure
10:41:31 gibi thanks
10:42:00 gibi http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025101.html
10:59:23 opendevreview Balazs Gibizer proposed openstack/nova master: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811394
11:00:39 opendevreview Balazs Gibizer proposed openstack/nova master: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811396
11:07:09 lyarwood gibi: I'm going to hold off posting to the thread, I think people are likely aware of the stable requirement here
11:07:34 lyarwood gibi: but I'm slightly confused by the way forward, it seems that people want to add the trailing / but that's breaking keystone at least on stable/xena
11:13:32 gibi lyarwood: ohh, I haven't checked the stable/xena result yet
11:17:22 gibi lyarwood: do you think a non trailing / version would pass?
11:17:40 lyarwood gibi: it passed on master for the non-grenade jobs
11:17:46 gibi hm, true

Earlier   Later