Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-14
17:09:01 sean-k-mooney which is the one that is runnign the backgound and it kill whatever was leaked
17:12:58 sean-k-mooney gibi: by the way would stoping the compute service help in this case.
17:14:29 sean-k-mooney gibi: we start the service here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/tests/functional/integrated_helpers.py#L1125-L1143
17:15:00 sean-k-mooney gibi: if we added a tear_down funciton implmeation that expiclty stop them would that help clean up any running eventlests
17:16:36 sean-k-mooney we have the service refernces so i feel like we shoudl be able to use them to invoke stop https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/service.py#L282-L296
17:16:56 sean-k-mooney that will at least stop the rpc server instnaces
17:18:37 sean-k-mooney hum ok i guess we are at least partly doing that https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/test.py#L438-L441
17:20:25 sean-k-mooney actully no that is not registring it to be involed automaticaly its patching the stop funciton
17:22:27 sean-k-mooney the service fixture calls kill as a cleanup function which in turn calls stop so we are already stoping the services when the serivce fixtuer is disposed of
19:14:21 opendevreview Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649
23:28:05 opendevreview Hang Yang proposed openstack/nova master: Support creating servers with RBAC SGs https://review.opendev.org/c/openstack/nova/+/811521
#openstack-nova - 2021-10-15
07:44:57 bauzas good morning Nova
08:46:35 kashyap gibi[m]: bauzas: Friday shameless plug: I recently summarized a KVM maintainer's talk for LWN on QEMU and software complexity. I think the lessons are interesting for OpenStack too:
08:46:43 kashyap gibi[m]: bauzas: "A QEMU case study in grappling with software complexity" — https://lwn.net/SubscriberLink/872321/221e8d48eb609a38/
08:46:47 bauzas ++
08:47:12 kashyap If you're short on time, read the intro "Sources of complexity", "Ways to fight back", and the short, one-para conclusion.
08:47:34 kashyap (Especially check out the idea of "incomplete transitions")
09:47:32 mdbooth My devstack is failing when it tries to run 'openstack --os-cloud devstack-system-admin registered limit create --service glance --default-limit 10000 --region RegionOne image_size_total' with 'Cloud devstack-system-admin was not found'. This is a fresh install. I don't know what devstack-system-admin is or what creates it, so I'm at a loss for
09:47:32 mdbooth where to look. There's a comment from dansmith that this is a hack: https://github.com/openstack/devstack/blob/82facd6edf7cefac1ab68de4fe9054d7c4cb50db/lib/glance#L291-L294 . Has something invalidated the hack? Does anybody know where 'devstack-system-admin' is supposed to come from?
09:49:19 mdbooth To the best of my knowledge there is no clouds.yaml anywhere on this system. If there is, it was created by devstack and put somewhere I don't know to look for it.
09:49:40 kashyap mdbooth: See this commit in DevStack: 56905820 (Add devstack-system-admin for system scoped actions, 2019-01-08)
09:51:12 mdbooth 👀
09:51:20 kashyap Heh
09:51:39 kashyap Also see the code under the comment "#admin with a system-scoped token -> devstack-system" in devstack/functions-common
09:52:04 kashyap Although the commit message isn't particularly descriptive; and assumes "inside knowledge"
09:54:55 mdbooth Hmm, that appears to be updating a clouds.yaml file
09:55:10 mdbooth As I don't have a clouds.yaml file, I wonder if this is an ordering thing
09:55:33 mdbooth Did devstack create the glance limit before creating clouds.yaml?
09:57:02 mdbooth I'm going to remove GLANCE_LIMIT_IMAGE_SIZE_TOTAL from my local.conf and re-run glance, then look to see what it put in clouds.yaml
09:57:22 mdbooth s/glance/stack.sh/
09:58:57 kashyap I don't know about the Glance limit ... but there are bunch of commits that might give a hint (git log --oneline | egrep -i 'glance.*limit')
10:06:37 frickler /etc/openstack/clouds.yaml is what devstack generates
10:07:13 mdbooth frickler: Yeah, that was my RTFS. It seems to be running this without having created it, though 🤔
10:07:41 mdbooth Although I just found GLANCE_ENABLE_QUOTAS. I wonder if I can sidestep this whole thing.
10:19:02 frickler mdbooth: are you using a reduced set of services? it might be a bug in the async dependencies
10:19:44 mdbooth Very possibly. I rewrite local.conf this morning to use ovn and I'm convinced I wasn't hitting this yesterday.
10:20:43 frickler mdbooth: if you can share your local.conf I can give it a spin
10:21:26 mdbooth Just re-provisioning. I'll have a fully hydrated one in a few minutes.
10:22:51 mdbooth frickler: Actually you can just get it here: https://github.com/shiftstack/cluster-api-provider-openstack/blob/devstack-on-openstack/hack/ci/cloud-init/default.yaml.tpl
10:23:47 mdbooth That's the bottom half of a cloud-init which runs devstack
10:25:26 mdbooth OPENSTACK_RELEASE is xena
10:27:53 frickler ok, having a meeting now, will try to run it afterwards
10:28:40 mdbooth frickler: Thanks. That version includes GLANCE_ENABLE_QUOTAS=False because I'm just testing that.
10:28:56 mdbooth But it previously used GLANCE_LIMIT_IMAGE_SIZE_TOTAL=10000 instead
11:02:02 mdbooth frickler: FWIW I've been in hacker mode on that config for a while (just look at the history!). I just disabled tempest and horizon which had accidentally become enabled again, and it seems to have completed. It's still using GLANCE_ENABLE_QUOTAS=False.
11:03:39 mdbooth Which is to say, if there's a dependency issue I'll bet it relates to tempest or horizon, but I haven't proven that.
11:29:38 frickler mdbooth: o.k., at least I could reproduce your failure with GLANCE_LIMIT_IMAGE_SIZE_TOTAL being set
11:50:49 frickler mdbooth: nice one, this actually only fails consistently with DEVSTACK_PARALLEL=False
11:52:25 frickler with async, https://github.com/openstack/devstack/blob/82facd6edf7cefac1ab68de4fe9054d7c4cb50db/stack.sh#L1107 runs in the background and write_clouds_yaml in L1122 has a fair chance of being fast enough
11:52:33 frickler dansmith: ^^
11:56:05 mdbooth frickler: Oh, wow! I only turned that on temporarily to rule it out as the potential cause of another issue!
12:27:01 kashyap mdbooth: TIL, "tpl" extension
12:27:27 mdbooth kashyap: Not mine in this case, but I'm pretty sure I've used it before.
12:27:44 kashyap (From your link. Probably it's just a convenient reference to refer to that YAML file as a "template")
12:27:56 kashyap mdbooth: I see
12:56:27 gibi sean-k-mooney: about stoping the services. you are right we are doing it already. that does not stop all the eventlets the service spawnd. I also tried to iterate all the eventlets and and call throw() on them to stop them but that did not help either.
12:57:10 gibi kashyap: thanks for the links, I added it as weekend reading :)
12:57:46 kashyap gibi: No prob. (It took 8 gruelling revisions. :D. But I always become a bit of a better person after writing for LWN)
12:57:56 sean-k-mooney gibi: ya, i have your review open on my other monitor. the more i read over it and look at it the more compleing it becomes.
12:58:21 sean-k-mooney gibi: its a little non obviious at first glance why we have to do this but its a nice solution when yuou did into it
12:58:29 gibi kashyap: I follow LWN but not a subscriber. I think it is a prestige to write there :)
12:59:13 kashyap gibi: I realize not everyone has a subscription; Red Hat has a group sub. Hence I created a "subscriber link", as I posted it in a community channel.
12:59:16 gibi sean-k-mooney: would be better to kill eventlets at the end of each testcase, but I did not find a way to do it
12:59:27 gibi kashyap: yeah I see and I thank you for it
12:59:42 kashyap No prob at all. (And sorry for the plug.)
13:00:23 kashyap But the main idea of essential vs. accidental complexity comes from the famous 1986 paper called "No Silver Bullet" by Fred Brooks - https://en.wikipedia.org/wiki/No_Silver_Bullet
13:00:53 sean-k-mooney gibi: well there might be a way to do it if we modifed the test setup so that each test used a seperate greenpool then we could stop all eventlets in the pool and discard it at the end of the test
13:01:07 kashyap (So it was nice to see concrete examples of it in QEMU.)
13:01:48 sean-k-mooney to do that i think we would have to modify the nova service deffintion and possible nova utils to use a non default eventlet pool
13:02:13 sean-k-mooney but if we did that we could extend the kill function to terminate the pool
13:03:38 frickler mdbooth: I wanted to move the write_clouds_yaml earlier anyway in https://review.opendev.org/c/openstack/devstack/+/780417, I guess I can just do that step in its own patch to fix your issue
13:05:02 gibi kashyap: ohh yeah essential and accidental complexity I like those topics
13:05:23 mdbooth frickler: I'd appreciate it
13:05:30 kashyap gibi: Yeah; the idea goes back 2000 years ago! (Aristotle++)
13:05:34 gibi ohh
13:05:37 gibi I did not know that
13:06:02 kashyap I linked to it in the intro too :)
13:06:22 mdbooth eventlet--
13:06:33 kashyap mdbooth: Heh, what a contrasting negative karma
13:06:41 kashyap (Sorry for your pain)
13:07:00 mdbooth I only have the scars now, and occasionally the nightmares.
13:07:03 gibi sean-k-mooney: if terminating the pool also just calls greenlet.throw() then that would have the same problem as I had when I manually called that at the end of the test on each greenlet
13:08:22 frickler mdbooth: https://review.opendev.org/c/openstack/devstack/+/814142
13:08:56 mdbooth It blows my mind that at some point there was a meeting and somebody said: "You know what, lets just monkey patch everything and replace it all with our own stuff, what could go wrong?". And somebody else in that meeting agreed with them, and they started doing it.
13:09:35 gibi mdbooth: I assume it was a single person project. :)
13:09:36 sean-k-mooney mdbooth: well the alternitive was to continue to use twisted so...
13:10:31 gibi our use threading until you scale too big where the overhead of threads are too much
13:10:55 gibi our use other than python without the GIL ;)
13:11:01 sean-k-mooney gibi: i was considering coudl we stop all the service we spawwned as greantreads and then etiehr call waitall to wait for the to finsih or loop over and call kill on the all running greenthreads
13:12:19 sean-k-mooney so stop service and call https://eventlet.net/doc/modules/greenpool.html#eventlet.greenpool.GreenPool.waitall or stop services and call https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.kill on all greentreads in the pool
13:12:31 gibi it is multiple eventlet per service, but yes, you are right. That should work. I did not call wait after throw, maybe that was the problem
13:12:55 gibi note that we not just have greenthreads, we have naked greenlets as well somehow
13:13:09 gibi I did not traced where they are coming from
13:15:23 sean-k-mooney i need to test something else today but i still think your current patch is likely a viable solution in the sort term and we could explore the green pool approche in parallel/after
13:25:02 bauzas folks, looking at the nova PTG agenda we have atm
13:25:41 bauzas it looks to me we don't have a lot of topics to discuss, so maybe we shouldn't have a schedule, ok ?
13:26:01 bauzas I'll just prioritize some topics
13:30:50 dansmith frickler: ah, need to wait for all those accounts to finish before write_clouds_yaml I guess huh?
13:30:51 sean-k-mooney we might want to keep one of the session free for an unconfrence/follow up dicussions
13:30:57 sean-k-mooney bauzas: ^

Earlier   Later