| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-14 | |||
| 16:36:23 | gibi | the lesson for me is that we probably should not run multiple testcases in a sequence in a same process if those testcases use some kind of parallelism | |
| 16:36:36 | gibi | but I have no viable alternative | |
| 16:36:45 | melwitt | I was completely stumped. I'm so happy y'all figured it out | |
| 16:37:56 | gibi | ... and I still not like eventlets ;) | |
| 16:40:51 | melwitt | haha :) | |
| 16:41:46 | melwitt | speaking of eventlet... | |
| 16:42:43 | melwitt | here's a thing I'd appreciate your eyes on https://review.opendev.org/c/openstack/nova/+/813114 to see if this is the right way to solve it or if I'm missing a better way | |
| 16:42:58 | melwitt | gibi ^ | |
| 16:43:20 | gibi | melwitt: added to my queue | |
| 16:43:26 | melwitt | danke | |
| 17:04:33 | sean-k-mooney | most of the work was by gibi but ok ill review that it looks promising | |
| 17:04:46 | sean-k-mooney | it its not that complex at first glance | |
| 17:06:00 | sean-k-mooney | ah you are intercepting spawn | |
| 17:06:07 | sean-k-mooney | and that is where your getting the testcase id | |
| 17:06:11 | sean-k-mooney | and propagating it | |
| 17:06:43 | sean-k-mooney | gibi: so the runtime error will that cause the test to fail? | |
| 17:07:15 | sean-k-mooney | i.e. when https://review.opendev.org/c/openstack/nova/+/814036/1/nova/tests/fixtures/notifications.py#168 is raised | |
| 17:07:28 | sean-k-mooney | which test will fail | |
| 17:08:01 | sean-k-mooney | or will we catch that and just loog it | |
| 17:08:40 | sean-k-mooney | oh the runtim error goes to the eventlet that called notify | |
| 17:09:01 | sean-k-mooney | which is the one that is runnign the backgound and it kill whatever was leaked | |
| 17:12:58 | sean-k-mooney | gibi: by the way would stoping the compute service help in this case. | |
| 17:14:29 | sean-k-mooney | gibi: we start the service here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/tests/functional/integrated_helpers.py#L1125-L1143 | |
| 17:15:00 | sean-k-mooney | gibi: if we added a tear_down funciton implmeation that expiclty stop them would that help clean up any running eventlests | |
| 17:16:36 | sean-k-mooney | we have the service refernces so i feel like we shoudl be able to use them to invoke stop https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/service.py#L282-L296 | |
| 17:16:56 | sean-k-mooney | that will at least stop the rpc server instnaces | |
| 17:18:37 | sean-k-mooney | hum ok i guess we are at least partly doing that https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/test.py#L438-L441 | |
| 17:20:25 | sean-k-mooney | actully no that is not registring it to be involed automaticaly its patching the stop funciton | |
| 17:22:27 | sean-k-mooney | the service fixture calls kill as a cleanup function which in turn calls stop so we are already stoping the services when the serivce fixtuer is disposed of | |
| 19:14:21 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649 | |
| 23:28:05 | opendevreview | Hang Yang proposed openstack/nova master: Support creating servers with RBAC SGs https://review.opendev.org/c/openstack/nova/+/811521 | |
| #openstack-nova - 2021-10-15 | |||
| 07:44:57 | bauzas | good morning Nova | |
| 08:46:35 | kashyap | gibi[m]: bauzas: Friday shameless plug: I recently summarized a KVM maintainer's talk for LWN on QEMU and software complexity. I think the lessons are interesting for OpenStack too: | |
| 08:46:43 | kashyap | gibi[m]: bauzas: "A QEMU case study in grappling with software complexity" — https://lwn.net/SubscriberLink/872321/221e8d48eb609a38/ | |
| 08:46:47 | bauzas | ++ | |
| 08:47:12 | kashyap | If you're short on time, read the intro "Sources of complexity", "Ways to fight back", and the short, one-para conclusion. | |
| 08:47:34 | kashyap | (Especially check out the idea of "incomplete transitions") | |
| 09:47:32 | mdbooth | My devstack is failing when it tries to run 'openstack --os-cloud devstack-system-admin registered limit create --service glance --default-limit 10000 --region RegionOne image_size_total' with 'Cloud devstack-system-admin was not found'. This is a fresh install. I don't know what devstack-system-admin is or what creates it, so I'm at a loss for | |
| 09:47:32 | mdbooth | where to look. There's a comment from dansmith that this is a hack: https://github.com/openstack/devstack/blob/82facd6edf7cefac1ab68de4fe9054d7c4cb50db/lib/glance#L291-L294 . Has something invalidated the hack? Does anybody know where 'devstack-system-admin' is supposed to come from? | |
| 09:49:19 | mdbooth | To the best of my knowledge there is no clouds.yaml anywhere on this system. If there is, it was created by devstack and put somewhere I don't know to look for it. | |
| 09:49:40 | kashyap | mdbooth: See this commit in DevStack: 56905820 (Add devstack-system-admin for system scoped actions, 2019-01-08) | |
| 09:51:12 | mdbooth | 👀 | |
| 09:51:20 | kashyap | Heh | |
| 09:51:39 | kashyap | Also see the code under the comment "#admin with a system-scoped token -> devstack-system" in devstack/functions-common | |
| 09:52:04 | kashyap | Although the commit message isn't particularly descriptive; and assumes "inside knowledge" | |
| 09:54:55 | mdbooth | Hmm, that appears to be updating a clouds.yaml file | |
| 09:55:10 | mdbooth | As I don't have a clouds.yaml file, I wonder if this is an ordering thing | |
| 09:55:33 | mdbooth | Did devstack create the glance limit before creating clouds.yaml? | |
| 09:57:02 | mdbooth | I'm going to remove GLANCE_LIMIT_IMAGE_SIZE_TOTAL from my local.conf and re-run glance, then look to see what it put in clouds.yaml | |
| 09:57:22 | mdbooth | s/glance/stack.sh/ | |
| 09:58:57 | kashyap | I don't know about the Glance limit ... but there are bunch of commits that might give a hint (git log --oneline | egrep -i 'glance.*limit') | |
| 10:06:37 | frickler | /etc/openstack/clouds.yaml is what devstack generates | |
| 10:07:13 | mdbooth | frickler: Yeah, that was my RTFS. It seems to be running this without having created it, though 🤔 | |
| 10:07:41 | mdbooth | Although I just found GLANCE_ENABLE_QUOTAS. I wonder if I can sidestep this whole thing. | |
| 10:19:02 | frickler | mdbooth: are you using a reduced set of services? it might be a bug in the async dependencies | |
| 10:19:44 | mdbooth | Very possibly. I rewrite local.conf this morning to use ovn and I'm convinced I wasn't hitting this yesterday. | |
| 10:20:43 | frickler | mdbooth: if you can share your local.conf I can give it a spin | |
| 10:21:26 | mdbooth | Just re-provisioning. I'll have a fully hydrated one in a few minutes. | |
| 10:22:51 | mdbooth | frickler: Actually you can just get it here: https://github.com/shiftstack/cluster-api-provider-openstack/blob/devstack-on-openstack/hack/ci/cloud-init/default.yaml.tpl | |
| 10:23:47 | mdbooth | That's the bottom half of a cloud-init which runs devstack | |
| 10:25:26 | mdbooth | OPENSTACK_RELEASE is xena | |
| 10:27:53 | frickler | ok, having a meeting now, will try to run it afterwards | |
| 10:28:40 | mdbooth | frickler: Thanks. That version includes GLANCE_ENABLE_QUOTAS=False because I'm just testing that. | |
| 10:28:56 | mdbooth | But it previously used GLANCE_LIMIT_IMAGE_SIZE_TOTAL=10000 instead | |
| 11:02:02 | mdbooth | frickler: FWIW I've been in hacker mode on that config for a while (just look at the history!). I just disabled tempest and horizon which had accidentally become enabled again, and it seems to have completed. It's still using GLANCE_ENABLE_QUOTAS=False. | |
| 11:03:39 | mdbooth | Which is to say, if there's a dependency issue I'll bet it relates to tempest or horizon, but I haven't proven that. | |
| 11:29:38 | frickler | mdbooth: o.k., at least I could reproduce your failure with GLANCE_LIMIT_IMAGE_SIZE_TOTAL being set | |
| 11:50:49 | frickler | mdbooth: nice one, this actually only fails consistently with DEVSTACK_PARALLEL=False | |
| 11:52:25 | frickler | with async, https://github.com/openstack/devstack/blob/82facd6edf7cefac1ab68de4fe9054d7c4cb50db/stack.sh#L1107 runs in the background and write_clouds_yaml in L1122 has a fair chance of being fast enough | |
| 11:52:33 | frickler | dansmith: ^^ | |
| 11:56:05 | mdbooth | frickler: Oh, wow! I only turned that on temporarily to rule it out as the potential cause of another issue! | |
| 12:27:01 | kashyap | mdbooth: TIL, "tpl" extension | |
| 12:27:27 | mdbooth | kashyap: Not mine in this case, but I'm pretty sure I've used it before. | |
| 12:27:44 | kashyap | (From your link. Probably it's just a convenient reference to refer to that YAML file as a "template") | |
| 12:27:56 | kashyap | mdbooth: I see | |
| 12:56:27 | gibi | sean-k-mooney: about stoping the services. you are right we are doing it already. that does not stop all the eventlets the service spawnd. I also tried to iterate all the eventlets and and call throw() on them to stop them but that did not help either. | |
| 12:57:10 | gibi | kashyap: thanks for the links, I added it as weekend reading :) | |
| 12:57:46 | kashyap | gibi: No prob. (It took 8 gruelling revisions. :D. But I always become a bit of a better person after writing for LWN) | |
| 12:57:56 | sean-k-mooney | gibi: ya, i have your review open on my other monitor. the more i read over it and look at it the more compleing it becomes. | |
| 12:58:21 | sean-k-mooney | gibi: its a little non obviious at first glance why we have to do this but its a nice solution when yuou did into it | |
| 12:58:29 | gibi | kashyap: I follow LWN but not a subscriber. I think it is a prestige to write there :) | |
| 12:59:13 | kashyap | gibi: I realize not everyone has a subscription; Red Hat has a group sub. Hence I created a "subscriber link", as I posted it in a community channel. | |
| 12:59:16 | gibi | sean-k-mooney: would be better to kill eventlets at the end of each testcase, but I did not find a way to do it | |
| 12:59:27 | gibi | kashyap: yeah I see and I thank you for it | |
| 12:59:42 | kashyap | No prob at all. (And sorry for the plug.) | |
| 13:00:23 | kashyap | But the main idea of essential vs. accidental complexity comes from the famous 1986 paper called "No Silver Bullet" by Fred Brooks - https://en.wikipedia.org/wiki/No_Silver_Bullet | |
| 13:00:53 | sean-k-mooney | gibi: well there might be a way to do it if we modifed the test setup so that each test used a seperate greenpool then we could stop all eventlets in the pool and discard it at the end of the test | |
| 13:01:07 | kashyap | (So it was nice to see concrete examples of it in QEMU.) | |
| 13:01:48 | sean-k-mooney | to do that i think we would have to modify the nova service deffintion and possible nova utils to use a non default eventlet pool | |
| 13:02:13 | sean-k-mooney | but if we did that we could extend the kill function to terminate the pool | |
| 13:03:38 | frickler | mdbooth: I wanted to move the write_clouds_yaml earlier anyway in https://review.opendev.org/c/openstack/devstack/+/780417, I guess I can just do that step in its own patch to fix your issue | |
| 13:05:02 | gibi | kashyap: ohh yeah essential and accidental complexity I like those topics | |
| 13:05:23 | mdbooth | frickler: I'd appreciate it | |
| 13:05:30 | kashyap | gibi: Yeah; the idea goes back 2000 years ago! (Aristotle++) | |
| 13:05:34 | gibi | ohh | |
| 13:05:37 | gibi | I did not know that | |
| 13:06:02 | kashyap | I linked to it in the intro too :) | |
| 13:06:22 | mdbooth | eventlet-- | |
| 13:06:33 | kashyap | mdbooth: Heh, what a contrasting negative karma | |
| 13:06:41 | kashyap | (Sorry for your pain) | |