| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-14 | |||
| 12:36:21 | sean-k-mooney | gibi: there is https://eventlet.net/doc/modules/corolocal.html | |
| 12:36:45 | gibi | for the partial: for that we need to attach the partial to a thing that is specific to the current test case execution | |
| 12:37:08 | sean-k-mooney | gibi: ya and i dont really know how to do that | |
| 12:37:10 | gibi | for corolocal that can be the storage, but then we probably need to patch eventlet.spawn* to fill it | |
| 12:37:25 | gibi | I will play around | |
| 12:37:27 | sean-k-mooney | certnely not in the notificaiotn fixture which is where we really want to do this | |
| 12:37:37 | sean-k-mooney | ya i might try and play with this too | |
| 12:38:40 | gibi | sean-k-mooney: eventlet patches threading.local to be corolocal.local() | |
| 12:38:46 | lajoskatona | gauzas, gibi: Hi, for rbac discussion do you think Neutron should join to the discussion? (see: https://etherpad.opendev.org/p/policy-popup-yoga-ptg ) | |
| 12:38:58 | lajoskatona | bauzas ---^ (sorry) | |
| 12:39:45 | gibi | lajoskatona: for the external event discussion would be good to have somebody from neutron as the client of that api | |
| 12:39:51 | sean-k-mooney | lajoskatona: i think there is work to be done with makeing nova capable of calling neturon where neutron is using scope enforcement | |
| 12:40:11 | sean-k-mooney | and ya the external events is the flip side of that | |
| 12:40:46 | lajoskatona | gibi, sean-k-mooney: thanks, than I add it to next week's shcedule | |
| 12:41:15 | sean-k-mooney | gibi: i still think we need to create some form of oslo.midelware so that we can dicorver a services policy programticaly form teh api | |
| 12:41:58 | sean-k-mooney | right now the operator will need to set the correct scopes ectra in our config file | |
| 12:42:05 | lajoskatona | gibi, bauzas, sean-k-mooney: we have edge session at the same time (1400-1600) try to fix that | |
| 12:42:25 | bauzas | sorry in a meeting | |
| 12:42:46 | bauzas | can you tl;dr ? | |
| 12:42:49 | bauzas | I'm hardly following | |
| 12:44:19 | sean-k-mooney | bauzas: there is a clash between the nova rbac popup session and a neutron? edge session | |
| 12:44:51 | sean-k-mooney | it woudl be good if we could adjust the schduler to accomidate that lajoskatona is that a correct summary | |
| 12:45:23 | lajoskatona | sean-k-mooney: yes | |
| 12:45:37 | lajoskatona | sean-k-mooney, bauzas, gibi: I try to fetch ildikov to see if wee need both hours for edge..... | |
| 12:45:39 | sean-k-mooney | we can also talk about neutorn rback issues in the nova neutron session too if we cant | |
| 12:46:08 | lajoskatona | sean-k-mooney: yeah, worst case | |
| 13:01:13 | bauzas | ah ok | |
| 13:01:33 | bauzas | let's then wait for ildikov but I'm pretty sure we can find other slots | |
| 13:30:44 | bauzas | and then I'm back | |
| 16:06:10 | melwitt | bauzas: sorry to bring this up again but do you think there's a chance you could look at https://review.opendev.org/c/openstack/nova/+/791807 and https://review.opendev.org/c/openstack/nova/+/806629 before the ptg? elodilles is +2 on them and I'm trying to avoid them getting lost | |
| 16:08:06 | bauzas | melwitt: yeah I remember I had to review your patches but I wasn't seeing them on my review priority list, adding them | |
| 16:08:30 | melwitt | thank you bauzas | |
| 16:08:46 | bauzas | melwitt: will be the first patches I look tomorrow | |
| 16:29:17 | opendevreview | Balazs Gibizer proposed openstack/nova master: Prevent leaked eventlets to send notifications https://review.opendev.org/c/openstack/nova/+/814036 | |
| 16:29:47 | gibi | sean-k-mooney: ^^ this fixes the local reproduction for me | |
| 16:34:11 | melwitt | gibi: omg, I spent some time looking at these failures yesterday and could not figure out how/why we got "no reply on conductor" and DBNonExistentTable. awesome find 🙌 | |
| 16:34:46 | gibi | melwitt: it was hard one, sean-k-mooney and I spent two day figuring it out | |
| 16:36:23 | gibi | the lesson for me is that we probably should not run multiple testcases in a sequence in a same process if those testcases use some kind of parallelism | |
| 16:36:36 | gibi | but I have no viable alternative | |
| 16:36:45 | melwitt | I was completely stumped. I'm so happy y'all figured it out | |
| 16:37:56 | gibi | ... and I still not like eventlets ;) | |
| 16:40:51 | melwitt | haha :) | |
| 16:41:46 | melwitt | speaking of eventlet... | |
| 16:42:43 | melwitt | here's a thing I'd appreciate your eyes on https://review.opendev.org/c/openstack/nova/+/813114 to see if this is the right way to solve it or if I'm missing a better way | |
| 16:42:58 | melwitt | gibi ^ | |
| 16:43:20 | gibi | melwitt: added to my queue | |
| 16:43:26 | melwitt | danke | |
| 17:04:33 | sean-k-mooney | most of the work was by gibi but ok ill review that it looks promising | |
| 17:04:46 | sean-k-mooney | it its not that complex at first glance | |
| 17:06:00 | sean-k-mooney | ah you are intercepting spawn | |
| 17:06:07 | sean-k-mooney | and that is where your getting the testcase id | |
| 17:06:11 | sean-k-mooney | and propagating it | |
| 17:06:43 | sean-k-mooney | gibi: so the runtime error will that cause the test to fail? | |
| 17:07:15 | sean-k-mooney | i.e. when https://review.opendev.org/c/openstack/nova/+/814036/1/nova/tests/fixtures/notifications.py#168 is raised | |
| 17:07:28 | sean-k-mooney | which test will fail | |
| 17:08:01 | sean-k-mooney | or will we catch that and just loog it | |
| 17:08:40 | sean-k-mooney | oh the runtim error goes to the eventlet that called notify | |
| 17:09:01 | sean-k-mooney | which is the one that is runnign the backgound and it kill whatever was leaked | |
| 17:12:58 | sean-k-mooney | gibi: by the way would stoping the compute service help in this case. | |
| 17:14:29 | sean-k-mooney | gibi: we start the service here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/tests/functional/integrated_helpers.py#L1125-L1143 | |
| 17:15:00 | sean-k-mooney | gibi: if we added a tear_down funciton implmeation that expiclty stop them would that help clean up any running eventlests | |
| 17:16:36 | sean-k-mooney | we have the service refernces so i feel like we shoudl be able to use them to invoke stop https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/service.py#L282-L296 | |
| 17:16:56 | sean-k-mooney | that will at least stop the rpc server instnaces | |
| 17:18:37 | sean-k-mooney | hum ok i guess we are at least partly doing that https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/test.py#L438-L441 | |
| 17:20:25 | sean-k-mooney | actully no that is not registring it to be involed automaticaly its patching the stop funciton | |
| 17:22:27 | sean-k-mooney | the service fixture calls kill as a cleanup function which in turn calls stop so we are already stoping the services when the serivce fixtuer is disposed of | |
| 19:14:21 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649 | |
| 23:28:05 | opendevreview | Hang Yang proposed openstack/nova master: Support creating servers with RBAC SGs https://review.opendev.org/c/openstack/nova/+/811521 | |
| #openstack-nova - 2021-10-15 | |||
| 07:44:57 | bauzas | good morning Nova | |
| 08:46:35 | kashyap | gibi[m]: bauzas: Friday shameless plug: I recently summarized a KVM maintainer's talk for LWN on QEMU and software complexity. I think the lessons are interesting for OpenStack too: | |
| 08:46:43 | kashyap | gibi[m]: bauzas: "A QEMU case study in grappling with software complexity" — https://lwn.net/SubscriberLink/872321/221e8d48eb609a38/ | |
| 08:46:47 | bauzas | ++ | |
| 08:47:12 | kashyap | If you're short on time, read the intro "Sources of complexity", "Ways to fight back", and the short, one-para conclusion. | |
| 08:47:34 | kashyap | (Especially check out the idea of "incomplete transitions") | |
| 09:47:32 | mdbooth | where to look. There's a comment from dansmith that this is a hack: https://github.com/openstack/devstack/blob/82facd6edf7cefac1ab68de4fe9054d7c4cb50db/lib/glance#L291-L294 . Has something invalidated the hack? Does anybody know where 'devstack-system-admin' is supposed to come from? | |
| 09:47:32 | mdbooth | My devstack is failing when it tries to run 'openstack --os-cloud devstack-system-admin registered limit create --service glance --default-limit 10000 --region RegionOne image_size_total' with 'Cloud devstack-system-admin was not found'. This is a fresh install. I don't know what devstack-system-admin is or what creates it, so I'm at a loss for | |
| 09:49:19 | mdbooth | To the best of my knowledge there is no clouds.yaml anywhere on this system. If there is, it was created by devstack and put somewhere I don't know to look for it. | |
| 09:49:40 | kashyap | mdbooth: See this commit in DevStack: 56905820 (Add devstack-system-admin for system scoped actions, 2019-01-08) | |
| 09:51:12 | mdbooth | 👀 | |
| 09:51:20 | kashyap | Heh | |
| 09:51:39 | kashyap | Also see the code under the comment "#admin with a system-scoped token -> devstack-system" in devstack/functions-common | |
| 09:52:04 | kashyap | Although the commit message isn't particularly descriptive; and assumes "inside knowledge" | |
| 09:54:55 | mdbooth | Hmm, that appears to be updating a clouds.yaml file | |
| 09:55:10 | mdbooth | As I don't have a clouds.yaml file, I wonder if this is an ordering thing | |
| 09:55:33 | mdbooth | Did devstack create the glance limit before creating clouds.yaml? | |
| 09:57:02 | mdbooth | I'm going to remove GLANCE_LIMIT_IMAGE_SIZE_TOTAL from my local.conf and re-run glance, then look to see what it put in clouds.yaml | |
| 09:57:22 | mdbooth | s/glance/stack.sh/ | |
| 09:58:57 | kashyap | I don't know about the Glance limit ... but there are bunch of commits that might give a hint (git log --oneline | egrep -i 'glance.*limit') | |
| 10:06:37 | frickler | /etc/openstack/clouds.yaml is what devstack generates | |
| 10:07:13 | mdbooth | frickler: Yeah, that was my RTFS. It seems to be running this without having created it, though 🤔 | |
| 10:07:41 | mdbooth | Although I just found GLANCE_ENABLE_QUOTAS. I wonder if I can sidestep this whole thing. | |
| 10:19:02 | frickler | mdbooth: are you using a reduced set of services? it might be a bug in the async dependencies | |
| 10:19:44 | mdbooth | Very possibly. I rewrite local.conf this morning to use ovn and I'm convinced I wasn't hitting this yesterday. | |
| 10:20:43 | frickler | mdbooth: if you can share your local.conf I can give it a spin | |
| 10:21:26 | mdbooth | Just re-provisioning. I'll have a fully hydrated one in a few minutes. | |
| 10:22:51 | mdbooth | frickler: Actually you can just get it here: https://github.com/shiftstack/cluster-api-provider-openstack/blob/devstack-on-openstack/hack/ci/cloud-init/default.yaml.tpl | |
| 10:23:47 | mdbooth | That's the bottom half of a cloud-init which runs devstack | |
| 10:25:26 | mdbooth | OPENSTACK_RELEASE is xena | |
| 10:27:53 | frickler | ok, having a meeting now, will try to run it afterwards | |
| 10:28:40 | mdbooth | frickler: Thanks. That version includes GLANCE_ENABLE_QUOTAS=False because I'm just testing that. | |