| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-19 | |||
| 16:42:42 | melwitt | when O | |
| 16:43:49 | melwitt | *when I've looked into nova-lvm job failures, what I have seen is that they fail due to the glance apache server responding to nova with a HTTP 502 bad gateway. meanwhile no errors in glance logs https://etherpad.opendev.org/p/nova-ci-failures-minimal#L11 | |
| 16:44:22 | melwitt | not really sure what to do next to debug | |
| 16:45:17 | sean-k-mooney | melwitt: this maybe be related to glance-api being OOM killed | |
| 16:45:45 | sean-k-mooney | melwitt: artom was debuging failure where that happend so it would be good to confirm that is not whats happening fro the nova-lvm case | |
| 16:46:12 | artom | "debugging" is a strong word | |
| 16:46:22 | artom | I chased a red herring only to figure out that glanceclient does chunking for us | |
| 16:46:27 | melwitt | hm ok. I didn't think there was OOM killer in these but let me check | |
| 16:46:35 | artom | So I still no have no idea why glance-api gets oom-killed | |
| 16:47:04 | melwitt | oh, yeah there is some OOM kill in here | |
| 16:47:05 | sean-k-mooney | melwitt: there may not be but if there is no error in the glance api log i would first check for an OOM event | |
| 16:47:36 | sean-k-mooney | so we chatted about turning on zswap on nova-next | |
| 16:47:50 | sean-k-mooney | but since we are having OOM events in nova-lvm i might turn it on there too | |
| 16:48:03 | sean-k-mooney | or increase the swap there to 8G if its still at 4G | |
| 16:48:05 | melwitt | yeah, g-api. hm | |
| 16:49:04 | melwitt | Feb 15 13:59:43 np0036770057 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/system-devstack.slice/devstack@g-api.service,task=uwsgi,pid=86510,uid=1001 | |
| 16:51:08 | sean-k-mooney | melwitt: so its runing with 4G of swap today | |
| 16:51:21 | dansmith | also, g-api might not be actually consuming lots of memory, it might just be that the OOM killer decided it was the best thing to kill to create some space | |
| 16:51:34 | dansmith | so look at the oom output to see if it was huge or not | |
| 16:55:55 | opendevreview | sean mooney proposed openstack/nova master: bump nova-lvm to use 8G of swap from 4G https://review.opendev.org/c/openstack/nova/+/909469 | |
| 16:56:01 | melwitt | Feb 15 13:59:43 np0036770057 kernel: Out of memory: Killed process 86510 (uwsgi) total-vm:2290700kB, anon-rss:1733784kB, file-rss:6324kB, shmem-rss:32kB, UID:1001 pgtables:3732kB oom_score_adj:0 4707 so I guess 2.2 GB? not sure if that's typical | |
| 16:56:24 | sean-k-mooney | melwitt: dansmith ^ thats the swap bump and ill look at adding zswap to that and nova-next in a seperate patch later | |
| 16:57:22 | sean-k-mooney | melwitt: so it should be maybe 150-200mb at most normally | |
| 16:57:27 | dansmith | makes sense to try it on nova-lvm too so that seems okay | |
| 16:57:31 | sean-k-mooney | that looks like it holding a guest image in memory | |
| 16:58:13 | sean-k-mooney | that or we need to add an exicit python garbage collector call in the image upload path | |
| 16:58:29 | melwitt | Feb 15 13:59:43 np0036770057 kernel: Free swap = 0kB | |
| 16:58:32 | dansmith | the image upload path doesn't store the image | |
| 16:58:35 | sean-k-mooney | i.e. to egerly run the garbage collector after straming the image | |
| 16:59:28 | dansmith | it's possible that python keeps allocating new buffers for each thing we read and needs a break to do garbage collection, that's certainly possible | |
| 17:00:24 | sean-k-mooney | i belive we have hadd issues with that in eithe glance or cinder before | |
| 17:01:55 | sean-k-mooney | maybe im thinking of swift | |
| 17:01:57 | sean-k-mooney | https://codesearch.opendev.org/?q=gc.collect&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 17:02:11 | sean-k-mooney | there are a few project that have an explcit gc.collect() call | |
| 17:03:00 | sean-k-mooney | hum although mainly in test code | |
| 17:03:11 | dansmith | there was also something about python never deflating the heap, which I think gorka can tell us about | |
| 17:03:49 | sean-k-mooney | ya thats what i was recalling. speaking to gorka about the agents growing there memory usage over time | |
| 17:03:50 | dansmith | but I would think that wouldn't be our problem here since we should be good to re-use those portions of the heap over and over, especially with allocations of exactly the same size each time | |
| 17:04:35 | sean-k-mooney | in anycase bumping to 8G of swap will mitigate the issue for now | |
| 17:05:26 | opendevreview | Jay Faulkner proposed openstack/nova-specs master: Sharding: More detailed test plan https://review.opendev.org/c/openstack/nova-specs/+/908673 | |
| 17:05:52 | JayF | https://review.opendev.org/c/openstack/nova-specs/+/908673 Heads up, I'm going to be implementing the plan laid out here this week; please make loud noises quickly if it's unacceptable :D | |
| 17:07:44 | sean-k-mooney | ack for 2024.2 or are you still hoping to merge the shard stuff for 2024.1 | |
| 17:08:05 | sean-k-mooney | i guess for 2024.1 | |
| 17:08:36 | sean-k-mooney | if you can get the testing working in time before FF i guess that is fine | |
| 17:09:56 | sean-k-mooney | for me the minium i think we need is a 2 node job with to set of vms with 2 ironnic compuge service partitioning the vms into shards within the same conducotr group | |
| 17:11:59 | sean-k-mooney | JayF: having tempest test that acttuly assert the toplogy is proably not in the scope of tempest. it would be in the scope of whitebox or could be done via a differnt tempest plugin | |
| 17:12:13 | gibi | Uggla: i responded in the manila series | |
| 17:12:17 | JayF | sean-k-mooney: It was going to be in ironic-tempest-plugin | |
| 17:12:27 | sean-k-mooney | ack thats proably ok then | |
| 17:12:33 | JayF | Two node job is not something we do in Ironic really | |
| 17:12:51 | JayF | And we are working to slim down our job set; try to get more cases in less VMs | |
| 17:13:14 | sean-k-mooney | JayF: can you deploy 2 nova-compute services on a singel node | |
| 17:13:15 | JayF | so adding more extensive jobs works against that -- but if we validate the resource pool only contains sharded nodes, we're golden (and already have code to do this in devstack that I can lift) | |
| 17:13:29 | JayF | sean-k-mooney: We have zero tests where we do >1 configured nova-compute in Ironic. | |
| 17:13:40 | JayF | sean-k-mooney: and the devstack machinery does not support it | |
| 17:14:09 | sean-k-mooney | right so that is why i was expecting to have a 2 node job | |
| 17:14:24 | sean-k-mooney | each runnign a single nova-compute with ironic | |
| 17:14:36 | sean-k-mooney | but ok we can get a similar effect with what your suggesting | |
| 17:14:44 | sean-k-mooney | it just sound like much more work | |
| 17:15:12 | JayF | No, much much less work. I've already looked into this with Julia and we have a good amount of confidence in this approach. | |
| 17:15:14 | sean-k-mooney | well not nessiarly much more just more then i would have expected | |
| 17:15:42 | sean-k-mooney | ack did you have time to respine the patches to nova for the pep8 error | |
| 17:15:46 | JayF | I just don't want to do it and someone come along and say "not good enough" :D | |
| 17:15:53 | JayF | yeah, I just did | |
| 17:15:57 | sean-k-mooney | cool | |
| 17:16:01 | JayF | right before talking in here because I went to link it and saw the V-1 :D | |
| 17:17:16 | dansmith | I guess I'm confused about how we can know the partitioning works with only one node? | |
| 17:17:18 | sean-k-mooney | ok i put the spec update on teh same gerrit topic as teh code | |
| 17:17:21 | sean-k-mooney | "ironic-shards" | |
| 17:17:53 | sean-k-mooney | dansmith: the job will create more then 1 ironic "vm" and only tag a subset of them with the shard key | |
| 17:18:05 | sean-k-mooney | dansmith: so the job will need to assert that not all the nodes in the ironic node list | |
| 17:18:14 | sean-k-mooney | show up in the RPs | |
| 17:18:16 | JayF | Yeah, we'll end up with e.g. 10 nodes configured (fake drivers, no backend), and only 10-N of them configured with a shard | |
| 17:18:25 | JayF | once the good nodes show up, we start a timer and look for any bad nodes | |
| 17:18:25 | sean-k-mooney | and then ones that do march the expected shard key | |
| 17:18:28 | dansmith | so just making sure that the one nova-ironic we have doesn't pick up the other fake resources outside the configured shard? | |
| 17:18:31 | JayF | exactly | |
| 17:19:23 | dansmith | okay the spec doesn't say ten.. so to be clear, it will be a bunch of fake ironic resources and not just .. one real and one fake, right? | |
| 17:19:38 | dansmith | because the existing hash ring thing will get 1/2 right half the time | |
| 17:19:41 | JayF | with fake nodes, the number can be as many as we want | |
| 17:19:44 | sean-k-mooney | which works and covers the gap we had. it just not how i would have approched it | |
| 17:19:53 | JayF | so if we need a large number of "N" to have confidence, it's trivial to do it | |
| 17:20:02 | JayF | where large is like, small two digit number | |
| 17:20:26 | sean-k-mooney | JayF: we dont but it would have been nice to have more then one nova-compute in the same ironic conductor group with diffent shard keys | |
| 17:20:28 | dansmith | okay I was hoping for better, tbh, but I guess it's better than nothing | |
| 17:20:34 | dansmith | right exactly | |
| 17:20:37 | JayF | sean-k-mooney: doing that same test for conductor groups is a follow on | |
| 17:20:46 | JayF | oh, not exactly what you said | |
| 17:20:54 | JayF | but ensuring we have nodes in good/bad conductor groups and testing there | |
| 17:21:24 | JayF | similarly to how we are planning for shards (so then we'll have CI coverage for conductor group based paritioning, too, which AIUI doesn't exist at all today either) | |
| 17:21:39 | sean-k-mooney | so from a nova point of view we want to ensure we can have multiple nova compute services with idffent shard keys in the same cloud | |
| 17:21:51 | sean-k-mooney | and for each to manage ther own slice | |
| 17:22:06 | sean-k-mooney | we kind of get that by what you propose | |
| 17:22:14 | sean-k-mooney | it just would have been nicer to actully test that directly | |
| 17:22:19 | JayF | We are validating up to the virt driver level, but not up to the compute level | |
| 17:22:46 | JayF | and you'd rather us go up a little bit further, which makes sense | |
| 17:23:18 | JayF | This is an MVP is good; but I will keep an eye out for opportunities to test >1 nova compute -- this is likely something to plan and do as a separate project b/c I suspect it'll be a massive change to the ironic devstack plugin to do so | |
| 17:23:40 | JayF | TheJulia: ^ WDYT; tl;dr how hard would it be to do an Ironic test job with >1 nova-compute pulling from the same Ironic? | |
| 17:23:43 | sean-k-mooney | i dont think it would need much work at all | |
| 17:24:25 | JayF | Maybe I'll reword it to be more personal: that is closer to the approach I was taking last year that kept me running into walls; if it's possible/easy I don't see that path myself | |