| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-19 | |||
| 16:56:24 | sean-k-mooney | melwitt: dansmith ^ thats the swap bump and ill look at adding zswap to that and nova-next in a seperate patch later | |
| 16:57:22 | sean-k-mooney | melwitt: so it should be maybe 150-200mb at most normally | |
| 16:57:27 | dansmith | makes sense to try it on nova-lvm too so that seems okay | |
| 16:57:31 | sean-k-mooney | that looks like it holding a guest image in memory | |
| 16:58:13 | sean-k-mooney | that or we need to add an exicit python garbage collector call in the image upload path | |
| 16:58:29 | melwitt | Feb 15 13:59:43 np0036770057 kernel: Free swap = 0kB | |
| 16:58:32 | dansmith | the image upload path doesn't store the image | |
| 16:58:35 | sean-k-mooney | i.e. to egerly run the garbage collector after straming the image | |
| 16:59:28 | dansmith | it's possible that python keeps allocating new buffers for each thing we read and needs a break to do garbage collection, that's certainly possible | |
| 17:00:24 | sean-k-mooney | i belive we have hadd issues with that in eithe glance or cinder before | |
| 17:01:55 | sean-k-mooney | maybe im thinking of swift | |
| 17:01:57 | sean-k-mooney | https://codesearch.opendev.org/?q=gc.collect&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 17:02:11 | sean-k-mooney | there are a few project that have an explcit gc.collect() call | |
| 17:03:00 | sean-k-mooney | hum although mainly in test code | |
| 17:03:11 | dansmith | there was also something about python never deflating the heap, which I think gorka can tell us about | |
| 17:03:49 | sean-k-mooney | ya thats what i was recalling. speaking to gorka about the agents growing there memory usage over time | |
| 17:03:50 | dansmith | but I would think that wouldn't be our problem here since we should be good to re-use those portions of the heap over and over, especially with allocations of exactly the same size each time | |
| 17:04:35 | sean-k-mooney | in anycase bumping to 8G of swap will mitigate the issue for now | |
| 17:05:26 | opendevreview | Jay Faulkner proposed openstack/nova-specs master: Sharding: More detailed test plan https://review.opendev.org/c/openstack/nova-specs/+/908673 | |
| 17:05:52 | JayF | https://review.opendev.org/c/openstack/nova-specs/+/908673 Heads up, I'm going to be implementing the plan laid out here this week; please make loud noises quickly if it's unacceptable :D | |
| 17:07:44 | sean-k-mooney | ack for 2024.2 or are you still hoping to merge the shard stuff for 2024.1 | |
| 17:08:05 | sean-k-mooney | i guess for 2024.1 | |
| 17:08:36 | sean-k-mooney | if you can get the testing working in time before FF i guess that is fine | |
| 17:09:56 | sean-k-mooney | for me the minium i think we need is a 2 node job with to set of vms with 2 ironnic compuge service partitioning the vms into shards within the same conducotr group | |
| 17:11:59 | sean-k-mooney | JayF: having tempest test that acttuly assert the toplogy is proably not in the scope of tempest. it would be in the scope of whitebox or could be done via a differnt tempest plugin | |
| 17:12:13 | gibi | Uggla: i responded in the manila series | |
| 17:12:17 | JayF | sean-k-mooney: It was going to be in ironic-tempest-plugin | |
| 17:12:27 | sean-k-mooney | ack thats proably ok then | |
| 17:12:33 | JayF | Two node job is not something we do in Ironic really | |
| 17:12:51 | JayF | And we are working to slim down our job set; try to get more cases in less VMs | |
| 17:13:14 | sean-k-mooney | JayF: can you deploy 2 nova-compute services on a singel node | |
| 17:13:15 | JayF | so adding more extensive jobs works against that -- but if we validate the resource pool only contains sharded nodes, we're golden (and already have code to do this in devstack that I can lift) | |
| 17:13:29 | JayF | sean-k-mooney: We have zero tests where we do >1 configured nova-compute in Ironic. | |
| 17:13:40 | JayF | sean-k-mooney: and the devstack machinery does not support it | |
| 17:14:09 | sean-k-mooney | right so that is why i was expecting to have a 2 node job | |
| 17:14:24 | sean-k-mooney | each runnign a single nova-compute with ironic | |
| 17:14:36 | sean-k-mooney | but ok we can get a similar effect with what your suggesting | |
| 17:14:44 | sean-k-mooney | it just sound like much more work | |
| 17:15:12 | JayF | No, much much less work. I've already looked into this with Julia and we have a good amount of confidence in this approach. | |
| 17:15:14 | sean-k-mooney | well not nessiarly much more just more then i would have expected | |
| 17:15:42 | sean-k-mooney | ack did you have time to respine the patches to nova for the pep8 error | |
| 17:15:46 | JayF | I just don't want to do it and someone come along and say "not good enough" :D | |
| 17:15:53 | JayF | yeah, I just did | |
| 17:15:57 | sean-k-mooney | cool | |
| 17:16:01 | JayF | right before talking in here because I went to link it and saw the V-1 :D | |
| 17:17:16 | dansmith | I guess I'm confused about how we can know the partitioning works with only one node? | |
| 17:17:18 | sean-k-mooney | ok i put the spec update on teh same gerrit topic as teh code | |
| 17:17:21 | sean-k-mooney | "ironic-shards" | |
| 17:17:53 | sean-k-mooney | dansmith: the job will create more then 1 ironic "vm" and only tag a subset of them with the shard key | |
| 17:18:05 | sean-k-mooney | dansmith: so the job will need to assert that not all the nodes in the ironic node list | |
| 17:18:14 | sean-k-mooney | show up in the RPs | |
| 17:18:16 | JayF | Yeah, we'll end up with e.g. 10 nodes configured (fake drivers, no backend), and only 10-N of them configured with a shard | |
| 17:18:25 | JayF | once the good nodes show up, we start a timer and look for any bad nodes | |
| 17:18:25 | sean-k-mooney | and then ones that do march the expected shard key | |
| 17:18:28 | dansmith | so just making sure that the one nova-ironic we have doesn't pick up the other fake resources outside the configured shard? | |
| 17:18:31 | JayF | exactly | |
| 17:19:23 | dansmith | okay the spec doesn't say ten.. so to be clear, it will be a bunch of fake ironic resources and not just .. one real and one fake, right? | |
| 17:19:38 | dansmith | because the existing hash ring thing will get 1/2 right half the time | |
| 17:19:41 | JayF | with fake nodes, the number can be as many as we want | |
| 17:19:44 | sean-k-mooney | which works and covers the gap we had. it just not how i would have approched it | |
| 17:19:53 | JayF | so if we need a large number of "N" to have confidence, it's trivial to do it | |
| 17:20:02 | JayF | where large is like, small two digit number | |
| 17:20:26 | sean-k-mooney | JayF: we dont but it would have been nice to have more then one nova-compute in the same ironic conductor group with diffent shard keys | |
| 17:20:28 | dansmith | okay I was hoping for better, tbh, but I guess it's better than nothing | |
| 17:20:34 | dansmith | right exactly | |
| 17:20:37 | JayF | sean-k-mooney: doing that same test for conductor groups is a follow on | |
| 17:20:46 | JayF | oh, not exactly what you said | |
| 17:20:54 | JayF | but ensuring we have nodes in good/bad conductor groups and testing there | |
| 17:21:24 | JayF | similarly to how we are planning for shards (so then we'll have CI coverage for conductor group based paritioning, too, which AIUI doesn't exist at all today either) | |
| 17:21:39 | sean-k-mooney | so from a nova point of view we want to ensure we can have multiple nova compute services with idffent shard keys in the same cloud | |
| 17:21:51 | sean-k-mooney | and for each to manage ther own slice | |
| 17:22:06 | sean-k-mooney | we kind of get that by what you propose | |
| 17:22:14 | sean-k-mooney | it just would have been nicer to actully test that directly | |
| 17:22:19 | JayF | We are validating up to the virt driver level, but not up to the compute level | |
| 17:22:46 | JayF | and you'd rather us go up a little bit further, which makes sense | |
| 17:23:18 | JayF | This is an MVP is good; but I will keep an eye out for opportunities to test >1 nova compute -- this is likely something to plan and do as a separate project b/c I suspect it'll be a massive change to the ironic devstack plugin to do so | |
| 17:23:40 | JayF | TheJulia: ^ WDYT; tl;dr how hard would it be to do an Ironic test job with >1 nova-compute pulling from the same Ironic? | |
| 17:23:43 | sean-k-mooney | i dont think it would need much work at all | |
| 17:24:25 | JayF | Maybe I'll reword it to be more personal: that is closer to the approach I was taking last year that kept me running into walls; if it's possible/easy I don't see that path myself | |
| 17:24:28 | sean-k-mooney | JayF: the way multi node plugins are ment to work is runnign the same plugin once per node just with bit enabled/disbale as needed | |
| 17:24:56 | sean-k-mooney | JayF: so to add a second nova-comptue with ironic we would jsut turn off all the bits the plugin does for the ironic api extra | |
| 17:25:03 | JayF | I think you're ignoring that we'd have to have the second node talk to Ironic on the first node, which I think would require some network magic | |
| 17:25:17 | sean-k-mooney | and only deploy nova-compute with the ironic driver with a difffen tshard key configured | |
| 17:25:43 | sean-k-mooney | JayF: yes but we have that for all the other multi node jobs | |
| 17:26:02 | sean-k-mooney | anyway when you have a job up for review | |
| 17:26:11 | sean-k-mooney | we can look at makeing a multi node variant later | |
| 17:26:38 | JayF | I am not sure Ironic community at large will be thrilled to add multinode to our gate complexities, but timing-wise that's probably PTG topic anyway. | |
| 17:27:09 | sean-k-mooney | if we are not crating any addtional fake nodes then we can likely do it with just a config overried to change the virt driver to ironic and populate the nova.conf with the ironic details | |
| 17:27:11 | dansmith | ironic has no multinode jobs? | |
| 17:27:44 | TheJulia | We have one, adding further complexity to it is not ideal | |
| 17:27:49 | Uggla | gibi, thx, I will have a look. | |
| 17:28:00 | dansmith | eesh | |
| 17:28:14 | JayF | oh, we do > ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode https://zuul.opendev.org/t/openstack/build/7a8be5f2987745f390f6078e8ac87d82 : SUCCESS in 1h 14m 03s (non-voting) | |
| 17:28:16 | JayF | non-voting | |
| 17:28:18 | TheJulia | JayF: to the orignal ask, that is how the job operates now | |
| 17:28:41 | JayF | Ack, so as a second step, I can look into making that job also test sharding | |
| 17:28:57 | TheJulia | I think it just would if set, fwiw | |
| 17:29:08 | dansmith | I guess I'm not sure what configuration is more deserving of a multinode layout than the configuration that arranges for more-than-trivial nodes | |
| 17:29:39 | JayF | I honestly don't know what that job tests; it'll be on my plate to look into it | |
| 17:30:12 | JayF | if I can just flip sharding on in that job and use it to test, that'd be even simpler (although I still think there's value in doing the "check resources" type job, because it'll fail more cleanly if we break things) | |