| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-19 | |||
| 17:13:29 | JayF | sean-k-mooney: We have zero tests where we do >1 configured nova-compute in Ironic. | |
| 17:13:40 | JayF | sean-k-mooney: and the devstack machinery does not support it | |
| 17:14:09 | sean-k-mooney | right so that is why i was expecting to have a 2 node job | |
| 17:14:24 | sean-k-mooney | each runnign a single nova-compute with ironic | |
| 17:14:36 | sean-k-mooney | but ok we can get a similar effect with what your suggesting | |
| 17:14:44 | sean-k-mooney | it just sound like much more work | |
| 17:15:12 | JayF | No, much much less work. I've already looked into this with Julia and we have a good amount of confidence in this approach. | |
| 17:15:14 | sean-k-mooney | well not nessiarly much more just more then i would have expected | |
| 17:15:42 | sean-k-mooney | ack did you have time to respine the patches to nova for the pep8 error | |
| 17:15:46 | JayF | I just don't want to do it and someone come along and say "not good enough" :D | |
| 17:15:53 | JayF | yeah, I just did | |
| 17:15:57 | sean-k-mooney | cool | |
| 17:16:01 | JayF | right before talking in here because I went to link it and saw the V-1 :D | |
| 17:17:16 | dansmith | I guess I'm confused about how we can know the partitioning works with only one node? | |
| 17:17:18 | sean-k-mooney | ok i put the spec update on teh same gerrit topic as teh code | |
| 17:17:21 | sean-k-mooney | "ironic-shards" | |
| 17:17:53 | sean-k-mooney | dansmith: the job will create more then 1 ironic "vm" and only tag a subset of them with the shard key | |
| 17:18:05 | sean-k-mooney | dansmith: so the job will need to assert that not all the nodes in the ironic node list | |
| 17:18:14 | sean-k-mooney | show up in the RPs | |
| 17:18:16 | JayF | Yeah, we'll end up with e.g. 10 nodes configured (fake drivers, no backend), and only 10-N of them configured with a shard | |
| 17:18:25 | JayF | once the good nodes show up, we start a timer and look for any bad nodes | |
| 17:18:25 | sean-k-mooney | and then ones that do march the expected shard key | |
| 17:18:28 | dansmith | so just making sure that the one nova-ironic we have doesn't pick up the other fake resources outside the configured shard? | |
| 17:18:31 | JayF | exactly | |
| 17:19:23 | dansmith | okay the spec doesn't say ten.. so to be clear, it will be a bunch of fake ironic resources and not just .. one real and one fake, right? | |
| 17:19:38 | dansmith | because the existing hash ring thing will get 1/2 right half the time | |
| 17:19:41 | JayF | with fake nodes, the number can be as many as we want | |
| 17:19:44 | sean-k-mooney | which works and covers the gap we had. it just not how i would have approched it | |
| 17:19:53 | JayF | so if we need a large number of "N" to have confidence, it's trivial to do it | |
| 17:20:02 | JayF | where large is like, small two digit number | |
| 17:20:26 | sean-k-mooney | JayF: we dont but it would have been nice to have more then one nova-compute in the same ironic conductor group with diffent shard keys | |
| 17:20:28 | dansmith | okay I was hoping for better, tbh, but I guess it's better than nothing | |
| 17:20:34 | dansmith | right exactly | |
| 17:20:37 | JayF | sean-k-mooney: doing that same test for conductor groups is a follow on | |
| 17:20:46 | JayF | oh, not exactly what you said | |
| 17:20:54 | JayF | but ensuring we have nodes in good/bad conductor groups and testing there | |
| 17:21:24 | JayF | similarly to how we are planning for shards (so then we'll have CI coverage for conductor group based paritioning, too, which AIUI doesn't exist at all today either) | |
| 17:21:39 | sean-k-mooney | so from a nova point of view we want to ensure we can have multiple nova compute services with idffent shard keys in the same cloud | |
| 17:21:51 | sean-k-mooney | and for each to manage ther own slice | |
| 17:22:06 | sean-k-mooney | we kind of get that by what you propose | |
| 17:22:14 | sean-k-mooney | it just would have been nicer to actully test that directly | |
| 17:22:19 | JayF | We are validating up to the virt driver level, but not up to the compute level | |
| 17:22:46 | JayF | and you'd rather us go up a little bit further, which makes sense | |
| 17:23:18 | JayF | This is an MVP is good; but I will keep an eye out for opportunities to test >1 nova compute -- this is likely something to plan and do as a separate project b/c I suspect it'll be a massive change to the ironic devstack plugin to do so | |
| 17:23:40 | JayF | TheJulia: ^ WDYT; tl;dr how hard would it be to do an Ironic test job with >1 nova-compute pulling from the same Ironic? | |
| 17:23:43 | sean-k-mooney | i dont think it would need much work at all | |
| 17:24:25 | JayF | Maybe I'll reword it to be more personal: that is closer to the approach I was taking last year that kept me running into walls; if it's possible/easy I don't see that path myself | |
| 17:24:28 | sean-k-mooney | JayF: the way multi node plugins are ment to work is runnign the same plugin once per node just with bit enabled/disbale as needed | |
| 17:24:56 | sean-k-mooney | JayF: so to add a second nova-comptue with ironic we would jsut turn off all the bits the plugin does for the ironic api extra | |
| 17:25:03 | JayF | I think you're ignoring that we'd have to have the second node talk to Ironic on the first node, which I think would require some network magic | |
| 17:25:17 | sean-k-mooney | and only deploy nova-compute with the ironic driver with a difffen tshard key configured | |
| 17:25:43 | sean-k-mooney | JayF: yes but we have that for all the other multi node jobs | |
| 17:26:02 | sean-k-mooney | anyway when you have a job up for review | |
| 17:26:11 | sean-k-mooney | we can look at makeing a multi node variant later | |
| 17:26:38 | JayF | I am not sure Ironic community at large will be thrilled to add multinode to our gate complexities, but timing-wise that's probably PTG topic anyway. | |
| 17:27:09 | sean-k-mooney | if we are not crating any addtional fake nodes then we can likely do it with just a config overried to change the virt driver to ironic and populate the nova.conf with the ironic details | |
| 17:27:11 | dansmith | ironic has no multinode jobs? | |
| 17:27:44 | TheJulia | We have one, adding further complexity to it is not ideal | |
| 17:27:49 | Uggla | gibi, thx, I will have a look. | |
| 17:28:00 | dansmith | eesh | |
| 17:28:14 | JayF | oh, we do > ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode https://zuul.opendev.org/t/openstack/build/7a8be5f2987745f390f6078e8ac87d82 : SUCCESS in 1h 14m 03s (non-voting) | |
| 17:28:16 | JayF | non-voting | |
| 17:28:18 | TheJulia | JayF: to the orignal ask, that is how the job operates now | |
| 17:28:41 | JayF | Ack, so as a second step, I can look into making that job also test sharding | |
| 17:28:57 | TheJulia | I think it just would if set, fwiw | |
| 17:29:08 | dansmith | I guess I'm not sure what configuration is more deserving of a multinode layout than the configuration that arranges for more-than-trivial nodes | |
| 17:29:39 | JayF | I honestly don't know what that job tests; it'll be on my plate to look into it | |
| 17:30:12 | JayF | if I can just flip sharding on in that job and use it to test, that'd be even simpler (although I still think there's value in doing the "check resources" type job, because it'll fail more cleanly if we break things) | |
| 17:31:03 | sean-k-mooney | there is some logic in devstack to run multiple fake computes on a single node | |
| 17:31:56 | sean-k-mooney | so we could (next cycle) explore doign that for ironic. but honestly i think all that is need to enable the comptue service on a secnd node and set the virt driver to ironci and set the other config options | |
| 17:31:59 | TheJulia | JayF: multiple ironic-conductors and nova-computes if I'm recalling correctly | |
| 17:32:15 | TheJulia | It has just been a while since I've done any work on it | |
| 17:32:30 | JayF | oooh, so is that the job I dind't know existed that does coverage for conductor_groups | |
| 17:32:33 | JayF | that would make a lot of sense, actually | |
| 17:33:57 | sean-k-mooney | thats one of the two jobs we have in our expermiental pipeline | |
| 17:33:59 | sean-k-mooney | https://github.com/openstack/nova/blob/master/.zuul.yaml#L981-L990 | |
| 17:34:15 | sean-k-mooney | we have ironic-tempest-bfv and ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode | |
| 17:35:04 | JayF | I think that is the whole mystery laid out there then | |
| 17:35:12 | JayF | looks like I have a template job, and need to just shardify it | |
| 17:35:56 | sean-k-mooney | ya so this is how hte second node is configured https://github.com/openstack/ironic/blob/master/zuul.d/ironic-jobs.yaml#L692-L733 | |
| 17:36:36 | JayF | those OVS config bits where the parts I was worried about figuring out, honestly | |
| 17:37:00 | sean-k-mooney | well thats common with the libvirt jobs | |
| 17:37:09 | sean-k-mooney | for the most part anyway | |
| 17:37:34 | JayF | My Ironic knowledge is concentrated closer to the endpoint :) I'm working to spread it up the stack more and more over time but networking is a big blind spot | |
| 17:39:00 | sean-k-mooney | JayF: what i would suggest is focus on the testing you had planed before | |
| 17:39:32 | sean-k-mooney | if we can speratelly add teh config seeting to tag the ironic nodes in that other job with a shard key | |
| 17:39:57 | sean-k-mooney | and configure the relevnet nova-computes on ech host to look at only the shard key for the mv on the relevent host | |
| 17:40:05 | sean-k-mooney | that will cover us for the multi node job | |
| 17:40:24 | JayF | Well, except for the "you should | |
| 17:40:32 | JayF | do the other one first" that's what I had in mind | |
| 17:41:14 | sean-k-mooney | well if you want to do the multi node one first that fine | |
| 17:41:15 | JayF | I'll take your advice and still keep on the path I added to the spec -- even though you did an ace job of making the multinode seem easier :D | |
| 17:41:30 | sean-k-mooney | i was just hoping you could provide an exmaple of configuring it | |
| 17:41:39 | sean-k-mooney | then we could help prot it to the multi node one | |
| 17:41:56 | opendevreview | Fabian Wiesel proposed openstack/nova master: Do not close returned image-chunk iterator https://review.opendev.org/c/openstack/nova/+/909474 | |
| 17:43:12 | sean-k-mooney | JayF: you can set teh shard key in the nova.conf trivially like this https://github.com/openstack/nova/blob/master/.zuul.yaml#L459-L485 | |
| 17:43:57 | sean-k-mooney | im not sure how you go about tagging the relevent fake ironics with a shard key | |
| 17:44:46 | sean-k-mooney | if really needed to hack thing quickly i would use a local.sh file created by a pre playbook | |
| 17:45:10 | sean-k-mooney | like this https://review.opendev.org/c/openstack/nova/+/679656/13/playbooks/nfv/nfv.yaml | |
| 17:45:24 | sean-k-mooney | but i assume you have a nicer way to do that in teh ironic plugin | |