| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-19 | |||
| 17:18:05 | sean-k-mooney | dansmith: so the job will need to assert that not all the nodes in the ironic node list | |
| 17:18:14 | sean-k-mooney | show up in the RPs | |
| 17:18:16 | JayF | Yeah, we'll end up with e.g. 10 nodes configured (fake drivers, no backend), and only 10-N of them configured with a shard | |
| 17:18:25 | JayF | once the good nodes show up, we start a timer and look for any bad nodes | |
| 17:18:25 | sean-k-mooney | and then ones that do march the expected shard key | |
| 17:18:28 | dansmith | so just making sure that the one nova-ironic we have doesn't pick up the other fake resources outside the configured shard? | |
| 17:18:31 | JayF | exactly | |
| 17:19:23 | dansmith | okay the spec doesn't say ten.. so to be clear, it will be a bunch of fake ironic resources and not just .. one real and one fake, right? | |
| 17:19:38 | dansmith | because the existing hash ring thing will get 1/2 right half the time | |
| 17:19:41 | JayF | with fake nodes, the number can be as many as we want | |
| 17:19:44 | sean-k-mooney | which works and covers the gap we had. it just not how i would have approched it | |
| 17:19:53 | JayF | so if we need a large number of "N" to have confidence, it's trivial to do it | |
| 17:20:02 | JayF | where large is like, small two digit number | |
| 17:20:26 | sean-k-mooney | JayF: we dont but it would have been nice to have more then one nova-compute in the same ironic conductor group with diffent shard keys | |
| 17:20:28 | dansmith | okay I was hoping for better, tbh, but I guess it's better than nothing | |
| 17:20:34 | dansmith | right exactly | |
| 17:20:37 | JayF | sean-k-mooney: doing that same test for conductor groups is a follow on | |
| 17:20:46 | JayF | oh, not exactly what you said | |
| 17:20:54 | JayF | but ensuring we have nodes in good/bad conductor groups and testing there | |
| 17:21:24 | JayF | similarly to how we are planning for shards (so then we'll have CI coverage for conductor group based paritioning, too, which AIUI doesn't exist at all today either) | |
| 17:21:39 | sean-k-mooney | so from a nova point of view we want to ensure we can have multiple nova compute services with idffent shard keys in the same cloud | |
| 17:21:51 | sean-k-mooney | and for each to manage ther own slice | |
| 17:22:06 | sean-k-mooney | we kind of get that by what you propose | |
| 17:22:14 | sean-k-mooney | it just would have been nicer to actully test that directly | |
| 17:22:19 | JayF | We are validating up to the virt driver level, but not up to the compute level | |
| 17:22:46 | JayF | and you'd rather us go up a little bit further, which makes sense | |
| 17:23:18 | JayF | This is an MVP is good; but I will keep an eye out for opportunities to test >1 nova compute -- this is likely something to plan and do as a separate project b/c I suspect it'll be a massive change to the ironic devstack plugin to do so | |
| 17:23:40 | JayF | TheJulia: ^ WDYT; tl;dr how hard would it be to do an Ironic test job with >1 nova-compute pulling from the same Ironic? | |
| 17:23:43 | sean-k-mooney | i dont think it would need much work at all | |
| 17:24:25 | JayF | Maybe I'll reword it to be more personal: that is closer to the approach I was taking last year that kept me running into walls; if it's possible/easy I don't see that path myself | |
| 17:24:28 | sean-k-mooney | JayF: the way multi node plugins are ment to work is runnign the same plugin once per node just with bit enabled/disbale as needed | |
| 17:24:56 | sean-k-mooney | JayF: so to add a second nova-comptue with ironic we would jsut turn off all the bits the plugin does for the ironic api extra | |
| 17:25:03 | JayF | I think you're ignoring that we'd have to have the second node talk to Ironic on the first node, which I think would require some network magic | |
| 17:25:17 | sean-k-mooney | and only deploy nova-compute with the ironic driver with a difffen tshard key configured | |
| 17:25:43 | sean-k-mooney | JayF: yes but we have that for all the other multi node jobs | |
| 17:26:02 | sean-k-mooney | anyway when you have a job up for review | |
| 17:26:11 | sean-k-mooney | we can look at makeing a multi node variant later | |
| 17:26:38 | JayF | I am not sure Ironic community at large will be thrilled to add multinode to our gate complexities, but timing-wise that's probably PTG topic anyway. | |
| 17:27:09 | sean-k-mooney | if we are not crating any addtional fake nodes then we can likely do it with just a config overried to change the virt driver to ironic and populate the nova.conf with the ironic details | |
| 17:27:11 | dansmith | ironic has no multinode jobs? | |
| 17:27:44 | TheJulia | We have one, adding further complexity to it is not ideal | |
| 17:27:49 | Uggla | gibi, thx, I will have a look. | |
| 17:28:00 | dansmith | eesh | |
| 17:28:14 | JayF | oh, we do > ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode https://zuul.opendev.org/t/openstack/build/7a8be5f2987745f390f6078e8ac87d82 : SUCCESS in 1h 14m 03s (non-voting) | |
| 17:28:16 | JayF | non-voting | |
| 17:28:18 | TheJulia | JayF: to the orignal ask, that is how the job operates now | |
| 17:28:41 | JayF | Ack, so as a second step, I can look into making that job also test sharding | |
| 17:28:57 | TheJulia | I think it just would if set, fwiw | |
| 17:29:08 | dansmith | I guess I'm not sure what configuration is more deserving of a multinode layout than the configuration that arranges for more-than-trivial nodes | |
| 17:29:39 | JayF | I honestly don't know what that job tests; it'll be on my plate to look into it | |
| 17:30:12 | JayF | if I can just flip sharding on in that job and use it to test, that'd be even simpler (although I still think there's value in doing the "check resources" type job, because it'll fail more cleanly if we break things) | |
| 17:31:03 | sean-k-mooney | there is some logic in devstack to run multiple fake computes on a single node | |
| 17:31:56 | sean-k-mooney | so we could (next cycle) explore doign that for ironic. but honestly i think all that is need to enable the comptue service on a secnd node and set the virt driver to ironci and set the other config options | |
| 17:31:59 | TheJulia | JayF: multiple ironic-conductors and nova-computes if I'm recalling correctly | |
| 17:32:15 | TheJulia | It has just been a while since I've done any work on it | |
| 17:32:30 | JayF | oooh, so is that the job I dind't know existed that does coverage for conductor_groups | |
| 17:32:33 | JayF | that would make a lot of sense, actually | |
| 17:33:57 | sean-k-mooney | thats one of the two jobs we have in our expermiental pipeline | |
| 17:33:59 | sean-k-mooney | https://github.com/openstack/nova/blob/master/.zuul.yaml#L981-L990 | |
| 17:34:15 | sean-k-mooney | we have ironic-tempest-bfv and ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode | |
| 17:35:04 | JayF | I think that is the whole mystery laid out there then | |
| 17:35:12 | JayF | looks like I have a template job, and need to just shardify it | |
| 17:35:56 | sean-k-mooney | ya so this is how hte second node is configured https://github.com/openstack/ironic/blob/master/zuul.d/ironic-jobs.yaml#L692-L733 | |
| 17:36:36 | JayF | those OVS config bits where the parts I was worried about figuring out, honestly | |
| 17:37:00 | sean-k-mooney | well thats common with the libvirt jobs | |
| 17:37:09 | sean-k-mooney | for the most part anyway | |
| 17:37:34 | JayF | My Ironic knowledge is concentrated closer to the endpoint :) I'm working to spread it up the stack more and more over time but networking is a big blind spot | |
| 17:39:00 | sean-k-mooney | JayF: what i would suggest is focus on the testing you had planed before | |
| 17:39:32 | sean-k-mooney | if we can speratelly add teh config seeting to tag the ironic nodes in that other job with a shard key | |
| 17:39:57 | sean-k-mooney | and configure the relevnet nova-computes on ech host to look at only the shard key for the mv on the relevent host | |
| 17:40:05 | sean-k-mooney | that will cover us for the multi node job | |
| 17:40:24 | JayF | Well, except for the "you should | |
| 17:40:32 | JayF | do the other one first" that's what I had in mind | |
| 17:41:14 | sean-k-mooney | well if you want to do the multi node one first that fine | |
| 17:41:15 | JayF | I'll take your advice and still keep on the path I added to the spec -- even though you did an ace job of making the multinode seem easier :D | |
| 17:41:30 | sean-k-mooney | i was just hoping you could provide an exmaple of configuring it | |
| 17:41:39 | sean-k-mooney | then we could help prot it to the multi node one | |
| 17:41:56 | opendevreview | Fabian Wiesel proposed openstack/nova master: Do not close returned image-chunk iterator https://review.opendev.org/c/openstack/nova/+/909474 | |
| 17:43:12 | sean-k-mooney | JayF: you can set teh shard key in the nova.conf trivially like this https://github.com/openstack/nova/blob/master/.zuul.yaml#L459-L485 | |
| 17:43:57 | sean-k-mooney | im not sure how you go about tagging the relevent fake ironics with a shard key | |
| 17:44:46 | sean-k-mooney | if really needed to hack thing quickly i would use a local.sh file created by a pre playbook | |
| 17:45:10 | sean-k-mooney | like this https://review.opendev.org/c/openstack/nova/+/679656/13/playbooks/nfv/nfv.yaml | |
| 17:45:24 | sean-k-mooney | but i assume you have a nicer way to do that in teh ironic plugin | |
| 17:45:37 | sean-k-mooney | ideally via a local.conf parmater | |
| 17:46:40 | sean-k-mooney | JayF: if your not familar with /opt/stack/devstack/local.sh if it exist it run right before devstack finishes so if a job creates on eit will be run after devstack has otherwise run fully but before any tests are run | |
| 17:47:31 | sean-k-mooney | so its a way to hack in funcitonaltiy quickly via a zuul job that hsould eventually be in a devstack plugin | |
| 17:50:27 | JayF | honestly that's as nifty for local dev setup as it is for a CI trick, ty | |
| 17:50:58 | sean-k-mooney | yes so the reason ci jobs should not use it hevialy | |
| 17:51:05 | sean-k-mooney | is because its ment to be reserved for local dev | |
| 17:51:12 | sean-k-mooney | but they can use it | |
| 17:51:28 | sean-k-mooney | plugins just should not require it to have stuff to work | |
| 17:51:46 | sean-k-mooney | since its for the user not the "framework/plugins" to use | |
| 17:52:32 | sean-k-mooney | so form a ci point of view its perfectly valid to use | |
| 17:52:46 | sean-k-mooney | but if its complex it proably shoudl be a feature in the plugin/devstack instead | |
| 17:54:00 | JayF | I mean, yeah, it's kinda obvious it'd be hacky to land a CI job using that | |
| 17:56:58 | sean-k-mooney | JayF: i was epecting to see something in enroll_nodes https://github.com/openstack/ironic/blob/245505dc04da6e401e1810812809fef3f3757f04/devstack/lib/ironic#L2431 | |
| 17:57:09 | sean-k-mooney | to set the shard key on the node | |
| 17:57:12 | sean-k-mooney | did i miss that | |
| 17:57:49 | JayF | shard key is optional; you only need it when setting things up | |
| 17:58:07 | JayF | and it's unused upstream until the nova change lands | |