Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-03-31
14:55:40 sean-k-mooney i think bauzas was just to busy to finish it
14:55:48 dansmith okay
14:55:54 sean-k-mooney i dont know of anything blockign completing the work
14:55:56 bauzas I'm pretty burned about vgpu but I can try to update my mtty series
14:56:49 sean-k-mooney the other thing gibi and i were hopign to do is after a coulple of the provider get to epoxy we would like to see if we can get image with igb enbaled
14:56:58 sean-k-mooney so we can test sriov
14:57:07 sean-k-mooney not sure when that will happen but its related.
14:57:18 Uggla bauzas, maybe you can offload some vgpu stuff to me.
14:59:12 sean-k-mooney dansmith: on a related note but changing the topic slightly https://review.opendev.org/c/openstack/placement/+/945465 is being tracked as a bugfix, but do you think its a backportable change or master only
14:59:31 gibi https://etherpad.opendev.org/p/nova-2025.2-ptg#L378
14:59:53 dansmith sean-k-mooney: I think it's backportable and would be a good candidate, personally, since it's very self-contained and the bug really sucks for operators, IMHO
15:00:18 dansmith sean-k-mooney: but, I have not been able to test that it actually fixes the nova migration problems yet since I need two computes to do it
15:00:35 sean-k-mooney ack my follow up is if that backportable do we need nova work for the existing nova bug or do you think cold migration ectra woudl just start working?
15:00:51 dansmith I believe this should make it just work on the nova side
15:01:42 sean-k-mooney ack, not that our downsteam qe have a lot of capsity for upstream work but we might want to talk to them about how we coudl test the previously broken operation in upstream tempest
15:02:23 sean-k-mooney well or whitebox
15:02:32 dansmith again, this requires putting the compute node into overcapacity state with an allocation_ratio change, so I don't really think we can do that in upstream parallel tempest
15:02:38 dansmith whitebox potentially
15:04:34 sean-k-mooney tempest has the ida of serial test that run one by one. that might help here but many of these edge cases have the potentil to conflict wth other test so i guess we would need to be carful about how we would test this
15:05:05 dansmith yeah, true.. it just seems sort of off the table for tempest by scope too, but perhaps not
15:05:38 sean-k-mooney the other concern i have is will this break any current functionl tests if we have repoducer for any of the nova bugs
15:05:44 sean-k-mooney i dont think we do
15:06:02 sean-k-mooney but if you fix it in placment it will presumable sto raising an error with the placement driect fixture
15:06:14 dansmith if we do, I'm not aware of them
15:06:51 gibi having a placement only tempest test for the overallocation fix could be written to be independent from any other test by creating a separate RP, but I'm not sure that worth it compared to a full nova migration test with overallocation.
15:07:15 sean-k-mooney ok in that case i realy dont have any other question related to this.
15:07:27 sean-k-mooney gibi: that could be done but gabit kind fo does that already right
15:07:38 gibi yeah
15:07:40 dansmith gibi: yeah could definitely do that but the placement functional tests feels just as good for that to me
15:08:01 gibi I a big believer in functional tests :)
15:08:08 gibi I'm
15:11:04 dansmith certainly for this they're plenty adequate I think
16:19:10 dansmith sean-k-mooney: didn't you have a recipe for using the upstream ansible playbooks to deploy multinode devstack locally?
16:54:23 ivveh hey, im looking to be pointed in the correct direction. i want to nova to build my custom xmls. like hooks or some other method that is supported by nova and preferably has a framework for that. i'd like to avoid modifying the libvirt driver as much as possible
17:05:32 sean-k-mooney dansmith: sorry yes i have
17:07:47 sean-k-mooney dansmith: https://github.com/SeanMooney/ard/blob/master/ansible/deploy_multinode_devstack.yaml is the playbook that does it that was driven by molecule with vagrant_libvirt to provdie the vms https://github.com/SeanMooney/ard/blob/master/molecule/default/molecule.yml
17:08:11 dansmith ah with vagrant
17:08:24 sean-k-mooney you didnt need vagrant
17:08:41 sean-k-mooney the playbooks could work with any servers once you hadd ssh access
17:09:28 sean-k-mooney so like i provisioned some server internally for vdpa testing and just poited the palybooks at it with an inventory file
17:09:45 sean-k-mooney https://github.com/SeanMooney/ard/tree/master/examples/vdpa
17:11:13 sean-k-mooney i wrote 3 roles devstack_{common,compute,controller} that wrappted the upstream devstack roles form teh gate jobs
17:11:17 sean-k-mooney https://github.com/SeanMooney/ard/tree/master/ansible/roles
17:12:49 sean-k-mooney dansmith: its been like 3+ years since i used any of this
17:12:59 sean-k-mooney it proably still wroks but i havent been maintining it
17:13:15 sean-k-mooney dansmith: was there a specific reason you asked
17:14:22 dansmith sean-k-mooney: I was struggling through re-re-remembering the multinode devstack rain dance
17:14:28 dansmith but I got it figured out whilst waiting
17:14:37 sean-k-mooney ah ok
17:14:44 dansmith sean-k-mooney: gibi: can confirm that just that placement change fixes the nova migrate-while-over-capacity bug
17:14:52 sean-k-mooney i have some downstream docs of the process with bash
17:15:09 dansmith I was able to repro, then literally swapped placement to that commit, restarted, retried the same migration and it went, confirmed, all good
17:15:18 sean-k-mooney and one of my team recently wrote an ansible playbook to deploy multi node devstack on our internal cloud i can ping you does internally
17:15:33 sean-k-mooney dansmith: oh nice
17:16:24 dansmith sean-k-mooney: thanks, I need to formalize my own notes, because I have some of my own scripting and making things work with those is most ideal,
17:16:31 dansmith I just suck at persisting the notes after I get it workin,
17:16:39 dansmith but I'm going to do that now
17:18:11 ivveh as more context to my previous question is that i want to include iothread stanza into the xml (for now)
17:19:58 sean-k-mooney dansmith: the main parts of multi node are syncing the tls/ceph data, exchantign the ssh keys/known_hosts, ntp sync and updating /etc/hosts before deploying the comptues. then adding a discover host at the end
17:20:28 dansmith yeah, it was just the CA stuff I forgot this time
17:20:56 dansmith as soon as it failed, I knew the problem, but then had to remember to copy the CA dir *and* the bundle, lest it would nuke the latter on stacking the subnode
17:21:01 sean-k-mooney did you know that tls is appreantly not on by default in devstack. i always used the ci jobs as a basis for mine
17:21:14 sean-k-mooney but if you use devstack by default it does not enabel the tls-proxy
17:21:31 dansmith yeah, I have it on my base node though, so..
17:21:43 gibi dansmith: nice test results
17:22:12 dansmith gibi: also works for the case where reserved is changed to cover the node, so that covers the OTU case too
17:22:17 sean-k-mooney ivveh: we do not have any hook point in nova to allow you to modify the xml. that is a very intetional decsions
17:23:05 sean-k-mooney ivveh: if you want to modify the xml you are required to modify the virt driver because we do not want to supprot external customisation fo the domain thoguh any kind of hooks
17:24:32 sean-k-mooney ivveh: what is it you are actully trying to do to the xml
17:25:40 gibi dansmith: cool
17:29:47 dansmith I was hoping to have tested the migration allocation thing for real last week but ran out of time. Luckily my monday morning meeting was canceled I could get a jump on it
17:46:34 ivveh sean-k-mooney: add <iothreads/>
17:48:06 ivveh (as i couldn't find it anywhere in the extraspecs or other methods of modifying the xml) but feel free to point me in the correct direcion if iothreads do exist, i need to point them out and pin on them, maybe some other stuff too
17:49:01 sean-k-mooney its not currently supproted. we had a blueprint to add it a year ago but the person working on it did actuly push any code
17:49:30 ivveh gotcha, would it be as an extraspec or some other method?
17:50:03 sean-k-mooney what i was suggeting was to allway allcoate 1 and make it float over the cpu_share_set
17:50:37 sean-k-mooney we discussed if we shoudl supprot more then one but the advice form our virt team was no
17:51:23 sean-k-mooney they suggested that in the future 1 per cinder volume might make sense but the high level degisn we had dicussed in teh past was 1 keep it simple and start with all vms having 1 iothread
17:51:38 sean-k-mooney 2 in the future add a flavor extra spec to opt into one per cinder volum eif needed
17:52:09 sean-k-mooney 3 possible allow this to be set on the image or config instead (TBD if/when we add the 1 per cinder volume feature)
17:52:49 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/939254 is the most recent attempt
17:53:03 sean-k-mooney https://blueprints.launchpad.net/nova/+spec/iothreads-for-instances
17:53:19 ivveh thanks, ill have a look
17:53:37 sean-k-mooney this version is adding a config option to contol the behvior
17:54:31 sean-k-mooney ah the blueprint has the link to the ptg dicssion form last cyle https://etherpad.opendev.org/p/nova-2025.1-ptg#L686
17:55:20 ivveh yeah, i wouldn't know how to do it at this point, i was thinking of having a pool of cores for this use and somehow allocate to those cores via vms (no matter how many volumes they had). i also need to do the same thing for virtiofsd (a different more complex story as i also have to figure out how to force it to use fifo or change some code)
17:56:19 sean-k-mooney ivveh: so this level of low level customstion of the domain is onlyt possibe via modifying the in tree driver
17:56:41 sean-k-mooney upstream we consier any out of band modification of the dmoain to make the vm unsupported
17:56:49 ivveh yeah, thats what i thought. i was hoping to avoid it, but..
17:57:23 sean-k-mooney ivveh: we only added the iniall use of virtiofs this cycle
17:57:43 sean-k-mooney so we have not looekd at any kind of affinity for vritio fs deamon
17:57:44 ivveh my usecase is very specific as i know how many and how large my vms can be and i wanna assign them a certain amount of cores
17:58:16 sean-k-mooney i see. that not really a usecase that is very compatible with cloud computeing :)
17:58:37 ivveh depends on what flavors you rent ;)
17:58:50 ivveh but i get your point
17:59:18 sean-k-mooney waht i mean is we allow you to express constratis like "i want this vm to have dedicated pinned cpus" and obviosly you can create falvors with X amount of cpus
17:59:32 sean-k-mooney but we intentioally do not allow you to do direct maping of logcial cpus to host cpus
17:59:46 ivveh yeah no, i understand openstack needs to have its constraints, its normal and acceptable
17:59:49 sean-k-mooney nova can do that internlly but its not exposed directly in the flavor
18:00:18 ivveh for shenenings you have to go custom

Earlier   Later