Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-03-31
17:07:47 sean-k-mooney dansmith: https://github.com/SeanMooney/ard/blob/master/ansible/deploy_multinode_devstack.yaml is the playbook that does it that was driven by molecule with vagrant_libvirt to provdie the vms https://github.com/SeanMooney/ard/blob/master/molecule/default/molecule.yml
17:08:11 dansmith ah with vagrant
17:08:24 sean-k-mooney you didnt need vagrant
17:08:41 sean-k-mooney the playbooks could work with any servers once you hadd ssh access
17:09:28 sean-k-mooney so like i provisioned some server internally for vdpa testing and just poited the palybooks at it with an inventory file
17:09:45 sean-k-mooney https://github.com/SeanMooney/ard/tree/master/examples/vdpa
17:11:13 sean-k-mooney i wrote 3 roles devstack_{common,compute,controller} that wrappted the upstream devstack roles form teh gate jobs
17:11:17 sean-k-mooney https://github.com/SeanMooney/ard/tree/master/ansible/roles
17:12:49 sean-k-mooney dansmith: its been like 3+ years since i used any of this
17:12:59 sean-k-mooney it proably still wroks but i havent been maintining it
17:13:15 sean-k-mooney dansmith: was there a specific reason you asked
17:14:22 dansmith sean-k-mooney: I was struggling through re-re-remembering the multinode devstack rain dance
17:14:28 dansmith but I got it figured out whilst waiting
17:14:37 sean-k-mooney ah ok
17:14:44 dansmith sean-k-mooney: gibi: can confirm that just that placement change fixes the nova migrate-while-over-capacity bug
17:14:52 sean-k-mooney i have some downstream docs of the process with bash
17:15:09 dansmith I was able to repro, then literally swapped placement to that commit, restarted, retried the same migration and it went, confirmed, all good
17:15:18 sean-k-mooney and one of my team recently wrote an ansible playbook to deploy multi node devstack on our internal cloud i can ping you does internally
17:15:33 sean-k-mooney dansmith: oh nice
17:16:24 dansmith sean-k-mooney: thanks, I need to formalize my own notes, because I have some of my own scripting and making things work with those is most ideal,
17:16:31 dansmith I just suck at persisting the notes after I get it workin,
17:16:39 dansmith but I'm going to do that now
17:18:11 ivveh as more context to my previous question is that i want to include iothread stanza into the xml (for now)
17:19:58 sean-k-mooney dansmith: the main parts of multi node are syncing the tls/ceph data, exchantign the ssh keys/known_hosts, ntp sync and updating /etc/hosts before deploying the comptues. then adding a discover host at the end
17:20:28 dansmith yeah, it was just the CA stuff I forgot this time
17:20:56 dansmith as soon as it failed, I knew the problem, but then had to remember to copy the CA dir *and* the bundle, lest it would nuke the latter on stacking the subnode
17:21:01 sean-k-mooney did you know that tls is appreantly not on by default in devstack. i always used the ci jobs as a basis for mine
17:21:14 sean-k-mooney but if you use devstack by default it does not enabel the tls-proxy
17:21:31 dansmith yeah, I have it on my base node though, so..
17:21:43 gibi dansmith: nice test results
17:22:12 dansmith gibi: also works for the case where reserved is changed to cover the node, so that covers the OTU case too
17:22:17 sean-k-mooney ivveh: we do not have any hook point in nova to allow you to modify the xml. that is a very intetional decsions
17:23:05 sean-k-mooney ivveh: if you want to modify the xml you are required to modify the virt driver because we do not want to supprot external customisation fo the domain thoguh any kind of hooks
17:24:32 sean-k-mooney ivveh: what is it you are actully trying to do to the xml
17:25:40 gibi dansmith: cool
17:29:47 dansmith I was hoping to have tested the migration allocation thing for real last week but ran out of time. Luckily my monday morning meeting was canceled I could get a jump on it
17:46:34 ivveh sean-k-mooney: add <iothreads/>
17:48:06 ivveh (as i couldn't find it anywhere in the extraspecs or other methods of modifying the xml) but feel free to point me in the correct direcion if iothreads do exist, i need to point them out and pin on them, maybe some other stuff too
17:49:01 sean-k-mooney its not currently supproted. we had a blueprint to add it a year ago but the person working on it did actuly push any code
17:49:30 ivveh gotcha, would it be as an extraspec or some other method?
17:50:03 sean-k-mooney what i was suggeting was to allway allcoate 1 and make it float over the cpu_share_set
17:50:37 sean-k-mooney we discussed if we shoudl supprot more then one but the advice form our virt team was no
17:51:23 sean-k-mooney they suggested that in the future 1 per cinder volume might make sense but the high level degisn we had dicussed in teh past was 1 keep it simple and start with all vms having 1 iothread
17:51:38 sean-k-mooney 2 in the future add a flavor extra spec to opt into one per cinder volum eif needed
17:52:09 sean-k-mooney 3 possible allow this to be set on the image or config instead (TBD if/when we add the 1 per cinder volume feature)
17:52:49 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/939254 is the most recent attempt
17:53:03 sean-k-mooney https://blueprints.launchpad.net/nova/+spec/iothreads-for-instances
17:53:19 ivveh thanks, ill have a look
17:53:37 sean-k-mooney this version is adding a config option to contol the behvior
17:54:31 sean-k-mooney ah the blueprint has the link to the ptg dicssion form last cyle https://etherpad.opendev.org/p/nova-2025.1-ptg#L686
17:55:20 ivveh yeah, i wouldn't know how to do it at this point, i was thinking of having a pool of cores for this use and somehow allocate to those cores via vms (no matter how many volumes they had). i also need to do the same thing for virtiofsd (a different more complex story as i also have to figure out how to force it to use fifo or change some code)
17:56:19 sean-k-mooney ivveh: so this level of low level customstion of the domain is onlyt possibe via modifying the in tree driver
17:56:41 sean-k-mooney upstream we consier any out of band modification of the dmoain to make the vm unsupported
17:56:49 ivveh yeah, thats what i thought. i was hoping to avoid it, but..
17:57:23 sean-k-mooney ivveh: we only added the iniall use of virtiofs this cycle
17:57:43 sean-k-mooney so we have not looekd at any kind of affinity for vritio fs deamon
17:57:44 ivveh my usecase is very specific as i know how many and how large my vms can be and i wanna assign them a certain amount of cores
17:58:16 sean-k-mooney i see. that not really a usecase that is very compatible with cloud computeing :)
17:58:37 ivveh depends on what flavors you rent ;)
17:58:50 ivveh but i get your point
17:59:18 sean-k-mooney waht i mean is we allow you to express constratis like "i want this vm to have dedicated pinned cpus" and obviosly you can create falvors with X amount of cpus
17:59:32 sean-k-mooney but we intentioally do not allow you to do direct maping of logcial cpus to host cpus
17:59:46 ivveh yeah no, i understand openstack needs to have its constraints, its normal and acceptable
17:59:49 sean-k-mooney nova can do that internlly but its not exposed directly in the flavor
18:00:18 ivveh for shenenings you have to go custom
18:00:41 sean-k-mooney we talked about having a pool for iothread. but we didnt want to have to schdule on them
18:01:32 ivveh i think either way, it would be great for nova to have a iothread pool that nova/cinder could take resources from (as these days defaults aren't enough)
18:03:00 ivveh would be great to have like flavors with iothread spec extraspec for example, to be able to utilize those resources, or not
18:03:21 sean-k-mooney when we dicussed it the idea was for it to just be an addtion set of core, taht woould default to cpu_shared_set if not defiend in the config
18:04:38 sean-k-mooney i.e. add a iothread_set to nova. those cpu could overlap with cpu_shared_set but now cpu_dedicated_set. we kind of felt that was too complex to start with
18:04:44 sean-k-mooney that does not mean we could not add that later
18:06:12 ivveh yea if you use cpu_shared_set and vms are pinned then they would always be free for stuff like iothreads or virtiofsd
18:06:54 sean-k-mooney so we already supprot moving the emulator thread to cpu_shared_set for pinned vms
18:07:11 sean-k-mooney so we were effectivly going to treat the extra iothread the same
18:07:49 ivveh or cpu_isolation if you wanna let things do whatever, i guess
#openstack-nova - 2025-04-01
01:56:08 opendevreview Yaguang Tang proposed openstack/nova stable/2023.2: Fix device type when booting from ISO image https://review.opendev.org/c/openstack/nova/+/945903
06:52:10 opendevreview Yaguang Tang proposed openstack/nova stable/2024.1: Fix device type when booting from ISO image https://review.opendev.org/c/openstack/nova/+/945816
09:28:55 sahid o/
09:29:26 sahid any chance to have review regarding that one, et the serie behind?
09:29:27 sahid https://review.opendev.org/c/openstack/neutron/+/940983/6
09:29:35 sahid it's waiting for a while now :-)
09:31:54 sean-k-mooney i assume you ment ot put that in the neutron channel
09:32:12 stephenfin sean-k-mooney: trivial README change here if you've 3.5 seconds https://review.opendev.org/c/openstack/nova/+/944994
09:32:25 sean-k-mooney sahid: but i would not recommend creating a theread pool dynmaiclly in a plugin
09:32:46 sean-k-mooney sahid: if your going to use thread pools you shoudl create one globaly for neutron to use and reuse it
09:32:50 sean-k-mooney like we do in nova
09:34:00 sahid sean-k-mooney: oh yes... my mistake
09:34:22 sahid sean-k-mooney: that is an interesting point yes
09:34:41 sean-k-mooney sahid: https://review.opendev.org/c/openstack/nova/+/922497
09:35:03 sahid i will discuss that with neutron, i think ralonsoh may also need it from it's change to the l3 agent
09:35:09 sahid its
09:35:09 sean-k-mooney actully that not quite the patch i wanted
09:35:36 sean-k-mooney well no that uses futureist
09:36:19 sean-k-mooney oh im not adding eventlet tpool there im just moving the import
09:36:26 sean-k-mooney thats what was confusing me
09:40:00 sahid in all cases I think that make sense to have one global thread pool and each pick on it, so each service will use its own thread pool
09:41:00 sahid but well some time we may need to have one context os threads so we can play with them, join them or kill them
09:44:11 sean-k-mooney sahid: you shoudl never kill a thread in python
09:44:54 sean-k-mooney you shoudl be using futures for the most part when you need to wait on results
09:45:11 sean-k-mooney if you need to kill something you should be using a process pool

Earlier   Later