Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-10-14
22:30:54 sean-k-mooney the other thing i brought up is https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L169-L172
22:31:43 sean-k-mooney the nova/cmd moduels for the proxy dont call eventlet but we do import that moduel here https://review.opendev.org/c/openstack/nova/+/904424/13/nova/cmd/standalone/baseproxy.py#29
22:32:05 melwitt ah, yeah. that one I haven't known what it's for, none of the work I did on the proxy was related to it
22:32:35 sean-k-mooney so it was added by https://github.com/openstack/nova/commit/cb25bc4530323aaa33d5c42eb01f998d463f2106
22:33:25 sean-k-mooney which is apprently for https://bugs.launchpad.net/nova/+bug/1193031
22:33:57 melwitt oh, huh.
22:34:06 sean-k-mooney The solution is to either not monkey_patch for /cmd/nova-novncproxy and do the rpc call that validates the token with the consolauth service in a separate process that is monkey_patched (the rpc won't work without it), or to make openstack/common/rpc/amqp/multicall not use green-threads for this particular call (ie. block the caller).
22:34:19 sean-k-mooney so consolauth is dead
22:34:26 sean-k-mooney and we are trying to not monkeypatch
22:34:39 sean-k-mooney so maybe we just delete that and we are good?
22:35:50 melwitt delete all of consoleauth? sure, what could go wrong
22:36:03 sean-k-mooney i ment the hub
22:36:12 sean-k-mooney but also didnt you remove the console auth service
22:36:19 sean-k-mooney we now do auth in the cell db
22:36:47 melwitt oh, the console auth service, yeah that is no longer used
22:37:51 melwitt for some reason I was thinking the db call but yeah that doesn't use rpc
22:38:24 sean-k-mooney well even if it did we could deal with this differently
22:38:45 melwitt oh wait, there is an rpc call for "validating the console port
22:39:50 melwitt https://github.com/openstack/nova/blob/741e1b38705efecc083f091c51164b3cafb9697c/nova/console/websocketproxy.py#L133
22:40:00 melwitt that's the only one AFAICT
22:40:22 sean-k-mooney so we are changing the oslo messaging executor to threading
22:40:42 sean-k-mooney so im not sure it that will be a problem or not but ok this is somethign we need to test
22:42:32 melwitt ok, yeah. I would think it should work fine if you delete the hub at the same time as the switch to native threads in oslo.messaging. I agree, definitely something to test
22:43:53 sean-k-mooney that is tomorrows problem o/
22:44:20 melwitt yes o/
23:14:16 sean-k-mooney[m] so i reinstalled my home openstack and entirly redidi my sotrage to use nova lvm instead of cinder lvm, most of the speedu is form other change but i wend form about an 80 minute devstack install to about 21 minutes
23:15:02 sean-k-mooney[m] im hoping with a liitlme more turning to shave another miniute or so but thats closer to the speed i would expect on this host
23:15:53 sean-k-mooney[m] i used to reliable be able to get a fresh vm deployed in about 18-20 mins so that pretty close already
23:19:52 sean-k-mooney[m] although there isnt much standing out as a bottle neck to be honest although i think i can speed up the disk io slightly more. beyond that i would likely need a local mirror/cache but the install time for package is already pretty good https://paste.opendev.org/show/825791/
#openstack-nova - 2024-10-15
01:40:49 opendevreview Takashi Kajinami proposed openstack/placement master: Drop db migration tool https://review.opendev.org/c/openstack/placement/+/932324
07:02:46 opendevreview Balazs Gibizer proposed openstack/nova stable/2024.2: Revert "Test live migration between hosts with differnet cpu_shared_sets" https://review.opendev.org/c/openstack/nova/+/932379
07:17:07 gibi sean-k-mooney[m]: melwitt: I skimmed the discussion above above about the threading in scatter gather. Yes, my comment about the timer was only about the proposed native thread based code.
07:19:37 gibi also please note in native threading there is no such thing that having a timer in an existing thread. The timer itself is always a seperate thread (with basically a sleep and a callback). So if we want a threading.timer based solution to signal an existing thread we need a communication channel between the timer thread that runs our callback on timeout and the existing thread that needs the timer
07:19:43 gibi signal. The most simple thing is to share an Event between the two threads. All the solution needs the existing thread to monitor some channel (e.g. the Event) periodically.
07:31:19 sean-k-mooney[m] an event wont work as far as im aware since the event wont raise an excpetion unless we check it
07:31:53 sean-k-mooney[m] also isint the time just a syscall to the OS to have the OS signal the thread after a timer interupt fires
07:36:15 sean-k-mooney[m] hum, maybe a https://docs.python.org/3/library/threading.html#threading.Barrier is closer to what we want
07:36:39 sean-k-mooney[m] the main issue is we dont really have a concpet of a frestandign cancelable task
07:36:46 sean-k-mooney[m] that what we really want
07:45:44 opendevreview Michael Still proposed openstack/nova-specs master: Repropose spice-direct console support. https://review.opendev.org/c/openstack/nova-specs/+/932387
07:46:31 sean-k-mooney[m] gibi: the first example here https://www.geeksforgeeks.org/python-different-ways-to-kill-a-thread/ could be adapted for our use
07:47:31 sean-k-mooney[m] we can wrap the function into a calss with an overloaded call operator and add a function to raise the excption
07:47:57 sean-k-mooney[m] then we can use the current timeout code to invoke that function if it expires
07:54:03 sean-k-mooney[m] ill see if i can come up with a restandign example of that and we can see what we think of the approch
09:29:49 gibi sean-k-mooney[m]: threading.Timer is a thread not an OS signal based construct https://github.com/python/cpython/blob/546dddca43a2a69dbe33d230e9e540636b403270/Lib/threading.py#L1318-L1343
09:30:18 sean-k-mooney gibi: ya i read the docs on it after
09:30:24 sean-k-mooney its a subclass of thread
09:30:25 gibi sean-k-mooney[m]: python has signal handling if we want but that require special care as signal handlers can only run in the main thread and can be tricky to write them properly
09:30:51 sean-k-mooney no im in two minds
09:31:03 gibi https://docs.python.org/3/library/signal.html#signal.SIGALRM
09:31:04 sean-k-mooney part of me wonders if the current approch is corect
09:31:21 sean-k-mooney i.e. are we trying to handle this in too generic a way
09:31:39 sean-k-mooney the only reason this timeout exist sis to timeout the db query to the cell db
09:31:56 sean-k-mooney so im partly wonderign if sqlachemy has a native way to do that
09:32:35 gibi yeah, we can step back and look at the current use case to see if we can reformulate it
09:32:37 sean-k-mooney on the other hand if we dont and we want a generic killable way to execute a task a process is better for that
09:33:27 gibi what we want is a to limit the API response time which a) is limited by the web server (haproxy) already I guess and b) avoid stuck threads waiting for a result that never arrives.
09:35:25 sean-k-mooney yep we have a few options, create a task obejct that will execute an arbitry funciton but kill itself after a timeout, run the function in a process and kill it outselves, convert it to a asyncio corutine and cancel it after a timeout or use somethign in sqlacment to kill it after a timeout at the low level
09:39:18 gibi Do we know how many places in nova we rely on the eventlet in process timer? If it is just 2-3 places then it make sense to do a specific solution for each, if it is more than 5 then I would vote for a creation of a cancellable task primitive and use that everywhere
09:40:21 sean-k-mooney thats a good question. This query_wrapper function is only used in one place i belive
09:40:31 sean-k-mooney im not sure if we have this patterin in ohter however
09:41:14 sean-k-mooney im currently trying to play with creating a BoundTask class to do that
09:41:38 sean-k-mooney maybe TimerTask naming is hard
09:41:40 gibi I guess we have timers in the compute for e.g. vif_plug events
09:42:15 sean-k-mooney yes but it depend on how they are used
09:42:33 sean-k-mooney what i want to do and other can disagree
09:42:51 sean-k-mooney is have our main loop handel pooling for RPC messages
09:43:02 sean-k-mooney but dispatch the message handler to a worker thread pool
09:43:36 sean-k-mooney so basiclaly each RPC handlelr woudl be spawned into its own thread which it can freely block on
09:43:46 gibi I'm not sure how oslo.messaging works in threading mode, I assumed it already does the dispatch of messages to separate threads (from a pool)
09:44:09 sean-k-mooney ya im kind fo hoping it does but not sure either
09:45:12 sean-k-mooney basicaly i want use to be able to safly dispatch connurent work to a seprate io_thread_pool in our functions and block on the returned future
09:45:21 gibi https://docs.openstack.org/oslo.messaging/latest/reference/executors.html#threading
09:45:26 gibi Executor that uses a thread pool to execute calls asynchronously.
09:45:51 sean-k-mooney yes bvut im not sure if that is just submiting it to rabbit
09:46:23 sean-k-mooney i.e. the read/write calls
09:46:28 gibi I think that is the receiving side
09:47:01 gibi the sending side is simply blocking for cast until rabbit takes the message, and blocking for call until the response arrives
09:47:57 sean-k-mooney so if we get this fro free then cool, this may also be externalise in oslo.service
09:48:18 sean-k-mooney this was on my todo list to look at after the removel of our explict use of eventlet.tpool
09:49:35 gibi yepp executors are for receving messages https://github.com/openstack/oslo.messaging/blob/930d97599c0048e84f7c47dca0d62d2cc0562166/doc/source/reference/executors.rst#L12
09:49:41 sean-k-mooney https://github.com/openstack/oslo.messaging/blob/930d97599c0048e84f7c47dca0d62d2cc0562166/oslo_messaging/server.py#L368
09:50:29 sean-k-mooney so ok _on_incomming is just submitting the callback to the executor which in threading mode is a threadpool executor form oslo messaging
09:50:37 sean-k-mooney so this is already done for us
09:50:53 gibi yepp it seems so
09:50:53 sean-k-mooney that is what i was hoping for but as i siad i had not confirmed that till now
09:52:14 gibi btw, as you are around, could you do a review round on the igb series https://review.opendev.org/q/topic:%22bp/igb-vif-model%22 the os-traits part is landed released an bumped into both global and nova reqs.
09:52:37 gibi so the nova series is ready to go
10:51:30 sean-k-mooney oh i was not aware we had approved the bluepirnt yet
10:51:58 sean-k-mooney sure i stood up a devstack yesterday so i can pull it in an try it out and give it a test
10:52:19 sean-k-mooney ill see if i can do that later today
10:52:33 gibi thanks. Yes the bp was approved couple of weeks ago
10:52:58 sean-k-mooney ack i vagule remember that but it didnt click that it was ready for review again
10:53:32 sean-k-mooney for some reasoin i tought we were waiting for the ptg but i was proably thinking about somethign else
11:14:07 WJeffs1 Hey, So I was sure I saw an option once, to run a script locally on the HV after a VM is created and the same for deletion? As I'm being overly hopeful or is this actually an option?
11:14:26 sean-k-mooney not in nova but it is in libvirt
11:14:30 sean-k-mooney libvirt has hooks
11:14:40 sean-k-mooney we dont supprot that but it would allow you to do that
11:14:50 WJeffs1 ahh that was probably where I saw it :) I'll dig there thanks!
11:50:03 opendevreview Stephen Finucane proposed openstack/placement master: requirements: Remove setuptools https://review.opendev.org/c/openstack/placement/+/932398

Earlier   Later