| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 18:05:27 | imacdonn | not sure that I can come up with every possible scenario off the top of my head | |
| 18:05:42 | dansmith | the other things we could do are: | |
| 18:06:05 | dansmith | 1. establish and drop connections before/after request | |
| 18:06:36 | dansmith | 2. leave heartbeats in place, as they are today so that rabbit will close the stale connections, which seems to be working for most people, | |
| 18:06:36 | efried | mriedem: n-cpu thinks the inventory is being set: http://logs.openstack.org/11/659611/1/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/9c644a2/controller/logs/screen-n-cpu.txt.gz#_May_16_17_17_55_788562 | |
| 18:06:43 | sean-k-mooney | if i do | |
| 18:06:46 | sean-k-mooney | >>> def func(): | |
| 18:06:47 | dansmith | 3. change our threading model for part of nova to run a heartbeat thread | |
| 18:06:48 | sean-k-mooney | ... print("test") | |
| 18:06:50 | sean-k-mooney | ... | |
| 18:06:52 | sean-k-mooney | >>> eventlet.greenthread.spawn(func) | |
| 18:06:54 | sean-k-mooney | it never prints anything | |
| 18:06:57 | dansmith | 4. split nova-api into the wsgi request processing side and the persistent bit | |
| 18:07:20 | sean-k-mooney | so if you dont monkeypatch it looks liek spawn does nothing but queue the fucntion to run | |
| 18:07:22 | sean-k-mooney | and never run it | |
| 18:07:25 | dansmith | sean-k-mooney: that makes me wonder how disabling monkeypatching does anything | |
| 18:07:49 | sean-k-mooney | dansmith: as long a s you dont call the scater gater code it should apper to work | |
| 18:08:10 | dansmith | sean-k-mooney: right, but we do that for instance list, and they say that disabling monkeypatching just makes everything work normally, right? | |
| 18:08:13 | sean-k-mooney | that is the only part of the api that actully uses eventlet | |
| 18:08:36 | efried | mriedem: oh, reserved is 1 and never gets set to 0 | |
| 18:08:49 | mriedem | efried: hmm, i wonder if it things the node isn't available | |
| 18:08:52 | mriedem | *thinks | |
| 18:08:54 | sean-k-mooney | dansmith: maybe its different on python 2 | |
| 18:08:58 | mriedem | b/c i think that's when the driver sets reserved=1 | |
| 18:09:04 | sean-k-mooney | i jsut checked with py36 | |
| 18:09:05 | efried | yup | |
| 18:09:17 | mriedem | Node 7c5e9503-1e20-42bf-9ae7-d14cef0f38a9 is not ready for a deployment, reporting resources as reserved for it. Node's provision state is cleaning, power state is power off and maintenance is False. | |
| 18:09:58 | efried | just so | |
| 18:10:50 | mriedem | yeah so the node is dead | |
| 18:10:50 | mriedem | http://logs.openstack.org/11/659611/1/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/9c644a2/controller/logs/screen-ir-cond.txt.gz#_May_16_17_28_09_311564 | |
| 18:10:55 | imacdonn | sean-k-mooney dansmith: side-note ... my "real" deployment is py2, devstack is py3 ... may be a factor in the failure to reconnect | |
| 18:13:42 | mriedem | yeah i just abandoned the change | |
| 18:15:45 | sean-k-mooney | dansmith: if you save teh greanthread object and then call wait() it runs all the queued greenthreads | |
| 18:16:07 | dansmith | sean-k-mooney: right that's the main loop part we will never do in wsgi mode | |
| 18:16:08 | sean-k-mooney | so if i do gt = eventlet.greenthread.spawn(func) | |
| 18:16:20 | sean-k-mooney | it prints after i call gt.wait() | |
| 18:16:40 | sean-k-mooney | dont we wait in the scater gather however | |
| 18:17:14 | dansmith | oh, sorry your point is maybe that's why it's working if they disable? | |
| 18:17:29 | dansmith | yeah, so we do that in sequence, which means it becomes serialized | |
| 18:17:58 | dansmith | so they just hit cell0, then cell1, etc | |
| 18:18:27 | dansmith | that's the behavior I had been assuming for non-patched, | |
| 18:19:01 | dansmith | I just wasn't sure if in reality the lower layers would actually use threading.Thread() unpatched for the work | |
| 18:19:08 | sean-k-mooney | well i think when we call wait it is entering a coperative green thread mode so we still get concurance | |
| 18:19:09 | dansmith | which hmm, I guess could still be | |
| 18:19:22 | sean-k-mooney | until that greenthread finishes at least | |
| 18:19:51 | sean-k-mooney | and sice we are doing it in a for it will complete the scater gather as we will wait on all the greadthread we queued | |
| 18:20:00 | sean-k-mooney | so we should not need to monkey patch | |
| 18:20:24 | dansmith | well, it depends on whether it's spawning a real thread and we're just waiting on it's future, or if we pass through and just exec the thread function during the wait | |
| 18:21:22 | sean-k-mooney | let me try a slightly less dumb test function that does a green trhead sleep and increment a counter and see if its serialsed or random | |
| 18:21:38 | dansmith | sean-k-mooney: >>> def func2(): | |
| 18:21:38 | dansmith | ... print(len(threading.enumerate())) | |
| 18:21:38 | dansmith | ... | |
| 18:21:38 | dansmith | >>> t = eventlet.greenthread.spawn(func2) | |
| 18:21:38 | dansmith | >>> t.wait() | |
| 18:21:39 | dansmith | test | |
| 18:21:40 | dansmith | 1 | |
| 18:21:48 | dansmith | sean-k-mooney: only one thread running from inside the thread function | |
| 18:21:53 | dansmith | so it's sequential | |
| 18:22:44 | dansmith | the test came from another thread I spawned first (like your example) which hadn't been waited on | |
| 18:22:53 | dansmith | normally enumerate() returns one for the main thread all the time, | |
| 18:23:08 | dansmith | so 1 means it didn't spawn something else for func2() | |
| 18:24:33 | dansmith | so one thing we could do in the interim is, | |
| 18:24:53 | dansmith | provide a knob to let them avoid the monkeypatch without having to actually patch the code, if they prefer synchronous behavior | |
| 18:25:22 | dansmith | that isn't too terrible for people with just the default two cells, and the rest of the people who don't suffer from this can continue to do what we do today | |
| 18:26:21 | melwitt | I think that's already available in the form of an env variable. which is what this tripleo patch is proposing to turn off monkey patching https://review.opendev.org/#/c/657168/1/deployment/nova/nova-api-container-puppet.yaml | |
| 18:26:38 | dansmith | melwitt: ah even better :) | |
| 18:27:00 | sean-k-mooney | dansmith: it is concurrent however | |
| 18:27:03 | sean-k-mooney | if i do | |
| 18:27:06 | sean-k-mooney | >>> def func(): | |
| 18:27:08 | sean-k-mooney | ... eventlet.greenthread.sleep(10) | |
| 18:27:10 | sean-k-mooney | ... global i | |
| 18:27:12 | sean-k-mooney | ... i+=1 | |
| 18:27:14 | sean-k-mooney | ... print(i) | |
| 18:27:16 | sean-k-mooney | ... | |
| 18:27:18 | sean-k-mooney | >>> i = 0 | |
| 18:27:20 | sean-k-mooney | >>> gt = eventlet.greenthread.spawn(func) | |
| 18:27:22 | sean-k-mooney | >>> gt = eventlet.greenthread.spawn(func) | |
| 18:27:24 | sean-k-mooney | >>> gt = eventlet.greenthread.spawn(func) | |
| 18:27:26 | sean-k-mooney | >>> gt = eventlet.greenthread.spawn(func) | |
| 18:27:28 | sean-k-mooney | >>> gt.wait() | |
| 18:27:30 | sean-k-mooney | it does not take 40 secods to print | |
| 18:27:50 | dansmith | sean-k-mooney: because you're using eventlet's sleep right? | |
| 18:27:58 | sean-k-mooney | yes | |
| 18:28:12 | imacdonn | mnaser (if still awake?), dansmith, sean-k-mooney: log from the failure-to-reconnect case: http://paste.openstack.org/show/751486/ | |
| 18:28:24 | dansmith | that does't mean it's concurrent, that means it's warping time for you :) | |
| 18:31:59 | mnaser | Looks like conductor yoking our | |
| 18:32:05 | mnaser | Timing out | |
| 18:32:31 | mnaser | I’ve seen really weird behaviour thy usually gets resolved by deleting every single queue in the vhost | |
| 18:33:34 | sean-k-mooney | dansmith: well in anycase i think we now now why its working if you dont monkeypatch | |
| 18:33:53 | dansmith | yup, as expected | |
| 18:33:54 | sean-k-mooney | there it concurant of fully synconous i dont know | |
| 18:35:51 | dansmith | sean-k-mooney: run this and tell me what you think :) https://pastebin.com/XHV9t9cM | |
| 18:36:01 | imacdonn | mnaser: this happens when the monkey patching is used in combination with a wsgi container ... and doesn't happen otherwise | |
| 18:36:32 | dansmith | sean-k-mooney: one single wait, but it clearly runs them in order, and blocking calls block everything | |
| 18:36:43 | melwitt | imacdonn: not sure what else to try other than a python2 devstack test maybe, to try and pinpoint what is the difference in your env vs the others. mnaser, are you running stein under py3 or py2? | |
| 18:37:28 | imacdonn | melwitt: yeah, thought about py2 devstack too | |
| 18:40:59 | sean-k-mooney | yes it took 53 seconds | |
| 18:41:04 | sean-k-mooney | but https://pastebin.com/QN5L7yDY took 5 | |
| 18:41:20 | sean-k-mooney | time.sleep will suppend the thread | |
| 18:41:44 | sean-k-mooney | eventlet.greenthread.sleep shoudl yeild to the next coroutine | |