| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 13:24:31 | mriedem | they did the naughty and enabled live migration for numa instances, 2 landed on the same host with conflicting pinned CPUs, and how the RT constantly fails but the scheduler continues to pick the host b/c it looks free since it's usage isn't getting reported | |
| 13:24:42 | mriedem | s/how/now/ | |
| 13:25:08 | dansmith | oye | |
| 13:25:23 | dansmith | isn't the build failure count going up in that case anyway? | |
| 13:25:42 | mriedem | maybe, i can ask in the bug report | |
| 13:26:01 | sean-k-mooney | mriedem: did we fix the typo so that that config option works yet | |
| 13:27:47 | mriedem | the qemu vs QEMU thing? | |
| 13:28:01 | mriedem | it wouldn't matter, | |
| 13:28:09 | mriedem | because in this case they enabled the workaround to allow live migrating instances with numa | |
| 13:28:10 | sean-k-mooney | ya we were checking for hypervior = kvm but its alway qemu | |
| 13:29:19 | sean-k-mooney | yes we fixed it https://review.opendev.org/#/c/635350/] | |
| 13:29:51 | sean-k-mooney | we never actully released with it broken so we shoudl be good | |
| 13:29:53 | mriedem | well, this person is on rocky | |
| 13:29:54 | mriedem | but yes https://review.opendev.org/#/q/I5127227a1e3d76dd413a820b048547ba578aff6b | |
| 13:31:38 | sean-k-mooney | mriedem: i would have hoped the numa toplogy filter would have prevented new vms from being scudled by the way if there was not space | |
| 13:31:50 | sean-k-mooney | the one that was mighrtated would not be fixed but ya | |
| 13:32:32 | aarents | Hi, mdbooth I updated https://review.opendev.org/#/c/659054/ if you want to have a look, maybe also mriedem ? | |
| 13:32:38 | sean-k-mooney | which reminding me i have a meeting then i need to updated docs for the sriov migration | |
| 13:34:10 | mdbooth | aarents: Thanks, still +1. | |
| 13:34:13 | adrianc | sean-k-mooney: Hi, id like to see if we can converge on the FUP for SR-IOV LM, would be great if you can take a quick look :) | |
| 13:41:31 | sean-k-mooney | adrianc: sure i think this was more or less fine so ill take alook at the latest version | |
| 13:44:43 | adrianc | sean-k-mooney, i did not upload a new PS as i wanted to reach an agreement on the propery thing, anyway... you will see | |
| 13:56:17 | efried | Nova meeting in 4 minutes in #openstack-meeting | |
| 14:02:06 | mriedem | tssurya: i'm going to remove the locked reason stuff from the runways etherpad since the nova and novaclient stuff is done https://etherpad.openstack.org/p/nova-runways-train | |
| 14:09:31 | efried | sean-k-mooney: Last week's nova meeting we got a request to review | |
| 14:09:31 | efried | https://review.opendev.org/#/c/643023/ | |
| 14:09:32 | efried | https://review.opendev.org/#/c/643024/ | |
| 14:09:32 | efried | which it looked like you were actively looking into. Would you be able to follow up please? | |
| 14:10:05 | sean-k-mooney | yes | |
| 14:10:16 | tssurya | mriedem: ack | |
| 14:10:33 | efried | thanks sean-k-mooney | |
| 14:18:53 | sean-k-mooney | adrianc: just replied on https://review.opendev.org/#/c/659101/2/nova/objects/instance_pci_requests.py@47 | |
| 14:19:43 | adrianc | thx sean-k-mooney, ill take a look shortly | |
| 14:20:22 | sean-k-mooney | im going to go review https://review.opendev.org/#/q/topic:bug/1809095+(status:open+OR+status:merged) before my next meeting | |
| 14:21:01 | sean-k-mooney | which is your fix for bug #1809095 | |
| 14:21:02 | openstack | bug 1809095 in OpenStack Compute (nova) "Wrong representor port was unplugged from OVS during cold migration" [Medium,In progress] https://launchpad.net/bugs/1809095 - Assigned to Adrian Chiris (adrian.chiris) | |
| 14:32:32 | mriedem | btw, i remember mtreinish trying to get glance-api moved to wsgi in pike and it got hung up on threading stuff with eventlet, hence https://review.opendev.org/#/c/549743/ | |
| 14:32:57 | dansmith | yup | |
| 14:34:52 | mriedem | oh and https://review.opendev.org/#/c/531498/ | |
| 14:35:04 | mriedem | ^ taskflow evenlet async stuff + wsgi + glance-api | |
| 14:36:21 | mtreinish | yep, but I couldn't get anyone to ever really review that or explain how to use the async glance APIs. And there is no dsvm coverage for any of it | |
| 14:37:55 | mriedem | mtreinish: for context we're going over similar in nova-api https://review.opendev.org/#/c/650172/ | |
| 14:41:14 | cdent | I reckon the root of the problems in both glance and nova is that we are trying to async workings in an environment (an HTTP API) where it doesn't make sense. The traditional python solution to this problem is to dump into an offboard async task queue (celery and the like). The scatter/gather thing is not quite in the same boat as the glance stuff (which is more amenable to the celery thing perhaps) | |
| 14:46:13 | mtreinish | cdent: yeah, that was the best way to fix it, to just do the async bits in another service | |
| 14:46:37 | mtreinish | but I assumed just launching another process or thread to handle it should work as a quick workaround | |
| 14:46:56 | cdent | it ought to | |
| 14:47:24 | cdent | but I would guess that zombie blocked processes would present the same concern as blocked threads | |
| 14:47:36 | cdent | which might suggest: that's the way unix goes, deal... | |
| 14:53:11 | sean-k-mooney | i mean we could just not set a limit on the worker trhead in the tread pool or make it a config option | |
| 14:53:42 | sean-k-mooney | python will grow and srink the thread pool automatically if you dont specify the number of workers | |
| 14:53:53 | dansmith | that's the problem, these are a threadpool, which means they're re-used and capped at some max number, so enough api requests come in during a cell down event, and you immediately exhaust the pool of workers until things time out | |
| 14:54:15 | dansmith | sean-k-mooney: only up to N*CPUs I think I read | |
| 14:54:27 | dansmith | where N is like 3 or something | |
| 14:54:53 | sean-k-mooney | dansmith: maybe not sure | |
| 14:55:09 | sean-k-mooney | but we could leave it as a config option and defult to -1 mean let python figure it out | |
| 14:55:26 | sean-k-mooney | or set it to X if you care and have mesured for your system | |
| 14:55:34 | mriedem | https://docs.python.org/3.6/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor | |
| 14:55:40 | mriedem | "If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor." | |
| 14:56:25 | dansmith | right, so with 5 cells, that becomes $CPUs | |
| 14:56:26 | dansmith | and everyone has at least two | |
| 14:57:15 | cdent | when calculating the number of threads that will be around, another thing to keep in mind is mod_wsgi will often run with multiple process and multiple threads in those processes. what is the pool associated with? | |
| 14:57:45 | cdent | every request or thread or process? | |
| 14:58:43 | dansmith | idk | |
| 14:58:44 | cdent | process | |
| 14:59:40 | sean-k-mooney | so we could revert to createing the thread pool per call and set it to one thread per cell | |
| 15:00:09 | sean-k-mooney | but the issue with that is a its expensive to spawn treads and b we nolonger have a limit on them | |
| 15:00:47 | sean-k-mooney | but we will not have the issue of a potention exausting of the thread pool | |
| 15:01:30 | sean-k-mooney | cdent: the pool i a module global in the current patch so it would associated with the process | |
| 15:03:13 | cdent | sean-k-mooney: [t 6MVa] | |
| 15:03:13 | purplerbot | <cdent> process [2019-05-16 14:58:44.001687] [n 6MVa] | |
| 15:03:55 | sean-k-mooney | ? | |
| 15:04:23 | cdent | [l 6MVa] | |
| 15:04:23 | purplerbot | http://p.anticdent.org/6MVa | |
| 15:04:39 | cdent | in other words, I figured it out by looking at the code a few minutes ago | |
| 15:04:50 | sean-k-mooney | oh right :) | |
| 15:06:13 | mriedem | efried: on that ironic job failure, it looks like that job has probably been broken since stein when we started reporting capability traits for nodes http://logs.openstack.org/32/634832/29/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/fba9197/controller/logs/devstacklog.txt.gz#_2019-05-16_03_28_21_590 | |
| 15:06:21 | jangutter | cdent: I really thought you were mistyping 6 million volt-ampere. | |
| 15:06:36 | mriedem | if i'm reading that correctly it's expecting only the CUSTOM_GOLD trait | |
| 15:06:40 | sean-k-mooney | dansmith: what was the behavior that motivated the scater gather code in the first place to you rememebr? was it to deal with down cells or was it a performce imporvoment? | |
| 15:07:12 | dansmith | sean-k-mooney: it's to avoid a sequential walk of all the cell databases, serializing their latencies | |
| 15:07:28 | sean-k-mooney | ah ok so it was to addres perfromace ok | |
| 15:07:51 | cdent | Are there numbers on those latencies? | |
| 15:07:57 | dansmith | sean-k-mooney: cern has 70 cells, some of which are remote | |
| 15:08:06 | cdent | ah. the remote part | |
| 15:08:08 | dansmith | as in, in another country | |
| 15:08:18 | efried | mriedem: looks like the fix is just to take out the square brackets, nah? | |
| 15:08:40 | sean-k-mooney | ya even if they wer all in the same data center it could be a proble as the number of cell grow | |
| 15:09:18 | dansmith | sean-k-mooney: especially if one in the middle is timing out, and we wait for that to complete before moving to the next one and get its response | |
| 15:09:52 | dansmith | and everyone has two of these at minimum so moving to sequential means 2N latency, which is twice as bad, regardless of what N is | |
| 15:10:00 | sean-k-mooney | i was just wondering if the initall approch to solving the problem was patinting us in to a corner that could be avoid by revisiting the originial usecase cool | |
| 15:11:04 | sean-k-mooney | effectivly waht we want to do is queue up a bunch of asyc request and wait for them to complete | |
| 15:11:24 | dansmith | ..a thing for which eventlet is ideal | |
| 15:11:24 | sean-k-mooney | eventlet and thread pools both allwo that with different tradeoffs | |
| 15:11:50 | sean-k-mooney | the issue isnt the use of eventlets its self | |
| 15:12:38 | sean-k-mooney | its that mod_wsgi and uwsgi both have exposed diffreent bugs caused by using eventlets with them | |
| 15:15:06 | sean-k-mooney | does wsgi provide a way to run request like this in the background that we could leverag and fall back to eventlest when not runnign under wsgi? | |
| 15:16:03 | sean-k-mooney | google found me https://pypi.org/project/uwsgi-tasks/ but no idea if that would help | |
| 15:18:52 | cdent | that could work for the uwsgi situation, but not mod_wsgi, presumably | |
| 15:19:18 | sean-k-mooney | i dont know i personally would prefer to try melwitt's approch first and maybe ask cern to try it and get feed back | |
| 15:20:38 | cdent | sean-k-mooney: you mean the current code under review? that seems reasonable. I guess the issue is that we don't know if the problem is a problem until it gets hit... | |