| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-16 | |||
| 09:38:56 | sean-k-mooney | the hosts will steal cpu time form the pinned instances | |
| 09:39:26 | sean-k-mooney | but again that is what happens today if you dont ommit cores for the host to use form the vcpu pin set | |
| 09:41:03 | bhagyashris | okay, So that case should be handle by code or by operator then | |
| 09:41:20 | sean-k-mooney | by the operator | |
| 09:41:30 | bhagyashris | okay | |
| 09:42:14 | bhagyashris | Thank you :) | |
| 10:08:59 | openstackgerrit | Arnaud Morin proposed openstack/nova master: Always Set dhcp_server in network_info https://review.opendev.org/658362 | |
| 10:25:05 | mdbooth | Any reason https://review.opendev.org/658845 shouldn't be backported to stable? | |
| 11:59:23 | rabel | hi. there seems to be a problem with api microversions. i have an openstack rocky (deployed with packstack), but my openstackclient (newest version, directly from git) tells me that the compute api version is 2.1 | |
| 12:02:40 | artom | rabel, it's an known osc behaviour/bug. While novaclient autonegotiates to the greatest common supported microversion, osc defaults to 2.1, and you have to manually specify which microversion you want | |
| 12:13:03 | rabel | artom thanks | |
| 12:56:35 | alex_xu | efried: I think the in_tree take care of all the cases for https://review.opendev.org/#/c/645520/ also | |
| 13:08:37 | efried | alex_xu: Okay. I'll take a closer look at that patch soon. | |
| 13:08:50 | alex_xu | efried: thx | |
| 13:16:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop logging traceback when skipping quiesce https://review.opendev.org/659374 | |
| 13:23:53 | mriedem | dansmith: random idea, but with something like https://bugs.launchpad.net/nova/+bug/1829349 where the RT is constantly failing b/c of conflicted pinned CPUs on the same host, we could leverage the build failure stats thing and if the RT fails after x hits or whatever we try to take that host out of scheduling | |
| 13:23:54 | openstack | Launchpad bug 1829349 in OpenStack Compute (nova) "Resource Tracker failed to update usage in case numa topology conflict happen" [Undecided,New] | |
| 13:24:31 | mriedem | they did the naughty and enabled live migration for numa instances, 2 landed on the same host with conflicting pinned CPUs, and how the RT constantly fails but the scheduler continues to pick the host b/c it looks free since it's usage isn't getting reported | |
| 13:24:42 | mriedem | s/how/now/ | |
| 13:25:08 | dansmith | oye | |
| 13:25:23 | dansmith | isn't the build failure count going up in that case anyway? | |
| 13:25:42 | mriedem | maybe, i can ask in the bug report | |
| 13:26:01 | sean-k-mooney | mriedem: did we fix the typo so that that config option works yet | |
| 13:27:47 | mriedem | the qemu vs QEMU thing? | |
| 13:28:01 | mriedem | it wouldn't matter, | |
| 13:28:09 | mriedem | because in this case they enabled the workaround to allow live migrating instances with numa | |
| 13:28:10 | sean-k-mooney | ya we were checking for hypervior = kvm but its alway qemu | |
| 13:29:19 | sean-k-mooney | yes we fixed it https://review.opendev.org/#/c/635350/] | |
| 13:29:51 | sean-k-mooney | we never actully released with it broken so we shoudl be good | |
| 13:29:53 | mriedem | well, this person is on rocky | |
| 13:29:54 | mriedem | but yes https://review.opendev.org/#/q/I5127227a1e3d76dd413a820b048547ba578aff6b | |
| 13:31:38 | sean-k-mooney | mriedem: i would have hoped the numa toplogy filter would have prevented new vms from being scudled by the way if there was not space | |
| 13:31:50 | sean-k-mooney | the one that was mighrtated would not be fixed but ya | |
| 13:32:32 | aarents | Hi, mdbooth I updated https://review.opendev.org/#/c/659054/ if you want to have a look, maybe also mriedem ? | |
| 13:32:38 | sean-k-mooney | which reminding me i have a meeting then i need to updated docs for the sriov migration | |
| 13:34:10 | mdbooth | aarents: Thanks, still +1. | |
| 13:34:13 | adrianc | sean-k-mooney: Hi, id like to see if we can converge on the FUP for SR-IOV LM, would be great if you can take a quick look :) | |
| 13:41:31 | sean-k-mooney | adrianc: sure i think this was more or less fine so ill take alook at the latest version | |
| 13:44:43 | adrianc | sean-k-mooney, i did not upload a new PS as i wanted to reach an agreement on the propery thing, anyway... you will see | |
| 13:56:17 | efried | Nova meeting in 4 minutes in #openstack-meeting | |
| 14:02:06 | mriedem | tssurya: i'm going to remove the locked reason stuff from the runways etherpad since the nova and novaclient stuff is done https://etherpad.openstack.org/p/nova-runways-train | |
| 14:09:31 | efried | sean-k-mooney: Last week's nova meeting we got a request to review | |
| 14:09:31 | efried | https://review.opendev.org/#/c/643023/ | |
| 14:09:32 | efried | https://review.opendev.org/#/c/643024/ | |
| 14:09:32 | efried | which it looked like you were actively looking into. Would you be able to follow up please? | |
| 14:10:05 | sean-k-mooney | yes | |
| 14:10:16 | tssurya | mriedem: ack | |
| 14:10:33 | efried | thanks sean-k-mooney | |
| 14:18:53 | sean-k-mooney | adrianc: just replied on https://review.opendev.org/#/c/659101/2/nova/objects/instance_pci_requests.py@47 | |
| 14:19:43 | adrianc | thx sean-k-mooney, ill take a look shortly | |
| 14:20:22 | sean-k-mooney | im going to go review https://review.opendev.org/#/q/topic:bug/1809095+(status:open+OR+status:merged) before my next meeting | |
| 14:21:01 | sean-k-mooney | which is your fix for bug #1809095 | |
| 14:21:02 | openstack | bug 1809095 in OpenStack Compute (nova) "Wrong representor port was unplugged from OVS during cold migration" [Medium,In progress] https://launchpad.net/bugs/1809095 - Assigned to Adrian Chiris (adrian.chiris) | |
| 14:32:32 | mriedem | btw, i remember mtreinish trying to get glance-api moved to wsgi in pike and it got hung up on threading stuff with eventlet, hence https://review.opendev.org/#/c/549743/ | |
| 14:32:57 | dansmith | yup | |
| 14:34:52 | mriedem | oh and https://review.opendev.org/#/c/531498/ | |
| 14:35:04 | mriedem | ^ taskflow evenlet async stuff + wsgi + glance-api | |
| 14:36:21 | mtreinish | yep, but I couldn't get anyone to ever really review that or explain how to use the async glance APIs. And there is no dsvm coverage for any of it | |
| 14:37:55 | mriedem | mtreinish: for context we're going over similar in nova-api https://review.opendev.org/#/c/650172/ | |
| 14:41:14 | cdent | I reckon the root of the problems in both glance and nova is that we are trying to async workings in an environment (an HTTP API) where it doesn't make sense. The traditional python solution to this problem is to dump into an offboard async task queue (celery and the like). The scatter/gather thing is not quite in the same boat as the glance stuff (which is more amenable to the celery thing perhaps) | |
| 14:46:13 | mtreinish | cdent: yeah, that was the best way to fix it, to just do the async bits in another service | |
| 14:46:37 | mtreinish | but I assumed just launching another process or thread to handle it should work as a quick workaround | |
| 14:46:56 | cdent | it ought to | |
| 14:47:24 | cdent | but I would guess that zombie blocked processes would present the same concern as blocked threads | |
| 14:47:36 | cdent | which might suggest: that's the way unix goes, deal... | |
| 14:53:11 | sean-k-mooney | i mean we could just not set a limit on the worker trhead in the tread pool or make it a config option | |
| 14:53:42 | sean-k-mooney | python will grow and srink the thread pool automatically if you dont specify the number of workers | |
| 14:53:53 | dansmith | that's the problem, these are a threadpool, which means they're re-used and capped at some max number, so enough api requests come in during a cell down event, and you immediately exhaust the pool of workers until things time out | |
| 14:54:15 | dansmith | sean-k-mooney: only up to N*CPUs I think I read | |
| 14:54:27 | dansmith | where N is like 3 or something | |
| 14:54:53 | sean-k-mooney | dansmith: maybe not sure | |
| 14:55:09 | sean-k-mooney | but we could leave it as a config option and defult to -1 mean let python figure it out | |
| 14:55:26 | sean-k-mooney | or set it to X if you care and have mesured for your system | |
| 14:55:34 | mriedem | https://docs.python.org/3.6/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor | |
| 14:55:40 | mriedem | "If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor." | |
| 14:56:25 | dansmith | right, so with 5 cells, that becomes $CPUs | |
| 14:56:26 | dansmith | and everyone has at least two | |
| 14:57:15 | cdent | when calculating the number of threads that will be around, another thing to keep in mind is mod_wsgi will often run with multiple process and multiple threads in those processes. what is the pool associated with? | |
| 14:57:45 | cdent | every request or thread or process? | |
| 14:58:43 | dansmith | idk | |
| 14:58:44 | cdent | process | |
| 14:59:40 | sean-k-mooney | so we could revert to createing the thread pool per call and set it to one thread per cell | |
| 15:00:09 | sean-k-mooney | but the issue with that is a its expensive to spawn treads and b we nolonger have a limit on them | |
| 15:00:47 | sean-k-mooney | but we will not have the issue of a potention exausting of the thread pool | |
| 15:01:30 | sean-k-mooney | cdent: the pool i a module global in the current patch so it would associated with the process | |
| 15:03:13 | cdent | sean-k-mooney: [t 6MVa] | |
| 15:03:13 | purplerbot | <cdent> process [2019-05-16 14:58:44.001687] [n 6MVa] | |
| 15:03:55 | sean-k-mooney | ? | |
| 15:04:23 | cdent | [l 6MVa] | |
| 15:04:23 | purplerbot | http://p.anticdent.org/6MVa | |
| 15:04:39 | cdent | in other words, I figured it out by looking at the code a few minutes ago | |
| 15:04:50 | sean-k-mooney | oh right :) | |
| 15:06:13 | mriedem | efried: on that ironic job failure, it looks like that job has probably been broken since stein when we started reporting capability traits for nodes http://logs.openstack.org/32/634832/29/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/fba9197/controller/logs/devstacklog.txt.gz#_2019-05-16_03_28_21_590 | |
| 15:06:21 | jangutter | cdent: I really thought you were mistyping 6 million volt-ampere. | |
| 15:06:36 | mriedem | if i'm reading that correctly it's expecting only the CUSTOM_GOLD trait | |
| 15:06:40 | sean-k-mooney | dansmith: what was the behavior that motivated the scater gather code in the first place to you rememebr? was it to deal with down cells or was it a performce imporvoment? | |
| 15:07:12 | dansmith | sean-k-mooney: it's to avoid a sequential walk of all the cell databases, serializing their latencies | |
| 15:07:28 | sean-k-mooney | ah ok so it was to addres perfromace ok | |
| 15:07:51 | cdent | Are there numbers on those latencies? | |
| 15:07:57 | dansmith | sean-k-mooney: cern has 70 cells, some of which are remote | |