| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-26 | |||
| 14:44:07 | mriedem | good night | |
| 14:48:21 | mriedem | stephenfin: are you aware of cold migration/resize support for numa topology changes ^ ? | |
| 14:48:27 | mriedem | i thought that was all baked in long ago | |
| 14:49:28 | kashyap | mriedem: IIRC, he's out for a few more days. | |
| 14:49:46 | mriedem | ok i'll wait for cfriesen then | |
| 14:54:53 | mriedem | dansmith: do you think https://review.openstack.org/#/c/607735/ is worth sending to rocky? | |
| 14:55:02 | mriedem | it's a pretty latent issue, so not really sure it's worth it | |
| 14:55:41 | dansmith | mriedem: it is, but it's a pretty trivial thing to push back and we know it's a problem for people | |
| 14:55:48 | dansmith | we also know we can't push it back any farther, | |
| 14:55:58 | dansmith | so it's not like it can go back to kilo or anything | |
| 14:56:02 | mriedem | sure | |
| 14:56:03 | mriedem | ok | |
| 15:04:41 | cfriesen | mriedem: alex_xu: pretty sure cold migration *was* working before all the placement stuff, but I seem to remember seeing at least one bug saying it's currently broken. | |
| 15:05:55 | cfriesen | we're on pike at the moment and it seems to be working, but we do have a few patches in that area for other features. | |
| 15:06:07 | sean-k-mooney | cold migration in what context? | |
| 15:06:41 | sean-k-mooney | mriedem: stephenfin is on PTO until next week | |
| 15:06:54 | sean-k-mooney | mriedem: he left his znc bouncer running | |
| 15:07:34 | mriedem | cfriesen: as alex pointed out, | |
| 15:07:45 | mriedem | i don't see the request spec numa topologies field updated before it goes through the scheduler | |
| 15:08:01 | sean-k-mooney | mriedem: oh alex_xu was asking about numa toplogy changes for resize/cold migration it "should" work upstream | |
| 15:08:29 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L74 | |
| 15:08:45 | mriedem | i don't see RequestSpec.numa_topology updated from the new flavor *before* we hit the numa filter | |
| 15:09:04 | mriedem | so i don't see how the scheduling is working | |
| 15:09:31 | sean-k-mooney | hum so you think its using the old flavor perhaps | |
| 15:09:47 | sean-k-mooney | i can try testing this in a hour or so | |
| 15:10:03 | jmlowe | Has anybody had trouble with the placement client in nova compute hanging? | |
| 15:10:04 | sean-k-mooney | i jsut need to get my dev enviornemt running after the weekend | |
| 15:10:25 | mriedem | jmlowe: never heard of that | |
| 15:10:45 | jmlowe | Specifically nova.compute.resource_tracker._update_available_resource was holding a lock for about 2 min | |
| 15:11:05 | jmlowe | Sometimes it would do it and sometimes not | |
| 15:11:33 | jmlowe | no errors anywhere | |
| 15:12:21 | cdent | jmlowe: how many instances on that compute node? | |
| 15:12:42 | jmlowe | It did wreak havoc with instance launches | |
| 15:12:43 | cdent | are you sure it was talking to placement where things were stuck, or just somewhere in _udpate*? | |
| 15:12:58 | cdent | alos | |
| 15:13:05 | cdent | ^walso | |
| 15:13:07 | cfriesen | mriedem: sorry, had to answer a call from my boss. let me take a quick look | |
| 15:14:22 | jmlowe | I got it down to 2 min to run _update_inventory in nova/scheduler/client/report.py | |
| 15:15:07 | mriedem | jmlowe: how many instances on that compute? | |
| 15:15:10 | jmlowe | it seems to have made the http put call correctly, but then just waits for tcp timeout | |
| 15:15:13 | mriedem | libvirt or vcenter or ironic? | |
| 15:15:36 | jmlowe | it's all of my 280 computes, so 0 - 24 | |
| 15:15:42 | jmlowe | libvirt | |
| 15:16:04 | mriedem | hmm, well there is a lock in the resource tracker when that is called, which will make things in nova-compute slow to a crawl, | |
| 15:16:11 | mriedem | but why the placement response would be so slow idk, | |
| 15:16:21 | mriedem | have you traced the request via request ID in the placement api logs? | |
| 15:16:27 | mriedem | also, which release? | |
| 15:16:44 | jmlowe | yes, nothing in placement takes more than a second or so | |
| 15:16:57 | jmlowe | queens | |
| 15:18:20 | jmlowe | seems like the there's a bug in some underlying client library that is occasionally not returning from a http call without a timeout | |
| 15:18:38 | mriedem | hmm, nova is using keystoneauth1 to send the requests to placement | |
| 15:18:57 | mriedem | you might be able to enable some debug logging there | |
| 15:19:45 | cdent | jmlowe: I think you should make jeremy find and fix this | |
| 15:20:28 | cdent | not you fungi | |
| 15:20:48 | jmlowe | I did, then went on a 6 week road trip | |
| 15:20:55 | fungi | yay! for once something's not my fault | |
| 15:20:56 | jmlowe | then he went | |
| 15:21:08 | cdent | The jeremy of which I speak is an old friend (on the order of 30 years) | |
| 15:21:13 | cdent | Typical of him. | |
| 15:21:17 | cfriesen | mriedem: alex_xu: I think it's this code that updates the flavor on a resize: https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L304-L316 | |
| 15:22:27 | mriedem | cfriesen: but that doesn't update the RequestSpec.numa_topology field | |
| 15:22:44 | mriedem | the else block | |
| 15:22:57 | mriedem | or RequestSpec.pci_requests for that matter | |
| 15:23:44 | mriedem | so if resize with a new numa topology works today, it's getting lucky b/c the scheduler picks a host that fits the original topology, and then the RT.move_claim on the compute fits the new requested numa topology | |
| 15:23:53 | mriedem | as far as i can tell anyway | |
| 15:24:42 | sean-k-mooney | mriedem: well we trow away all the host pinning info and recalulate it on the compute node anyway so it is likely working becaue it gets lucky or we retry | |
| 15:24:54 | cdent | jmlowe: do you have load balancers or proxies between n-cpu placement? | |
| 15:25:03 | jmlowe | I do | |
| 15:25:10 | cdent | connections perhaps not closing? | |
| 15:25:13 | mriedem | cdent: fwiw, that ENABLED_PYTHON3_PACKAGES variable in devstack looks like it should be killed now | |
| 15:25:26 | mriedem | "# Special case some services that have experimental | |
| 15:25:27 | mriedem | # support for python3 in progress, but don't claim support | |
| 15:25:27 | mriedem | # in their classifier" | |
| 15:25:28 | cdent | mriedem: it does come into play | |
| 15:25:33 | mriedem | yeah i see where it's used | |
| 15:25:39 | mriedem | but i'm just thinking it shouldn't exist anymore | |
| 15:25:44 | jmlowe | probably, I'd expect the http client to close the connection once it received a 200 responxe | |
| 15:25:49 | mriedem | given the py3 first goal | |
| 15:26:02 | mriedem | e.g. neutron isn't in that list | |
| 15:26:04 | mriedem | nor keystone | |
| 15:26:17 | sean-k-mooney | mriedem: well it either shouldn not exists or default to ture right | |
| 15:26:21 | cdent | mriedem: yes, I agree that it appears to be an anachronism that shouldn't be used at all, but that it sometimes does get used | |
| 15:26:33 | cdent | it is currently what makes my fix work | |
| 15:26:33 | mriedem | i shall put out the dhellmann signal | |
| 15:26:48 | sean-k-mooney | mriedem: there are some project that dont rung on python3 i belive | |
| 15:26:57 | jmlowe | Testing now, but it seems I can mitigate all of this with the timeout setting in the placement section of nova.conf | |
| 15:27:46 | mriedem | cdent: a peach? | |
| 15:28:27 | mriedem | jmlowe: do you have to set connection timeouts for other nova interactions, like with glance? | |
| 15:28:42 | jmlowe | I've never needed them before | |
| 15:29:39 | mriedem | weird. i think in queens we were going through ksa for glance interactions as well | |
| 15:29:45 | mriedem | so it should all be the same | |
| 15:29:54 | mriedem | from a client perspective i mean, | |
| 15:29:57 | mriedem | just different api servers | |
| 15:31:31 | jmlowe | I have been seeing the occasional ssl error in neutron clients, so could even be as simple as some sort of bug in haproxy | |
| 15:31:39 | cfriesen | mriedem: sean-k-mooney: yeah, I think you might be right, and most of the time it gets fixed up by the claim. ick. | |
| 15:33:23 | sean-k-mooney | cfriesen: mriedem we need to do a cleanup/audit of the migraion codepaths in general. artoms live migration spec will help but we likely have some cleanup to do for resize/cold migrate too. | |
| 15:34:07 | cdent | jmlowe: what's hosting your placement service? mod_wsgi, uwsgi, something else? | |
| 15:35:05 | sean-k-mooney | cfriesen: mriedem i know in pratcice cold migrate and resize "usually" work and results in the numa toplogy being recaluated but it may be just getting lucky or relying on retries. when we model numa in placemnt this will cange and we will have to correct the behavior for that | |
| 15:36:00 | sean-k-mooney | cfriesen: by the way i dont know if you saw my ML post regarding the pci numa affitiy policies | |
| 15:36:40 | cdent | efried: thanks for the +w on the external placement in nova change. I assume we want to let its child ( https://review.openstack.org/#/c/618215/ ) sit until more things have settled | |