| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-26 | |||
| 12:01:40 | sean-k-mooney | cdent: have a good weekend | |
| 12:05:43 | cdent | sean-k-mooney: I made a pact with myself to not do work over the weekend, but that ended up being easy because I seem to have a cold. So I sat in front of the netflix watching junk tv and feeling like junk. | |
| 12:08:25 | sean-k-mooney | well the cold suck but the fact you didnt work on your weekend is good. i spent a significat part of the weekend just listening to audio books and trying not to do anything technical | |
| 12:09:07 | sean-k-mooney | until lik 10 PM sunday when i decided to deploy kubernetes on an old laptop but i almost made it lol | |
| 12:11:16 | cdent | I think if I hadn't had the cold I probably wouldn't have made it | |
| 12:34:58 | tobias-urdin | oh boy, after reading api-ref for nova when searching for some help, seeing all the red boxes for the proxy apis deprecation almost gave me tears, nova will be so clean in the future | |
| 12:39:51 | cdent | relatively speaking :) | |
| 13:30:46 | sean-k-mooney | tobias-urdin: nova will only be clean if we actully remove the proxy apips form the codebase | |
| 13:31:04 | sean-k-mooney | tobias-urdin: to do that would require us to increase teh minium microversion we support | |
| 13:31:27 | sean-k-mooney | that is not something we have done since we introduced microverions as far as i am aware | |
| 13:32:19 | cdent | sean-k-mooney: there was talk at summit about doing such a thing | |
| 13:32:40 | cdent | but there are _many_ prereqs to make it possible. | |
| 13:32:57 | sean-k-mooney | cdent: it would be nice if we could but ya i assumed we would have a lot of work to do first | |
| 13:33:36 | sean-k-mooney | cdent: was there any progress on a common way of reporting errors at the summig | |
| 13:33:41 | sean-k-mooney | *summit | |
| 13:34:04 | cdent | not really, no | |
| 13:34:27 | sean-k-mooney | oh well i knew that was a long shot :) | |
| 13:40:51 | tobias-urdin | sean-k-mooney: true, i'm just happy about all the effort on the structure and cleanup of nova :) | |
| 13:55:43 | efried | n-sch meeting in 5 minutes in #openstack-meeting-alt | |
| 14:04:58 | mriedem | yikun: fyi https://blueprints.launchpad.net/nova/+spec/initial-allocation-ratios is back in a runway slot | |
| 14:05:03 | mriedem | https://etherpad.openstack.org/p/nova-runways-stein | |
| 14:05:22 | mriedem | dansmith: we should update the topic at some point for the current runways | |
| 14:27:37 | dansmith | mriedem: ack | |
| 14:29:34 | alex_xu | mriedem: do you know resize works with numa topology change in the new flavor? I can't found we parse the numa topo from new flavor. Try to figure out it maybe a bug or something we don't support well yet. | |
| 14:31:31 | mriedem | alex_xu: i think it's supported...we'll do the move_claim during resize which gets the new_flavor off the migration record in the resource tracker | |
| 14:31:53 | mriedem | and the request spec uses the new flavor (i think) when it runs through the scheduler (numa topo filter) during scheduling for the resize | |
| 14:32:08 | alex_xu | mriedem: yea...I saw the move claim code also. | |
| 14:32:40 | alex_xu | mriedem: but the request spec wont extract new numa topo from the new flavor. I guess we missed something | |
| 14:34:28 | mriedem | cfriesen might know off the top of his head faster than me | |
| 14:34:38 | mriedem | but i thought cold migration worked | |
| 14:35:40 | mriedem | alex_xu: you're saying we don't update https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L55 for the new flavor right? | |
| 14:36:24 | alex_xu | mriedem: yes | |
| 14:36:53 | alex_xu | mriedem: we generate new numa_topology in move claim. but we still use old numa_topology in the scheduler | |
| 14:38:40 | mriedem | now i'm having a hard time finding where we set the new_flavor on the request spec before calling the scheduler | |
| 14:39:11 | alex_xu | hah, probably because we don't have that :) | |
| 14:40:25 | mriedem | heh, no i think that happens b/c i just wrote a functional regression test that relies on it | |
| 14:40:37 | mriedem | https://review.openstack.org/#/c/619123/ | |
| 14:41:58 | alex_xu | oh, yea, I miss read that. but still not parse the numa stuff from flavor | |
| 14:43:47 | alex_xu | mriedem: it's late for me, I will dig into more tomorrow. thanks for the info, good to know that isn't something we don't support, then it is probably a bug... | |
| 14:43:50 | mriedem | alex_xu: https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L316 is where we set the new flavor on reqspec | |
| 14:44:01 | mriedem | alex_xu: sure, i'll ping you if i figure something out :) | |
| 14:44:07 | mriedem | good night | |
| 14:48:21 | mriedem | stephenfin: are you aware of cold migration/resize support for numa topology changes ^ ? | |
| 14:48:27 | mriedem | i thought that was all baked in long ago | |
| 14:49:28 | kashyap | mriedem: IIRC, he's out for a few more days. | |
| 14:49:46 | mriedem | ok i'll wait for cfriesen then | |
| 14:54:53 | mriedem | dansmith: do you think https://review.openstack.org/#/c/607735/ is worth sending to rocky? | |
| 14:55:02 | mriedem | it's a pretty latent issue, so not really sure it's worth it | |
| 14:55:41 | dansmith | mriedem: it is, but it's a pretty trivial thing to push back and we know it's a problem for people | |
| 14:55:48 | dansmith | we also know we can't push it back any farther, | |
| 14:55:58 | dansmith | so it's not like it can go back to kilo or anything | |
| 14:56:02 | mriedem | sure | |
| 14:56:03 | mriedem | ok | |
| 15:04:41 | cfriesen | mriedem: alex_xu: pretty sure cold migration *was* working before all the placement stuff, but I seem to remember seeing at least one bug saying it's currently broken. | |
| 15:05:55 | cfriesen | we're on pike at the moment and it seems to be working, but we do have a few patches in that area for other features. | |
| 15:06:07 | sean-k-mooney | cold migration in what context? | |
| 15:06:41 | sean-k-mooney | mriedem: stephenfin is on PTO until next week | |
| 15:06:54 | sean-k-mooney | mriedem: he left his znc bouncer running | |
| 15:07:34 | mriedem | cfriesen: as alex pointed out, | |
| 15:07:45 | mriedem | i don't see the request spec numa topologies field updated before it goes through the scheduler | |
| 15:08:01 | sean-k-mooney | mriedem: oh alex_xu was asking about numa toplogy changes for resize/cold migration it "should" work upstream | |
| 15:08:29 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L74 | |
| 15:08:45 | mriedem | i don't see RequestSpec.numa_topology updated from the new flavor *before* we hit the numa filter | |
| 15:09:04 | mriedem | so i don't see how the scheduling is working | |
| 15:09:31 | sean-k-mooney | hum so you think its using the old flavor perhaps | |
| 15:09:47 | sean-k-mooney | i can try testing this in a hour or so | |
| 15:10:03 | jmlowe | Has anybody had trouble with the placement client in nova compute hanging? | |
| 15:10:04 | sean-k-mooney | i jsut need to get my dev enviornemt running after the weekend | |
| 15:10:25 | mriedem | jmlowe: never heard of that | |
| 15:10:45 | jmlowe | Specifically nova.compute.resource_tracker._update_available_resource was holding a lock for about 2 min | |
| 15:11:05 | jmlowe | Sometimes it would do it and sometimes not | |
| 15:11:33 | jmlowe | no errors anywhere | |
| 15:12:21 | cdent | jmlowe: how many instances on that compute node? | |
| 15:12:42 | jmlowe | It did wreak havoc with instance launches | |
| 15:12:43 | cdent | are you sure it was talking to placement where things were stuck, or just somewhere in _udpate*? | |
| 15:12:58 | cdent | alos | |
| 15:13:05 | cdent | ^walso | |
| 15:13:07 | cfriesen | mriedem: sorry, had to answer a call from my boss. let me take a quick look | |
| 15:14:22 | jmlowe | I got it down to 2 min to run _update_inventory in nova/scheduler/client/report.py | |
| 15:15:07 | mriedem | jmlowe: how many instances on that compute? | |
| 15:15:10 | jmlowe | it seems to have made the http put call correctly, but then just waits for tcp timeout | |
| 15:15:13 | mriedem | libvirt or vcenter or ironic? | |
| 15:15:36 | jmlowe | it's all of my 280 computes, so 0 - 24 | |
| 15:15:42 | jmlowe | libvirt | |
| 15:16:04 | mriedem | hmm, well there is a lock in the resource tracker when that is called, which will make things in nova-compute slow to a crawl, | |
| 15:16:11 | mriedem | but why the placement response would be so slow idk, | |
| 15:16:21 | mriedem | have you traced the request via request ID in the placement api logs? | |
| 15:16:27 | mriedem | also, which release? | |
| 15:16:44 | jmlowe | yes, nothing in placement takes more than a second or so | |
| 15:16:57 | jmlowe | queens | |
| 15:18:20 | jmlowe | seems like the there's a bug in some underlying client library that is occasionally not returning from a http call without a timeout | |
| 15:18:38 | mriedem | hmm, nova is using keystoneauth1 to send the requests to placement | |
| 15:18:57 | mriedem | you might be able to enable some debug logging there | |
| 15:19:45 | cdent | jmlowe: I think you should make jeremy find and fix this | |
| 15:20:28 | cdent | not you fungi | |
| 15:20:48 | jmlowe | I did, then went on a 6 week road trip | |
| 15:20:55 | fungi | yay! for once something's not my fault | |
| 15:20:56 | jmlowe | then he went | |
| 15:21:08 | cdent | The jeremy of which I speak is an old friend (on the order of 30 years) | |
| 15:21:13 | cdent | Typical of him. | |