| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 21:36:05 | artom | mriedem, <3 | |
| 21:36:14 | stephenfin | VPMEM conflicted with LM but efried fixed it | |
| 21:36:20 | mriedem | ...because it's been a thing people have wanted forever | |
| 21:36:23 | stephenfin | PCPU and VPMEM is a horror show | |
| 21:36:40 | sean-k-mooney | mriedem: sicne we intoduced numa in like juno | |
| 21:36:42 | artom | mriedem, and here I thought it was because of my pretty eyes | |
| 21:36:42 | stephenfin | in terms of number of merge conflicts, not complexity | |
| 21:36:54 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 21:36:54 | artom | Incoming | |
| 21:36:54 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | |
| 21:36:55 | mriedem | artom: you do have nice brown (?) eyes | |
| 21:36:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 21:36:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 21:36:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 21:36:58 | mriedem | i got lost in them | |
| 21:37:09 | efried | hazel is my vote | |
| 21:37:18 | mriedem | +Wing | |
| 21:37:20 | stephenfin | I could have worked on k8s... | |
| 21:37:35 | sean-k-mooney | stephenfin: i hear you like yaml | |
| 21:37:45 | sean-k-mooney | stephenfin: becasue that is all k8s is as a user | |
| 21:38:45 | mriedem | soon we'll all only code in yaml | |
| 21:39:07 | efried | mriedem: "I'll push a devstack change to put a required trait that I know will be on all libvirt computes and that can show the failure for us quicker than writing some functional libvirt test." I assume this is a joke. | |
| 21:39:22 | mriedem | efried: nope | |
| 21:39:29 | mriedem | it's a 2 line change in devstack, | |
| 21:39:33 | mriedem | yes i know it will take awhile to get a node | |
| 21:39:36 | efried | yeah, and an 11h... | |
| 21:39:44 | mriedem | but as melwitt pointed out earlier, nova gets punished by zuul | |
| 21:39:52 | mriedem | devstack probably doesn't | |
| 21:39:55 | efried | true story | |
| 21:40:03 | melwitt | yeah, a devstack change might make it in a few hours. or less | |
| 21:40:11 | sean-k-mooney | or you can cheat. notice that all my jobs completed in like 2 yours | |
| 21:40:13 | efried | I thought you meant push a change to nova/devstack | |
| 21:40:19 | mriedem | i heard fungi takes a devstack change manually and runs it on his own hardware :) | |
| 21:40:43 | sean-k-mooney | the FN special lables im using have there own pool and if you put them in the experimental queue run almost right away | |
| 21:40:58 | melwitt | let's cheat | |
| 21:41:55 | fungi | mriedem: i take devstack changes amd manually rub them on my feet. it's better than a pedicure | |
| 21:42:19 | mriedem | i was not expecting that response | |
| 21:42:39 | melwitt | that is some fungi CI indeed | |
| 21:42:53 | mriedem | now i just need to look up the magic incantation to configure a flavor with a required trait | |
| 21:42:53 | fungi | i no longer have a basement full of server racks | |
| 21:42:57 | sean-k-mooney | melwitt: i mean if we want to repoduce quickly we could. also im ment to be wringin a greneade job.... | |
| 21:42:59 | fungi | nor, you know, a basement | |
| 21:43:33 | sean-k-mooney | efried: stephenfin so what patches are where currently | |
| 21:43:39 | sean-k-mooney | vpmem is pending | |
| 21:43:47 | efried | mriedem: extra spec trait:YOUR_TRAIT_HERE='required' ? | |
| 21:43:49 | sean-k-mooney | and stephenfin you have a version of pcpu on that | |
| 21:43:58 | mriedem | efried: WRONG | |
| 21:44:02 | mriedem | trait:<trait>=required | |
| 21:44:06 | mriedem | quotes would kill you | |
| 21:44:15 | melwitt | sean-k-mooney: I only vaguely understand what y'all were talking about earlier. was just saying funny [to me] unhelpful things | |
| 21:44:20 | efried | mriedem: I was being syntax-y | |
| 21:44:34 | stephenfin | sean-k-mooney: I do, yeah. Just running tests locally before I push it up | |
| 21:44:35 | efried | if you're doing it on an osc cli, the quotes will go away | |
| 21:44:59 | efried | s/n osc// | |
| 21:45:25 | mriedem | https://review.opendev.org/681938 | |
| 21:45:44 | sean-k-mooney | melwitt: :) i mean if one queue is slow and the other is fast sometimes cheating in the gate is for the greater good | |
| 21:46:18 | efried | sean-k-mooney: keeping in mind that shoving stuff in the queue from another project will push all the nova stuff down a slot too. | |
| 21:46:54 | sean-k-mooney | ya i know that is why os-vif is generaly way quicker to test stuff in the nova | |
| 21:46:54 | fungi | yep. the change scheduling round-robin allocates nodes to changes by project, so the more changes there are for a given project requesting resources, the longer the later ones will wait for a turn | |
| 21:46:59 | mriedem | https://bugs.launchpad.net/nova/+bug/1843836 tagged for train-rc-potential so we can start using that tag | |
| 21:47:00 | openstack | Launchpad bug 1843836 in OpenStack Compute (nova) "Failure to schedule if flavor contains non-CPU flag traits" [Undecided,In progress] - Assigned to Stephen Finucane (stephenfinucane) | |
| 21:47:37 | sean-k-mooney | fungi: isnt that there to stop all gate capastity been eaten by triplo | |
| 21:47:48 | mriedem | sean-k-mooney: and nova and neutron | |
| 21:47:57 | fungi | that was a big part of it, but yes also nova and neutron ;) | |
| 21:48:09 | mriedem | https://bugs.launchpad.net/nova/+bugs?field.tag=train-rc-potential | |
| 21:48:28 | sean-k-mooney | yes but i think triplo still uses like the equivalent of nova and neutron combined | |
| 21:48:39 | sean-k-mooney | anyway it makes it fairer for everyone else | |
| 21:49:19 | fungi | under the old first-come-first-served scheduling, if a project has someone push a 30-change series in one shot then those all got priority and one-off changes for other projects got to wait until that entire series got the requested resources | |
| 21:50:28 | sean-k-mooney | we nver have 30+ long seriese in nova | |
| 21:50:32 | efried | never | |
| 21:50:59 | sean-k-mooney | only ever have 3 of them together | |
| 21:51:06 | sean-k-mooney | its never just 1 | |
| 21:51:31 | sean-k-mooney | also our downstream ci is supper dumb and really does not like it when that is down to it | |
| 21:52:24 | sean-k-mooney | our downstream ci does not first apply the previous patch before the current one | |
| 21:52:34 | fungi | i'll be the first to admit that the round-robin job scheduler algorithm is still painful, but in general being backlogged on builds is going to be painful for someone regardless | |
| 21:53:11 | sean-k-mooney | fungi: yes but at least zuul make series actully work | |
| 21:53:20 | sean-k-mooney | jenkins on the other hand... | |
| 21:53:49 | fungi | if people have ideas for less painful scheduling algorithms/rules we can consider those too | |
| 21:54:21 | sean-k-mooney | the only one i have come up with that i thought was beeter we cant do. | |
| 21:54:40 | fungi | still trying to figure out if some of the packet scheduling algorithms used for adaptive rate limiting in the network space could be applied to job scheduling | |
| 21:54:41 | sean-k-mooney | which is split check into fast-check and check | |
| 21:55:09 | sean-k-mooney | and stick the tox jobs and docs in fast check | |
| 21:55:34 | fungi | we already have an analog of qos in place by prioritizing different pipelines | |
| 21:55:51 | fungi | and perform windowing and exponential backoff on failure in dependent pipelines | |
| 21:56:09 | fungi | so it's not out of the realm of possibility that networking ideas have still more we can steal from | |
| 21:56:16 | sean-k-mooney | fungi: just dont follow tcp's window algortioum | |
| 21:56:29 | sean-k-mooney | but ya | |
| 21:56:35 | fungi | well, yeah, it's not the exact same algorithm, just the basic idea | |
| 21:57:19 | fungi | but basically we only allocate resources to a subset of changes in dependent pipelines like the gate, and then increase or decrease that window based on how many changes pass or fail | |
| 21:57:45 | sean-k-mooney | there are some intersting algortiom in the cache/task execution domain too | |
| 21:58:06 | fungi | so that even though the gate pipeline has priority over the check pipeline, a perpetually failing gate load won't monopolize all available resources and leave check starved entirely | |
| 21:58:48 | sean-k-mooney | well to get to gate you have to go through check | |
| 21:58:53 | sean-k-mooney | so that should not happen anyway | |
| 21:58:59 | sean-k-mooney | it would be self regualting | |
| 21:59:27 | sean-k-mooney | this would poably be better conversation to have on infra or zuul channel | |
| 21:59:33 | fungi | except dependent pipelines can eat orders of magnitude more resources due to gate resets from failures near the front of the queue | |
| 21:59:42 | efried | fungi: would be neat to have a fast-fail toggle... somewhere, somehow. | |
| 22:00:08 | sean-k-mooney | efried: its a conflict between fail fast and report as much info as possibel | |
| 22:00:15 | efried | yeah, I know | |
| 22:00:36 | sean-k-mooney | fungi: how hard would it be to have each job report independly | |