| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-10 | |||
| 17:24:56 | dansmith | if comments were addressed it's probably easy for other people to confirm, | |
| 17:25:08 | efried | if you're okay with that, then sure. | |
| 17:25:21 | dansmith | but for the record, I'm scared of both pcpu and vpmems at this point | |
| 17:25:24 | efried | imo the changes are simple and in line with what you requested | |
| 17:25:48 | efried | I don't blame you. But hey, it's just FF. We have *weeks* to fix bugs :P | |
| 17:43:02 | mriedem | i'll call it, the first major bugs from pmem and pcpu regressions will be after vexxhost upgrades to train, and then in about 18-24 months when other deployments start upgrading to train :) | |
| 17:43:16 | mriedem | maybe cern in a year | |
| 17:43:41 | dansmith | heh, yeah, we won't find any of the bugs in the FF->release window | |
| 17:45:39 | mriedem | is there any way to test pcpu in a gate job if we ran tempest smoke tests in serial or something? like wouldn't that just be a matter of creating a flavor with PCPU, single node devstack? | |
| 17:45:52 | mriedem | and configuring n-cpu for the dedicated CPUs on the host? | |
| 17:45:56 | mriedem | basically all of them | |
| 17:46:53 | dansmith | I'm less concerned about if it actually works in a contrived scenario, and more about an existing deployment trying to get through the upgrade and/or being able to use it without regressions or other issues | |
| 17:47:18 | mriedem | sure, and functional tests are good, they just aren't a replacement for the real thing | |
| 17:47:23 | dansmith | presumably vpmem testing is pretty much impossive | |
| 17:47:27 | dansmith | *impossible | |
| 17:47:43 | mriedem | i assume so, hence the 3rd party ci | |
| 17:57:11 | artom | Which, btw, came up super fast (though I dunno if it had been in the works for a long time before) | |
| 17:57:25 | artom | RH can learn a thing or 2 | |
| 18:02:25 | dansmith | mriedem: so numa LM should be ready for you, if I understand the current state right? | |
| 18:02:50 | dansmith | I hit the object patch again a bit ago | |
| 18:03:30 | dansmith | I think the third patch should be good too, but you wanted some verification from the NUMA boyz which sent it off into that cpu pinning tangent, so not sure if you're still waiting for something there | |
| 18:10:07 | mriedem | artom: came up super fast? you mean the 3rd party CI did? | |
| 18:10:13 | artom | mriedem, yeah | |
| 18:10:20 | artom | Intel_Zuul appeared a few days ago | |
| 18:10:36 | mriedem | dansmith: yeah it's sitting in a tab, was going through gibi's series until i hit a stopping point, which i just did | |
| 18:10:58 | mriedem | gibi: https://review.opendev.org/#/c/676972/ re-introduces the race you just fixed, so i'll rebase the series and fix that and +W once your other change is merged | |
| 18:11:23 | artom | I talked with efried a few days ago, mention the apparent disconnect between the "burden of proof" on my for NUMA LM, vs the VPMEM stuff that was apparently ready to go in without any public demonstrations of it working | |
| 18:11:37 | mriedem | and "Intel NFV CI" comments on everything immediately just to say it was skipped, which is annoying | |
| 18:11:48 | mriedem | efried: any way you can get "Intel NFV CI" to shut up if it's not going to do anything? | |
| 18:11:51 | mriedem | it hasn't done anything for years | |
| 18:11:55 | artom | Dunno if Intel_Zuul popping up was related, but the coincidence was interesting :) | |
| 18:12:19 | mriedem | artom: fwiw i'm not ready for vpmem to go in | |
| 18:12:21 | dansmith | artom: well, one difference is that yours has the potential to break existing functionality, whereas vpmem hopefully won't break anything existing, only people that try to use it | |
| 18:12:21 | efried | mriedem, artom: The Intel_Zuul is actually running. It's only running pmem, three tests. Unfortunately at the moment it's hardcoded to pull down an old version of the pmem series. | |
| 18:12:30 | efried | right ^ | |
| 18:12:38 | artom | dansmith, yeah, efried made the same argument - and it's true | |
| 18:12:50 | dansmith | artom: not that it's no risk at all, but it is a teensy bit different | |
| 18:13:03 | artom | dansmith, yep, I get you | |
| 18:13:08 | mriedem | except all of the resource tracker weird side bugs refactoring that code will probably introduce on everyone | |
| 18:13:17 | dansmith | well, that's true | |
| 18:13:35 | efried | mriedem: retrieving the allocations earlier than we were before. That's the only difference. | |
| 18:13:49 | efried | All of it runs under COMPUTE_RESOURCE_SEMAPHORE | |
| 18:13:55 | artom | Anyways, my point was: Intel spun up a CI for their RFE in a matter of days (apparently). RH sucks if we can't do the same | |
| 18:13:58 | efried | did before, does now. | |
| 18:14:08 | mriedem | artom: the guy was working on that since denver i think | |
| 18:14:18 | artom | mriedem, on the CI? | |
| 18:14:18 | mriedem | efried: btw i was harassing you about "Intel NFV CI" not Intel_Zuul | |
| 18:14:25 | efried | yeah, I can go ask, but it wasn't like days | |
| 18:14:28 | mriedem | "Intel NFV CI" is an old thing that no longer does anything except comment that it's not doing anything | |
| 18:14:37 | mriedem | artom: yeah | |
| 18:14:43 | mriedem | we were talking about 3rd party CI in denver | |
| 18:14:44 | mriedem | for vpmem | |
| 18:14:48 | artom | mriedem, OK, I'll eat my words then | |
| 18:15:05 | mriedem | you and the other red hat bros might have been hungover still from the bar the night before :P | |
| 18:15:07 | efried | mriedem: fwiw Intel NFV CI they're trying to resurrect to do the thing it was originally intended for. | |
| 18:15:17 | mriedem | efried: i've been hearing that for months | |
| 18:15:22 | efried | first step was turning it back on to be a no-op | |
| 18:15:24 | mriedem | i'd like it to shut up until it actually delivers | |
| 18:15:26 | efried | yeah I know. | |
| 18:15:40 | artom | mriedem, I have Russian roots, calling me an alcoholic is a noop ;) | |
| 18:16:26 | sean-k-mooney | mriedem: we shoudl be able to test the pcpu stuff in the gate yes | |
| 18:16:30 | mriedem | https://www.youtube.com/watch?v=soNcOfRvOtg | |
| 18:17:00 | dansmith | <3 | |
| 18:17:19 | dansmith | I love me some early 'priest | |
| 18:17:50 | sean-k-mooney | i was talking to a few people on the infra channel and i think we migth be able to replace teh intel nfv ci with first party ci. | |
| 18:18:28 | sean-k-mooney | we might be abel to get multi numa nested virt lables form limestone and vexhost in the future | |
| 18:18:55 | sean-k-mooney | we can do non numa testing in the gate already as we have 3 providers with nested virt capablity | |
| 18:19:10 | sean-k-mooney | or rather singel numa | |
| 18:21:05 | sean-k-mooney | the testing i set up for the numa live migration was/is testing with cpu pinning, multiple numa nodes and hugepages | |
| 18:23:51 | mriedem | hmm, it seems weird to me that the intel pmem job is running and passing on patches that don't have anything to do with pmem when pmem isn't merged | |
| 18:23:56 | mriedem | like, how does that even work? | |
| 18:24:50 | sean-k-mooney | mriedem: they have hardocde a specifc version of the patch to be merged in | |
| 18:25:17 | sean-k-mooney | they were tring to un do that earlier | |
| 18:25:47 | mriedem | 2019-09-10 03:27:01.729430 | TASK [upgrade-libvirt-qemu : apply vpmem patch] | |
| 18:25:49 | mriedem | aha | |
| 18:25:55 | efried | yeah | |
| 18:26:04 | mriedem | refs/changes/70/678470/13 -> FETCH_HEAD | |
| 18:26:05 | efried | unfortunately it's applying PS13, not the latest | |
| 18:26:11 | mriedem | that patch is up to PS27 now | |
| 18:26:12 | sean-k-mooney | yep | |
| 18:26:24 | efried | known, Rui is supposed to unwind that asap. | |
| 18:26:42 | mriedem | i'm not sure why you wouldn't just only run it on patches in that series and skip for anything else | |
| 18:26:54 | efried | that would have been another way to do it. | |
| 18:27:05 | efried | though it wouldn't have made sense to do it for patches at the bottom either | |
| 18:27:13 | mriedem | artom: sean-k-mooney: so this is the numa lm patch/job we care about right? https://review.opendev.org/#/c/680739/ | |
| 18:27:40 | sean-k-mooney | yes more or less | |
| 18:27:49 | mriedem | which hasn't run on the latest series of patches | |
| 18:28:42 | sean-k-mooney | yes so if you recheck it | |
| 18:28:49 | sean-k-mooney | it will just run the singel job we want | |
| 18:29:11 | sean-k-mooney | shall i do it | |
| 18:29:15 | artom | mriedem, it hasn't - what's FN's status, are we back up? | |
| 18:29:15 | mriedem | ye shall | |
| 18:29:19 | artom | donnyd ^^? | |
| 18:29:32 | donnyd | yea | |
| 18:29:40 | donnyd | it was back up yesterday | |
| 18:29:51 | artom | 👍 | |
| 18:29:56 | sean-k-mooney | ya there is ocationally a network issue | |
| 18:30:27 | sean-k-mooney | becaue we curently cant fail over to another cloud it more obvios with this job | |
| 18:31:49 | donnyd | im not sure why this particular job keeps doing that too... it seems to be failure more often than not | |
| 18:32:14 | donnyd | every now and again i get something that pukes... but by and large it works | |