| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-10 | |||
| 18:10:20 | artom | Intel_Zuul appeared a few days ago | |
| 18:10:36 | mriedem | dansmith: yeah it's sitting in a tab, was going through gibi's series until i hit a stopping point, which i just did | |
| 18:10:58 | mriedem | gibi: https://review.opendev.org/#/c/676972/ re-introduces the race you just fixed, so i'll rebase the series and fix that and +W once your other change is merged | |
| 18:11:23 | artom | I talked with efried a few days ago, mention the apparent disconnect between the "burden of proof" on my for NUMA LM, vs the VPMEM stuff that was apparently ready to go in without any public demonstrations of it working | |
| 18:11:37 | mriedem | and "Intel NFV CI" comments on everything immediately just to say it was skipped, which is annoying | |
| 18:11:48 | mriedem | efried: any way you can get "Intel NFV CI" to shut up if it's not going to do anything? | |
| 18:11:51 | mriedem | it hasn't done anything for years | |
| 18:11:55 | artom | Dunno if Intel_Zuul popping up was related, but the coincidence was interesting :) | |
| 18:12:19 | mriedem | artom: fwiw i'm not ready for vpmem to go in | |
| 18:12:21 | dansmith | artom: well, one difference is that yours has the potential to break existing functionality, whereas vpmem hopefully won't break anything existing, only people that try to use it | |
| 18:12:21 | efried | mriedem, artom: The Intel_Zuul is actually running. It's only running pmem, three tests. Unfortunately at the moment it's hardcoded to pull down an old version of the pmem series. | |
| 18:12:30 | efried | right ^ | |
| 18:12:38 | artom | dansmith, yeah, efried made the same argument - and it's true | |
| 18:12:50 | dansmith | artom: not that it's no risk at all, but it is a teensy bit different | |
| 18:13:03 | artom | dansmith, yep, I get you | |
| 18:13:08 | mriedem | except all of the resource tracker weird side bugs refactoring that code will probably introduce on everyone | |
| 18:13:17 | dansmith | well, that's true | |
| 18:13:35 | efried | mriedem: retrieving the allocations earlier than we were before. That's the only difference. | |
| 18:13:49 | efried | All of it runs under COMPUTE_RESOURCE_SEMAPHORE | |
| 18:13:55 | artom | Anyways, my point was: Intel spun up a CI for their RFE in a matter of days (apparently). RH sucks if we can't do the same | |
| 18:13:58 | efried | did before, does now. | |
| 18:14:08 | mriedem | artom: the guy was working on that since denver i think | |
| 18:14:18 | artom | mriedem, on the CI? | |
| 18:14:18 | mriedem | efried: btw i was harassing you about "Intel NFV CI" not Intel_Zuul | |
| 18:14:25 | efried | yeah, I can go ask, but it wasn't like days | |
| 18:14:28 | mriedem | "Intel NFV CI" is an old thing that no longer does anything except comment that it's not doing anything | |
| 18:14:37 | mriedem | artom: yeah | |
| 18:14:43 | mriedem | we were talking about 3rd party CI in denver | |
| 18:14:44 | mriedem | for vpmem | |
| 18:14:48 | artom | mriedem, OK, I'll eat my words then | |
| 18:15:05 | mriedem | you and the other red hat bros might have been hungover still from the bar the night before :P | |
| 18:15:07 | efried | mriedem: fwiw Intel NFV CI they're trying to resurrect to do the thing it was originally intended for. | |
| 18:15:17 | mriedem | efried: i've been hearing that for months | |
| 18:15:22 | efried | first step was turning it back on to be a no-op | |
| 18:15:24 | mriedem | i'd like it to shut up until it actually delivers | |
| 18:15:26 | efried | yeah I know. | |
| 18:15:40 | artom | mriedem, I have Russian roots, calling me an alcoholic is a noop ;) | |
| 18:16:26 | sean-k-mooney | mriedem: we shoudl be able to test the pcpu stuff in the gate yes | |
| 18:16:30 | mriedem | https://www.youtube.com/watch?v=soNcOfRvOtg | |
| 18:17:00 | dansmith | <3 | |
| 18:17:19 | dansmith | I love me some early 'priest | |
| 18:17:50 | sean-k-mooney | i was talking to a few people on the infra channel and i think we migth be able to replace teh intel nfv ci with first party ci. | |
| 18:18:28 | sean-k-mooney | we might be abel to get multi numa nested virt lables form limestone and vexhost in the future | |
| 18:18:55 | sean-k-mooney | we can do non numa testing in the gate already as we have 3 providers with nested virt capablity | |
| 18:19:10 | sean-k-mooney | or rather singel numa | |
| 18:21:05 | sean-k-mooney | the testing i set up for the numa live migration was/is testing with cpu pinning, multiple numa nodes and hugepages | |
| 18:23:51 | mriedem | hmm, it seems weird to me that the intel pmem job is running and passing on patches that don't have anything to do with pmem when pmem isn't merged | |
| 18:23:56 | mriedem | like, how does that even work? | |
| 18:24:50 | sean-k-mooney | mriedem: they have hardocde a specifc version of the patch to be merged in | |
| 18:25:17 | sean-k-mooney | they were tring to un do that earlier | |
| 18:25:47 | mriedem | 2019-09-10 03:27:01.729430 | TASK [upgrade-libvirt-qemu : apply vpmem patch] | |
| 18:25:49 | mriedem | aha | |
| 18:25:55 | efried | yeah | |
| 18:26:04 | mriedem | refs/changes/70/678470/13 -> FETCH_HEAD | |
| 18:26:05 | efried | unfortunately it's applying PS13, not the latest | |
| 18:26:11 | mriedem | that patch is up to PS27 now | |
| 18:26:12 | sean-k-mooney | yep | |
| 18:26:24 | efried | known, Rui is supposed to unwind that asap. | |
| 18:26:42 | mriedem | i'm not sure why you wouldn't just only run it on patches in that series and skip for anything else | |
| 18:26:54 | efried | that would have been another way to do it. | |
| 18:27:05 | efried | though it wouldn't have made sense to do it for patches at the bottom either | |
| 18:27:13 | mriedem | artom: sean-k-mooney: so this is the numa lm patch/job we care about right? https://review.opendev.org/#/c/680739/ | |
| 18:27:40 | sean-k-mooney | yes more or less | |
| 18:27:49 | mriedem | which hasn't run on the latest series of patches | |
| 18:28:42 | sean-k-mooney | yes so if you recheck it | |
| 18:28:49 | sean-k-mooney | it will just run the singel job we want | |
| 18:29:11 | sean-k-mooney | shall i do it | |
| 18:29:15 | artom | mriedem, it hasn't - what's FN's status, are we back up? | |
| 18:29:15 | mriedem | ye shall | |
| 18:29:19 | artom | donnyd ^^? | |
| 18:29:32 | donnyd | yea | |
| 18:29:40 | donnyd | it was back up yesterday | |
| 18:29:51 | artom | 👍 | |
| 18:29:56 | sean-k-mooney | ya there is ocationally a network issue | |
| 18:30:27 | sean-k-mooney | becaue we curently cant fail over to another cloud it more obvios with this job | |
| 18:31:49 | donnyd | im not sure why this particular job keeps doing that too... it seems to be failure more often than not | |
| 18:32:14 | donnyd | every now and again i get something that pukes... but by and large it works | |
| 18:32:54 | sean-k-mooney | well this job cant retry on another provider and im not sure if zuul will retry on the same one | |
| 18:33:16 | sean-k-mooney | so i think its more a case of it works our your out of luck | |
| 18:34:59 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329 | |
| 18:35:05 | donnyd | Well if FN was having this kind of failure rate we are seeing with this job... I am pretty sure the other projects would have already kilt me | |
| 18:36:03 | sean-k-mooney | well i admit it does look like there is something else going on but im at a loss as to what | |
| 18:36:55 | donnyd | I don't think we were having this much of an issue when it was in a seperate pool on the other label | |
| 18:37:42 | sean-k-mooney | well we have hit a few different issues. 1 was quotas when we change pool | |
| 18:37:50 | sean-k-mooney | then the other issue is the ssh connection | |
| 18:38:19 | sean-k-mooney | i think the quota issue has gone away sicne you are nolonger managin that on your end and contoling it via nodepool | |
| 18:38:26 | donnyd | well there is no more quota (within reason) max-servers is set to 70 and quota for instances set to 100 | |
| 18:38:33 | donnyd | everything else on quota is -1 | |
| 18:38:38 | sean-k-mooney | ya | |
| 18:39:02 | sean-k-mooney | the pools were both the same project on the same cloud right | |
| 18:39:12 | donnyd | correct | |
| 18:39:12 | sean-k-mooney | so it should not affect this behavior | |
| 18:39:20 | donnyd | should is the opportune word | |
| 18:39:36 | sean-k-mooney | yes this also should work :) | |
| 18:39:44 | donnyd | LOL sean-k-mooney | |
| 18:39:55 | mriedem | https://review.opendev.org/#/c/634606/ went from PS75 to PS83 in a hurry | |
| 18:40:04 | donnyd | So the ssh thing I have some theories on and a fix in flight | |
| 18:40:16 | donnyd | my edge router could be a little (lot) better | |
| 18:40:40 | donnyd | so its possible that its struggling with all of the connections... or i have a knob i need to turn | |
| 18:41:07 | donnyd | load is not high according to cpu / mem / network... but that doesn't mean something else isn't borked... so I am digging | |