| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-10 | |||
| 18:12:38 | artom | dansmith, yeah, efried made the same argument - and it's true | |
| 18:12:50 | dansmith | artom: not that it's no risk at all, but it is a teensy bit different | |
| 18:13:03 | artom | dansmith, yep, I get you | |
| 18:13:08 | mriedem | except all of the resource tracker weird side bugs refactoring that code will probably introduce on everyone | |
| 18:13:17 | dansmith | well, that's true | |
| 18:13:35 | efried | mriedem: retrieving the allocations earlier than we were before. That's the only difference. | |
| 18:13:49 | efried | All of it runs under COMPUTE_RESOURCE_SEMAPHORE | |
| 18:13:55 | artom | Anyways, my point was: Intel spun up a CI for their RFE in a matter of days (apparently). RH sucks if we can't do the same | |
| 18:13:58 | efried | did before, does now. | |
| 18:14:08 | mriedem | artom: the guy was working on that since denver i think | |
| 18:14:18 | artom | mriedem, on the CI? | |
| 18:14:18 | mriedem | efried: btw i was harassing you about "Intel NFV CI" not Intel_Zuul | |
| 18:14:25 | efried | yeah, I can go ask, but it wasn't like days | |
| 18:14:28 | mriedem | "Intel NFV CI" is an old thing that no longer does anything except comment that it's not doing anything | |
| 18:14:37 | mriedem | artom: yeah | |
| 18:14:43 | mriedem | we were talking about 3rd party CI in denver | |
| 18:14:44 | mriedem | for vpmem | |
| 18:14:48 | artom | mriedem, OK, I'll eat my words then | |
| 18:15:05 | mriedem | you and the other red hat bros might have been hungover still from the bar the night before :P | |
| 18:15:07 | efried | mriedem: fwiw Intel NFV CI they're trying to resurrect to do the thing it was originally intended for. | |
| 18:15:17 | mriedem | efried: i've been hearing that for months | |
| 18:15:22 | efried | first step was turning it back on to be a no-op | |
| 18:15:24 | mriedem | i'd like it to shut up until it actually delivers | |
| 18:15:26 | efried | yeah I know. | |
| 18:15:40 | artom | mriedem, I have Russian roots, calling me an alcoholic is a noop ;) | |
| 18:16:26 | sean-k-mooney | mriedem: we shoudl be able to test the pcpu stuff in the gate yes | |
| 18:16:30 | mriedem | https://www.youtube.com/watch?v=soNcOfRvOtg | |
| 18:17:00 | dansmith | <3 | |
| 18:17:19 | dansmith | I love me some early 'priest | |
| 18:17:50 | sean-k-mooney | i was talking to a few people on the infra channel and i think we migth be able to replace teh intel nfv ci with first party ci. | |
| 18:18:28 | sean-k-mooney | we might be abel to get multi numa nested virt lables form limestone and vexhost in the future | |
| 18:18:55 | sean-k-mooney | we can do non numa testing in the gate already as we have 3 providers with nested virt capablity | |
| 18:19:10 | sean-k-mooney | or rather singel numa | |
| 18:21:05 | sean-k-mooney | the testing i set up for the numa live migration was/is testing with cpu pinning, multiple numa nodes and hugepages | |
| 18:23:51 | mriedem | hmm, it seems weird to me that the intel pmem job is running and passing on patches that don't have anything to do with pmem when pmem isn't merged | |
| 18:23:56 | mriedem | like, how does that even work? | |
| 18:24:50 | sean-k-mooney | mriedem: they have hardocde a specifc version of the patch to be merged in | |
| 18:25:17 | sean-k-mooney | they were tring to un do that earlier | |
| 18:25:47 | mriedem | 2019-09-10 03:27:01.729430 | TASK [upgrade-libvirt-qemu : apply vpmem patch] | |
| 18:25:49 | mriedem | aha | |
| 18:25:55 | efried | yeah | |
| 18:26:04 | mriedem | refs/changes/70/678470/13 -> FETCH_HEAD | |
| 18:26:05 | efried | unfortunately it's applying PS13, not the latest | |
| 18:26:11 | mriedem | that patch is up to PS27 now | |
| 18:26:12 | sean-k-mooney | yep | |
| 18:26:24 | efried | known, Rui is supposed to unwind that asap. | |
| 18:26:42 | mriedem | i'm not sure why you wouldn't just only run it on patches in that series and skip for anything else | |
| 18:26:54 | efried | that would have been another way to do it. | |
| 18:27:05 | efried | though it wouldn't have made sense to do it for patches at the bottom either | |
| 18:27:13 | mriedem | artom: sean-k-mooney: so this is the numa lm patch/job we care about right? https://review.opendev.org/#/c/680739/ | |
| 18:27:40 | sean-k-mooney | yes more or less | |
| 18:27:49 | mriedem | which hasn't run on the latest series of patches | |
| 18:28:42 | sean-k-mooney | yes so if you recheck it | |
| 18:28:49 | sean-k-mooney | it will just run the singel job we want | |
| 18:29:11 | sean-k-mooney | shall i do it | |
| 18:29:15 | artom | mriedem, it hasn't - what's FN's status, are we back up? | |
| 18:29:15 | mriedem | ye shall | |
| 18:29:19 | artom | donnyd ^^? | |
| 18:29:32 | donnyd | yea | |
| 18:29:40 | donnyd | it was back up yesterday | |
| 18:29:51 | artom | 👍 | |
| 18:29:56 | sean-k-mooney | ya there is ocationally a network issue | |
| 18:30:27 | sean-k-mooney | becaue we curently cant fail over to another cloud it more obvios with this job | |
| 18:31:49 | donnyd | im not sure why this particular job keeps doing that too... it seems to be failure more often than not | |
| 18:32:14 | donnyd | every now and again i get something that pukes... but by and large it works | |
| 18:32:54 | sean-k-mooney | well this job cant retry on another provider and im not sure if zuul will retry on the same one | |
| 18:33:16 | sean-k-mooney | so i think its more a case of it works our your out of luck | |
| 18:34:59 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329 | |
| 18:35:05 | donnyd | Well if FN was having this kind of failure rate we are seeing with this job... I am pretty sure the other projects would have already kilt me | |
| 18:36:03 | sean-k-mooney | well i admit it does look like there is something else going on but im at a loss as to what | |
| 18:36:55 | donnyd | I don't think we were having this much of an issue when it was in a seperate pool on the other label | |
| 18:37:42 | sean-k-mooney | well we have hit a few different issues. 1 was quotas when we change pool | |
| 18:37:50 | sean-k-mooney | then the other issue is the ssh connection | |
| 18:38:19 | sean-k-mooney | i think the quota issue has gone away sicne you are nolonger managin that on your end and contoling it via nodepool | |
| 18:38:26 | donnyd | well there is no more quota (within reason) max-servers is set to 70 and quota for instances set to 100 | |
| 18:38:33 | donnyd | everything else on quota is -1 | |
| 18:38:38 | sean-k-mooney | ya | |
| 18:39:02 | sean-k-mooney | the pools were both the same project on the same cloud right | |
| 18:39:12 | donnyd | correct | |
| 18:39:12 | sean-k-mooney | so it should not affect this behavior | |
| 18:39:20 | donnyd | should is the opportune word | |
| 18:39:36 | sean-k-mooney | yes this also should work :) | |
| 18:39:44 | donnyd | LOL sean-k-mooney | |
| 18:39:55 | mriedem | https://review.opendev.org/#/c/634606/ went from PS75 to PS83 in a hurry | |
| 18:40:04 | donnyd | So the ssh thing I have some theories on and a fix in flight | |
| 18:40:16 | donnyd | my edge router could be a little (lot) better | |
| 18:40:40 | donnyd | so its possible that its struggling with all of the connections... or i have a knob i need to turn | |
| 18:41:07 | donnyd | load is not high according to cpu / mem / network... but that doesn't mean something else isn't borked... so I am digging | |
| 18:42:06 | sean-k-mooney | ya perhaps you said your using bgp to advertiese the block right but i assume you dont need to adverteise each /64 you are delegating to the vms and have a /48 or something from your isp | |
| 18:43:49 | donnyd | each vm is on the same /64 that is advertised (i know... not cloudy) at the edge | |
| 18:44:15 | donnyd | the zuul tenant has one /64 and its already advertised.. so if we had a routing issue it would be pretty big | |
| 18:44:20 | sean-k-mooney | donnyd: oh you have a /64 for the sight and the vms are getting a /128? form that subnet | |
| 18:44:34 | donnyd | its also possible my state table is too small | |
| 18:44:49 | donnyd | I have tweaked some knobs to start | |
| 18:44:57 | donnyd | so hopefully the issue goes away | |
| 18:45:36 | sean-k-mooney | i know that alot of hardware router assume that endpoing are /64 so i know use /128 or more stict routes can cause issue somethimes | |
| 18:45:38 | donnyd | yea, i tried to get the functionality to work pretty much like v4 so people don't get too confused | |
| 18:46:06 | donnyd | The way its setup is exactly like if your isp provided you a /64 | |
| 18:46:20 | sean-k-mooney | right | |
| 18:46:26 | sean-k-mooney | ok makes sense | |