| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-10 | |||
| 18:28:49 | sean-k-mooney | it will just run the singel job we want | |
| 18:29:11 | sean-k-mooney | shall i do it | |
| 18:29:15 | artom | mriedem, it hasn't - what's FN's status, are we back up? | |
| 18:29:15 | mriedem | ye shall | |
| 18:29:19 | artom | donnyd ^^? | |
| 18:29:32 | donnyd | yea | |
| 18:29:40 | donnyd | it was back up yesterday | |
| 18:29:51 | artom | 👍 | |
| 18:29:56 | sean-k-mooney | ya there is ocationally a network issue | |
| 18:30:27 | sean-k-mooney | becaue we curently cant fail over to another cloud it more obvios with this job | |
| 18:31:49 | donnyd | im not sure why this particular job keeps doing that too... it seems to be failure more often than not | |
| 18:32:14 | donnyd | every now and again i get something that pukes... but by and large it works | |
| 18:32:54 | sean-k-mooney | well this job cant retry on another provider and im not sure if zuul will retry on the same one | |
| 18:33:16 | sean-k-mooney | so i think its more a case of it works our your out of luck | |
| 18:34:59 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329 | |
| 18:35:05 | donnyd | Well if FN was having this kind of failure rate we are seeing with this job... I am pretty sure the other projects would have already kilt me | |
| 18:36:03 | sean-k-mooney | well i admit it does look like there is something else going on but im at a loss as to what | |
| 18:36:55 | donnyd | I don't think we were having this much of an issue when it was in a seperate pool on the other label | |
| 18:37:42 | sean-k-mooney | well we have hit a few different issues. 1 was quotas when we change pool | |
| 18:37:50 | sean-k-mooney | then the other issue is the ssh connection | |
| 18:38:19 | sean-k-mooney | i think the quota issue has gone away sicne you are nolonger managin that on your end and contoling it via nodepool | |
| 18:38:26 | donnyd | well there is no more quota (within reason) max-servers is set to 70 and quota for instances set to 100 | |
| 18:38:33 | donnyd | everything else on quota is -1 | |
| 18:38:38 | sean-k-mooney | ya | |
| 18:39:02 | sean-k-mooney | the pools were both the same project on the same cloud right | |
| 18:39:12 | donnyd | correct | |
| 18:39:12 | sean-k-mooney | so it should not affect this behavior | |
| 18:39:20 | donnyd | should is the opportune word | |
| 18:39:36 | sean-k-mooney | yes this also should work :) | |
| 18:39:44 | donnyd | LOL sean-k-mooney | |
| 18:39:55 | mriedem | https://review.opendev.org/#/c/634606/ went from PS75 to PS83 in a hurry | |
| 18:40:04 | donnyd | So the ssh thing I have some theories on and a fix in flight | |
| 18:40:16 | donnyd | my edge router could be a little (lot) better | |
| 18:40:40 | donnyd | so its possible that its struggling with all of the connections... or i have a knob i need to turn | |
| 18:41:07 | donnyd | load is not high according to cpu / mem / network... but that doesn't mean something else isn't borked... so I am digging | |
| 18:42:06 | sean-k-mooney | ya perhaps you said your using bgp to advertiese the block right but i assume you dont need to adverteise each /64 you are delegating to the vms and have a /48 or something from your isp | |
| 18:43:49 | donnyd | each vm is on the same /64 that is advertised (i know... not cloudy) at the edge | |
| 18:44:15 | donnyd | the zuul tenant has one /64 and its already advertised.. so if we had a routing issue it would be pretty big | |
| 18:44:20 | sean-k-mooney | donnyd: oh you have a /64 for the sight and the vms are getting a /128? form that subnet | |
| 18:44:34 | donnyd | its also possible my state table is too small | |
| 18:44:49 | donnyd | I have tweaked some knobs to start | |
| 18:44:57 | donnyd | so hopefully the issue goes away | |
| 18:45:36 | sean-k-mooney | i know that alot of hardware router assume that endpoing are /64 so i know use /128 or more stict routes can cause issue somethimes | |
| 18:45:38 | donnyd | yea, i tried to get the functionality to work pretty much like v4 so people don't get too confused | |
| 18:46:06 | donnyd | The way its setup is exactly like if your isp provided you a /64 | |
| 18:46:20 | sean-k-mooney | right | |
| 18:46:26 | sean-k-mooney | ok makes sense | |
| 18:46:28 | donnyd | all the systems on your network could get addresses off that /64 and the isp routes it | |
| 18:46:43 | donnyd | that is exactly how I have FN setup | |
| 18:47:05 | donnyd | not great for billing (don't have billing at FN), but great for actual networky things | |
| 18:47:29 | donnyd | so if two instances needed to talk in the same tenant... they could just start talking.. | |
| 18:47:36 | sean-k-mooney | im currently trying to get ipv6 working via a HE.net ipv4->ipv6 tunnel but i have only got as far as my router has ipv6 | |
| 18:47:49 | sean-k-mooney | when i had my client get ipv6 i had mtu issue | |
| 18:48:11 | sean-k-mooney | so your doing better then me at getting ipv6 to work | |
| 18:48:55 | donnyd | i have my /48 routed to me via HE and then I am able to pass out /64's to tenants | |
| 18:49:27 | sean-k-mooney | ya that is what i setup on my router too but couldnt get the mtu clamping to work | |
| 18:49:59 | sean-k-mooney | so i could get to ipv6 sight but packtes over aboutu 1350-1400 bytes were droped | |
| 18:50:10 | donnyd | that is strange | |
| 18:50:21 | donnyd | what are you using at your edge? | |
| 18:50:33 | donnyd | could be ISP not doing you any favors | |
| 18:50:45 | donnyd | i have a business connection, so they pretty much leave me alone | |
| 18:50:51 | sean-k-mooney | a ubiquite edgerouter-x well actully that is not my edge router | |
| 18:50:58 | sean-k-mooney | my isp router is infront of it | |
| 18:51:12 | sean-k-mooney | so you my isp router could be messing it up | |
| 18:51:47 | dansmith | also a business connection here and they'll give me multiple /64s for my subnets | |
| 18:52:12 | sean-k-mooney | i pay extra for a static ip but they wont let me pay for a buisness connection and that is the only way to get native ipv6 | |
| 18:52:33 | dansmith | our residential connections here all have native ipv6, | |
| 18:52:43 | dansmith | but not sure those people can get multiple /64s like I can | |
| 18:52:53 | sean-k-mooney | most are on cable broadband here but not vsdl | |
| 18:53:01 | donnyd | dansmith: verizon fios business doesn't even have v6 to offer | |
| 18:53:08 | dansmith | donnyd: sucks | |
| 18:53:15 | sean-k-mooney | all the fiber to home stuff is ipv6 enabled | |
| 18:53:16 | dansmith | mine is cable | |
| 18:53:22 | donnyd | yea.. its pretty frustrating | |
| 18:53:43 | dansmith | I don't like to say nice things about comcast, but they do have the ipv6 stuff pretty well sorted at this point (and have for a couple years) | |
| 18:53:44 | donnyd | the network is pretty quick... so the HE overlay doesn't really seem to be hurting for performance | |
| 18:54:00 | dansmith | yeah, I loved my fios business for speed when I had it | |
| 18:54:14 | dansmith | but then moved outside their area and they mostly stopped expanding it | |
| 18:54:14 | donnyd | yea I had them when i was in CoSprings and their business class was pretty good | |
| 18:55:58 | sean-k-mooney | artom: sorry this took so long but the latest version is updating the numa toplogy blob in the db correctly | |
| 18:56:22 | sean-k-mooney | so puting back in the instance.refresh() or whatever you did fixed that issue | |
| 18:58:35 | sean-k-mooney | im going to try testing a bunch of different cases but are there any in partcalar people want me to test | |
| 19:05:26 | mriedem | artom: so don't respin now since we need to get a result from that ci job, but queue this locally https://review.opendev.org/#/c/634606/83 | |
| 19:06:28 | artom | mriedem, ack | |
| 19:06:32 | artom | sean-k-mooney, cool, thank you :) | |
| 19:07:07 | artom | I managed to hit it func tests, as I said. But... that was by cheating and setting do_cleanup = True in the code itself, to trigger the driver.cleanup() call | |
| 19:08:25 | mriedem | smells like a networking orgy in here | |
| 19:08:33 | artom | So I'm trying to do it correctly by forcing making is_shared_instance_path false | |
| 19:08:42 | artom | Which leads to a whole other rabbit hole... | |
| 19:08:55 | dansmith | mriedem: efried: not sure if you saw this, but alex_xu confirmed that we just stop caring about quotas on pcpu instances (IIUC) per my question about how it works: https://review.opendev.org/#/c/674895/32/nova/virt/libvirt/driver.py | |
| 19:09:27 | mriedem | stephenfin said something about dealing with quota needs to happen yet | |
| 19:09:38 | dansmith | mriedem: efried: I can see the solution being a new quota class (ick) or lumping them together (which may be confusing) but just leaving them ignored doesn't seem like a good plan to me, especially since it differs for the user based on whether or not the deployment is configured for placement | |
| 19:11:10 | sean-k-mooney | dansmith: for what its worth i point out the PCPU quota issue in the unified limts spec | |
| 19:12:03 | sean-k-mooney | if we implement it next cycle we either will have 2 quotas or have to use both vcpu and pcpu count when looking at the cpu quota | |
| 19:13:12 | dansmith | we have to do something other than just pretend they're not there regardless of when we land it | |
| 19:13:41 | sean-k-mooney | well we did not intend to pretend they are not there | |
| 19:13:49 | dansmith | since it's all about consuming whole cpus, I'm pretty sure that operators won't be okay with just allowing anyone to boot enough pcpu guests to exhaust the whole deployment | |
| 19:13:50 | sean-k-mooney | but yes | |
| 19:13:55 | dansmith | I know, but... | |
| 19:14:37 | sean-k-mooney | personally i prefer having two spereate quotas one per RC but i can see why some wont want to distiguiush | |