| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-01 | |||
| 18:40:27 | sean-k-mooney | you can pass a cpuset | |
| 18:40:38 | mnaser | https://github.com/openstack/zun/commit/7b3b1c5e07eb8b9e81031fa6ee4625ff6870b41d | |
| 18:40:44 | mnaser | this is the big commit and i dont think they have that (yet) | |
| 18:41:19 | sean-k-mooney | ya so without that it would not be safe to run zun on a nova host that has a PCPU inventory | |
| 18:41:50 | sean-k-mooney | s/safe/correct/ as the contianer could steal cpu time from the pinned instnace | |
| 18:44:00 | sean-k-mooney | mnaser: i also see this being an issue if we delet the nova compute service on that host but not the zun service | |
| 18:44:19 | mnaser | yeah i think that's an operator problem at that point because setting the zun setting would be invalid | |
| 18:44:24 | sean-k-mooney | unless they intentally took acount of that | |
| 18:44:35 | sean-k-mooney | not really | |
| 18:44:38 | mnaser | but then i think if you swap out that setting in flight | |
| 18:44:40 | sean-k-mooney | if i deploy with both | |
| 18:44:41 | mnaser | you probably lose all your allocation | |
| 18:44:54 | sean-k-mooney | and then later wanted to remvoe nova to just have zun | |
| 18:45:12 | sean-k-mooney | then wen we delete the compute node rp as part of service deleteion it might cause issue for zuun | |
| 18:45:37 | sean-k-mooney | right we will try to delete all the allcoation against the compute node ro | |
| 18:45:41 | sean-k-mooney | *RP | |
| 18:46:49 | sean-k-mooney | i dont recal the zun team discussing this with the nova team at the denver PTG did i miss a session on this? | |
| 18:47:34 | mnaser | yeah i think i remember talking about it with dansmith saying "it would be cool if" but i think they mentioned that "nova assumes it owns the rp, there is no such thing as rp sharing" | |
| 18:48:09 | sean-k-mooney | mnaser: right as of today it does | |
| 18:48:25 | sean-k-mooney | back in the dublin ptg i was suggestign nova should not own the compute node RP | |
| 18:48:45 | mnaser | ok so compute node rp sharing is a thing that isn't an antipattern | |
| 18:48:59 | sean-k-mooney | and each service should have there own nested RP subtree runder a shared compute node RP with no inventories | |
| 18:49:16 | sean-k-mooney | mnaser: right now it not a thing | |
| 18:50:00 | sean-k-mooney | well not really a thing im not 100% sure that neutorn bandwith based schduleing is not shareing an RP with nova | |
| 18:50:11 | sean-k-mooney | gibi: ^ do you know the answer to that | |
| 18:50:55 | sean-k-mooney | mnaser: im not aware of share where not is not the owner of the compute node RP. | |
| 18:54:43 | sean-k-mooney | looking at the zun patch tehy seam to have ported alot of code form nova which is not nessisarlly a good thing. | |
| 18:55:34 | efried | mnaser, sean-k-mooney: For some definition of "sharing". It doesn't matter who creates the RP (neutron vs nova for bw, cyborg vs nova for accels, etc) -- the important thing is that, if you're creating an instance, the entire *claim* needs to be atomic, and therefore done by nova. | |
| 18:55:56 | sean-k-mooney | or zun | |
| 18:56:16 | efried | as long as nova doesn't think it's going to come in afterwards and do some other claim, sure. | |
| 18:56:33 | sean-k-mooney | well zun has its own top level api | |
| 18:56:41 | efried | I don't really know what zun is or what it does | |
| 18:57:19 | sean-k-mooney | its a container as a service project that started as a replacment for nova-docker | |
| 18:57:29 | efried | I do recall seeing they were porting in a bunch of nova's resource tracker and scheduler gorp | |
| 18:57:32 | sean-k-mooney | so it can be used as a nova virt driver or as a standalone service | |
| 18:57:36 | efried | basically inheriting a mountain of technical debt | |
| 18:57:39 | efried | and we told them not to do that | |
| 18:57:42 | efried | and they did it anyway. | |
| 18:57:52 | sean-k-mooney | yes hence my comment above | |
| 18:59:00 | sean-k-mooney | when i see patches merge with TODO comments from jaypipes i assume that was not a recent TODO and there for form nova | |
| 19:00:57 | sean-k-mooney | the bit that worres my is fi the share with nova flag is set | |
| 19:02:15 | sean-k-mooney | the pass the full provider tree for the nova compute node RP to there update_provider_tree function | |
| 19:03:46 | sean-k-mooney | they are also modifing trait possible on the root RP | |
| 19:17:42 | mriedem | sean-k-mooney: bw providers are nested under the compute node rp | |
| 19:18:48 | mriedem | the case that sean-k-mooney mentioned about nova deleting the compute node and root resource provider when you delete a compute service, i can definitely see that screwing with zun if they assume the same resource provider with the same uuid is going to exist, i don't know how they link that up, maybe via hostname | |
| 19:19:03 | mriedem | we have our own problems managing that stuff, i.e. the orphaned allocations/providers bug we have right now, | |
| 19:19:11 | mriedem | let alone worrying about how we're going to be breaking external things | |
| 19:19:21 | efried | mriedem: Did you have something to do with https://review.opendev.org/#/c/684326/ not being combined with its (current) successor https://review.opendev.org/#/c/685937/ ? (USE_NEUTRON refactoring in tests) | |
| 19:19:37 | sean-k-mooney | ya so depending on the valude of "share with nova" they either create there own root RP or share nova | |
| 19:20:13 | sean-k-mooney | and they use the hostname to look up the one created by nova if "shareing" is enabled | |
| 19:20:14 | mriedem | efried: you can see my comments on PS1, i don't know why it changed like it did, i haven't looked yet, but it definitely isn't what i was expecting | |
| 19:20:29 | mriedem | efried: he had https://review.opendev.org/#/c/684326/1 | |
| 19:20:44 | mriedem | and then in the next patch in the series reverted a bunch of what he added with https://review.opendev.org/#/c/684327/ | |
| 19:20:45 | efried | mriedem: yeah, the earlier comments have been addressed to squash together a now-abandoned change | |
| 19:20:48 | mriedem | i said just squash those | |
| 19:21:02 | mriedem | sure, he did but then there were other things tacked on that i wasn't expecting | |
| 19:21:05 | efried | I'm just wondering why there's still two patches, one that sets False and the other that removes True | |
| 19:21:09 | mriedem | and haven't taken the time to grok the diff | |
| 19:21:29 | mriedem | iow, i said "just do x" and he did x + 4 | |
| 19:22:43 | mriedem | btw, unrelated to this, but we have another train rc2 fix on master with a +2 and ack from the bug reporter that it fixes the regression https://review.opendev.org/#/c/685194/ | |
| 19:23:28 | efried | o, I was reviewing that and then... didn't finish for some reason. | |
| 19:23:30 | efried | stand by. | |
| 19:24:19 | sean-k-mooney | i was going to say we shoudl navoe nova serial testing in the gate but it actuly just i nthe unit tests | |
| 19:25:01 | sean-k-mooney | getting spice testing should be pretty simple we jus tneed to pick one of our exisitg jobs to swap over ot it | |
| 19:25:12 | sean-k-mooney | im not sure if devstack deploy nova-serial | |
| 19:25:19 | sean-k-mooney | it proably does but i have never used it | |
| 19:25:27 | sean-k-mooney | melwitt: ^ | |
| 19:27:47 | efried | mriedem, melwitt: +A | |
| 19:27:54 | mriedem | hot | |
| 19:28:10 | sean-k-mooney | if we change greande-py3 to nova-serial-proxy and sawped nova-greade-multione to spice and kept the neturon-greade multinoe as the default novnc it should cover all three? | |
| 19:38:55 | efried | above my pay grade, sounds like a melwitt question. | |
| 19:42:13 | zigo | Heya ! Got my first VM running on Train with all the RC1 packaged and uploaded to Debian already ! :) | |
| 19:42:29 | zigo | It's never been that smooth ... :P | |
| 19:46:24 | mriedem | zigo: that's good news, thanks for sharing | |
| 19:46:32 | mriedem | we'll have an RC2, just fyi | |
| 19:46:53 | zigo | mriedem: For the moment, there's no "but x is broken" part, as much as I know... :P | |
| 19:47:32 | zigo | Still waiting for Glance RC1 though, it still not out. | |
| 19:47:53 | zigo | Looks like it Glance from Stein is enough. :) | |
| 20:23:22 | mnaser | hmm | |
| 20:23:29 | mnaser | i remember there was the rabbitmq api stuff we talked about a while ago | |
| 20:23:52 | mnaser | is it possible that at some point if nova heartbeats get messed up, all processes seem to be forever stuck (heartbeat misses accumulate) | |
| 20:24:58 | mnaser | >missed heartbeats from client, timeout: 60s | |
| 20:25:07 | mnaser | with some "client unexpectedly closed TCP connection" | |
| 20:25:26 | mnaser | and a whole bunch of: Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed | |
| 20:25:44 | mnaser | but also "Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer" (assuming this is when rabbitmq kills it cause it hasnt gotten heartbeat) | |
| 20:25:57 | mnaser | and for context this is a very busy api so its not sitting idle | |
| 20:26:13 | mriedem | are you thinking of this? https://review.opendev.org/#/c/662095/5/releasenotes/notes/eventlet-monkey-patch-5f734ef581aa550e.yaml | |
| 20:27:11 | mnaser | hmm, threads=1 in this deployment (osa with uwsgi) | |
| 20:27:26 | mnaser | and again this is a quite busy api endpoint so i dont imagine it sat idle for more than.. 5-10s at a time | |
| 20:28:16 | mriedem | maybe check this out https://review.opendev.org/#/c/663074/ | |
| 20:28:30 | mnaser | simple math shows 1000 requests per last 26 minutes so 38 requests per minute.. doubt thats enough to trigger something lik ethis | |
| 20:31:13 | mnaser | hmm | |
| 20:31:17 | mnaser | it'll be hard to run that | |
| 20:31:27 | mnaser | but im also a bit curious why its triggering here at all | |
| 20:36:48 | mnaser | also we run HEAD / every few seconds too | |
| 20:43:02 | mriedem | efried: explain to me why this wouldn't work in a test: | |
| 20:43:04 | mriedem | 1. do something, | |
| 20:43:07 | mriedem | 2. patch x | |
| 20:43:11 | mriedem | 3. do something else | |
| 20:43:19 | mriedem | 4. stop the patch on | |