| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 14:05:10 | bauzas | there is still 24 hours left | |
| 14:05:15 | bauzas | -ish | |
| 14:05:30 | sean-k-mooney | artom: let me know when you push your local version so i can test. im still getting the same error with the same vcpu_pin_set on each host | |
| 14:05:36 | bauzas | oh dear no | |
| 14:05:38 | donnyd | https://www.irccloud.com/pastebin/nh6VDxYb/ | |
| 14:05:42 | bauzas | PTL Nominations Aug 27, 2019 23:45 UTC Sep 03, 2019 23:45 UTC | |
| 14:05:51 | sean-k-mooney | bauzas: there are like 9 hours left | |
| 14:05:55 | donnyd | https://www.irccloud.com/pastebin/1dwaNShp/ | |
| 14:05:57 | bauzas | well, we can afford a PTL designation | |
| 14:06:00 | donnyd | sean-k-mooney: ^^^ | |
| 14:06:09 | donnyd | LMK if those look right to you | |
| 14:06:24 | stephenfin | bauzas: I didn't, but looking at it now I don't think that's we should do that there | |
| 14:06:31 | sean-k-mooney | donnyd: yep they look fine | |
| 14:06:46 | stephenfin | That's a much bigger discussion that what that patch is trying to fix (namely, making sure it's enabled for all the libvirt functional tests) | |
| 14:06:57 | donnyd | project config stuff will be up in a minute | |
| 14:07:16 | bauzas | stephenfin: I don't disagree, I'm just saying that enabling NUMATopologyFilter on *all* libvirt tests is a massive hammer with little justification | |
| 14:07:42 | sean-k-mooney | donnyd: cool ping me when you push it. | |
| 14:07:58 | bauzas | stephenfin: so, honestly, that's why the question is more widely like : if we go with enabling the filter for all tests, then why not enabling it by default ? | |
| 14:08:12 | sean-k-mooney | donnyd: i can proably make the job work on the smaller memory node too or use one of each | |
| 14:08:17 | bauzas | and I know this is a debatable question :) | |
| 14:08:24 | stephenfin | bauzas: It's already enabled for all but two of them, are it _should_ be enabled for one of those two | |
| 14:08:46 | dansmith | artom: so the multi-numa nfv thing is running live migrations with your patches yeah? | |
| 14:08:48 | efried | sean-k-mooney: I haven't decided yet. | |
| 14:09:14 | cdent | dansmith for ptl! | |
| 14:09:14 | sean-k-mooney | dansmith: yes im using depens on to pull in his patches | |
| 14:09:27 | stephenfin | It was enabled for all these tests in test_pci_sriov_servers.py and test_numa_servers.py. It should have been enabled for test_rt_servers.py. The only other one is test_reshape.py. I can disable it for that one, if you'd like? | |
| 14:09:42 | sean-k-mooney | dansmith: if you use depends-on in the same repo it create a merge commit assumeing there is not merge conflcits and runs on that | |
| 14:09:48 | stephenfin | or set it explicitly on the other three. Either one is good with me | |
| 14:09:49 | bauzas | stephenfin: uh | |
| 14:09:51 | dansmith | sean-k-mooney: yeah, I see that's mechanically true, but ... you guys have gone through and seen that it's working? | |
| 14:09:53 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | |
| 14:09:54 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 14:09:54 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 14:09:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 14:10:00 | sean-k-mooney | dansmith: yes | |
| 14:10:01 | artom | sean-k-mooney, ^^ | |
| 14:10:11 | sean-k-mooney | i have used it in the past | |
| 14:10:17 | artom | dansmith, yeah, huge props to sean-k-mooney for getting it set up (and donnyd's hardware!) | |
| 14:10:23 | dansmith | artom: does it look like it's all working? | |
| 14:10:23 | donnyd | https://review.opendev.org/679789 | |
| 14:10:35 | donnyd | I don't think I forgot anything | |
| 14:10:39 | sean-k-mooney | dansmith: hugepages are not fully working yet | |
| 14:10:44 | artom | dansmith, needs to run against the latest PSes, as it found issues with hugepages | |
| 14:10:50 | dansmith | artom: okay cool | |
| 14:11:11 | bauzas | stephenfin: honestly, dunno what to say | |
| 14:11:51 | bauzas | stephenfin: maybe just add a little more explanation in the commit msg and then I'll consider having the option set for all libvirt tests to be good enough | |
| 14:12:02 | sean-k-mooney | artom: does that patch have the fixes for hugepages. ill test it locally while we wait for ci | |
| 14:12:32 | donnyd | cdent: which part of FN do you think is faster. I know disk access should be in orders of magnitude faster from local NVME (in raid0) | |
| 14:13:01 | sean-k-mooney | damb raid0 i guess for ci it makes sense | |
| 14:13:02 | stephenfin | bauzas: Sure, lemme do that | |
| 14:13:19 | stephenfin | efried: yo, any chance you could sign these off for me? https://review.opendev.org/#/c/678902 https://review.opendev.org/#/c/678861 ? | |
| 14:13:24 | donnyd | sean-k-mooney: yea if something fails.. it just restarts the job on another hypervisor | |
| 14:13:34 | donnyd | so speed is the name of the game for CI... | |
| 14:13:40 | cdent | donnyd: it does seem like it mostly related to disk speed. Where I see it the most is: a) getting log files from the swift hosted on FN, b) the placement performance are consistently faster there | |
| 14:13:40 | sean-k-mooney | donnyd: devstack is hugely io bound | |
| 14:13:55 | efried | stephenfin: looking | |
| 14:14:07 | stephenfin | bauzas: Need any help with https://review.opendev.org/#/c/678902, btw? That's probably the trickiest patch of the series | |
| 14:14:17 | donnyd | I would never do that for prod things - but for CI.. go as fast as possible is where I went with it | |
| 14:14:37 | sean-k-mooney | donnyd: well ci is kind of prod. | |
| 14:14:42 | stephenfin | Well, it looks tricky cos it's big. It's not really though | |
| 14:14:53 | sean-k-mooney | donnyd: but i have no issue with raid0 as long as you dont put your database on it | |
| 14:15:00 | bauzas | stephenfin: not sure I understand your question ? Why should I need help on something I reviewed ? | |
| 14:15:05 | luyao | dansmith: Could you look at https://review.opendev.org/#/c/678448/9 and https://review.opendev.org/#/c/678447/https://review.opendev.org/635229. Thanks. | |
| 14:15:12 | donnyd | sean-k-mooney: but if I lose something its not like losing a customers long running instance I can't get back | |
| 14:15:22 | stephenfin | bauzas: Damn, I meant https://review.opendev.org/#/c/671800/ | |
| 14:15:24 | sean-k-mooney | donnyd: ya that is ture | |
| 14:15:47 | donnyd | umm.. no.. the controller is setup a little bit more robust | |
| 14:15:52 | sean-k-mooney | donnyd: the could model however is to keep data in sepereate data volumes that are attached to a stateles instance. | |
| 14:16:02 | bauzas | stephenfin: oh this one is too hairy for now, I'll look at it tomorrow morning :) | |
| 14:16:05 | stephenfin | If you modify the address bar in Firefox, it doesn't change as you navigate pages until you refresh the page | |
| 14:16:10 | luyao | dansmith: sorry, they are https://review.opendev.org/#/c/678447/ and https://review.opendev.org/#/c/678448/9 | |
| 14:16:12 | sean-k-mooney | *cloud model | |
| 14:16:21 | stephenfin | So you end up copy-pasting the same thing multiple times | |
| 14:16:25 | stephenfin | bauzas: Heh, cool | |
| 14:17:21 | donnyd | I need to get HA controllers running, however I have to make a choice between # of instances to run and potential for downtime... Heat is my battle here | |
| 14:17:36 | donnyd | I use forced air - NO A/C | |
| 14:17:50 | stephenfin | efried: I know you don't like adding config options without any users, but what about model fields? I feel I should split https://review.opendev.org/#/c/671800/ up so people can focus on the functional changes rather than the noise introduced by the additional field | |
| 14:17:53 | bauzas | oh shit, dadtaxi, yeah | |
| 14:18:34 | bauzas | stephenfin: I'm not worried by the size of the change in question | |
| 14:18:55 | bauzas | anyway, bbiab | |
| 14:19:06 | donnyd | If you think about it - I can push a button and redeploy the whole control plane.. so is two hours of downtime once a year or so worth losing 30 instances in CI | |
| 14:19:34 | donnyd | I made my bets on short downtime being more acceptable than losing CI resources | |
| 14:19:43 | sean-k-mooney | donnyd: well contoler can run vms too | |
| 14:19:51 | sean-k-mooney | donnyd: what are you using to deploy | |
| 14:20:04 | donnyd | Its custom for my infra | |
| 14:20:10 | sean-k-mooney | ah ok | |
| 14:20:30 | donnyd | I was using tripleo - but I couldn't get rabbit to act right | |
| 14:21:01 | sean-k-mooney | donnyd: ya i dont like how tripleo deplfoys rabbit | |
| 14:21:26 | sean-k-mooney | well i dont like how tripleo deploys things in general | |
| 14:21:42 | donnyd | well from my perspective there is a performance hit from running in containers - small but noticeable at my scale | |
| 14:21:43 | sean-k-mooney | but that is another mater | |
| 14:22:01 | donnyd | https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&refresh=30s | |
| 14:22:03 | sean-k-mooney | really i have nver notticed any performac ehit | |
| 14:22:16 | donnyd | You can see I don't have much left on my controller for VM's | |
| 14:22:34 | sean-k-mooney | oh you mean from a ram proint of view | |
| 14:23:39 | luyao | efried: Would you like to test the patches about virtual persistent memory support? I have tested it on real hardware, and I'm preparing another OpenStack with pmems, so you can test on it if you want. I can send you how to access those servers after I set up the env. | |
| 14:24:08 | donnyd | Yea, memory is short in the controller | |
| 14:24:37 | efried | luyao: It would be nice if *someone* could do that testing in addition to you, just to get another pair of eyes on it. But I'm not sure I have the time to do it myself. | |
| 14:24:38 | sean-k-mooney | donnyd: thats becasue heat is partly because of heat | |
| 14:24:39 | donnyd | only has 96G | |