| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 14:10:23 | dansmith | artom: does it look like it's all working? | |
| 14:10:23 | donnyd | https://review.opendev.org/679789 | |
| 14:10:35 | donnyd | I don't think I forgot anything | |
| 14:10:39 | sean-k-mooney | dansmith: hugepages are not fully working yet | |
| 14:10:44 | artom | dansmith, needs to run against the latest PSes, as it found issues with hugepages | |
| 14:10:50 | dansmith | artom: okay cool | |
| 14:11:11 | bauzas | stephenfin: honestly, dunno what to say | |
| 14:11:51 | bauzas | stephenfin: maybe just add a little more explanation in the commit msg and then I'll consider having the option set for all libvirt tests to be good enough | |
| 14:12:02 | sean-k-mooney | artom: does that patch have the fixes for hugepages. ill test it locally while we wait for ci | |
| 14:12:32 | donnyd | cdent: which part of FN do you think is faster. I know disk access should be in orders of magnitude faster from local NVME (in raid0) | |
| 14:13:01 | sean-k-mooney | damb raid0 i guess for ci it makes sense | |
| 14:13:02 | stephenfin | bauzas: Sure, lemme do that | |
| 14:13:19 | stephenfin | efried: yo, any chance you could sign these off for me? https://review.opendev.org/#/c/678902 https://review.opendev.org/#/c/678861 ? | |
| 14:13:24 | donnyd | sean-k-mooney: yea if something fails.. it just restarts the job on another hypervisor | |
| 14:13:34 | donnyd | so speed is the name of the game for CI... | |
| 14:13:40 | cdent | donnyd: it does seem like it mostly related to disk speed. Where I see it the most is: a) getting log files from the swift hosted on FN, b) the placement performance are consistently faster there | |
| 14:13:40 | sean-k-mooney | donnyd: devstack is hugely io bound | |
| 14:13:55 | efried | stephenfin: looking | |
| 14:14:07 | stephenfin | bauzas: Need any help with https://review.opendev.org/#/c/678902, btw? That's probably the trickiest patch of the series | |
| 14:14:17 | donnyd | I would never do that for prod things - but for CI.. go as fast as possible is where I went with it | |
| 14:14:37 | sean-k-mooney | donnyd: well ci is kind of prod. | |
| 14:14:42 | stephenfin | Well, it looks tricky cos it's big. It's not really though | |
| 14:14:53 | sean-k-mooney | donnyd: but i have no issue with raid0 as long as you dont put your database on it | |
| 14:15:00 | bauzas | stephenfin: not sure I understand your question ? Why should I need help on something I reviewed ? | |
| 14:15:05 | luyao | dansmith: Could you look at https://review.opendev.org/#/c/678448/9 and https://review.opendev.org/#/c/678447/https://review.opendev.org/635229. Thanks. | |
| 14:15:12 | donnyd | sean-k-mooney: but if I lose something its not like losing a customers long running instance I can't get back | |
| 14:15:22 | stephenfin | bauzas: Damn, I meant https://review.opendev.org/#/c/671800/ | |
| 14:15:24 | sean-k-mooney | donnyd: ya that is ture | |
| 14:15:47 | donnyd | umm.. no.. the controller is setup a little bit more robust | |
| 14:15:52 | sean-k-mooney | donnyd: the could model however is to keep data in sepereate data volumes that are attached to a stateles instance. | |
| 14:16:02 | bauzas | stephenfin: oh this one is too hairy for now, I'll look at it tomorrow morning :) | |
| 14:16:05 | stephenfin | If you modify the address bar in Firefox, it doesn't change as you navigate pages until you refresh the page | |
| 14:16:10 | luyao | dansmith: sorry, they are https://review.opendev.org/#/c/678447/ and https://review.opendev.org/#/c/678448/9 | |
| 14:16:12 | sean-k-mooney | *cloud model | |
| 14:16:21 | stephenfin | So you end up copy-pasting the same thing multiple times | |
| 14:16:25 | stephenfin | bauzas: Heh, cool | |
| 14:17:21 | donnyd | I need to get HA controllers running, however I have to make a choice between # of instances to run and potential for downtime... Heat is my battle here | |
| 14:17:36 | donnyd | I use forced air - NO A/C | |
| 14:17:50 | stephenfin | efried: I know you don't like adding config options without any users, but what about model fields? I feel I should split https://review.opendev.org/#/c/671800/ up so people can focus on the functional changes rather than the noise introduced by the additional field | |
| 14:17:53 | bauzas | oh shit, dadtaxi, yeah | |
| 14:18:34 | bauzas | stephenfin: I'm not worried by the size of the change in question | |
| 14:18:55 | bauzas | anyway, bbiab | |
| 14:19:06 | donnyd | If you think about it - I can push a button and redeploy the whole control plane.. so is two hours of downtime once a year or so worth losing 30 instances in CI | |
| 14:19:34 | donnyd | I made my bets on short downtime being more acceptable than losing CI resources | |
| 14:19:43 | sean-k-mooney | donnyd: well contoler can run vms too | |
| 14:19:51 | sean-k-mooney | donnyd: what are you using to deploy | |
| 14:20:04 | donnyd | Its custom for my infra | |
| 14:20:10 | sean-k-mooney | ah ok | |
| 14:20:30 | donnyd | I was using tripleo - but I couldn't get rabbit to act right | |
| 14:21:01 | sean-k-mooney | donnyd: ya i dont like how tripleo deplfoys rabbit | |
| 14:21:26 | sean-k-mooney | well i dont like how tripleo deploys things in general | |
| 14:21:42 | donnyd | well from my perspective there is a performance hit from running in containers - small but noticeable at my scale | |
| 14:21:43 | sean-k-mooney | but that is another mater | |
| 14:22:01 | donnyd | https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&refresh=30s | |
| 14:22:03 | sean-k-mooney | really i have nver notticed any performac ehit | |
| 14:22:16 | donnyd | You can see I don't have much left on my controller for VM's | |
| 14:22:34 | sean-k-mooney | oh you mean from a ram proint of view | |
| 14:23:39 | luyao | efried: Would you like to test the patches about virtual persistent memory support? I have tested it on real hardware, and I'm preparing another OpenStack with pmems, so you can test on it if you want. I can send you how to access those servers after I set up the env. | |
| 14:24:08 | donnyd | Yea, memory is short in the controller | |
| 14:24:37 | efried | luyao: It would be nice if *someone* could do that testing in addition to you, just to get another pair of eyes on it. But I'm not sure I have the time to do it myself. | |
| 14:24:38 | sean-k-mooney | donnyd: thats becasue heat is partly because of heat | |
| 14:24:39 | donnyd | only has 96G | |
| 14:24:47 | efried | luyao: sounds like the kind of thing sean-k-mooney would enjoy :P | |
| 14:25:22 | sean-k-mooney | efried: i dont have pmem servers in my hallway | |
| 14:25:25 | efried | luyao: But I assume it requires specialized hardware, so needs to be inside Intel's firewall. | |
| 14:25:26 | efried | yeah. | |
| 14:25:47 | sean-k-mooney | so i would have to use qemu to fake pmem in a vm and then thest with nested virt | |
| 14:25:52 | donnyd | Due to the nature of forced air cooling, I can only do so much in the way of servers that can be turned on at a single point in time | |
| 14:26:02 | luyao | efried: yes, I will check if others can access the servers | |
| 14:26:19 | sean-k-mooney | efried: i could proably set that up but not until after i test all the other stuff on my list | |
| 14:26:40 | luyao | sean-k-mooney: you can use a normal file as a fake pmem | |
| 14:26:41 | sean-k-mooney | luyao: any progress on the intel ci for pmem | |
| 14:26:58 | sean-k-mooney | luyao: yes so if i spin up a vm with pme via the file backend | |
| 14:27:15 | sean-k-mooney | i could deploy devstask inside and test "hardware" pmem supprot | |
| 14:27:28 | luyao | sean-k-mooney: CI is in good process, they are testing the tempest plugin | |
| 14:27:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Make scheduler filters customizable https://review.opendev.org/679745 | |
| 14:27:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 14:27:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | |
| 14:27:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 14:27:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | |
| 14:27:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 14:27:39 | sean-k-mooney | are we adding file backed pmem support this cycle to nova | |
| 14:27:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | |
| 14:27:48 | sean-k-mooney | or jsut hardware pmem | |
| 14:28:34 | efried | luyao: I'm pretty sure there's no reasonable way to get an outsider access to the internal lab. At least not in a reasonable time frame to make it worthwhile. | |
| 14:28:40 | efried | stephenfin: those two reviews are done | |
| 14:29:01 | stephenfin | ta | |
| 14:29:29 | efried | stephenfin: unused model fields are okay as long as we've got the series baked enough to be pretty sure we're going to use them. | |
| 14:29:58 | efried | stephenfin: We had a bit of a miscue on vpmem and merged a field we ended up redesigning out, so had to do an ugly migration patch. | |
| 14:29:58 | artom | efried, FWIW, given the amount of (very justifiable, I should say) grief I got over NUMA LM CI, I'd feel betrayed if something else got merged without seeing it run in public CI somewhere | |
| 14:30:59 | stephenfin | bauzas: Have respun https://review.opendev.org/#/c/679745/ | |
| 14:31:13 | efried | artom: heard and understood. Though arguably vpmem (niche specialized hardware) is different from NUMA+LM. | |
| 14:33:09 | artom | efried, that's also a good point - still, releasing "we support vpmem now!" only to find out it doesn't work because the dev testing it missed a thing (no shade on luyao or you) doesn't look good :) | |
| 14:33:31 | artom | Anyways, just I'm just grumpy. We all know there's a pressing lack of specialized hardware in CI | |
| 14:33:46 | artom | And with things going more telco (at least from an RH customers POV), that's only going to get worse | |
| 14:34:08 | efried | agreed on all counts. | |
| 14:34:21 | luyao | sean-k-mooney: We don't support normal file backend. So there will be so many details. You can use a part of memory to simulate a pmem device, and specific kernel , libvirt and qemu are needed, so it may take long time to setup. | |
| 14:34:40 | sean-k-mooney | artom: if we support file backed pmem in nova we can emulate the specialised hardware in ci using file backed pemem on the node pool vms | |
| 14:34:47 | artom | I was in an interesting convo with an RH consultant posted at Bell, our local massive telco | |
| 14:34:58 | artom | There was some interest on their part to become more than a pure consumer | |