| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 14:15:05 | luyao | dansmith: Could you look at https://review.opendev.org/#/c/678448/9 and https://review.opendev.org/#/c/678447/https://review.opendev.org/635229. Thanks. | |
| 14:15:12 | donnyd | sean-k-mooney: but if I lose something its not like losing a customers long running instance I can't get back | |
| 14:15:22 | stephenfin | bauzas: Damn, I meant https://review.opendev.org/#/c/671800/ | |
| 14:15:24 | sean-k-mooney | donnyd: ya that is ture | |
| 14:15:47 | donnyd | umm.. no.. the controller is setup a little bit more robust | |
| 14:15:52 | sean-k-mooney | donnyd: the could model however is to keep data in sepereate data volumes that are attached to a stateles instance. | |
| 14:16:02 | bauzas | stephenfin: oh this one is too hairy for now, I'll look at it tomorrow morning :) | |
| 14:16:05 | stephenfin | If you modify the address bar in Firefox, it doesn't change as you navigate pages until you refresh the page | |
| 14:16:10 | luyao | dansmith: sorry, they are https://review.opendev.org/#/c/678447/ and https://review.opendev.org/#/c/678448/9 | |
| 14:16:12 | sean-k-mooney | *cloud model | |
| 14:16:21 | stephenfin | So you end up copy-pasting the same thing multiple times | |
| 14:16:25 | stephenfin | bauzas: Heh, cool | |
| 14:17:21 | donnyd | I need to get HA controllers running, however I have to make a choice between # of instances to run and potential for downtime... Heat is my battle here | |
| 14:17:36 | donnyd | I use forced air - NO A/C | |
| 14:17:50 | stephenfin | efried: I know you don't like adding config options without any users, but what about model fields? I feel I should split https://review.opendev.org/#/c/671800/ up so people can focus on the functional changes rather than the noise introduced by the additional field | |
| 14:17:53 | bauzas | oh shit, dadtaxi, yeah | |
| 14:18:34 | bauzas | stephenfin: I'm not worried by the size of the change in question | |
| 14:18:55 | bauzas | anyway, bbiab | |
| 14:19:06 | donnyd | If you think about it - I can push a button and redeploy the whole control plane.. so is two hours of downtime once a year or so worth losing 30 instances in CI | |
| 14:19:34 | donnyd | I made my bets on short downtime being more acceptable than losing CI resources | |
| 14:19:43 | sean-k-mooney | donnyd: well contoler can run vms too | |
| 14:19:51 | sean-k-mooney | donnyd: what are you using to deploy | |
| 14:20:04 | donnyd | Its custom for my infra | |
| 14:20:10 | sean-k-mooney | ah ok | |
| 14:20:30 | donnyd | I was using tripleo - but I couldn't get rabbit to act right | |
| 14:21:01 | sean-k-mooney | donnyd: ya i dont like how tripleo deplfoys rabbit | |
| 14:21:26 | sean-k-mooney | well i dont like how tripleo deploys things in general | |
| 14:21:42 | donnyd | well from my perspective there is a performance hit from running in containers - small but noticeable at my scale | |
| 14:21:43 | sean-k-mooney | but that is another mater | |
| 14:22:01 | donnyd | https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&refresh=30s | |
| 14:22:03 | sean-k-mooney | really i have nver notticed any performac ehit | |
| 14:22:16 | donnyd | You can see I don't have much left on my controller for VM's | |
| 14:22:34 | sean-k-mooney | oh you mean from a ram proint of view | |
| 14:23:39 | luyao | efried: Would you like to test the patches about virtual persistent memory support? I have tested it on real hardware, and I'm preparing another OpenStack with pmems, so you can test on it if you want. I can send you how to access those servers after I set up the env. | |
| 14:24:08 | donnyd | Yea, memory is short in the controller | |
| 14:24:37 | efried | luyao: It would be nice if *someone* could do that testing in addition to you, just to get another pair of eyes on it. But I'm not sure I have the time to do it myself. | |
| 14:24:38 | sean-k-mooney | donnyd: thats becasue heat is partly because of heat | |
| 14:24:39 | donnyd | only has 96G | |
| 14:24:47 | efried | luyao: sounds like the kind of thing sean-k-mooney would enjoy :P | |
| 14:25:22 | sean-k-mooney | efried: i dont have pmem servers in my hallway | |
| 14:25:25 | efried | luyao: But I assume it requires specialized hardware, so needs to be inside Intel's firewall. | |
| 14:25:26 | efried | yeah. | |
| 14:25:47 | sean-k-mooney | so i would have to use qemu to fake pmem in a vm and then thest with nested virt | |
| 14:25:52 | donnyd | Due to the nature of forced air cooling, I can only do so much in the way of servers that can be turned on at a single point in time | |
| 14:26:02 | luyao | efried: yes, I will check if others can access the servers | |
| 14:26:19 | sean-k-mooney | efried: i could proably set that up but not until after i test all the other stuff on my list | |
| 14:26:40 | luyao | sean-k-mooney: you can use a normal file as a fake pmem | |
| 14:26:41 | sean-k-mooney | luyao: any progress on the intel ci for pmem | |
| 14:26:58 | sean-k-mooney | luyao: yes so if i spin up a vm with pme via the file backend | |
| 14:27:15 | sean-k-mooney | i could deploy devstask inside and test "hardware" pmem supprot | |
| 14:27:28 | luyao | sean-k-mooney: CI is in good process, they are testing the tempest plugin | |
| 14:27:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Make scheduler filters customizable https://review.opendev.org/679745 | |
| 14:27:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 14:27:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | |
| 14:27:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 14:27:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | |
| 14:27:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 14:27:39 | sean-k-mooney | are we adding file backed pmem support this cycle to nova | |
| 14:27:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | |
| 14:27:48 | sean-k-mooney | or jsut hardware pmem | |
| 14:28:34 | efried | luyao: I'm pretty sure there's no reasonable way to get an outsider access to the internal lab. At least not in a reasonable time frame to make it worthwhile. | |
| 14:28:40 | efried | stephenfin: those two reviews are done | |
| 14:29:01 | stephenfin | ta | |
| 14:29:29 | efried | stephenfin: unused model fields are okay as long as we've got the series baked enough to be pretty sure we're going to use them. | |
| 14:29:58 | efried | stephenfin: We had a bit of a miscue on vpmem and merged a field we ended up redesigning out, so had to do an ugly migration patch. | |
| 14:29:58 | artom | efried, FWIW, given the amount of (very justifiable, I should say) grief I got over NUMA LM CI, I'd feel betrayed if something else got merged without seeing it run in public CI somewhere | |
| 14:30:59 | stephenfin | bauzas: Have respun https://review.opendev.org/#/c/679745/ | |
| 14:31:13 | efried | artom: heard and understood. Though arguably vpmem (niche specialized hardware) is different from NUMA+LM. | |
| 14:33:09 | artom | efried, that's also a good point - still, releasing "we support vpmem now!" only to find out it doesn't work because the dev testing it missed a thing (no shade on luyao or you) doesn't look good :) | |
| 14:33:31 | artom | Anyways, just I'm just grumpy. We all know there's a pressing lack of specialized hardware in CI | |
| 14:33:46 | artom | And with things going more telco (at least from an RH customers POV), that's only going to get worse | |
| 14:34:08 | efried | agreed on all counts. | |
| 14:34:21 | luyao | sean-k-mooney: We don't support normal file backend. So there will be so many details. You can use a part of memory to simulate a pmem device, and specific kernel , libvirt and qemu are needed, so it may take long time to setup. | |
| 14:34:40 | sean-k-mooney | artom: if we support file backed pmem in nova we can emulate the specialised hardware in ci using file backed pemem on the node pool vms | |
| 14:34:47 | artom | I was in an interesting convo with an RH consultant posted at Bell, our local massive telco | |
| 14:34:58 | artom | There was some interest on their part to become more than a pure consumer | |
| 14:35:22 | sean-k-mooney | luyao: ok | |
| 14:35:32 | artom | This is all very very preliminary, not even sure I should be talking about it, but I'd love it if that kinds of thing a. actually went somewhere and b. was more widespread in the telco world | |
| 14:36:00 | donnyd | sean-k-mooney: couldn't we attach like an ephemeral disk to the instance and use that as "fake pmem" | |
| 14:36:23 | sean-k-mooney | donnyd: no at least not the way you suggest | |
| 14:36:34 | donnyd | How about block storage? | |
| 14:36:45 | sean-k-mooney | donnyd: pmem device are exposed as dimms to the guest kernel | |
| 14:37:10 | donnyd | yea I know nothing about how it all works - /me goes to the corner to be quiet | |
| 14:37:16 | sean-k-mooney | so attaching block device ephemer or volumes wont present it to the guest correclty | |
| 14:37:17 | aspiers | kashyap: you around? | |
| 14:37:23 | kashyap | aspiers: Yep | |
| 14:37:36 | aspiers | kashyap: I've split out _guest_machine_type_for_sev(), but ... | |
| 14:37:36 | sean-k-mooney | donnyd: what we can do is have wemu use a file on the host disk to emulate pmem | |
| 14:37:43 | kashyap | (Although, I need to head out in an hour) | |
| 14:37:44 | sean-k-mooney | donnyd: libvirt/qemu support that | |
| 14:37:50 | aspiers | kashyap: now it occurs to me that this and the caller probably both belong in libvirt.utils | |
| 14:37:51 | sean-k-mooney | donnyd: nova does not currently | |
| 14:37:57 | kashyap | aspiers: :-) | |
| 14:38:05 | aspiers | does that make sense? | |
| 14:38:11 | kashyap | aspiers: I was also thinking something like that, sitting on _guest_machine_type_for_secure_boot() :D | |
| 14:38:12 | aspiers | all the rest of the machine type calculation is already in utils | |
| 14:38:15 | sean-k-mooney | donnyd: if we had that support we would jsut set some flave extra specls liek we did for numa sutff | |
| 14:38:37 | aspiers | kashyap: hell, we could even have machine_types.py | |
| 14:38:40 | kashyap | aspiers: Yeah, I think it does make sense -- keeps all the machine type-related context in the same place. | |
| 14:38:59 | aspiers | utils.py is a random mix of crap anyway | |