| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-20 | |||
| 13:59:06 | bauzas | (UTC) | |
| 13:59:28 | stephenfin | bauzas: just another PTG so | |
| 13:59:29 | stephenfin | :) | |
| 13:59:36 | belmoreira | bauzas 3pm utc works for me. thanks | |
| 14:00:41 | bauzas | just hope melwitt can be present at this time | |
| 14:00:56 | bauzas | belmoreira: ack, will flip the sessions | |
| 14:11:49 | bauzas | stephenfin: we'll do neutron things until 3pm UTC | |
| 15:07:48 | bauzas | dansmith: are you available ? belmoreira is discussing about moving instances between projects and we're talking of os-chown | |
| 15:08:12 | dansmith | bauzas: not right this moment, but I can try to join soon | |
| 15:08:19 | bauzas | dansmith: cool, appreciated. | |
| 15:11:24 | dansmith | bauzas: actually, this next topic is something I have to stay for, so it'll be a bit yet | |
| 15:11:37 | bauzas | ok | |
| 15:11:47 | bauzas | no worries | |
| 15:14:35 | dansmith | bauzas: I'm here for 15 mins | |
| 15:14:49 | bauzas | ta | |
| 15:17:34 | dansmith | which line is the vnc proxy thing? | |
| 15:17:48 | stephenfin | dansmith: 394 | |
| 15:18:02 | dansmith | ah, in the "after 3pm" thing | |
| 15:18:32 | stephenfin | yeah, I've lost track of how we're picking stuff/what we've covered :) | |
| 15:58:06 | bauzas | side note: Jabra Evolve 2 65 are terrible headsets for a day long wearing them | |
| 15:58:29 | bauzas | my ears are pretty done with it | |
| 15:58:59 | bauzas | stephenfin: yeah, we cherry-picked topics | |
| 15:59:24 | bauzas | I don't wanna move topics too much as it's also confusing | |
| 17:26:48 | bauzas | brinzhang: when you're up, tell me when you want to discuss the cyborg suspend/resume spec ? we can do this either this Thursday 1pm or Friday 1pm if that suits your TZ | |
| 17:27:48 | bauzas | for the moment, placing it at the top of the discussions we have left so it would be discuss this at 1pm, but we can punt it for later | |
| 17:28:40 | bauzas | to nova folks, I flipped topics so the agenda should reflect the first topics to discuss, starting at L357 | |
| #openstack-nova - 2021-10-21 | |||
| 00:22:54 | brinzhang | bauzas:hi, about the suspend/resume an accelerator instance feature, there is no spec, before we completed rebuild/evacuate, shelve/unshelve, I just want to continue | |
| 00:24:33 | brinzhang | bauzas: today, I am not sure that time(1pm) which is ok for me, but if possiable, I think I can, thanks | |
| 08:29:32 | bauzas | brinzhang: OK, then we'll start with your topic, but in case you're not around, I'll punt it until tomorrow | |
| 08:30:39 | gibi | morning Nova | |
| 10:09:43 | sean-k-mooney | for what its worth i think it make sense to proceed with suspend/resume for cyborg without a spec in the same way i hope to do the same for vdpa | |
| 10:10:29 | sean-k-mooney | its just the completion of the existing work which does nto really require much design work so i think specless blueprint and code review should be suffincet in both cases | |
| 10:18:11 | gibi | I'm ok with a specless bp too | |
| 12:11:50 | bauzas | reminder : nova sessions start in 50 mins on https://www.openstack.org/ptg/rooms/newton | |
| 12:15:46 | bauzas | sean-k-mooney: gibi: yup I agree with both of you but given brinzhang asked to discuss for it, let's wait until we do | |
| 12:31:44 | gibi | sure | |
| 12:37:33 | tbarron | bauzas: do you have a window (however large) when you are targeting the virtios topic? manila folks plan to discuss at 23:15 this morning (I know Lee is out of course) | |
| 12:37:40 | frickler | with virt_type=qemu on qemu>=5.2 (bullseye, jammy, centos-stream-8) instances (i.e. rss of the qemu process) seem to use 3x as much ram as the flavor allocates, has anyone seen this behaviour before? it is causing failures in devstack CI with things just going OOM | |
| 12:39:06 | tbarron | bauzas: 13:15 :) | |
| 12:39:30 | tbarron | which isn't morning for lots of people, I know | |
| 12:43:46 | sean-k-mooney | frickler: no but i do know that in qemu 6.0 the -m attibute is nolonger required when using mem-backing args | |
| 12:44:09 | sean-k-mooney | frickler: so in 5.2 where we were seeing libvirt use both way that was still required | |
| 12:45:35 | sean-k-mooney | frickler: actully that chang is in 6.1 https://wiki.qemu.org/ChangeLog/6.1#Memory_backends | |
| 12:46:10 | sean-k-mooney | frickler: so the qemu command line generated form the libvirt xml that i looked at before i think was correct so this sound like a qemu bug | |
| 12:57:31 | bauzas | tbarron: others, sorry was taking a bit of timeoff before we start | |
| 12:57:41 | frickler | sean-k-mooney: do you have some contact at qemu to push that to? | |
| 12:57:50 | bauzas | tbarron: not sure we'll have time to discuss about the Manila spec today | |
| 12:58:12 | bauzas | tbarron: we can start to discuss it for tomorrow 1pm (or later if you prefer) | |
| 12:58:37 | sean-k-mooney | frickler: good question kashyap: do you know who could help root cause this | |
| 12:59:02 | kashyap | What's the context? /me catches up. (In a meeting now) | |
| 12:59:04 | sean-k-mooney | kashyap: context is that qemu on debian seams to be using 2-3x the ram that is allocated to the vm | |
| 12:59:13 | kashyap | What version of QEMU? | |
| 12:59:16 | sean-k-mooney | 5.2 | |
| 12:59:32 | kashyap | Fairly recent (Dec 2020) | |
| 12:59:52 | sean-k-mooney | kashyap: we are seeign this result in OOM issues in the ci hence frickler concern | |
| 12:59:56 | tbarron | bauzas: *bit* is right :D. 1300UTC tomorrow would be great, manila sessions don't start till 1400. | |
| 13:00:26 | stephenfin | I'll be 20/30 minutes late to the PTG sessions again today | |
| 13:00:46 | kashyap | frickler: Very odd; is this something sudden? I can check w/ the upstream, but before that, I'd need a bug (with proper details) | |
| 13:01:05 | kashyap | frickler: Oh, wait | |
| 13:01:26 | kashyap | frickler: virt_type=qemu is already deadly slow: as you might know it's not using any hardware acceleration | |
| 13:02:24 | kashyap | frickler: What is the host OS? (I'm assuming this is a nested env; i.e. the "host" is a level-1 guest. And the baremetal is some cloud-vendor provided) | |
| 13:02:39 | bauzas | nova sessions start by now, people can join with https://www.openstack.org/ptg/rooms/newton | |
| 13:02:57 | kashyap | frickler: I'd need these, to start with: QEMU version and the complete QEMU command-line of the guest. You'll find it here: /var/log/libvirt/qemu/instance-yyyyyyyy.log | |
| 13:04:54 | bauzas | tbarron: okay then let's discuss this at 1300UTC tomorrow | |
| 13:05:30 | bauzas | sean-k-mooney: joing | |
| 13:05:33 | bauzas | joining ? | |
| 13:06:30 | sean-k-mooney | yes sorry be right there | |
| 13:07:07 | bauzas | stephenfin: are you able to join us ? | |
| 13:07:21 | bauzas | stephenfin: we would discuss about your topics next | |
| 13:24:39 | stephenfin | bauzas: in 10 minutes | |
| 14:16:17 | bauzas | dansmith: fyi, we're discussing the healthcheck proposal (as you provided comments in there) | |
| 14:18:01 | dansmith | bauzas: arg, okay, in the tc right now | |
| 14:18:57 | bauzas | dansmith: I guess we collected your thoughts | |
| 14:20:08 | dansmith | bauzas: I can be there in a few I think | |
| 14:20:32 | bauzas | ok we'll continue to discuss the design before you join | |
| 14:32:10 | bauzas | dansmith: tl;dr: | |
| 14:32:32 | bauzas | 1/ discuss a library like oslo.healthcheck providing the HT knobs | |
| 14:32:46 | bauzas | 2/ having a global cache object knowing the state of the service | |
| 14:33:09 | bauzas | 3/ (I can't remember it, sean-k-mooney ? ) | |
| 14:33:27 | bauzas | oh, the API thing | |
| 14:34:59 | sean-k-mooney | 3 was allow /healtcheck at the api to call the helath check for that api process | |
| 14:35:17 | bauzas | yeah, I remembered it :) | |
| 14:47:45 | frickler | kashyap: https://paste.opendev.org/show/810150/ we know that qemu is slow, but we cannot use kvm reliably in CI. happening for different host OSes (i.e. what devstack runs in, no idea what host OS is in place on the cloud side) | |
| 14:51:04 | kashyap | Exactly; that's the peril here w/ nested probs ... not knowing the host OS :-( | |
| 14:59:41 | clarkb | it isn't jsut a reliability problem some of our providers don't expose nested virt at all. So it is a two tier problem. In some places we cannot use kvm. And in others we may but with the risk of poor reliability | |
| 15:00:19 | clarkb | kashyap: on the x86_64-v2 thing (^ related because qemu) does qemu not provide a predefined compatible cpu? | |
| 15:00:36 | kashyap | clarkb: Hiya | |
| 15:00:52 | clarkb | seems like it should? I remember when we first set up nova live migration testing one of the things I tried was defining a custom cpu since we have heterogenous resources in the clouds we use (and have no way to request consistent cpus) | |
| 15:01:02 | clarkb | but that didn't work at all whcih is how we ened up using he qemu64 model | |
| 15:01:07 | kashyap | Yeah, I know :-( | |
| 15:01:39 | clarkb | I want to say at the time we identified bugs in nova and friends and they were getting fixed so it is possible this is no longer an issue today but I think that if centos is saying that you need this minimum cpu now it is reasonable for there to be that cpu predefined in qemu | |
| 15:01:46 | kashyap | clarkb: Sadly, QEMU does not provide a model that will work on (a) TCG, i.e. plain emulation, *and* KVM; and (b) that works on Intel and AMD | |
| 15:01:52 | kashyap | clarkb: But! | |
| 15:02:01 | kashyap | There is an option I just tested. And this works: | |
| 15:02:15 | kashyap | clarkb: Using "Nehalem" satisfies both the above conditions. | |
| 15:02:21 | kashyap | And it works on both Intel and AMD. | |
| 15:02:34 | kashyap | Nehalem is the oldest comatible model that works with the x86_64-v2 | |
| 15:03:00 | clarkb | got it so there is a predefined model that addresses things. That is great news. I guess we should consider defaulting devstack to that then? | |
| 15:03:06 | kashyap | Yes! Indeed | |
| 15:03:17 | clarkb | I can try pushing a devstack change later today that does that | |