| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-21 | |||
| 00:22:54 | brinzhang | bauzas:hi, about the suspend/resume an accelerator instance feature, there is no spec, before we completed rebuild/evacuate, shelve/unshelve, I just want to continue | |
| 00:24:33 | brinzhang | bauzas: today, I am not sure that time(1pm) which is ok for me, but if possiable, I think I can, thanks | |
| 08:29:32 | bauzas | brinzhang: OK, then we'll start with your topic, but in case you're not around, I'll punt it until tomorrow | |
| 08:30:39 | gibi | morning Nova | |
| 10:09:43 | sean-k-mooney | for what its worth i think it make sense to proceed with suspend/resume for cyborg without a spec in the same way i hope to do the same for vdpa | |
| 10:10:29 | sean-k-mooney | its just the completion of the existing work which does nto really require much design work so i think specless blueprint and code review should be suffincet in both cases | |
| 10:18:11 | gibi | I'm ok with a specless bp too | |
| 12:11:50 | bauzas | reminder : nova sessions start in 50 mins on https://www.openstack.org/ptg/rooms/newton | |
| 12:15:46 | bauzas | sean-k-mooney: gibi: yup I agree with both of you but given brinzhang asked to discuss for it, let's wait until we do | |
| 12:31:44 | gibi | sure | |
| 12:37:33 | tbarron | bauzas: do you have a window (however large) when you are targeting the virtios topic? manila folks plan to discuss at 23:15 this morning (I know Lee is out of course) | |
| 12:37:40 | frickler | with virt_type=qemu on qemu>=5.2 (bullseye, jammy, centos-stream-8) instances (i.e. rss of the qemu process) seem to use 3x as much ram as the flavor allocates, has anyone seen this behaviour before? it is causing failures in devstack CI with things just going OOM | |
| 12:39:06 | tbarron | bauzas: 13:15 :) | |
| 12:39:30 | tbarron | which isn't morning for lots of people, I know | |
| 12:43:46 | sean-k-mooney | frickler: no but i do know that in qemu 6.0 the -m attibute is nolonger required when using mem-backing args | |
| 12:44:09 | sean-k-mooney | frickler: so in 5.2 where we were seeing libvirt use both way that was still required | |
| 12:45:35 | sean-k-mooney | frickler: actully that chang is in 6.1 https://wiki.qemu.org/ChangeLog/6.1#Memory_backends | |
| 12:46:10 | sean-k-mooney | frickler: so the qemu command line generated form the libvirt xml that i looked at before i think was correct so this sound like a qemu bug | |
| 12:57:31 | bauzas | tbarron: others, sorry was taking a bit of timeoff before we start | |
| 12:57:41 | frickler | sean-k-mooney: do you have some contact at qemu to push that to? | |
| 12:57:50 | bauzas | tbarron: not sure we'll have time to discuss about the Manila spec today | |
| 12:58:12 | bauzas | tbarron: we can start to discuss it for tomorrow 1pm (or later if you prefer) | |
| 12:58:37 | sean-k-mooney | frickler: good question kashyap: do you know who could help root cause this | |
| 12:59:02 | kashyap | What's the context? /me catches up. (In a meeting now) | |
| 12:59:04 | sean-k-mooney | kashyap: context is that qemu on debian seams to be using 2-3x the ram that is allocated to the vm | |
| 12:59:13 | kashyap | What version of QEMU? | |
| 12:59:16 | sean-k-mooney | 5.2 | |
| 12:59:32 | kashyap | Fairly recent (Dec 2020) | |
| 12:59:52 | sean-k-mooney | kashyap: we are seeign this result in OOM issues in the ci hence frickler concern | |
| 12:59:56 | tbarron | bauzas: *bit* is right :D. 1300UTC tomorrow would be great, manila sessions don't start till 1400. | |
| 13:00:26 | stephenfin | I'll be 20/30 minutes late to the PTG sessions again today | |
| 13:00:46 | kashyap | frickler: Very odd; is this something sudden? I can check w/ the upstream, but before that, I'd need a bug (with proper details) | |
| 13:01:05 | kashyap | frickler: Oh, wait | |
| 13:01:26 | kashyap | frickler: virt_type=qemu is already deadly slow: as you might know it's not using any hardware acceleration | |
| 13:02:24 | kashyap | frickler: What is the host OS? (I'm assuming this is a nested env; i.e. the "host" is a level-1 guest. And the baremetal is some cloud-vendor provided) | |
| 13:02:39 | bauzas | nova sessions start by now, people can join with https://www.openstack.org/ptg/rooms/newton | |
| 13:02:57 | kashyap | frickler: I'd need these, to start with: QEMU version and the complete QEMU command-line of the guest. You'll find it here: /var/log/libvirt/qemu/instance-yyyyyyyy.log | |
| 13:04:54 | bauzas | tbarron: okay then let's discuss this at 1300UTC tomorrow | |
| 13:05:30 | bauzas | sean-k-mooney: joing | |
| 13:05:33 | bauzas | joining ? | |
| 13:06:30 | sean-k-mooney | yes sorry be right there | |
| 13:07:07 | bauzas | stephenfin: are you able to join us ? | |
| 13:07:21 | bauzas | stephenfin: we would discuss about your topics next | |
| 13:24:39 | stephenfin | bauzas: in 10 minutes | |
| 14:16:17 | bauzas | dansmith: fyi, we're discussing the healthcheck proposal (as you provided comments in there) | |
| 14:18:01 | dansmith | bauzas: arg, okay, in the tc right now | |
| 14:18:57 | bauzas | dansmith: I guess we collected your thoughts | |
| 14:20:08 | dansmith | bauzas: I can be there in a few I think | |
| 14:20:32 | bauzas | ok we'll continue to discuss the design before you join | |
| 14:32:10 | bauzas | dansmith: tl;dr: | |
| 14:32:32 | bauzas | 1/ discuss a library like oslo.healthcheck providing the HT knobs | |
| 14:32:46 | bauzas | 2/ having a global cache object knowing the state of the service | |
| 14:33:09 | bauzas | 3/ (I can't remember it, sean-k-mooney ? ) | |
| 14:33:27 | bauzas | oh, the API thing | |
| 14:34:59 | sean-k-mooney | 3 was allow /healtcheck at the api to call the helath check for that api process | |
| 14:35:17 | bauzas | yeah, I remembered it :) | |
| 14:47:45 | frickler | kashyap: https://paste.opendev.org/show/810150/ we know that qemu is slow, but we cannot use kvm reliably in CI. happening for different host OSes (i.e. what devstack runs in, no idea what host OS is in place on the cloud side) | |
| 14:51:04 | kashyap | Exactly; that's the peril here w/ nested probs ... not knowing the host OS :-( | |
| 14:59:41 | clarkb | it isn't jsut a reliability problem some of our providers don't expose nested virt at all. So it is a two tier problem. In some places we cannot use kvm. And in others we may but with the risk of poor reliability | |
| 15:00:19 | clarkb | kashyap: on the x86_64-v2 thing (^ related because qemu) does qemu not provide a predefined compatible cpu? | |
| 15:00:36 | kashyap | clarkb: Hiya | |
| 15:00:52 | clarkb | seems like it should? I remember when we first set up nova live migration testing one of the things I tried was defining a custom cpu since we have heterogenous resources in the clouds we use (and have no way to request consistent cpus) | |
| 15:01:02 | clarkb | but that didn't work at all whcih is how we ened up using he qemu64 model | |
| 15:01:07 | kashyap | Yeah, I know :-( | |
| 15:01:39 | clarkb | I want to say at the time we identified bugs in nova and friends and they were getting fixed so it is possible this is no longer an issue today but I think that if centos is saying that you need this minimum cpu now it is reasonable for there to be that cpu predefined in qemu | |
| 15:01:46 | kashyap | clarkb: Sadly, QEMU does not provide a model that will work on (a) TCG, i.e. plain emulation, *and* KVM; and (b) that works on Intel and AMD | |
| 15:01:52 | kashyap | clarkb: But! | |
| 15:02:01 | kashyap | There is an option I just tested. And this works: | |
| 15:02:15 | kashyap | clarkb: Using "Nehalem" satisfies both the above conditions. | |
| 15:02:21 | kashyap | And it works on both Intel and AMD. | |
| 15:02:34 | kashyap | Nehalem is the oldest comatible model that works with the x86_64-v2 | |
| 15:03:00 | clarkb | got it so there is a predefined model that addresses things. That is great news. I guess we should consider defaulting devstack to that then? | |
| 15:03:06 | kashyap | Yes! Indeed | |
| 15:03:17 | clarkb | I can try pushing a devstack change later today that does that | |
| 15:03:21 | kashyap | clarkb: I was going to write an email to openstack-discuss list with this recommendation | |
| 15:03:27 | clarkb | kashyap: that would be great, thank you | |
| 15:03:42 | kashyap | clarkb: I'm drafting as we speak. So your timing couldn't be perfect :-) | |
| 15:04:05 | kashyap | s/be perfect/be more perfect/ | |
| 15:04:34 | kashyap | clarkb: The only assumption, which I think you'll agree is reasonable is: | |
| 15:04:57 | kashyap | clarkb: ... I imagine any hardware older than Nehalem is not capable of running OpenStack. | |
| 15:05:03 | kashyap | Say "yes", please :D | |
| 15:05:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Prevent leaked greenlets to interact with later tests https://review.opendev.org/c/openstack/nova/+/815017 | |
| 15:05:56 | clarkb | kashyap: well thats the next thing we need to figure out because we get virtual resources with their own cpu models being managed. Currently we can't boot fedora-34 in half of our clouds and are wondering if this is the same issue | |
| 15:06:18 | clarkb | kashyap: it isn't just real hardware to consider but also virtual hardware | |
| 15:06:20 | kashyap | I don't know of F34 switched to the -v2, lemme check | |
| 15:07:41 | kashyap | clarkb: Wait, pretty sure F34 is switched to it too | |
| 15:08:01 | kashyap | How do I know? By inference :D RHEL9 is based on F33/F34, so I presume it did too | |
| 15:08:04 | kashyap | I just need to double-check | |
| 15:08:44 | clarkb | kashyap: ya and half our clouds can't boot it | |
| 15:09:01 | clarkb | so there is possibility that we can't actually switch to Nehalem, but we can make a change and test it | |
| 15:10:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: Prevent leaked greenlets to interact with later tests https://review.opendev.org/c/openstack/nova/+/815017 | |
| 15:19:08 | kashyap | clarkb: Yeah, I'm actually quickly pushing a test-only patch to DevStack to see where it fails. If you haven't already done so | |
| 15:19:45 | clarkb | kashyap: remote: https://review.opendev.org/c/openstack/devstack/+/815020 Use Nehalem CPU model by default I just pushed that to have the CI system check things. But thats because I'm in meetings and don't have a current devstack install anywhere to check with | |
| 15:20:09 | kashyap | Ah-ha, thank you | |
| 15:20:18 | kashyap | clarkb: I don't have it either right now. But: | |
| 15:20:42 | kashyap | clarkb: I have tested today outside of DevStack: Nehalem model works with virt_type=qemu *and* with Intel and AMD | |
| 15:21:24 | clarkb | kashyap: ok, ya I suspect the question becomes whether or not the CPUs we get in the host VMs are new enough to support Nehalem | |
| 15:22:01 | clarkb | since they too get virtual capabilities to enable live mgiration in clouds and if they were too conservative we'll have problems. The good news if there is any is that they won't be able to boot centos/rhel 9 either so clouds seem likely to update them | |
| 15:22:19 | clarkb | we can recheck 815020 if it generally works to get it to run across a bunch of clouds and double check that | |
| 15:24:38 | kashyap | clarkb: Nehalem is nearly 13 years old ... I hope there are no such hosts : | |