| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-21 | |||
| 14:32:10 | bauzas | dansmith: tl;dr: | |
| 14:32:32 | bauzas | 1/ discuss a library like oslo.healthcheck providing the HT knobs | |
| 14:32:46 | bauzas | 2/ having a global cache object knowing the state of the service | |
| 14:33:09 | bauzas | 3/ (I can't remember it, sean-k-mooney ? ) | |
| 14:33:27 | bauzas | oh, the API thing | |
| 14:34:59 | sean-k-mooney | 3 was allow /healtcheck at the api to call the helath check for that api process | |
| 14:35:17 | bauzas | yeah, I remembered it :) | |
| 14:47:45 | frickler | kashyap: https://paste.opendev.org/show/810150/ we know that qemu is slow, but we cannot use kvm reliably in CI. happening for different host OSes (i.e. what devstack runs in, no idea what host OS is in place on the cloud side) | |
| 14:51:04 | kashyap | Exactly; that's the peril here w/ nested probs ... not knowing the host OS :-( | |
| 14:59:41 | clarkb | it isn't jsut a reliability problem some of our providers don't expose nested virt at all. So it is a two tier problem. In some places we cannot use kvm. And in others we may but with the risk of poor reliability | |
| 15:00:19 | clarkb | kashyap: on the x86_64-v2 thing (^ related because qemu) does qemu not provide a predefined compatible cpu? | |
| 15:00:36 | kashyap | clarkb: Hiya | |
| 15:00:52 | clarkb | seems like it should? I remember when we first set up nova live migration testing one of the things I tried was defining a custom cpu since we have heterogenous resources in the clouds we use (and have no way to request consistent cpus) | |
| 15:01:02 | clarkb | but that didn't work at all whcih is how we ened up using he qemu64 model | |
| 15:01:07 | kashyap | Yeah, I know :-( | |
| 15:01:39 | clarkb | I want to say at the time we identified bugs in nova and friends and they were getting fixed so it is possible this is no longer an issue today but I think that if centos is saying that you need this minimum cpu now it is reasonable for there to be that cpu predefined in qemu | |
| 15:01:46 | kashyap | clarkb: Sadly, QEMU does not provide a model that will work on (a) TCG, i.e. plain emulation, *and* KVM; and (b) that works on Intel and AMD | |
| 15:01:52 | kashyap | clarkb: But! | |
| 15:02:01 | kashyap | There is an option I just tested. And this works: | |
| 15:02:15 | kashyap | clarkb: Using "Nehalem" satisfies both the above conditions. | |
| 15:02:21 | kashyap | And it works on both Intel and AMD. | |
| 15:02:34 | kashyap | Nehalem is the oldest comatible model that works with the x86_64-v2 | |
| 15:03:00 | clarkb | got it so there is a predefined model that addresses things. That is great news. I guess we should consider defaulting devstack to that then? | |
| 15:03:06 | kashyap | Yes! Indeed | |
| 15:03:17 | clarkb | I can try pushing a devstack change later today that does that | |
| 15:03:21 | kashyap | clarkb: I was going to write an email to openstack-discuss list with this recommendation | |
| 15:03:27 | clarkb | kashyap: that would be great, thank you | |
| 15:03:42 | kashyap | clarkb: I'm drafting as we speak. So your timing couldn't be perfect :-) | |
| 15:04:05 | kashyap | s/be perfect/be more perfect/ | |
| 15:04:34 | kashyap | clarkb: The only assumption, which I think you'll agree is reasonable is: | |
| 15:04:57 | kashyap | clarkb: ... I imagine any hardware older than Nehalem is not capable of running OpenStack. | |
| 15:05:03 | kashyap | Say "yes", please :D | |
| 15:05:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Prevent leaked greenlets to interact with later tests https://review.opendev.org/c/openstack/nova/+/815017 | |
| 15:05:56 | clarkb | kashyap: well thats the next thing we need to figure out because we get virtual resources with their own cpu models being managed. Currently we can't boot fedora-34 in half of our clouds and are wondering if this is the same issue | |
| 15:06:18 | clarkb | kashyap: it isn't just real hardware to consider but also virtual hardware | |
| 15:06:20 | kashyap | I don't know of F34 switched to the -v2, lemme check | |
| 15:07:41 | kashyap | clarkb: Wait, pretty sure F34 is switched to it too | |
| 15:08:01 | kashyap | How do I know? By inference :D RHEL9 is based on F33/F34, so I presume it did too | |
| 15:08:04 | kashyap | I just need to double-check | |
| 15:08:44 | clarkb | kashyap: ya and half our clouds can't boot it | |
| 15:09:01 | clarkb | so there is possibility that we can't actually switch to Nehalem, but we can make a change and test it | |
| 15:10:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: Prevent leaked greenlets to interact with later tests https://review.opendev.org/c/openstack/nova/+/815017 | |
| 15:19:08 | kashyap | clarkb: Yeah, I'm actually quickly pushing a test-only patch to DevStack to see where it fails. If you haven't already done so | |
| 15:19:45 | clarkb | kashyap: remote: https://review.opendev.org/c/openstack/devstack/+/815020 Use Nehalem CPU model by default I just pushed that to have the CI system check things. But thats because I'm in meetings and don't have a current devstack install anywhere to check with | |
| 15:20:09 | kashyap | Ah-ha, thank you | |
| 15:20:18 | kashyap | clarkb: I don't have it either right now. But: | |
| 15:20:42 | kashyap | clarkb: I have tested today outside of DevStack: Nehalem model works with virt_type=qemu *and* with Intel and AMD | |
| 15:21:24 | clarkb | kashyap: ok, ya I suspect the question becomes whether or not the CPUs we get in the host VMs are new enough to support Nehalem | |
| 15:22:01 | clarkb | since they too get virtual capabilities to enable live mgiration in clouds and if they were too conservative we'll have problems. The good news if there is any is that they won't be able to boot centos/rhel 9 either so clouds seem likely to update them | |
| 15:22:19 | clarkb | we can recheck 815020 if it generally works to get it to run across a bunch of clouds and double check that | |
| 15:24:38 | kashyap | clarkb: Nehalem is nearly 13 years old ... I hope there are no such hosts : | |
| 15:26:00 | clarkb | kashyap: the issue is the clouds provide virtual CPUs with custom models too | |
| 15:26:13 | clarkb | because they want to do live mgiration. And if they were too conservative you have this problem. I don't think the actual CPUs are that old | |
| 15:26:23 | clarkb | but the CPUs we get in our instances may be | |
| 15:27:34 | kashyap | Right; I see what you mean. Even if they are conservative w/ the virtual CPUs, I'd be really surprised (wouldn't be the first time) if they're more conservative than Nehalem | |
| 15:29:50 | kashyap | clarkb: Bad me. I was _wrong_ earlier on my stupid "inference" about Fedora: Fedora *did not* switch the baseline ABI to x86-64-v2 | |
| 15:29:54 | kashyap | So we can rule that out | |
| 15:30:56 | clarkb | ok good, our problems for booting are differen then :) | |
| 15:32:25 | kashyap | Yeah; grr, now to find out the actual cause | |
| 15:32:31 | kashyap | Do you have the boot console log? | |
| 15:33:36 | clarkb | kashyap: not right now. It is something I can probably dig into tomorrow morning if you want to dive into it (have ptg stuff now then real world school meeting stuff after and by the time I'm done you should be enjoying your evening) | |
| 15:34:06 | clarkb | kashyap: also ianw is interested in the fedora 34 issue and his timezone overlap might be better? I guess it depends on how much of a morning person you are :) | |
| 15:34:24 | kashyap | No problem at all | |
| 15:34:52 | kashyap | Go handle what you need to. This can wait. And yes, will check with ianw. I'm in CEST; he's in Australia...so there should be some overlap :) | |
| 15:35:03 | clarkb | sounds good, thanks again! | |
| 15:56:00 | gibi | stephenfin, artom: another way to test is to move heat from novaclient to sdk ;) | |
| 15:56:15 | artom | gibi, you're an evil, evil man | |
| 15:56:30 | stephenfin | gibi: or tempest. That uses novaclient under the hood, I assume rather than subprocessing to the shell? | |
| 15:56:49 | artom | stephenfin, no, tempest reimplements all the client stuff from scratch in Python in-tree | |
| 15:56:57 | stephenfin | oh, so it does | |
| 15:56:57 | artom | By design | |
| 15:58:11 | artom | All I was saying is - we need to make https://opendev.org/openstack/openstacksdk/src/branch/master/.zuul.yaml#L116 work with 2 nodes do we can test how sdk does the client-side stuff for migrations and such | |
| 15:58:13 | sean-k-mooney | by the way when i was mentioning tempst i ment recreeate teh senario test using sdk in the sdk func test | |
| 15:58:17 | tosky | but in general for non testing stuff gibi's proposed solution is correct (assuming the sdk implementation of orchestration is complete enough) | |
| 15:58:17 | gibi | artom: neutron just realized that they have to do the same switch in heat | |
| 15:59:18 | sean-k-mooney | gibi: well heat need to do it rather then neutron but im sure it will be the same people in either case | |
| 15:59:50 | gibi | sean-k-mooney: except if neutron people want heat support for new neutron features where the only client support is in sdk :) | |
| 16:00:10 | gibi | anyhow starting the transition does not seem scarry https://review.opendev.org/c/openstack/heat/+/813425 | |
| 16:00:22 | sean-k-mooney | gibi: but do they want that support :) | |
| 16:00:28 | stephenfin | gibi: artom: I hate to be that guy but we also need to move our internal use of cinderclient, glanceclient and neutronclient to SDK at some point :) | |
| 16:00:36 | stephenfin | I think ironicclient was done a few cycles ago | |
| 16:00:41 | sean-k-mooney | gibi: you are assuming neutron care about heat | |
| 16:01:04 | sean-k-mooney | they proably do but maybe not | |
| 16:01:10 | stephenfin | wait, no, there's still ironicclient. It's just not a mandatory import | |
| 16:01:13 | stephenfin | *requirements | |
| 16:01:28 | artom | stephenfin, yeah... | |
| 16:01:55 | artom | Like, I'm taking part in this debat, but who am I kidding, I won't be the one doing the work - I tried to start, then just ran out of steam | |
| 16:02:29 | gibi | stephenfin: ouch, you are right :) | |
| 16:02:47 | artom | OTOH, what's kind of annoying is that the problem that I'm fixing in https://review.opendev.org/c/openstack/openstacksdk/+/741688 is still a problem. | |
| 16:02:57 | artom | image is *still* optional, when it isn't in the PAI | |
| 16:02:59 | artom | *API | |
| 16:03:50 | artom | And it's *still* missing the kwargs things for all the other params that are in the API | |
| 16:04:02 | stephenfin | artom: I should have looked at that. My bad :( If you got time to rebase it onto the feature/r1 branch (which will be merged into master soon enough) I'll review it in the AM | |
| 16:04:19 | artom | stephenfin, I need to fix the unit test below it and rebase | |
| 16:04:31 | artom | stephenfin, it's just such a slog, and I'm lazy and easily distracted | |
| 16:04:42 | stephenfin | you should try cocaine | |
| 16:04:50 | artom | Speaking from experience? | |
| 16:04:58 | stephenfin | all the bankers I know swear by it | |
| 16:05:29 | artom | Explains the state of the financial system | |
| 16:07:53 | gibi | happy hours already? | |