| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-10-16 | |||
| 22:47:39 | admiyo | kube at the bottom. openstack in the middle. Kube at the top | |
| 22:48:50 | JayF | admiyo: also metal3 is secretly Ironic too :) | |
| 22:49:12 | JayF | admiyo: it's pretty funny how everyone is slowly realizing hardware is a pain and just integrating with us instead of building it out from scratch lol | |
| 22:49:28 | sean-k-mooney[m] | admiyo: yep we support deploy openshift on openstack which is istelf on a differnet openshift which uses ironci via metal3 to deploy the openshift 0’s workers | |
| 22:49:51 | JayF | you must pay architects by the layer over there :P | |
| 22:50:11 | JayF | each time we add an abstraction layer, a sales engineer gets their wings? | |
| 22:50:16 | sean-k-mooney[m] | JayF: the thing that makes me sad about metal3 is it limits the scalablity of ironic because the operator cant keep up | |
| 22:50:19 | JayF | (/s obviously) | |
| 22:50:45 | JayF | sean-k-mooney[m]: What I hear you say is that OpenStack is more scalable than K8s CAPI, nice to hear /s | |
| 22:50:50 | sean-k-mooney[m] | im sure there are ways to fix it but i belive they were having to do batches for 25 phsyical hosts when scaling out | |
| 22:51:03 | JayF | metal3 just doesn't expose any of our scaling features, really | |
| 22:51:39 | sean-k-mooney[m] | right but even with one conductor and no sharding i would expct ironci to be able to provision more then 25 servers at a time | |
| 22:51:46 | sean-k-mooney[m] | liek biforst can | |
| 22:52:43 | sean-k-mooney[m] | so im just sad metal3 cant, when direct ironic or even ironci via nova can scale better | |
| 22:54:11 | JayF | sean-k-mooney[m]: it really can, I'm curious what the limitation is there | |
| 22:54:20 | JayF | sean-k-mooney[m]: oh, probably sqlite? | |
| 22:54:27 | JayF | if metal3 still uses sqlite by defaultt | |
| 22:55:47 | sean-k-mooney[m] | JayF: i think its ectd and how its monitoring the proress of the deployment | |
| 22:56:06 | JayF | oh, so *completely* outside of Ironic's pervue then | |
| 22:56:26 | JayF | and likely in a way that could be fixed (e.g. notifications->some consuming service which updates etcd via push instead of polling | |
| 22:56:34 | sean-k-mooney[m] | right ironic can scale more but the baremetal operator that is managing it cant keep up | |
| 22:56:49 | sean-k-mooney[m] | if you ask it to do 100 ndoes it will eventually catch up | |
| 22:57:40 | JayF | when onmetal was first being created; the principle engineer on our team would just drop 100 builds on us occassionally as a test | |
| 22:58:00 | JayF | that is actually the reason we redid the compute manager in ironic to do peer list, which we are just now ripping out | |
| 22:58:11 | JayF | because the old clusteredcomputemanager we used for ironic was so racey | |
| 23:00:57 | sean-k-mooney[m] | right but that predats palcement | |
| 23:01:28 | sean-k-mooney[m] | and like the admin roles in highnsigt we proably never would have done the peer list | |
| 23:02:24 | sean-k-mooney[m] | anyway i should calll it a day and go to sleep o/ | |
| 23:02:51 | JayF | o/ | |
| #openstack-nova - 2024-10-17 | |||
| 09:51:11 | pas-ha[m] | I failed to read the whole backlog of discussion on admin-ness etc, but some my thoughts: you want 'admin' or 'reader' or whatever only in one service? as admin create app creds with access rules limiting those to API of this service only. Done. | |
| 09:53:22 | pas-ha[m] | The actual problem I faced is that in all the services, 'adminness' means 2 things - admin API (possibility to perform certain action, like create a flavor') and admin for DB (possibility to read other projects data). And currently these are conflated. | |
| 09:55:53 | pas-ha[m] | This is why for now the only way to create a 'global reader' using project-scoped token is to create app creds with admin role and access rules scoped to GET APIs of all services only. This way all non-read request will be blocked by keystonemiddleware, but once the request reaches the actual service, it can do (read) whatever admin can. | |
| 09:57:56 | pas-ha[m] | What I also found suboptimal, is that in most services the actual 'personas' are defined as constants in code, not the rule: that can be easily overridden or extended by the cloud owner, requiring modification of many-many policy rules instead just several... | |
| 09:59:59 | pas-ha[m] | And I do agree that 'manager' will be a better solution instead of making a role do different things depending on the project it is assigned to. Leave 'admin' to be the admin of the whole cloud, period, and watch whom you give it carefully. Replace 'project admins' and even 'domain admins' with project and domain managers. Should make things more clear IMO. | |
| 12:26:31 | sean-k-mooney | pas-ha[m]: scoping accees base on the service has never been supported | |
| 12:26:52 | sean-k-mooney | it was discussed and delcared out of scope of the current srbac work | |
| 12:27:17 | sean-k-mooney | pas-ha[m]: part of the probelm there is keyston maps roles to user in a given project | |
| 12:27:51 | sean-k-mooney | btu to my konowladge there is not concept of a role assingment mapping between a user a a (service,project) pair | |
| 12:28:30 | sean-k-mooney | pas-ha[m]: so you cant express service (or endpoint) based access contol in the keystone data model without just crating many custom roles | |
| 12:28:38 | pas-ha[m] | yeah, I know, and I am fine with that, just pointing out how one can achieve that right now w/o any hacks, with APIs already available, and that is fine I guess | |
| 12:28:39 | sean-k-mooney | which was rejected for interop reasons | |
| 12:28:57 | pas-ha[m] | app creds + access rules = profit | |
| 12:29:10 | sean-k-mooney | oh you can scope app creds | |
| 12:29:14 | sean-k-mooney | good to know | |
| 12:29:29 | pas-ha[m] | down to specific API path and verb | |
| 12:29:45 | sean-k-mooney | personall i prefer using applciation creds even as a human | |
| 12:30:04 | pas-ha[m] | so like can give a get access to a specific server only (by uuid) and nothing else | |
| 12:30:26 | sean-k-mooney | that very powerful and useful to know indeed | |
| 12:30:59 | pas-ha[m] | switching the subject to scheduling, may I bring your attention to https://bugs.launchpad.net/nova/+bug/2076089 ? is there other way currently for an admin to boot an instance on a disabled compute host? | |
| 12:31:05 | sean-k-mooney | i knew i coudl scope by roles when using them but i have never looked at "access rules" before | |
| 12:31:31 | sean-k-mooney | pas-ha[m]: boot or spwan | |
| 12:31:44 | pas-ha[m] | boot | |
| 12:31:54 | sean-k-mooney | disableing computes is expictly for makign it imporssible to schdule a new worklaod | |
| 12:32:02 | sean-k-mooney | so https://bugs.launchpad.net/nova/+bug/2076089 is not valid | |
| 12:32:13 | pas-ha[m] | but it was working before 😕 | |
| 12:32:18 | sean-k-mooney | the disables feature is descided to explcliy block that | |
| 12:32:23 | pas-ha[m] | and is a normal part of node onboarding | |
| 12:32:30 | sean-k-mooney | you can start existing vms on a disbale compute | |
| 12:32:51 | pas-ha[m] | how does one test a node w/o exposing it to the whole cloud? | |
| 12:33:24 | sean-k-mooney | in general you dont but in practice va a host aggreate and something like the tenant filter | |
| 12:33:25 | pas-ha[m] | short of adding a new az of course... that users still can see and attempt to boot on | |
| 12:33:53 | sean-k-mooney | you and do it in a way that users wont see using a number of methods | |
| 12:33:57 | pas-ha[m] | yeah... sad ☹️ though still can migrate to a disabled host using old API | |
| 12:34:14 | sean-k-mooney | you can use a private flavor and required trait | |
| 12:34:24 | sean-k-mooney | the tenat isolation filter or some other methods | |
| 12:34:55 | sean-k-mooney | pas-ha[m]: basiclaly set https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.enable_new_services to false | |
| 12:34:57 | pas-ha[m] | with required trait that does not prevent from other flavors seeing it as available | |
| 12:35:30 | sean-k-mooney | that will cause it to be enroled as disabeld then you can put it into a seperate aggret that only your test proejct can use and test it after you enable it | |
| 12:37:53 | sean-k-mooney | pas-ha[m]: the basic workflow is enrole new nodes as disabled, put them in a restircted test aggreate, enable them, test them and finally remove them for the aggrate | |
| 12:38:44 | sean-k-mooney | pas-ha[m]: https://docs.openstack.org/nova/latest/admin/aggregates.html#tenant-isolation-with-placement is what i generally recommend fo this | |
| 12:39:40 | sean-k-mooney | but there are other approchs that work too | |
| 12:40:12 | sean-k-mooney | i prefer doing it with tenant isolation so that you can use the actual flavors/images you intent do use and just delete the placment aggreate when your done | |
| 12:41:01 | sean-k-mooney | pas-ha[m]: you cant migrate via the old api by the way, the way disabel works will filter out the host so --foce wont work | |
| 12:41:24 | pas-ha[m] | nope,it does work, tested on Antelope | |
| 12:41:49 | sean-k-mooney | it should not because the placment prefilter cant be turned off | |
| 12:42:01 | pas-ha[m] | ``` | |
| 12:42:03 | sean-k-mooney | if it works its a bug | |
| 12:42:13 | pas-ha[m] | * import openstack... (full message at <https://matrix.org/oftc/media/v1/media/download/AUsfx3faFN1prujTkMr2mB2XbtUHmg98qQVs5w-ulEaFy1UTw3_HViUOe21mF6pw3ijesukW_RFxPoO-qfRuuFpCeS4pkHBQAG1hdHJpeC5vcmcvWEJQRlZGcGRCc0JuWWFiZmdocmlKaGlw>) | |
| 12:43:02 | pas-ha[m] | API ref says the same btw. if you use older that 2.30 api, the scheduler does not apply when migrating to a specific host | |
| 12:43:02 | sean-k-mooney | pas-ha[m] https://github.com/openstack/nova/blob/master/nova/scheduler/request_filter.py#L242-L255 | |
| 12:43:29 | sean-k-mooney | pas-ha[m]: every request we make to placmenet unconditon fobids host with the COMPUTE_STAUS_DISBALED trait | |
| 12:44:12 | pas-ha[m] | afaiu in older api scheduler is not involved at all I guess... | |
| 12:44:28 | pas-ha[m] | https://docs.openstack.org/api-ref/compute/#live-migrate-server-os-migratelive-action | |
| 12:44:29 | pas-ha[m] | Prior to microversion 2.30, specifying a host will bypass validation by the scheduler, which could result in failures to actually migrate the instance to the specified host, or over-subscription of the host. | |
| 12:44:29 | sean-k-mooney | it is but it does very minimal checks | |
| 12:44:40 | sean-k-mooney | but the palcment one i did nto think was skiped | |
| 12:44:54 | sean-k-mooney | basically it does nto run any fitler or weighers | |
| 12:45:00 | sean-k-mooney | but it does confirm the host extist | |
| 12:45:28 | sean-k-mooney | its possibel we have a bug and we are early outing before check with placment | |
| 12:49:21 | pas-ha[m] | sean-k-mooney: and, changing to other scheduling problems, what would you think on https://bugs.launchpad.net/nova/+bug/2078669 (and https://review.opendev.org/c/openstack/nova/+/927744)? | |
| 12:49:21 | pas-ha[m] | this is also also something that was working before the AZFilter was removed (we had to re-instate it back in downstream 😕 ) | |
| 12:50:29 | pas-ha[m] | how can I boot an instance 'not in any of those AZs that I see' ? | |
| 12:52:27 | sean-k-mooney | thats a diffent issue and strictly speakign also not supproted | |
| 12:53:02 | sean-k-mooney | pas-ha[m]: see the red warining box in https://docs.openstack.org/nova/latest/admin/availability-zones.html | |
| 12:54:20 | sean-k-mooney | i am more open to supproting the defautl nova az but bauzas is stongly against it. we have docuemented taht you should not ever request the nova az effectivly since AZs were first added | |
| 12:57:42 | bauzas | sean-k-mooney: hmmm, what is the question ? | |
| 12:57:49 | pas-ha[m] | yeah, I know that doc. Still, this is in admin part of docs, and users rarely are aware of it, and they do see 'nova' as in az list... and again, if I have hosts in az 'foo' and hosts in no explicit az ('nova'), previously it was possible to boot "not in foo" == boot in 'nova' az. Now its not possible any more 😕 | |
| 12:58:28 | pas-ha[m] | btw, the api ref says that regular users should not see 'nova' az in az list, but I am pretty sure that's still the case even on master | |
| 12:58:48 | pas-ha[m] | bauzas: https://bugs.launchpad.net/nova/+bug/2078669 and the patch to that | |
| 12:59:07 | sean-k-mooney | horizon adds it to the lsit if no azs are retruned i belive | |