| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-10-16 | |||
| 22:56:34 | sean-k-mooney[m] | right ironic can scale more but the baremetal operator that is managing it cant keep up | |
| 22:56:49 | sean-k-mooney[m] | if you ask it to do 100 ndoes it will eventually catch up | |
| 22:57:40 | JayF | when onmetal was first being created; the principle engineer on our team would just drop 100 builds on us occassionally as a test | |
| 22:58:00 | JayF | that is actually the reason we redid the compute manager in ironic to do peer list, which we are just now ripping out | |
| 22:58:11 | JayF | because the old clusteredcomputemanager we used for ironic was so racey | |
| 23:00:57 | sean-k-mooney[m] | right but that predats palcement | |
| 23:01:28 | sean-k-mooney[m] | and like the admin roles in highnsigt we proably never would have done the peer list | |
| 23:02:24 | sean-k-mooney[m] | anyway i should calll it a day and go to sleep o/ | |
| 23:02:51 | JayF | o/ | |
| #openstack-nova - 2024-10-17 | |||
| 09:51:11 | pas-ha[m] | I failed to read the whole backlog of discussion on admin-ness etc, but some my thoughts: you want 'admin' or 'reader' or whatever only in one service? as admin create app creds with access rules limiting those to API of this service only. Done. | |
| 09:53:22 | pas-ha[m] | The actual problem I faced is that in all the services, 'adminness' means 2 things - admin API (possibility to perform certain action, like create a flavor') and admin for DB (possibility to read other projects data). And currently these are conflated. | |
| 09:55:53 | pas-ha[m] | This is why for now the only way to create a 'global reader' using project-scoped token is to create app creds with admin role and access rules scoped to GET APIs of all services only. This way all non-read request will be blocked by keystonemiddleware, but once the request reaches the actual service, it can do (read) whatever admin can. | |
| 09:57:56 | pas-ha[m] | What I also found suboptimal, is that in most services the actual 'personas' are defined as constants in code, not the rule: that can be easily overridden or extended by the cloud owner, requiring modification of many-many policy rules instead just several... | |
| 09:59:59 | pas-ha[m] | And I do agree that 'manager' will be a better solution instead of making a role do different things depending on the project it is assigned to. Leave 'admin' to be the admin of the whole cloud, period, and watch whom you give it carefully. Replace 'project admins' and even 'domain admins' with project and domain managers. Should make things more clear IMO. | |
| 12:26:31 | sean-k-mooney | pas-ha[m]: scoping accees base on the service has never been supported | |
| 12:26:52 | sean-k-mooney | it was discussed and delcared out of scope of the current srbac work | |
| 12:27:17 | sean-k-mooney | pas-ha[m]: part of the probelm there is keyston maps roles to user in a given project | |
| 12:27:51 | sean-k-mooney | btu to my konowladge there is not concept of a role assingment mapping between a user a a (service,project) pair | |
| 12:28:30 | sean-k-mooney | pas-ha[m]: so you cant express service (or endpoint) based access contol in the keystone data model without just crating many custom roles | |
| 12:28:38 | pas-ha[m] | yeah, I know, and I am fine with that, just pointing out how one can achieve that right now w/o any hacks, with APIs already available, and that is fine I guess | |
| 12:28:39 | sean-k-mooney | which was rejected for interop reasons | |
| 12:28:57 | pas-ha[m] | app creds + access rules = profit | |
| 12:29:10 | sean-k-mooney | oh you can scope app creds | |
| 12:29:14 | sean-k-mooney | good to know | |
| 12:29:29 | pas-ha[m] | down to specific API path and verb | |
| 12:29:45 | sean-k-mooney | personall i prefer using applciation creds even as a human | |
| 12:30:04 | pas-ha[m] | so like can give a get access to a specific server only (by uuid) and nothing else | |
| 12:30:26 | sean-k-mooney | that very powerful and useful to know indeed | |
| 12:30:59 | pas-ha[m] | switching the subject to scheduling, may I bring your attention to https://bugs.launchpad.net/nova/+bug/2076089 ? is there other way currently for an admin to boot an instance on a disabled compute host? | |
| 12:31:05 | sean-k-mooney | i knew i coudl scope by roles when using them but i have never looked at "access rules" before | |
| 12:31:31 | sean-k-mooney | pas-ha[m]: boot or spwan | |
| 12:31:44 | pas-ha[m] | boot | |
| 12:31:54 | sean-k-mooney | disableing computes is expictly for makign it imporssible to schdule a new worklaod | |
| 12:32:02 | sean-k-mooney | so https://bugs.launchpad.net/nova/+bug/2076089 is not valid | |
| 12:32:13 | pas-ha[m] | but it was working before 😕 | |
| 12:32:18 | sean-k-mooney | the disables feature is descided to explcliy block that | |
| 12:32:23 | pas-ha[m] | and is a normal part of node onboarding | |
| 12:32:30 | sean-k-mooney | you can start existing vms on a disbale compute | |
| 12:32:51 | pas-ha[m] | how does one test a node w/o exposing it to the whole cloud? | |
| 12:33:24 | sean-k-mooney | in general you dont but in practice va a host aggreate and something like the tenant filter | |
| 12:33:25 | pas-ha[m] | short of adding a new az of course... that users still can see and attempt to boot on | |
| 12:33:53 | sean-k-mooney | you and do it in a way that users wont see using a number of methods | |
| 12:33:57 | pas-ha[m] | yeah... sad ☹️ though still can migrate to a disabled host using old API | |
| 12:34:14 | sean-k-mooney | you can use a private flavor and required trait | |
| 12:34:24 | sean-k-mooney | the tenat isolation filter or some other methods | |
| 12:34:55 | sean-k-mooney | pas-ha[m]: basiclaly set https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.enable_new_services to false | |
| 12:34:57 | pas-ha[m] | with required trait that does not prevent from other flavors seeing it as available | |
| 12:35:30 | sean-k-mooney | that will cause it to be enroled as disabeld then you can put it into a seperate aggret that only your test proejct can use and test it after you enable it | |
| 12:37:53 | sean-k-mooney | pas-ha[m]: the basic workflow is enrole new nodes as disabled, put them in a restircted test aggreate, enable them, test them and finally remove them for the aggrate | |
| 12:38:44 | sean-k-mooney | pas-ha[m]: https://docs.openstack.org/nova/latest/admin/aggregates.html#tenant-isolation-with-placement is what i generally recommend fo this | |
| 12:39:40 | sean-k-mooney | but there are other approchs that work too | |
| 12:40:12 | sean-k-mooney | i prefer doing it with tenant isolation so that you can use the actual flavors/images you intent do use and just delete the placment aggreate when your done | |
| 12:41:01 | sean-k-mooney | pas-ha[m]: you cant migrate via the old api by the way, the way disabel works will filter out the host so --foce wont work | |
| 12:41:24 | pas-ha[m] | nope,it does work, tested on Antelope | |
| 12:41:49 | sean-k-mooney | it should not because the placment prefilter cant be turned off | |
| 12:42:01 | pas-ha[m] | ``` | |
| 12:42:03 | sean-k-mooney | if it works its a bug | |
| 12:42:13 | pas-ha[m] | * import openstack... (full message at <https://matrix.org/oftc/media/v1/media/download/AUsfx3faFN1prujTkMr2mB2XbtUHmg98qQVs5w-ulEaFy1UTw3_HViUOe21mF6pw3ijesukW_RFxPoO-qfRuuFpCeS4pkHBQAG1hdHJpeC5vcmcvWEJQRlZGcGRCc0JuWWFiZmdocmlKaGlw>) | |
| 12:43:02 | pas-ha[m] | API ref says the same btw. if you use older that 2.30 api, the scheduler does not apply when migrating to a specific host | |
| 12:43:02 | sean-k-mooney | pas-ha[m] https://github.com/openstack/nova/blob/master/nova/scheduler/request_filter.py#L242-L255 | |
| 12:43:29 | sean-k-mooney | pas-ha[m]: every request we make to placmenet unconditon fobids host with the COMPUTE_STAUS_DISBALED trait | |
| 12:44:12 | pas-ha[m] | afaiu in older api scheduler is not involved at all I guess... | |
| 12:44:28 | pas-ha[m] | https://docs.openstack.org/api-ref/compute/#live-migrate-server-os-migratelive-action | |
| 12:44:29 | pas-ha[m] | Prior to microversion 2.30, specifying a host will bypass validation by the scheduler, which could result in failures to actually migrate the instance to the specified host, or over-subscription of the host. | |
| 12:44:29 | sean-k-mooney | it is but it does very minimal checks | |
| 12:44:40 | sean-k-mooney | but the palcment one i did nto think was skiped | |
| 12:44:54 | sean-k-mooney | basically it does nto run any fitler or weighers | |
| 12:45:00 | sean-k-mooney | but it does confirm the host extist | |
| 12:45:28 | sean-k-mooney | its possibel we have a bug and we are early outing before check with placment | |
| 12:49:21 | pas-ha[m] | sean-k-mooney: and, changing to other scheduling problems, what would you think on https://bugs.launchpad.net/nova/+bug/2078669 (and https://review.opendev.org/c/openstack/nova/+/927744)? | |
| 12:49:21 | pas-ha[m] | this is also also something that was working before the AZFilter was removed (we had to re-instate it back in downstream 😕 ) | |
| 12:50:29 | pas-ha[m] | how can I boot an instance 'not in any of those AZs that I see' ? | |
| 12:52:27 | sean-k-mooney | thats a diffent issue and strictly speakign also not supproted | |
| 12:53:02 | sean-k-mooney | pas-ha[m]: see the red warining box in https://docs.openstack.org/nova/latest/admin/availability-zones.html | |
| 12:54:20 | sean-k-mooney | i am more open to supproting the defautl nova az but bauzas is stongly against it. we have docuemented taht you should not ever request the nova az effectivly since AZs were first added | |
| 12:57:42 | bauzas | sean-k-mooney: hmmm, what is the question ? | |
| 12:57:49 | pas-ha[m] | yeah, I know that doc. Still, this is in admin part of docs, and users rarely are aware of it, and they do see 'nova' as in az list... and again, if I have hosts in az 'foo' and hosts in no explicit az ('nova'), previously it was possible to boot "not in foo" == boot in 'nova' az. Now its not possible any more 😕 | |
| 12:58:28 | pas-ha[m] | btw, the api ref says that regular users should not see 'nova' az in az list, but I am pretty sure that's still the case even on master | |
| 12:58:48 | pas-ha[m] | bauzas: https://bugs.launchpad.net/nova/+bug/2078669 and the patch to that | |
| 12:59:07 | sean-k-mooney | horizon adds it to the lsit if no azs are retruned i belive | |
| 12:59:28 | pas-ha[m] | openstack cli does show it too | |
| 13:00:30 | bauzas | pas-ha[m]: tbc, the concern is that by default 'nova' isn't an AZ | |
| 13:01:06 | bauzas | if you want, you can change the default value for instances so there you could create a specific aggregate with a 'nova' AZ | |
| 13:01:34 | bauzas | but if you set an instance with '-az nova' then we'll pin into it | |
| 13:03:17 | pas-ha[m] | I know, but it is represented as such to the users still. And the user sees az 'expensive' and az 'nova', user thinks 'i definitely do not need 'expensive', and uses 'nova'. Before, with AZFilter being available, the instances landed where user intended. Now they are not. | |
| 13:05:18 | sean-k-mooney | ubuntu@devstack-iso:~$ openstack --os-cloud=devstack --debug availability zone list --compute 2>&1| nc termbin.com 9999 | |
| 13:05:20 | sean-k-mooney | https://termbin.com/b8q9 | |
| 13:05:26 | sean-k-mooney | bauzas: we do have a bug ^ | |
| 13:06:38 | pas-ha[m] | I proposed a patch that would make az 'nova' (or whatever default az is) be treated as 'not any host that belongs to any explicit az' (if there's no actual explicit az nova of course). IMO that should be exaclty what the user intended. | |
| 13:06:53 | sean-k-mooney | pas-ha[m]: that would break things | |
| 13:07:31 | pas-ha[m] | this was effectively working like that until AZFilter was removed recently | |
| 13:07:43 | sean-k-mooney | the problem with that is we do not allow moving host betwene azs | |
| 13:07:43 | pas-ha[m] | (in bobcat or caracal) | |
| 13:08:15 | sean-k-mooney | if they have instnace on them that is | |
| 13:08:23 | pas-ha[m] | yeah, but I am not forcing any AZ here. I am only forcing placement to discard any other candidates | |
| 13:09:04 | sean-k-mooney | you cant do that effiectnly without pasically passing all az aggreatd in a not | |
| 13:09:11 | sean-k-mooney | which does not scale | |
| 13:09:19 | sean-k-mooney | and will hit the max url limit in large clouds | |
| 13:09:44 | sean-k-mooney | the only way to make this work woudl be to group all hosts not in an az into a seperate aggreate | |
| 13:09:58 | sean-k-mooney | and then explcitly request that | |