Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-18
16:34:46 sean-k-mooney we do already cache all the metadata once built today btu its time based with no way to triger the invalidation
16:34:51 dansmith network-changed doesn't dispatch for adding a floating right?
16:35:20 sean-k-mooney it might but i dont think applciationcation should be relaying on the metadata contnet to be up to date
16:35:47 sean-k-mooney we do get network-chaged for a number of things but i dont knwo of a list that is written down anywhere to check
16:35:52 dansmith I was talking about server show
16:36:06 dansmith I don't think applications can rely on anything other than metadata for that info right?
16:36:25 sean-k-mooney they shoudl really call neutorn if the need the athoritive info
16:36:38 sean-k-mooney but if they dont know they are on openstack then no
16:36:41 sean-k-mooney this is all they have
16:36:46 dansmith applications inside guests can't do that unless they have creds, which we also don't provide them with :)
16:37:13 dansmith like some thing that needs to provision on boot and know it's external IP so it can offer it up to clients needs to ask metadata for that info
16:37:46 sean-k-mooney yep ops have asked for that JWS tokens speficically
16:38:51 sean-k-mooney dansmith: so the boot case shoudl be up to date today beacue its the first request but yes if it change during the boot
16:39:11 sean-k-mooney i.e. a floting ip was added while it was booting it could be racy
16:40:25 sean-k-mooney what would be nice at somepoint is to be able to povide a short lived bootsrap took like kubernbetes does but thats a much bigger feature
16:41:11 sean-k-mooney dansmith: may main concern with the lazy loading spec is it does not explain how its going to do this
16:42:00 sean-k-mooney liekk is the plan ot create a url router within the metadata code to fetch only the conent for that file via decompoing things into lots of funcitons
16:42:51 sean-k-mooney or do some kind of properlty based laze loading on a class
16:44:19 sean-k-mooney there is really just https://review.opendev.org/c/openstack/nova-specs/+/922201/9/specs/2024.2/approved/lazy-metadata-loading.rst#79
16:53:38 dansmith sean-k-mooney: yeah, if there's a race in the boot, add floating, cloud-init sequence it could be wrong
17:08:13 sean-k-mooney so to me the metadata api comes under the banner of "no more proxy apis" in nova
17:08:55 sean-k-mooney i.e. its not a data socue that shoudl be proxiying up to date info form other service its just a cached view of some useful info
17:09:22 sean-k-mooney so the idea that applciations are writen to poll it at a rate
17:09:29 dansmith idk, I don't think that's reasonable if there is literally no other way to get the info and we're holding the guest in our hand
17:09:35 sean-k-mooney that is enough to cause load issues is kind of bizar to me
17:09:43 dansmith like, if we start provisioning them with credentials and a directory, then I'm all for it
17:09:59 dansmith (provisioning with credentials that can be renewed I should say)
17:10:23 sean-k-mooney ya so at some point i think that should be done like how pods in k8s get service credital tokens to be able to talk to k8s
17:10:46 sean-k-mooney but short term if we can optimise it then ok im just concerned by the lack of detial on how
17:10:48 dansmith right
17:10:53 dansmith people have been asking for that for a long time
17:11:12 opendevreview Sylvain Bauza proposed openstack/nova master: cpu: Only check governor type on online cores https://review.opendev.org/c/openstack/nova/+/924427
17:11:17 sean-k-mooney ya i think its a reason able thing to od at least if the user opts into it
17:11:33 sean-k-mooney but its not what the spec is adressing
17:12:10 bauzas sean-k-mooney: I had to drop due to a bugfix, but please just capture your concerns on that lazy-loading metadata spec so you can move on
17:12:28 bauzas and we'll revisit that spec at the PTG
17:16:55 sean-k-mooney bauzas: i rased most of them in the spec already, mainily i dont know how they are planning to implemnte the lazy loading and im concured it will break cloud-init or cirros-init due to the fact they dont retry by default and some recuest take longer then other to formulate
17:17:04 bauzas ++
17:17:27 sean-k-mooney bauzas: bascily cloud-init and cirros-init today only retry the first request and assuem the rest will be fast and work first try by default
17:17:39 sean-k-mooney cloud init cna be configured to retry its just not the default
17:18:13 sean-k-mooney so if this is on by defualt im concerned it woudl regression previous bugs we had where vms would not get an ssh key inejcted ectra
17:20:14 dansmith sean-k-mooney: yeah, that's exactly my concern about making this less obvious by having some requests work and some not because they're all independent on the backend
17:20:22 dansmith I missed your comment about that but +100
17:20:49 dansmith I think if we want to defeat caching in certain circumstances we *should* make cloud-init declare when it wants uncached data or not,
17:20:53 dansmith and otherwise we're just guessing
17:21:06 dansmith but splitting this into N backend requests doesn't seem to improve things at all to me
17:26:02 sean-k-mooney i was wondering if there was soem way to use etag or just support triggerting the cache invalidation form the clinet side i.e. cluod init
17:26:11 sean-k-mooney or curl
17:26:32 sean-k-mooney i dont know what the standard convetion is for that at a http level but im sure there is one
17:30:16 sean-k-mooney sound like there is a semi standardisaed HTTP PURGE method that cloudflare, varnish and squid all support to do this
17:31:31 dansmith cache-control, I mentioned it in my review
17:31:49 sean-k-mooney cache-contol is slightly different
17:32:07 sean-k-mooney that says if caching layers are allowed to cache it or not and for how long
17:32:22 dansmith it's usable on request as well
17:32:25 sean-k-mooney that works with etag to allow the caching server to know if the content changed
17:32:31 sean-k-mooney oh really
17:32:39 dansmith https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
17:32:46 sean-k-mooney so you can send a cache-contole no-store or similr
17:32:49 dansmith max-age, no-cache, etc would be perfect here I think
17:33:03 sean-k-mooney ya i tought that was responce only
17:33:09 sean-k-mooney but if it can be in the request
17:33:11 sean-k-mooney then yes
17:34:28 sean-k-mooney ya so that seams quite reasonabel to look into
17:56:16 opendevreview Merged openstack/nova-specs master: Propose a new API microversion for a spice-direct console. https://review.opendev.org/c/openstack/nova-specs/+/915190
20:40:14 opendevreview melanie witt proposed openstack/nova master: nova-manage: Add flavor scanning to migrate_to_unified_limits https://review.opendev.org/c/openstack/nova/+/924110
23:40:47 opendevreview melanie witt proposed openstack/nova master: nova-manage: Add flavor scanning to migrate_to_unified_limits https://review.opendev.org/c/openstack/nova/+/924110
#openstack-nova - 2024-07-19
06:39:47 mikal bauzas: I am fine with working on adding tests too. Without Kerbside deployed it would still be meaningful to have a functional test which called the console create call to get an auth token and then looked up connection details from that.
06:40:14 mikal One wart there is that the openstacksdk doesn't actually support the console auth token API at the moment, so I need to add that too unless I am going to write a different client for that test.
07:49:59 bauzas mikal: fine by me, we'll discuss this on the implementation then
08:52:37 bauzas sean-k-mooney: when you're up, could you please +2/+W https://review.opendev.org/c/openstack/os-traits/+/908966 ?
08:52:46 bauzas then I'll create a os-traits release patch
08:54:27 sean-k-mooney bauzas: done
08:54:31 bauzas ++
08:54:39 sean-k-mooney i have been thinking about the cpu power managment
08:54:56 sean-k-mooney i have a simple quick fix and hten a longer term fix
08:54:58 bauzas all approved specs are now in the dalmatian etherpad
08:55:08 bauzas and Launchpad is now correct
08:55:14 bauzas sean-k-mooney: kk, tell me
08:55:28 bauzas I just need to go to a cycling shop in 30 mins
08:55:46 sean-k-mooney the simple quick fix is when nova-compute is stopped we can just power on all core
08:56:33 sean-k-mooney and when we start up we can just cache the core info with the runonce decorator before we turn off any of the cores
08:57:34 bauzas it would be a behaviour change, but why not
08:58:24 sean-k-mooney the longer term soltuion is we may want to add a local key value store or some other form of data persitince
08:59:21 sean-k-mooney we have had a few feature lately where having a very simple key value store avaibel o nthe compute like https://docs.python.org/3/library/dbm.html
08:59:27 sean-k-mooney would have been useful
08:59:40 sean-k-mooney so we shoudl disucss that at some point
09:00:14 bauzas for remembering the governor strategy ?
09:00:27 sean-k-mooney there are several other good option like sqlite
09:00:31 sean-k-mooney bauzas: not just for that
09:00:42 sean-k-mooney for data persitance on the comptue node locally in general
09:01:05 sean-k-mooney bauzas: for example for storing information about the image in the image cache
09:01:05 bauzas then this is a way larger context but I don't disagree
09:01:25 bauzas I just think this is useless for just caching the governor state
09:01:26 sean-k-mooney bauzas: right that why i said long term for that approhc
09:01:49 sean-k-mooney bauzas: right i never mention those becaus im not suggestin we do
09:02:29 sean-k-mooney what we need to cache is strictly the numa info for the cores before we turn them off so that the numa toplgoy blob has the correct numa node and socket
09:02:35 bauzas what we can also do is when the instance is about to spawn, we can doublecheck whether the core is poweroff if this is asking to get a governor
09:02:47 bauzas to set*
09:02:54 sean-k-mooney bauzas: this is not related to the govenor
09:03:05 sean-k-mooney im not talkign about that bug im talking about the other one

Earlier   Later