Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-11
09:14:58 sean-k-mooney live migrate defintly does not i dont thik coldmigrate/resize does either
09:15:07 sean-k-mooney build does but im not sure about unshelve
09:15:28 sean-k-mooney well that not entirly true
09:15:47 sean-k-mooney we got rid of reschdule entirly and replaced it with alternate_host 5 or 6 years ago
09:15:50 sean-k-mooney maybe more
09:16:23 sean-k-mooney i think if pre_live migtrate failes we may check the alternate hosts and the same mighgt be true of other move op
09:16:41 sean-k-mooney but we never retry and reschdule once we attemet to start the migration of data
09:16:48 songwenping__ build still entired reschedule?
09:16:56 opendevreview Balazs Gibizer proposed openstack/nova stable/2024.1: Stabilize iso format unit tests https://review.opendev.org/c/openstack/nova/+/923935
09:17:31 sean-k-mooney no its not a rescule because we dont go back to the scheduler it uses the alternitve hosts the scheduler orgially provided
09:17:43 sean-k-mooney by default the schduler gives us 3 hosts
09:17:55 sean-k-mooney the condocutor tries those in order
09:18:23 opendevreview Balazs Gibizer proposed openstack/nova stable/2023.2: Stabilize iso format unit tests https://review.opendev.org/c/openstack/nova/+/923936
09:18:26 sean-k-mooney provided the allocation candiate can still be claimed in placement
09:18:41 songwenping__ got it.
09:18:51 sean-k-mooney if we fail early enouch in evacuate/resize we may do the same
09:20:16 opendevreview Balazs Gibizer proposed openstack/nova stable/2023.1: Stabilize iso format unit tests https://review.opendev.org/c/openstack/nova/+/923937
09:22:19 sean-k-mooney songwenping__: https://docs.openstack.org/nova/latest/contributor/resize-and-cold-migrate.html#resource-claims
09:22:21 songwenping__ as nova-scheduler workers has same host_state info, if two evacuations come at the same time, the resource is enough in nova-scheduler, but nova-compute claim failed as two vms required resources exceed the remain resources on nova-compute.
09:22:53 sean-k-mooney songwenping__: that wont happen in general
09:23:12 sean-k-mooney it will only happen if the vm is using a resouce not tracked in placment or with numa toplogy
09:23:51 sean-k-mooney so before we call teh compute we the conductor creates an atomic claim in placement
09:24:02 sean-k-mooney for cpu/ram/disk and some other resouces
09:24:15 songwenping__ yeah, but in our HA mechanism, it occurs.
09:24:59 sean-k-mooney the vm should never get to the comptue part, are you runnign placement on an Active Active galera by any chance
09:25:36 sean-k-mooney active/active meaning multi master/multi writer setup
09:31:20 songwenping__ maybe our nova-scheduler use host_state cache info, not get ar from placement.
09:31:59 sean-k-mooney thats not how that works
09:32:12 sean-k-mooney unless this is pre rocky
09:32:56 songwenping__ yeah, we use rocky :)
09:35:03 songwenping__ the cache mechanism is deprecated after rocky?
09:35:49 sean-k-mooney there is a cache but we first filter in placement before the schduler filtere run and then claim atotmicaly after to avoid races
09:38:01 songwenping__ sean-k-mooney: got it, thanks.
10:47:10 frickler elodilles: you offered help some days ago, additional reviews for https://review.opendev.org/q/topic:%22format-inspector%22+status:open would be nice I think
10:55:28 elodilles frickler: ACK, thanks for the ping. i did not say it upstream, but i'm mostly off today. so i might only get there to review the patches tomorrow ~afternoonish
11:09:26 frickler elodilles: ah, ok, thx for the update, let's hope things might be resolved until then
12:14:26 elodilles frickler: i've reviewed and +2+W'd the stable/2024.1 patches (except the last patch as that is not merged on master yet)
13:07:08 sean-k-mooney the last 2 patches that sables the unit test we belive are just ened for old veriosn of the tools
13:07:41 sean-k-mooney frickler: elodilles basically we were backportign those downstream too train/wallaby and had ci failures
13:09:54 opendevreview Arnaud Morin proposed openstack/nova master: Avoid failing if compute_id file is empty https://review.opendev.org/c/openstack/nova/+/923953
13:19:18 frickler sean-k-mooney: yes, I'm not too worried about those, but we are running with the major iso format-inspector patch downstream now, so it would be comforting to see it merged
13:33:37 sean-k-mooney frickler: oh good to know. have you foudn any issue with it i should be aware of?
13:33:55 sean-k-mooney if you do just ping and ill take a look
13:36:55 frickler sean-k-mooney: sure, but no issues found so far
13:38:09 mnaser sean-k-mooney: btw https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/M3BBTNE22VMMZSFMGFWMJJDYEPW7VNHL/ of interest based on our discussion the other day :)
13:38:49 mnaser when you have a second id appreciate feeedback
13:41:47 sean-k-mooney you realise that is O(hosts*aggreates^2) right
13:42:47 sean-k-mooney im not saying that wont work
13:43:11 sean-k-mooney but we might need to be a littel smarter if we were to do that in nova proper in terms of performce
13:44:47 sean-k-mooney like see that there is a server group in teh request spec, grap all the relevent host for that server group and lookup the aggerst once in a map and pass in the relevent agrrate i the host state object ot each filter
13:45:34 sean-k-mooney im not sure if that will work but lookign up every aggrete reated to every other host that has an instance in the same server group per host is expensive...
13:57:52 mnaser sean-k-mooney: I realize it is expensive, but i couldn't think of a better approach, but also I am looking up aggregate for server groups only, so it is limiting it to the scope of hosts that are part of that host aggregate
13:58:09 mnaser so potentially a single lookup if there is one other node for example
13:59:29 mnaser and I think I can't really do a cache inside of the filter for a spec because its invoked for each host
14:00:55 sean-k-mooney the filter runs per host however
14:01:27 sean-k-mooney so you multiple that by the number of host reteruend from placment * the number of vm in the build request
14:01:51 opendevreview Merged openstack/nova stable/2024.1: port format inspector tests from glance https://review.opendev.org/c/openstack/nova/+/923722
14:02:44 opendevreview Merged openstack/nova stable/2024.1: Reproduce iso regression with deep format inspection https://review.opendev.org/c/openstack/nova/+/923723
14:03:01 sean-k-mooney mnaser: what i was thinkign is if we were to do this we would have the hostmanager get ll the host in the group and the aggreates for thos groups once before running the filter on each host and storing the info in either the hoststate object or request sepc so we can share that info acrros each vm and across each host
14:03:51 sean-k-mooney mnaser: we do somethign simialr for groups already in terms of maintianign a mapping of instance uuids to hosts
14:05:08 mnaser yeah we have host_state.instances which anti affinity can easily use
14:05:23 sean-k-mooney yep thats why its there
14:05:34 sean-k-mooney its only optionally populated based on a config option
14:05:48 sean-k-mooney as it uses potentally a lost of memory
14:06:05 sean-k-mooney we also have a list of aggreats in the host_state too
14:06:22 mnaser I think I end up using that
14:06:51 mnaser but then I have to check what is the failure domain set to _this_ host aggregate, and lookup what it is for all the other VMs
14:07:09 sean-k-mooney yep
14:07:19 sean-k-mooney so im not saying your silter wont work
14:07:25 mnaser oh yeah no it does work its just.. eww.
14:07:53 sean-k-mooney but if we were to supprot this in nova properly we likely would need to do somethign else to optimise this
14:08:08 mnaser sean-k-mooney: I think maybe an extra kwarg for host_passes to include all of the HostState for all the nodes still eligible in the filter
14:08:56 sean-k-mooney we try not to do that
14:08:58 mnaser so that reduces how much I need to lookup
14:09:05 sean-k-mooney it breaks all out of tree filters when we do
14:09:10 sean-k-mooney not that we technilay supprot those
14:09:25 sean-k-mooney but adding files to the host state object does not break the api
14:09:31 mnaser hmm ok valid
14:09:45 sean-k-mooney so we woudl pass this either via the host_state object or request spec
14:10:31 mnaser I mean if the failure domain concept becomes a little bit more concrete then there are other place we can leverage it in other places to optimize the way its looked up fo rsure
14:10:53 mnaser and maybe just maybe also expose it as a sha'd string in nova's api... but thats stretching my luck :)
14:11:35 mnaser the goal here is to expose this to k8s cluster provisioners and so that the failure domain can be exposed to the user so they can make sure they schedule their pods on nodes with different failure domains
16:41:56 sean-k-mooney mnaser: you asked for feedback :)
16:58:55 opendevreview Merged openstack/nova stable/2024.1: Add iso file format inspector https://review.opendev.org/c/openstack/nova/+/923724
16:59:00 opendevreview Merged openstack/nova stable/2024.1: fix qemu-img version dependent tests https://review.opendev.org/c/openstack/nova/+/923878
17:28:44 opendevreview keerthivasan proposed openstack/nova master: Move nova-manage db purge to nova-audit https://review.opendev.org/c/openstack/nova/+/708783
17:44:52 opendevreview keerthivasan proposed openstack/nova master: Move nova-manage db purge to nova-audit https://review.opendev.org/c/openstack/nova/+/708783
17:57:11 opendevreview Merged openstack/nova master: Stabilize iso format unit tests https://review.opendev.org/c/openstack/nova/+/923924
19:24:57 opendevreview Jens Harbott proposed openstack/nova master: DNM: Test devstack change https://review.opendev.org/c/openstack/nova/+/923759
#openstack-nova - 2024-07-12
02:21:55 opendevreview melanie witt proposed openstack/nova master: Add config option [quota]strict_unified_limits https://review.opendev.org/c/openstack/nova/+/924025
07:32:49 opendevreview melanie witt proposed openstack/nova master: Add config option [quota]strict_unified_limits https://review.opendev.org/c/openstack/nova/+/924025
07:35:55 melwitt sean-k-mooney: ^ POC attempt at the different logic we chatted about the other day
08:04:53 opendevreview Balazs Gibizer proposed openstack/nova stable/2023.2: fix qemu-img version dependent tests https://review.opendev.org/c/openstack/nova/+/923907
08:04:54 opendevreview Balazs Gibizer proposed openstack/nova stable/2023.2: Stabilize iso format unit tests https://review.opendev.org/c/openstack/nova/+/923936
08:09:01 opendevreview Balazs Gibizer proposed openstack/nova stable/2023.1: fix qemu-img version dependent tests https://review.opendev.org/c/openstack/nova/+/923908
08:09:02 opendevreview Balazs Gibizer proposed openstack/nova stable/2023.1: Stabilize iso format unit tests https://review.opendev.org/c/openstack/nova/+/923937
09:38:01 sean-k-mooney melwitt: oh cool. after creating that how do you feel about one approch vs the other. no need to reply now but let me know after you get some sleep
10:09:04 gibi sean-k-mooney: could you approve https://review.opendev.org/c/openstack/nova/+/923935 to continue landing the series?
10:09:48 gibi most of the 2023.2 is also on the gate now thanks to elodilles
10:13:59 sean-k-mooney i for got to clice the buttons and only left a commet ...

Earlier   Later