Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-02-19
15:06:19 bauzas and I just don't want to create a new virt driver API method just for this
15:06:21 dansmith right, okay, yeah that makes sense
15:06:45 dansmith I'm just wondering what bauzas meant by "so, we would need to inject the db to get the migrations in _update_provider_tree_for_vgpu() "
15:07:13 bauzas sean-k-mooney: I wouldn't check the allocations, I would just verify that all the dict keys have a current migration
15:07:28 sean-k-mooney well that required addtional db lookups
15:07:36 sean-k-mooney which can only be done in the compute manager or resouce tracker
15:07:38 sean-k-mooney not the driver
15:07:44 bauzas dansmith: I'm saying that I want to get the current migrations
15:07:44 dansmith bauzas: but "has an allocation for gpu on this host" is good enough right?
15:07:51 sean-k-mooney so that why im trying to use the allcotions instead
15:08:15 dansmith it also means the reservation mirrors the allocation in placement, which seems right
15:08:16 bauzas dansmith: no, because you could have an allocation for an instance while it's not because of a migration
15:08:36 bauzas if we want to check the dict, we need to look at the *migrations*
15:08:41 dansmith bauzas: okay, but that would be for an instance that migrated in successfully in which case it doesn't hurt right?
15:08:45 bauzas dansmith: that's what you asked me
15:09:00 bauzas dansmith: no, it would only be for current migrations
15:09:10 bauzas once the migration is done, then the dict key is removed
15:09:19 bauzas (or aborted)
15:09:24 dansmith bauzas: actually, what I asked is for it to not to get out of sync silently so that things stop working and a restart is the only solution :)
15:09:25 bauzas so we need to compare
15:09:43 sean-k-mooney bauzas: not what dansmith is saying is if we have an entry in the dict and the instance ahs an allcoation agaisnt the host and there is no migration
15:09:50 dansmith bauzas: if that happens, then we're fine, this is just for cleanup if the migration does *not* get rolled back successfully right?
15:09:59 sean-k-mooney then its can only happy because the migration succeeded
15:10:07 dansmith sean-k-mooney: well, not exactly
15:10:10 sean-k-mooney and we failed to remove it form the dict for some bug
15:10:23 bauzas dansmith: that's what I'm saying, we need to compare migrations with the dict
15:10:38 bauzas if a dict key is not related to a current migration, then it's stale
15:10:38 dansmith sean-k-mooney: what I'm saying is that if there's a reservation in the dict and there's an allocation, then it's either still in progress, has yet to be rolled back, or it succeeded in which case keeping the reservation in the dict doesn't prevent other stuff from working
15:11:01 dansmith once the instance is deleted or migrated away, healing based on the allocation would work itself out too
15:11:02 sean-k-mooney dansmith: ya i agree with that
15:11:20 sean-k-mooney and yes to the healing too
15:11:28 sean-k-mooney also bauzas if we ever use consumer_types
15:11:39 dansmith bauzas: a dict key for a completed migration where the instance is on our host is not incorrect, even if it is stale, and won't harm anything yeah?
15:11:40 sean-k-mooney you can actully tell if the allocation is a migration if you really want too
15:12:24 sean-k-mooney so if we start using consumer types for that and we do this based on the allcoation if you want to filter you will be able to in the future
15:12:30 bauzas dansmith: I wouldn't be super happy, as we should have removed it by https://review.opendev.org/c/openstack/nova/+/904209/16/nova/virt/libvirt/driver.py#11355
15:12:43 dansmith can we gmeet?
15:12:49 dansmith I feel like we're going in circles here
15:12:56 bauzas sure, I can
15:12:58 bauzas sec
15:13:18 bauzas https://meet.google.com/cyg-ucsv-xri
15:14:08 dansmith sean-k-mooney: are you able to join?
15:14:50 opendevreview Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824
15:17:34 opendevreview Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824
15:28:41 sean-k-mooney dansmith: sorry no i was in/am in anohter call
15:29:05 sean-k-mooney dansmith:i can join quickly now since this call is just ending
16:01:51 opendevreview Mateusz Klejn proposed openstack/nova master: Fixed: unit tests for force delete instance https://review.opendev.org/c/openstack/nova/+/909464
16:01:52 opendevreview Mateusz Klejn proposed openstack/nova master: Changed: adjust shelving for shelved image protection https://review.opendev.org/c/openstack/nova/+/909465
16:04:17 mnaser Good morning! https://review.opendev.org/c/openstack/nova/+/909228 + https://review.opendev.org/c/openstack/nova/+/909229 are ready for review today :)
16:42:42 melwitt when O
16:43:49 melwitt *when I've looked into nova-lvm job failures, what I have seen is that they fail due to the glance apache server responding to nova with a HTTP 502 bad gateway. meanwhile no errors in glance logs https://etherpad.opendev.org/p/nova-ci-failures-minimal#L11
16:44:22 melwitt not really sure what to do next to debug
16:45:17 sean-k-mooney melwitt: this maybe be related to glance-api being OOM killed
16:45:45 sean-k-mooney melwitt: artom was debuging failure where that happend so it would be good to confirm that is not whats happening fro the nova-lvm case
16:46:12 artom "debugging" is a strong word
16:46:22 artom I chased a red herring only to figure out that glanceclient does chunking for us
16:46:27 melwitt hm ok. I didn't think there was OOM killer in these but let me check
16:46:35 artom So I still no have no idea why glance-api gets oom-killed
16:47:04 melwitt oh, yeah there is some OOM kill in here
16:47:05 sean-k-mooney melwitt: there may not be but if there is no error in the glance api log i would first check for an OOM event
16:47:36 sean-k-mooney so we chatted about turning on zswap on nova-next
16:47:50 sean-k-mooney but since we are having OOM events in nova-lvm i might turn it on there too
16:48:03 sean-k-mooney or increase the swap there to 8G if its still at 4G
16:48:05 melwitt yeah, g-api. hm
16:49:04 melwitt Feb 15 13:59:43 np0036770057 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/system-devstack.slice/devstack@g-api.service,task=uwsgi,pid=86510,uid=1001
16:51:08 sean-k-mooney melwitt: so its runing with 4G of swap today
16:51:21 dansmith also, g-api might not be actually consuming lots of memory, it might just be that the OOM killer decided it was the best thing to kill to create some space
16:51:34 dansmith so look at the oom output to see if it was huge or not
16:55:55 opendevreview sean mooney proposed openstack/nova master: bump nova-lvm to use 8G of swap from 4G https://review.opendev.org/c/openstack/nova/+/909469
16:56:01 melwitt Feb 15 13:59:43 np0036770057 kernel: Out of memory: Killed process 86510 (uwsgi) total-vm:2290700kB, anon-rss:1733784kB, file-rss:6324kB, shmem-rss:32kB, UID:1001 pgtables:3732kB oom_score_adj:0 4707 so I guess 2.2 GB? not sure if that's typical
16:56:24 sean-k-mooney melwitt: dansmith ^ thats the swap bump and ill look at adding zswap to that and nova-next in a seperate patch later
16:57:22 sean-k-mooney melwitt: so it should be maybe 150-200mb at most normally
16:57:27 dansmith makes sense to try it on nova-lvm too so that seems okay
16:57:31 sean-k-mooney that looks like it holding a guest image in memory
16:58:13 sean-k-mooney that or we need to add an exicit python garbage collector call in the image upload path
16:58:29 melwitt Feb 15 13:59:43 np0036770057 kernel: Free swap = 0kB
16:58:32 dansmith the image upload path doesn't store the image
16:58:35 sean-k-mooney i.e. to egerly run the garbage collector after straming the image
16:59:28 dansmith it's possible that python keeps allocating new buffers for each thing we read and needs a break to do garbage collection, that's certainly possible
17:00:24 sean-k-mooney i belive we have hadd issues with that in eithe glance or cinder before
17:01:55 sean-k-mooney maybe im thinking of swift
17:01:57 sean-k-mooney https://codesearch.opendev.org/?q=gc.collect&i=nope&literal=nope&files=&excludeFiles=&repos=
17:02:11 sean-k-mooney there are a few project that have an explcit gc.collect() call
17:03:00 sean-k-mooney hum although mainly in test code
17:03:11 dansmith there was also something about python never deflating the heap, which I think gorka can tell us about
17:03:49 sean-k-mooney ya thats what i was recalling. speaking to gorka about the agents growing there memory usage over time
17:03:50 dansmith but I would think that wouldn't be our problem here since we should be good to re-use those portions of the heap over and over, especially with allocations of exactly the same size each time
17:04:35 sean-k-mooney in anycase bumping to 8G of swap will mitigate the issue for now
17:05:26 opendevreview Jay Faulkner proposed openstack/nova-specs master: Sharding: More detailed test plan https://review.opendev.org/c/openstack/nova-specs/+/908673
17:05:52 JayF https://review.opendev.org/c/openstack/nova-specs/+/908673 Heads up, I'm going to be implementing the plan laid out here this week; please make loud noises quickly if it's unacceptable :D
17:07:44 sean-k-mooney ack for 2024.2 or are you still hoping to merge the shard stuff for 2024.1
17:08:05 sean-k-mooney i guess for 2024.1
17:08:36 sean-k-mooney if you can get the testing working in time before FF i guess that is fine
17:09:56 sean-k-mooney for me the minium i think we need is a 2 node job with to set of vms with 2 ironnic compuge service partitioning the vms into shards within the same conducotr group
17:11:59 sean-k-mooney JayF: having tempest test that acttuly assert the toplogy is proably not in the scope of tempest. it would be in the scope of whitebox or could be done via a differnt tempest plugin
17:12:13 gibi Uggla: i responded in the manila series
17:12:17 JayF sean-k-mooney: It was going to be in ironic-tempest-plugin
17:12:27 sean-k-mooney ack thats proably ok then
17:12:33 JayF Two node job is not something we do in Ironic really
17:12:51 JayF And we are working to slim down our job set; try to get more cases in less VMs

Earlier   Later