Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-02-19
15:02:14 dansmith I thought the periodic was to clean up reservations that are now stale for migrations that never happened?
15:02:40 dansmith in the driver memory-resiident dict
15:03:22 sean-k-mooney yes
15:03:32 sean-k-mooney so i think we can do that in _update_provider_tree_for_vgpu
15:03:34 dansmith so what injecting into the DB needs to happen?
15:03:50 sean-k-mooney nothing has to happen in the db
15:04:05 dansmith ack, okay, I'm confused by bauzas' statement then
15:04:08 sean-k-mooney dansmith: is the destionation allcotion held by the instance uuid or the migration uuid
15:04:18 sean-k-mooney i think the instance_uuid?
15:04:27 dansmith instance IIRC yeah
15:04:43 sean-k-mooney ack, assuming that is correct all _update_provider_tree_for_vgpu need to do
15:05:05 sean-k-mooney is loop over the in memory dict
15:05:12 dansmith maybe bauzas is talking about _querying_ for canceled migration objects to know what to clean from the dict?
15:05:15 sean-k-mooney and remove any key that does not have an allocation
15:05:17 bauzas dansmith: btw. I added a new check by https://review.opendev.org/c/openstack/nova/+/904209/16/nova/virt/libvirt/driver.py#1740
15:05:41 dansmith sean-k-mooney: you can't do that I think because you might be racing with the start of the migration?
15:05:55 bauzas dansmith: my question was about where we could verify the dict
15:06:02 dansmith sean-k-mooney: or are you thinking the instance already has an allocation for that host, until it's canceled and we can go on the presence of that allocation?
15:06:08 sean-k-mooney we should have claimed the placment allcotion before calling migrate correct
15:06:19 bauzas and I just don't want to create a new virt driver API method just for this
15:06:21 dansmith right, okay, yeah that makes sense
15:06:45 dansmith I'm just wondering what bauzas meant by "so, we would need to inject the db to get the migrations in _update_provider_tree_for_vgpu() "
15:07:13 bauzas sean-k-mooney: I wouldn't check the allocations, I would just verify that all the dict keys have a current migration
15:07:28 sean-k-mooney well that required addtional db lookups
15:07:36 sean-k-mooney which can only be done in the compute manager or resouce tracker
15:07:38 sean-k-mooney not the driver
15:07:44 dansmith bauzas: but "has an allocation for gpu on this host" is good enough right?
15:07:44 bauzas dansmith: I'm saying that I want to get the current migrations
15:07:51 sean-k-mooney so that why im trying to use the allcotions instead
15:08:15 dansmith it also means the reservation mirrors the allocation in placement, which seems right
15:08:16 bauzas dansmith: no, because you could have an allocation for an instance while it's not because of a migration
15:08:36 bauzas if we want to check the dict, we need to look at the *migrations*
15:08:41 dansmith bauzas: okay, but that would be for an instance that migrated in successfully in which case it doesn't hurt right?
15:08:45 bauzas dansmith: that's what you asked me
15:09:00 bauzas dansmith: no, it would only be for current migrations
15:09:10 bauzas once the migration is done, then the dict key is removed
15:09:19 bauzas (or aborted)
15:09:24 dansmith bauzas: actually, what I asked is for it to not to get out of sync silently so that things stop working and a restart is the only solution :)
15:09:25 bauzas so we need to compare
15:09:43 sean-k-mooney bauzas: not what dansmith is saying is if we have an entry in the dict and the instance ahs an allcoation agaisnt the host and there is no migration
15:09:50 dansmith bauzas: if that happens, then we're fine, this is just for cleanup if the migration does *not* get rolled back successfully right?
15:09:59 sean-k-mooney then its can only happy because the migration succeeded
15:10:07 dansmith sean-k-mooney: well, not exactly
15:10:10 sean-k-mooney and we failed to remove it form the dict for some bug
15:10:23 bauzas dansmith: that's what I'm saying, we need to compare migrations with the dict
15:10:38 dansmith sean-k-mooney: what I'm saying is that if there's a reservation in the dict and there's an allocation, then it's either still in progress, has yet to be rolled back, or it succeeded in which case keeping the reservation in the dict doesn't prevent other stuff from working
15:10:38 bauzas if a dict key is not related to a current migration, then it's stale
15:11:01 dansmith once the instance is deleted or migrated away, healing based on the allocation would work itself out too
15:11:02 sean-k-mooney dansmith: ya i agree with that
15:11:20 sean-k-mooney and yes to the healing too
15:11:28 sean-k-mooney also bauzas if we ever use consumer_types
15:11:39 dansmith bauzas: a dict key for a completed migration where the instance is on our host is not incorrect, even if it is stale, and won't harm anything yeah?
15:11:40 sean-k-mooney you can actully tell if the allocation is a migration if you really want too
15:12:24 sean-k-mooney so if we start using consumer types for that and we do this based on the allcoation if you want to filter you will be able to in the future
15:12:30 bauzas dansmith: I wouldn't be super happy, as we should have removed it by https://review.opendev.org/c/openstack/nova/+/904209/16/nova/virt/libvirt/driver.py#11355
15:12:43 dansmith can we gmeet?
15:12:49 dansmith I feel like we're going in circles here
15:12:56 bauzas sure, I can
15:12:58 bauzas sec
15:13:18 bauzas https://meet.google.com/cyg-ucsv-xri
15:14:08 dansmith sean-k-mooney: are you able to join?
15:14:50 opendevreview Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824
15:17:34 opendevreview Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824
15:28:41 sean-k-mooney dansmith: sorry no i was in/am in anohter call
15:29:05 sean-k-mooney dansmith:i can join quickly now since this call is just ending
16:01:51 opendevreview Mateusz Klejn proposed openstack/nova master: Fixed: unit tests for force delete instance https://review.opendev.org/c/openstack/nova/+/909464
16:01:52 opendevreview Mateusz Klejn proposed openstack/nova master: Changed: adjust shelving for shelved image protection https://review.opendev.org/c/openstack/nova/+/909465
16:04:17 mnaser Good morning! https://review.opendev.org/c/openstack/nova/+/909228 + https://review.opendev.org/c/openstack/nova/+/909229 are ready for review today :)
16:42:42 melwitt when O
16:43:49 melwitt *when I've looked into nova-lvm job failures, what I have seen is that they fail due to the glance apache server responding to nova with a HTTP 502 bad gateway. meanwhile no errors in glance logs https://etherpad.opendev.org/p/nova-ci-failures-minimal#L11
16:44:22 melwitt not really sure what to do next to debug
16:45:17 sean-k-mooney melwitt: this maybe be related to glance-api being OOM killed
16:45:45 sean-k-mooney melwitt: artom was debuging failure where that happend so it would be good to confirm that is not whats happening fro the nova-lvm case
16:46:12 artom "debugging" is a strong word
16:46:22 artom I chased a red herring only to figure out that glanceclient does chunking for us
16:46:27 melwitt hm ok. I didn't think there was OOM killer in these but let me check
16:46:35 artom So I still no have no idea why glance-api gets oom-killed
16:47:04 melwitt oh, yeah there is some OOM kill in here
16:47:05 sean-k-mooney melwitt: there may not be but if there is no error in the glance api log i would first check for an OOM event
16:47:36 sean-k-mooney so we chatted about turning on zswap on nova-next
16:47:50 sean-k-mooney but since we are having OOM events in nova-lvm i might turn it on there too
16:48:03 sean-k-mooney or increase the swap there to 8G if its still at 4G
16:48:05 melwitt yeah, g-api. hm
16:49:04 melwitt Feb 15 13:59:43 np0036770057 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/system-devstack.slice/devstack@g-api.service,task=uwsgi,pid=86510,uid=1001
16:51:08 sean-k-mooney melwitt: so its runing with 4G of swap today
16:51:21 dansmith also, g-api might not be actually consuming lots of memory, it might just be that the OOM killer decided it was the best thing to kill to create some space
16:51:34 dansmith so look at the oom output to see if it was huge or not
16:55:55 opendevreview sean mooney proposed openstack/nova master: bump nova-lvm to use 8G of swap from 4G https://review.opendev.org/c/openstack/nova/+/909469
16:56:01 melwitt Feb 15 13:59:43 np0036770057 kernel: Out of memory: Killed process 86510 (uwsgi) total-vm:2290700kB, anon-rss:1733784kB, file-rss:6324kB, shmem-rss:32kB, UID:1001 pgtables:3732kB oom_score_adj:0 4707 so I guess 2.2 GB? not sure if that's typical
16:56:24 sean-k-mooney melwitt: dansmith ^ thats the swap bump and ill look at adding zswap to that and nova-next in a seperate patch later
16:57:22 sean-k-mooney melwitt: so it should be maybe 150-200mb at most normally
16:57:27 dansmith makes sense to try it on nova-lvm too so that seems okay
16:57:31 sean-k-mooney that looks like it holding a guest image in memory
16:58:13 sean-k-mooney that or we need to add an exicit python garbage collector call in the image upload path
16:58:29 melwitt Feb 15 13:59:43 np0036770057 kernel: Free swap = 0kB
16:58:32 dansmith the image upload path doesn't store the image
16:58:35 sean-k-mooney i.e. to egerly run the garbage collector after straming the image
16:59:28 dansmith it's possible that python keeps allocating new buffers for each thing we read and needs a break to do garbage collection, that's certainly possible
17:00:24 sean-k-mooney i belive we have hadd issues with that in eithe glance or cinder before
17:01:55 sean-k-mooney maybe im thinking of swift

Earlier   Later