Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-02-19
15:12:24 sean-k-mooney so if we start using consumer types for that and we do this based on the allcoation if you want to filter you will be able to in the future
15:12:30 bauzas dansmith: I wouldn't be super happy, as we should have removed it by https://review.opendev.org/c/openstack/nova/+/904209/16/nova/virt/libvirt/driver.py#11355
15:12:43 dansmith can we gmeet?
15:12:49 dansmith I feel like we're going in circles here
15:12:56 bauzas sure, I can
15:12:58 bauzas sec
15:13:18 bauzas https://meet.google.com/cyg-ucsv-xri
15:14:08 dansmith sean-k-mooney: are you able to join?
15:14:50 opendevreview Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824
15:17:34 opendevreview Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824
15:28:41 sean-k-mooney dansmith: sorry no i was in/am in anohter call
15:29:05 sean-k-mooney dansmith:i can join quickly now since this call is just ending
16:01:51 opendevreview Mateusz Klejn proposed openstack/nova master: Fixed: unit tests for force delete instance https://review.opendev.org/c/openstack/nova/+/909464
16:01:52 opendevreview Mateusz Klejn proposed openstack/nova master: Changed: adjust shelving for shelved image protection https://review.opendev.org/c/openstack/nova/+/909465
16:04:17 mnaser Good morning! https://review.opendev.org/c/openstack/nova/+/909228 + https://review.opendev.org/c/openstack/nova/+/909229 are ready for review today :)
16:42:42 melwitt when O
16:43:49 melwitt *when I've looked into nova-lvm job failures, what I have seen is that they fail due to the glance apache server responding to nova with a HTTP 502 bad gateway. meanwhile no errors in glance logs https://etherpad.opendev.org/p/nova-ci-failures-minimal#L11
16:44:22 melwitt not really sure what to do next to debug
16:45:17 sean-k-mooney melwitt: this maybe be related to glance-api being OOM killed
16:45:45 sean-k-mooney melwitt: artom was debuging failure where that happend so it would be good to confirm that is not whats happening fro the nova-lvm case
16:46:12 artom "debugging" is a strong word
16:46:22 artom I chased a red herring only to figure out that glanceclient does chunking for us
16:46:27 melwitt hm ok. I didn't think there was OOM killer in these but let me check
16:46:35 artom So I still no have no idea why glance-api gets oom-killed
16:47:04 melwitt oh, yeah there is some OOM kill in here
16:47:05 sean-k-mooney melwitt: there may not be but if there is no error in the glance api log i would first check for an OOM event
16:47:36 sean-k-mooney so we chatted about turning on zswap on nova-next
16:47:50 sean-k-mooney but since we are having OOM events in nova-lvm i might turn it on there too
16:48:03 sean-k-mooney or increase the swap there to 8G if its still at 4G
16:48:05 melwitt yeah, g-api. hm
16:49:04 melwitt Feb 15 13:59:43 np0036770057 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/system-devstack.slice/devstack@g-api.service,task=uwsgi,pid=86510,uid=1001
16:51:08 sean-k-mooney melwitt: so its runing with 4G of swap today
16:51:21 dansmith also, g-api might not be actually consuming lots of memory, it might just be that the OOM killer decided it was the best thing to kill to create some space
16:51:34 dansmith so look at the oom output to see if it was huge or not
16:55:55 opendevreview sean mooney proposed openstack/nova master: bump nova-lvm to use 8G of swap from 4G https://review.opendev.org/c/openstack/nova/+/909469
16:56:01 melwitt Feb 15 13:59:43 np0036770057 kernel: Out of memory: Killed process 86510 (uwsgi) total-vm:2290700kB, anon-rss:1733784kB, file-rss:6324kB, shmem-rss:32kB, UID:1001 pgtables:3732kB oom_score_adj:0 4707 so I guess 2.2 GB? not sure if that's typical
16:56:24 sean-k-mooney melwitt: dansmith ^ thats the swap bump and ill look at adding zswap to that and nova-next in a seperate patch later
16:57:22 sean-k-mooney melwitt: so it should be maybe 150-200mb at most normally
16:57:27 dansmith makes sense to try it on nova-lvm too so that seems okay
16:57:31 sean-k-mooney that looks like it holding a guest image in memory
16:58:13 sean-k-mooney that or we need to add an exicit python garbage collector call in the image upload path
16:58:29 melwitt Feb 15 13:59:43 np0036770057 kernel: Free swap = 0kB
16:58:32 dansmith the image upload path doesn't store the image
16:58:35 sean-k-mooney i.e. to egerly run the garbage collector after straming the image
16:59:28 dansmith it's possible that python keeps allocating new buffers for each thing we read and needs a break to do garbage collection, that's certainly possible
17:00:24 sean-k-mooney i belive we have hadd issues with that in eithe glance or cinder before
17:01:55 sean-k-mooney maybe im thinking of swift
17:01:57 sean-k-mooney https://codesearch.opendev.org/?q=gc.collect&i=nope&literal=nope&files=&excludeFiles=&repos=
17:02:11 sean-k-mooney there are a few project that have an explcit gc.collect() call
17:03:00 sean-k-mooney hum although mainly in test code
17:03:11 dansmith there was also something about python never deflating the heap, which I think gorka can tell us about
17:03:49 sean-k-mooney ya thats what i was recalling. speaking to gorka about the agents growing there memory usage over time
17:03:50 dansmith but I would think that wouldn't be our problem here since we should be good to re-use those portions of the heap over and over, especially with allocations of exactly the same size each time
17:04:35 sean-k-mooney in anycase bumping to 8G of swap will mitigate the issue for now
17:05:26 opendevreview Jay Faulkner proposed openstack/nova-specs master: Sharding: More detailed test plan https://review.opendev.org/c/openstack/nova-specs/+/908673
17:05:52 JayF https://review.opendev.org/c/openstack/nova-specs/+/908673 Heads up, I'm going to be implementing the plan laid out here this week; please make loud noises quickly if it's unacceptable :D
17:07:44 sean-k-mooney ack for 2024.2 or are you still hoping to merge the shard stuff for 2024.1
17:08:05 sean-k-mooney i guess for 2024.1
17:08:36 sean-k-mooney if you can get the testing working in time before FF i guess that is fine
17:09:56 sean-k-mooney for me the minium i think we need is a 2 node job with to set of vms with 2 ironnic compuge service partitioning the vms into shards within the same conducotr group
17:11:59 sean-k-mooney JayF: having tempest test that acttuly assert the toplogy is proably not in the scope of tempest. it would be in the scope of whitebox or could be done via a differnt tempest plugin
17:12:13 gibi Uggla: i responded in the manila series
17:12:17 JayF sean-k-mooney: It was going to be in ironic-tempest-plugin
17:12:27 sean-k-mooney ack thats proably ok then
17:12:33 JayF Two node job is not something we do in Ironic really
17:12:51 JayF And we are working to slim down our job set; try to get more cases in less VMs
17:13:14 sean-k-mooney JayF: can you deploy 2 nova-compute services on a singel node
17:13:15 JayF so adding more extensive jobs works against that -- but if we validate the resource pool only contains sharded nodes, we're golden (and already have code to do this in devstack that I can lift)
17:13:29 JayF sean-k-mooney: We have zero tests where we do >1 configured nova-compute in Ironic.
17:13:40 JayF sean-k-mooney: and the devstack machinery does not support it
17:14:09 sean-k-mooney right so that is why i was expecting to have a 2 node job
17:14:24 sean-k-mooney each runnign a single nova-compute with ironic
17:14:36 sean-k-mooney but ok we can get a similar effect with what your suggesting
17:14:44 sean-k-mooney it just sound like much more work
17:15:12 JayF No, much much less work. I've already looked into this with Julia and we have a good amount of confidence in this approach.
17:15:14 sean-k-mooney well not nessiarly much more just more then i would have expected
17:15:42 sean-k-mooney ack did you have time to respine the patches to nova for the pep8 error
17:15:46 JayF I just don't want to do it and someone come along and say "not good enough" :D
17:15:53 JayF yeah, I just did
17:15:57 sean-k-mooney cool
17:16:01 JayF right before talking in here because I went to link it and saw the V-1 :D
17:17:16 dansmith I guess I'm confused about how we can know the partitioning works with only one node?
17:17:18 sean-k-mooney ok i put the spec update on teh same gerrit topic as teh code
17:17:21 sean-k-mooney "ironic-shards"
17:17:53 sean-k-mooney dansmith: the job will create more then 1 ironic "vm" and only tag a subset of them with the shard key
17:18:05 sean-k-mooney dansmith: so the job will need to assert that not all the nodes in the ironic node list
17:18:14 sean-k-mooney show up in the RPs
17:18:16 JayF Yeah, we'll end up with e.g. 10 nodes configured (fake drivers, no backend), and only 10-N of them configured with a shard
17:18:25 JayF once the good nodes show up, we start a timer and look for any bad nodes
17:18:25 sean-k-mooney and then ones that do march the expected shard key
17:18:28 dansmith so just making sure that the one nova-ironic we have doesn't pick up the other fake resources outside the configured shard?
17:18:31 JayF exactly
17:19:23 dansmith okay the spec doesn't say ten.. so to be clear, it will be a bunch of fake ironic resources and not just .. one real and one fake, right?
17:19:38 dansmith because the existing hash ring thing will get 1/2 right half the time
17:19:41 JayF with fake nodes, the number can be as many as we want
17:19:44 sean-k-mooney which works and covers the gap we had. it just not how i would have approched it
17:19:53 JayF so if we need a large number of "N" to have confidence, it's trivial to do it
17:20:02 JayF where large is like, small two digit number
17:20:26 sean-k-mooney JayF: we dont but it would have been nice to have more then one nova-compute in the same ironic conductor group with diffent shard keys
17:20:28 dansmith okay I was hoping for better, tbh, but I guess it's better than nothing

Earlier   Later