| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-22 | |||
| 11:23:34 | kashyap | It'll help with bisecting. | |
| 11:24:08 | frickler | kashyap: well I replicated with a local devstack deployment. I can also try to just create an instance with virt-manager | |
| 11:24:34 | kashyap | Yes, that'll be more preferable, if possible. | |
| 11:25:24 | frickler | kashyap: ok, thx for your feedback so far | |
| 11:25:55 | kashyap | No problem. These TCG bugs (if it is indeed a bug) are hard to suss out. | |
| 11:37:05 | sean-k-mooney1 | frickler: you are seeing this just with a normal boot right | |
| 11:37:15 | sean-k-mooney1 | you dont need to boot many vms to trigger it | |
| 11:37:27 | sean-k-mooney1 | you are seeing the large memory usage with just a singel instnace | |
| 11:37:44 | sean-k-mooney1 | so this should not be hard to replicate right | |
| 11:37:53 | kashyap | sean-k-mooney1: I don't think it's just one boot. He said "CIs that try to start a larger no. of small instances" | |
| 11:37:57 | sean-k-mooney1 | frickler: out of interest do you have swap avaiable in these hosts | |
| 11:38:11 | kashyap | sean-k-mooney1: Good question ;-) The "ghost of swap"... | |
| 11:38:29 | sean-k-mooney1 | kashyap: its only an issue for cis because those small instance that used to fit nolonger do | |
| 11:38:56 | sean-k-mooney1 | kashyap: when i first spoke to frickler about this i think they mentioned it hapens for any small vms created | |
| 11:39:46 | sean-k-mooney | i.e. in ci the vm used to take say 128mb for ram is now using 600 so we cant run 4 of them in parallel anymore | |
| 11:40:36 | sean-k-mooney | so what im wondiring is in environment without swap are we seeing more resident memory usage | |
| 11:40:42 | frickler | sean-k-mooney: yes, I see this with a single instance. the CI example is just where we noticed it first, with jobs OOMing with tempest running parallel tests | |
| 11:41:35 | sean-k-mooney | frickler: we had a very weird customer issue where we saw python process have very large resident memory usage when no swap was presnet but when it was allocated they did not have high memory usage and also did not use any swap | |
| 11:42:27 | sean-k-mooney | it was like have swap avaiabel stop the memory allcoator preallcoating the memory | |
| 11:42:59 | frickler | hmm, indeed I have no swap on my test host. but we do have swap enabled on CI instances | |
| 11:43:25 | sean-k-mooney | ah ok i was going to say could you add a 1G swap file temporay to your devstack and see if it change behavior | |
| 11:43:35 | sean-k-mooney | if its in the ci then no point its not relatted | |
| 12:49:29 | bauzas | (late) reminder: final PTG day for nova sessions starting in 12 mins at https://www.openstack.org/ptg/rooms/newton | |
| 12:57:39 | sean-k-mooney | dansmith: for the health check you want me to support http over tcp ranther then just a tcp socket right. assuming yes im assuming if it make sense ot make this a real wsgi application or just use https://eventlet.net/doc/modules/wsgi.html#eventlet.wsgi.server to call a binay specifci health check function | |
| 12:58:17 | dansmith | sean-k-mooney: yes http and I'd keep it uuber simple (so the latter) | |
| 12:58:45 | sean-k-mooney | ok | |
| 13:00:24 | sean-k-mooney | brb just going to make a coffee and ill join | |
| 13:02:23 | bauzas | nova session started | |
| 13:03:08 | dansmith | sean-k-mooney: out of curiosity, does haproxy support some sort of bare tcp socket health check/ | |
| 13:03:12 | dansmith | I would kinda expect not | |
| 13:03:44 | dansmith | I thought even systemd wanted http, but can use a script too | |
| 13:10:03 | sean-k-mooney | dansmith: i think i tcan but orginaly haproxy was not part of my orignal usecases | |
| 13:10:14 | dansmith | okay | |
| 13:11:04 | sean-k-mooney | i was orginally thining of this as a camand/contol interface with comand objects echanged more like the rpc bus | |
| 13:11:39 | sean-k-mooney | with nc as or nova-manage as the cli | |
| 13:12:21 | gibi | sean-k-mooney: fyi, sriov agent only unrealiably sends vif plug for VFs, as it polls the hypervisor. If the unplug/plug is fast enough then the agent might miss the state when the device was down | |
| 13:12:42 | sean-k-mooney | gibi: more fun | |
| 13:12:46 | sean-k-mooney | ok | |
| 13:12:52 | gibi | it is fun all the way down :D | |
| 13:13:04 | sean-k-mooney | so we might want to only wait for vnic_type=normal then | |
| 13:13:15 | gibi | it seems soo | |
| 13:13:53 | sean-k-mooney | long term we really do need to fix this interface and enforce a stricter contract | |
| 13:14:05 | sean-k-mooney | rather then guessing cause there are so many factors to consider | |
| 13:14:58 | gibi | yes, we need neturon to be either enforce that events always sent, or declare in the port what event can be expected. There is no way nova can maintain a sane mapping alone | |
| 13:25:58 | bauzas | dmitriis: saw the chat ? | |
| 13:26:13 | dmitriis | bauzas: looking, 1 sec | |
| 13:26:21 | bauzas | dmitriis: I'm about to propose to postpone your topic after 3pm UTC | |
| 13:27:16 | dmitriis | bauzas: got a conflict at 3PM but I can make it work | |
| 13:27:26 | bauzas | dmitriis: maybe later then ? | |
| 13:27:43 | bauzas | the idea is just to avoid discussing your stuff *before* :) | |
| 13:28:11 | dmitriis | bauzas: let's do it at 3PM, later is more complicated :^) | |
| 13:28:27 | bauzas | ack, moving your topic then :) | |
| 13:28:52 | dmitriis | bauzas: ack, ty for pinging | |
| 13:33:57 | stephenfin | Is it just me or is tbarron's sound clipping real bad? I can understand him though (i.e. can be fixed later) | |
| 13:38:28 | tbarron | stephenfin: It may be from my end, sorry. Yesterday zoom was having trouble with my rural location. | |
| 13:39:01 | stephenfin | tbarron: nw, I was just concerned something was broken on my end :) I could understand everything just fine | |
| 13:39:19 | tbarron | cool | |
| 13:43:09 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove models that were moved to the API database https://review.opendev.org/c/openstack/nova/+/812149 | |
| 13:43:10 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove nova-network models https://review.opendev.org/c/openstack/nova/+/812151 | |
| 13:43:10 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove models for removed services, features https://review.opendev.org/c/openstack/nova/+/812150 | |
| 13:45:59 | opendevreview | Balazs Gibizer proposed openstack/nova stable/pike: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813437 | |
| 13:56:59 | kashyap | clarkb: I'm innundated with a few things; I will reply to your email on the list on Monday. Hope that's okay | |
| 15:15:59 | clarkb | kashyap: yup no worries | |
| 15:16:02 | clarkb | have a good weekend | |
| 17:04:44 | bauzas | dansmith: dunno if you're fancy happy joining us, but we're discussing a topic your knowledge could be helpful : instance v3.0 object bump | |
| 17:04:58 | dansmith | sorry, I'm tied up | |
| 17:05:01 | bauzas | or you're stuck in the TC meeting | |
| 17:05:06 | bauzas | heh, no worries | |
| 17:05:15 | dansmith | I definitely have opinions | |
| 17:05:18 | bauzas | dansmith: notes on the etherpad could be appreciated tho :) | |
| 17:05:28 | bauzas | https://etherpad.opendev.org/p/nova-yoga-ptg L646 | |
| 17:26:43 | sean-k-mooney | https://etherpad.opendev.org/p/nova-yoga-ptg-backup | |
| 17:26:54 | sean-k-mooney | no colors but there is a snapshot ^ | |
| 17:28:04 | clarkb | I just restored it to the version that bauzas identified as good (the original etherpad url I mean) | |
| 17:28:22 | bauzas | yeah \o/ | |
| 17:28:24 | sean-k-mooney | yep that looks ok | |
| 17:28:39 | bauzas | I just wrote our last conclusions | |
| 17:28:43 | bauzas | we can call it a wrap | |
| 17:28:51 | sean-k-mooney | clarkb++ thanks | |
| 17:28:57 | bauzas | sean-k-mooney: just copy it again, so we have a backup | |
| 17:29:22 | sean-k-mooney | i just did an plain text export form the timeline and imported it again in adifferent page | |
| 17:29:31 | bauzas | yeah that works | |
| 17:29:41 | sean-k-mooney | but ill try that in html form and see if it can keep colours | |
| 17:29:44 | bauzas | we have the highlights with the history | |
| 17:29:57 | bauzas | so nothing is technically lost | |
| 17:30:00 | sean-k-mooney | in etherpad format it goes to the latest point in history not the one you have selected in the timeline | |
| 17:31:07 | sean-k-mooney | ya so html format does not keep colors either | |
| 17:31:18 | sean-k-mooney | but we have teh backup in any case | |
| 17:31:28 | sean-k-mooney | and the orginal is restored so we are good | |
| 17:32:24 | bauzas | yup | |
| 17:32:33 | bauzas | on that note, /me calls it a week | |
| 17:32:36 | bauzas | \o | |
| 17:32:50 | gibi | me too | |
| 17:32:52 | gibi | o/ | |
| 17:33:00 | bauzas | I'm just sad to hear that the next PTG will still be virtual, but that's life | |
| 17:33:27 | bauzas | I'm just exhausted and I miss our whiteboards and hallway discussions | |
| 17:34:00 | bauzas | but I'll open a beer to enjoy the last day of the PTG as if it was physical :) | |
| 19:22:37 | mnaser_ | hm | |
| 19:22:52 | mnaser_ | anyone would have an idea as to why metadata service seems to be stalling / very slow | |
| 19:23:25 | mnaser_ | 2021-10-22 19:23:13.422 11 INFO nova.metadata.wsgi.server [req-f65469d1-4082-4b9e-ab3b-3c710c1f0366 - - - - -] 10.30.107.187,10.101.2.141 "GET /latest/meta-data/block-device-mapping/root HTTP/1.1" status: 200 len: 148 time: 25.8355188 | |