| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-21 | |||
| 15:42:57 | sean-k-mooney | ildikov: the history is we used to track priority the nova wiki, then that moved to the spec repo about 10 years ago, then we decied that was too heavy weight and we moved to "runways" in etherpad 6-8 years ago, then that hadd too much load on the ptl to we went to review priority lable about 3 years ago and back to a lighter weigher etherpad last year | |
| 15:43:39 | sean-k-mooney | over all that time specs/bluepirnts/bug usage has not changed. just where we keep the list of items ready to review and the prioriteis to review | |
| 15:44:16 | ildikov | I understand, it seems the team went through quite a few rounds of simplifications | |
| 15:44:54 | sean-k-mooney | as the team size got smaller and smaller and contonutions dropped off the cost of the more complex approches just didnt make sense | |
| 15:45:06 | sean-k-mooney | *contibutions | |
| 15:45:18 | ildikov | understandable | |
| 15:46:01 | ildikov | even with a bigger team size, I'm always a fan of simpler processes, when possible | |
| 15:46:01 | sean-k-mooney | https://etherpad.opendev.org/p/nova-2025.1-status so we call it the status etherpad | |
| 15:46:36 | sean-k-mooney | basically because it a light weight way to highlight the progress or lack there of of somethign that peropsl are activly working on | |
| 15:46:48 | sean-k-mooney | but its in etherpad so that anyone can edit it | |
| 15:47:24 | ildikov | oh nice! | |
| 15:47:53 | ildikov | it looks like a good overview of what the team is tracking throughout the release cycle | |
| 15:48:20 | ildikov | gives the opportunity to do a bit of project management, but without being too heavy on the tooling | |
| 15:48:54 | sean-k-mooney | yep this is the dalmaition one https://etherpad.opendev.org/p/nova-dalmatian-status and caracal https://etherpad.opendev.org/p/nova-caracal-status | |
| 15:49:14 | sean-k-mooney | its useful wehn lookign for cycle highlights or releasenotes too | |
| 15:49:38 | sean-k-mooney | that why we have the complete bugs and features section | |
| 15:50:01 | sean-k-mooney | you dont need ot have your review listed on the eptherpad for it to get reviewed or merged | |
| 15:50:24 | sean-k-mooney | it just help publicise it so its simpler for reviewers to find | |
| 15:52:02 | ildikov | I love that! | |
| 15:52:51 | ildikov | sometimes it's hard to fish for cycle highlights, etc, so it's nice y'all have the pointers in the etherpad | |
| 15:53:30 | ildikov | I have a meeting coming up, so will step away from IRC a bit | |
| 15:53:46 | ildikov | I don't know enough about the processes to update the contributor docs myself | |
| 15:54:20 | ildikov | what other ways can I help to make improvements and help new and casual contributors looking into Nova? | |
| 15:57:53 | sean-k-mooney | ildikov: its litrally just a normal review. so if you think you can rupdate it to help in anyway and have time to submit a reivew to the docs then please do, you could track it as a docs bug if you wanted too. | |
| 15:58:47 | sean-k-mooney | creating a bug for this and addign it to the meeting adgenda (the open discussion section) woudl proably be a good way forward | |
| 15:59:11 | sean-k-mooney | im not sure that ill have time to go do the doc update but perhaps that is somethign bauzas can help drive as ptl | |
| 16:00:14 | ildikov | sean-k-mooney: Sounds good. I'll create a docs bug later today, that is easy to do. We can then figure out next steps from there. | |
| 16:00:34 | ildikov | sean-k-mooney: thank you for the nice chat and the guidance! | |
| 16:00:49 | sean-k-mooney | no worries, enjoy your meeting :P | |
| 16:01:28 | sean-k-mooney | ill revert https://review.opendev.org/c/openstack/project-config/+/787523 in the interim | |
| 16:02:05 | sean-k-mooney | a striat revert wont work becasue we updated them in https://review.opendev.org/c/openstack/project-config/+/837595 but it wont take long to create a patch to do this | |
| 16:29:09 | bauzas | ildikov: sean-k-mooney : sorry I was awfully busy those days (and I don't wanna speak about last weeks-months) | |
| 16:29:37 | bauzas | that's still a TODO on myself to rejuvanate the docs but I'll try to make it soon as I can | |
| 16:52:39 | ildikov | bauzas: no worries | |
| 16:52:50 | melwitt | sean-k-mooney: ok so with an invalid config for the resource list, the error is raised during startup but it is not causing the nova-api process to exit https://paste.openstack.org/show/b7s0zWslkXznihw1Y3yB/ | |
| 16:53:16 | ildikov | bauzas: is it helpful to you if I create the docs bug that we talked about with Sean earlier? | |
| 16:57:37 | melwitt | sean-k-mooney: the oslo.service based processes do exit with a bad config (like nova-conductor). so the wsgi app case needs a bit more to cause an exit of the process | |
| 17:13:28 | opendevreview | Merged openstack/nova master: nova-manage: Add flavor scanning to migrate_to_unified_limits https://review.opendev.org/c/openstack/nova/+/924110 | |
| 17:15:33 | sean-k-mooney | is the wsgi app that is reloading | |
| 17:22:26 | melwitt | you're saying the wsgi app is reloading? | |
| 17:24:51 | sean-k-mooney | i was asking if it was | |
| 17:25:08 | sean-k-mooney | so is it not exeitign becuase its failign but reloading on a new request ectra | |
| 17:25:42 | sean-k-mooney | for example if you run it under nova-api console script it proably will exit like the other console scripts | |
| 17:26:52 | melwitt | I wonder if this is a matter of wsgi app config like "exit-on-reload = false" | |
| 17:27:09 | melwitt | yes I believe so | |
| 17:28:04 | sean-k-mooney | melwitt: so to me that past loks like the app exited | |
| 17:28:10 | sean-k-mooney | bhttps://paste.openstack.org/show/b7s0zWslkXznihw1Y3yB/ | |
| 17:28:30 | sean-k-mooney | but i woudl guess apache/uwsgi will try and reload it on the next request | |
| 17:28:32 | melwitt | yeah ... so why the process stays? I don't know much about this stuff | |
| 17:28:42 | melwitt | oh | |
| 17:28:45 | sean-k-mooney | which process is staying | |
| 17:28:56 | sean-k-mooney | we have 2 apache and then uwsign behind it | |
| 17:29:04 | melwitt | nova-apiuWSGI master and the workers | |
| 17:29:45 | melwitt | nova-apiuWSGI worker 1 nova-apiuWSGI worker 2 nova-apiuWSGI worker 3 | |
| 17:30:04 | sean-k-mooney | so i think those are the python interpreteres | |
| 17:30:16 | sean-k-mooney | that will be used to load the app on requets | |
| 17:31:21 | melwitt | yeah.. but we would want the interpreter to exit if the config raises ValueError or no? | |
| 17:42:44 | sean-k-mooney | i think that is the same behaivor we have for any other early check | |
| 17:42:50 | sean-k-mooney | so to me this is not unexpected | |
| 17:43:18 | melwitt | ok. yeah I'm slowly realizing this too | |
| 17:43:46 | sean-k-mooney | we may even have a exisitng "bug" for that as this feels vagly like familar ground | |
| 17:43:52 | melwitt | I was concerned if the app failing to load but the interpreter continues to run would cause any problems for an operator knowing that nova-api did not start properly or what | |
| 17:44:22 | sean-k-mooney | so it may | |
| 17:44:29 | sean-k-mooney | what is the status of the systemd service | |
| 17:44:43 | melwitt | it still says active and running :\ | |
| 17:44:46 | sean-k-mooney | (when i say it may i think this is already the case) | |
| 17:45:19 | sean-k-mooney | ack so without a real healthcheck i dont think this woudl be visable | |
| 17:45:31 | sean-k-mooney | but i woudl assuem the micro version endpoint does not work | |
| 17:45:42 | sean-k-mooney | so if you were actully monitoring the api by calling it | |
| 17:45:48 | sean-k-mooney | then you would see it as down | |
| 17:45:52 | sean-k-mooney | proably with a 503 | |
| 17:46:46 | melwitt | yeah, that makes sense | |
| 17:49:40 | melwitt | yeah when I try to curl GET it hangs for a little bit then returns 500 | |
| 17:50:16 | sean-k-mooney | ack ok i was not sure if it would be a 500 or 503 | |
| 17:52:22 | melwitt | ok, so it sounds like there isn't more to do (in that this is a universal thing across the board that we have not created a way to deal with) | |
| 17:52:24 | sean-k-mooney | https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html#preforking-vs-lazy-apps-vs-lazy | |
| 17:52:31 | sean-k-mooney | so i think this is also relevent | |
| 17:52:51 | sean-k-mooney | no i dont thik there is anything we can do | |
| 17:53:04 | melwitt | "lazy is discouraged!" | |
| 17:53:08 | sean-k-mooney | there might be a uwsgi option like you said to make it exit | |
| 17:53:16 | melwitt | in our config lazy-apps = true heh | |
| 17:53:24 | sean-k-mooney | lol | |
| 17:53:37 | melwitt | yeah I found some things while googling like the lazy-apps setting, need-app setting, etc | |
| 17:53:59 | sean-k-mooney | oh so lazy not lazy-app is discussraged | |
| 17:54:21 | sean-k-mooney | """Remember: lazy-apps is different from lazy, the first one only instructs uWSGI to load the application one time per worker, while the second is more invasive (and generally discouraged) as it changes a lot of internal defaults.""" | |
| 17:54:53 | melwitt | oh ok | |
| 17:54:58 | sean-k-mooney | so we are not using lazy so its fine | |
| 17:55:12 | sean-k-mooney | but also this si configurable by the deployer | |
| 17:55:17 | sean-k-mooney | so they can make there own choices | |
| 17:55:27 | melwitt | I see | |
| 17:55:30 | sean-k-mooney | melwitt: the lifecycle under appache is also differnt | |
| 17:55:36 | sean-k-mooney | i.e. mod_wsgi | |
| 17:55:56 | melwitt | that makes sense | |
| 17:56:12 | sean-k-mooney | in devstack we run the applicaiton under uwsgin and use apache mod_proxy to allow us to enabel tls | |
| 17:57:56 | sean-k-mooney | we shoudl change to "Zerg mode¶" just because i like the name | |
| 17:58:19 | sean-k-mooney | it even brings topics liek "The Zerg Dance" | |
| 17:58:31 | melwitt | oh yeah, that has to be good right | |
| 17:59:08 | sean-k-mooney | im not a starcraft person but a zerg rush is a thing. it totally cant be bad in any way | |
| 17:59:30 | melwitt | I'm not either but I do what the zerg rush is | |
| 17:59:58 | melwitt | haha yeah | |