| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-21 | |||
| 16:52:50 | melwitt | sean-k-mooney: ok so with an invalid config for the resource list, the error is raised during startup but it is not causing the nova-api process to exit https://paste.openstack.org/show/b7s0zWslkXznihw1Y3yB/ | |
| 16:53:16 | ildikov | bauzas: is it helpful to you if I create the docs bug that we talked about with Sean earlier? | |
| 16:57:37 | melwitt | sean-k-mooney: the oslo.service based processes do exit with a bad config (like nova-conductor). so the wsgi app case needs a bit more to cause an exit of the process | |
| 17:13:28 | opendevreview | Merged openstack/nova master: nova-manage: Add flavor scanning to migrate_to_unified_limits https://review.opendev.org/c/openstack/nova/+/924110 | |
| 17:15:33 | sean-k-mooney | is the wsgi app that is reloading | |
| 17:22:26 | melwitt | you're saying the wsgi app is reloading? | |
| 17:24:51 | sean-k-mooney | i was asking if it was | |
| 17:25:08 | sean-k-mooney | so is it not exeitign becuase its failign but reloading on a new request ectra | |
| 17:25:42 | sean-k-mooney | for example if you run it under nova-api console script it proably will exit like the other console scripts | |
| 17:26:52 | melwitt | I wonder if this is a matter of wsgi app config like "exit-on-reload = false" | |
| 17:27:09 | melwitt | yes I believe so | |
| 17:28:04 | sean-k-mooney | melwitt: so to me that past loks like the app exited | |
| 17:28:10 | sean-k-mooney | bhttps://paste.openstack.org/show/b7s0zWslkXznihw1Y3yB/ | |
| 17:28:30 | sean-k-mooney | but i woudl guess apache/uwsgi will try and reload it on the next request | |
| 17:28:32 | melwitt | yeah ... so why the process stays? I don't know much about this stuff | |
| 17:28:42 | melwitt | oh | |
| 17:28:45 | sean-k-mooney | which process is staying | |
| 17:28:56 | sean-k-mooney | we have 2 apache and then uwsign behind it | |
| 17:29:04 | melwitt | nova-apiuWSGI master and the workers | |
| 17:29:45 | melwitt | nova-apiuWSGI worker 1 nova-apiuWSGI worker 2 nova-apiuWSGI worker 3 | |
| 17:30:04 | sean-k-mooney | so i think those are the python interpreteres | |
| 17:30:16 | sean-k-mooney | that will be used to load the app on requets | |
| 17:31:21 | melwitt | yeah.. but we would want the interpreter to exit if the config raises ValueError or no? | |
| 17:42:44 | sean-k-mooney | i think that is the same behaivor we have for any other early check | |
| 17:42:50 | sean-k-mooney | so to me this is not unexpected | |
| 17:43:18 | melwitt | ok. yeah I'm slowly realizing this too | |
| 17:43:46 | sean-k-mooney | we may even have a exisitng "bug" for that as this feels vagly like familar ground | |
| 17:43:52 | melwitt | I was concerned if the app failing to load but the interpreter continues to run would cause any problems for an operator knowing that nova-api did not start properly or what | |
| 17:44:22 | sean-k-mooney | so it may | |
| 17:44:29 | sean-k-mooney | what is the status of the systemd service | |
| 17:44:43 | melwitt | it still says active and running :\ | |
| 17:44:46 | sean-k-mooney | (when i say it may i think this is already the case) | |
| 17:45:19 | sean-k-mooney | ack so without a real healthcheck i dont think this woudl be visable | |
| 17:45:31 | sean-k-mooney | but i woudl assuem the micro version endpoint does not work | |
| 17:45:42 | sean-k-mooney | so if you were actully monitoring the api by calling it | |
| 17:45:48 | sean-k-mooney | then you would see it as down | |
| 17:45:52 | sean-k-mooney | proably with a 503 | |
| 17:46:46 | melwitt | yeah, that makes sense | |
| 17:49:40 | melwitt | yeah when I try to curl GET it hangs for a little bit then returns 500 | |
| 17:50:16 | sean-k-mooney | ack ok i was not sure if it would be a 500 or 503 | |
| 17:52:22 | melwitt | ok, so it sounds like there isn't more to do (in that this is a universal thing across the board that we have not created a way to deal with) | |
| 17:52:24 | sean-k-mooney | https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html#preforking-vs-lazy-apps-vs-lazy | |
| 17:52:31 | sean-k-mooney | so i think this is also relevent | |
| 17:52:51 | sean-k-mooney | no i dont thik there is anything we can do | |
| 17:53:04 | melwitt | "lazy is discouraged!" | |
| 17:53:08 | sean-k-mooney | there might be a uwsgi option like you said to make it exit | |
| 17:53:16 | melwitt | in our config lazy-apps = true heh | |
| 17:53:24 | sean-k-mooney | lol | |
| 17:53:37 | melwitt | yeah I found some things while googling like the lazy-apps setting, need-app setting, etc | |
| 17:53:59 | sean-k-mooney | oh so lazy not lazy-app is discussraged | |
| 17:54:21 | sean-k-mooney | """Remember: lazy-apps is different from lazy, the first one only instructs uWSGI to load the application one time per worker, while the second is more invasive (and generally discouraged) as it changes a lot of internal defaults.""" | |
| 17:54:53 | melwitt | oh ok | |
| 17:54:58 | sean-k-mooney | so we are not using lazy so its fine | |
| 17:55:12 | sean-k-mooney | but also this si configurable by the deployer | |
| 17:55:17 | sean-k-mooney | so they can make there own choices | |
| 17:55:27 | melwitt | I see | |
| 17:55:30 | sean-k-mooney | melwitt: the lifecycle under appache is also differnt | |
| 17:55:36 | sean-k-mooney | i.e. mod_wsgi | |
| 17:55:56 | melwitt | that makes sense | |
| 17:56:12 | sean-k-mooney | in devstack we run the applicaiton under uwsgin and use apache mod_proxy to allow us to enabel tls | |
| 17:57:56 | sean-k-mooney | we shoudl change to "Zerg mode¶" just because i like the name | |
| 17:58:19 | sean-k-mooney | it even brings topics liek "The Zerg Dance" | |
| 17:58:31 | melwitt | oh yeah, that has to be good right | |
| 17:59:08 | sean-k-mooney | im not a starcraft person but a zerg rush is a thing. it totally cant be bad in any way | |
| 17:59:30 | melwitt | I'm not either but I do what the zerg rush is | |
| 17:59:58 | melwitt | haha yeah | |
| 18:00:27 | sean-k-mooney | i mean the pros secation specaks for its self "potentially the silver bullet" | |
| 18:00:41 | melwitt | lol | |
| 18:00:46 | sean-k-mooney | we never get to have those and shuold embrace them when they happen | |
| 18:01:30 | sean-k-mooney | this is why software will never help with the warewolfe apocalapes | |
| 18:01:43 | melwitt | 😆 | |
| 18:02:30 | melwitt | only because of this | |
| 18:03:03 | sean-k-mooney | its ironic becase most software engineers are clearly vampires, outside what is that :) | |
| 18:03:51 | sean-k-mooney | ok im goign to get somethign to drink and ill see if i can review your chanage when i get back | |
| 18:04:40 | melwitt | haha yeah | |
| 18:05:16 | melwitt | kk | |
| 18:07:19 | melwitt | bleh, recheck is already going to fail with a couple of rando errors | |
| 18:09:36 | melwitt | NoValidHost and urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='200.225.47.58', port=443): Read timed out. (read timeout=90) | |
| 18:29:31 | sean-k-mooney | thats presumable the placement call | |
| 18:29:41 | sean-k-mooney | melwitt: instead of recheckign it hit the rebase button | |
| 18:30:35 | sean-k-mooney | that will create a new revisison and kick the jobs out of the queue automatiically | |
| 18:48:31 | opendevreview | Merged openstack/placement master: Drop db migration tool https://review.opendev.org/c/openstack/placement/+/932324 | |
| 19:06:06 | yosef | sean-k-mooney: I dont get the by design part, would it create inconsistency if we add a command or an api to re-sync to the changed flavor? | |
| 19:08:10 | sean-k-mooney | yosef: yes, so flavor are ment to be immutable | |
| 19:08:34 | sean-k-mooney | the only reason you can change the extra specs is to allow you to create it adn add them wihtout doing it with a single command | |
| 19:08:51 | sean-k-mooney | but once they are used the expectation is that you will not modify them | |
| 19:08:55 | opendevreview | Merged openstack/nova master: Amend ShareMappingStatus due to asynchronous call https://review.opendev.org/c/openstack/nova/+/908864 | |
| 19:09:05 | opendevreview | Merged openstack/nova master: Amend DB model add a unique constraint. https://review.opendev.org/c/openstack/nova/+/912518 | |
| 19:09:14 | sean-k-mooney | yosef: the intented workflow is you create new flavor and retire the old one when you need to make changes | |
| 19:12:52 | yosef | sean-k-mooney: we have libvirt network and qos quotas as legacy on our production with thosands of VMs, we are migrating to new neutron qos api by changing database by raw sql queries, as we can not resize every one and take down time, I thought if we could add something so anybody else would have a simpler migration | |
| 19:13:50 | sean-k-mooney | its somethign that can be discssed nad could be done, but it would still be considerd bad pratice | |
| 19:14:06 | sean-k-mooney | that does not mean it cant be facilated in a more graceful way | |
| 19:14:37 | sean-k-mooney | unst that its a get our of jail card rather then how you should plan to operate yoru cloud on a day to day basis | |
| 19:18:11 | yosef | Yep that is the end of discussion, I wanted to know if it is a good Idea and an issue I could contribute on, we have almost done on our migration... | |
| 19:27:51 | sean-k-mooney | it can cause issue withthe exisitng instnace. i.e. if you change some parmaters the curent placmenet if the isntance may become inviald based on the other factors consitred by the schduler | |
| 19:28:30 | sean-k-mooney | also if you dont do it correctly and update both the request spec and the copy in the cell db in the instance_extra table to have the same content then that will cause issues | |
| 19:28:50 | sean-k-mooney | yosef: your not the first operator to do this however | |
| 21:55:23 | sean-k-mooney | o/ night folks | |
| 23:02:45 | opendevreview | melanie witt proposed openstack/nova master: WIP Retry _schedule if AllocationUpdatedFailed https://review.opendev.org/c/openstack/nova/+/935988 | |
| 23:50:04 | ildikov | sean-k-mooney: bauzas: here's the bug report for the contributor docs updates: https://bugs.launchpad.net/nova/+bug/2089325 | |