Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-24
14:02:30 mdbooth Unless we're saying that mod_wsgi is architecturally broken and shouldn't be used by anyone?
14:02:42 cdent WSGI exists so that http server-related concerns can exist outside the wsgi application code
14:02:47 mdbooth That could be true, I wouldn't know
14:02:53 cdent I do say that, these days :)
14:03:01 cdent and so do many other people
14:03:23 mdbooth cdent: Are those the same folks using undefined behaviour? :P
14:03:54 cdent they are people who don't want the wsgi server having any impact on the wsgi application, which mod-wsgi always has
14:04:09 cdent but for a long time it was the only performant choice
14:04:15 sean-k-mooney mdbooth: how mod wsgi is work i would say is falling int the same camp as undefiend behavior in c/c++
14:05:19 cdent I'm happy to consider changes in placement for all of this stuff, but it would make me much happier if there was also visible effort to inquire with Graham about whether there are ways to achieve the same thing in mod-wsgi
14:05:34 cdent we seem to be trying to take the local path, when a more global path _might_ be the right thing
14:07:01 sean-k-mooney cdent: well long term i think relaoding a script by spawning an entirely seperate prociess with its own pyton interperet would be the correct thing to do in mod_wsgi
14:07:40 sean-k-mooney that said trippleo also shoudl not be starting the placemetn api continer while upgradeing tht database
14:08:12 mdbooth sean-k-mooney: Yep, of course. But that's not the only reason why might fail on startup. This is still an issue.
14:08:13 sean-k-mooney short term we proably need a local solution
14:08:28 mdbooth Super short term is the tripleo fix.
14:09:00 sean-k-mooney mdbooth: well the tripleo fix would have been correct in any case
14:09:05 cdent I don't feel like I have all the info
14:09:08 mdbooth sean-k-mooney: ack
14:09:37 mdbooth cdent: We have a downstream failure because we're starting placement while still running db_sync on its db
14:09:39 cdent but I also don't feel like we (the three of us) have all the info about how mod-wsgi works
14:10:28 mdbooth When we restart it, we get a failure because placement_context_manager is already configured.
14:11:07 mdbooth So obviously we shouldn't be doing that, but it highlights that *any* restart of placement like this will fail for the same reason.
14:11:21 cdent what kind of 'restart' is being done?
14:11:26 sean-k-mooney mdbooth: well we are not exactly restarting it. the application is crashing somehow and reloading
14:11:46 mdbooth cdent: We're not restarting apache. I did ask about that.
14:11:53 sean-k-mooney mdbooth: if we were to restart the container then we definetly would have had a clean env
14:12:05 mdbooth sean-k-mooney: ack
14:12:18 cdent another option is to try touching the nova-placement-api or placement-api file
14:12:28 openstackgerrit Gaudenz Steinlin proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039
14:12:34 cdent that _may_ cause the daemon process (if that is what you are using) to clean itself up
14:12:52 cdent that's the "normal" way to do code reload and process reload with daemon process based mod-wsgi
14:13:17 sean-k-mooney cdent: updating the time stamp on an "imutable" continer feel kind of hacky
14:13:25 cdent long term: use uwsgi in the container, and have a FEP in some other container
14:13:36 cdent sean-k-mooney: i agree, in the case, just start the container back up
14:14:01 cdent the rules about how containers operate seems to be being selected sort of randomly
14:14:12 cdent in normal container life: if it doesn't work, you kill it and try again
14:14:15 sean-k-mooney so part of the issue is i think httpd does not exit but the apllicaiton cannoth relaod properly so the container wont restart
14:14:30 cdent and you don't _ever_ run something as heavy as apache2 in a container
14:14:48 sean-k-mooney e.g. if the whole thing exploed when we hit the unrecoverable error then docker would just restat the container and we woudl be fine
14:15:21 cdent you might be able to achieve that by no using daemon mode with mod-wsgi
14:15:30 cdent but uwsgi would be easier :) ;)
14:15:31 sean-k-mooney this is happening on Rocky by the way so in osp13 for us downstream
14:16:16 cdent for future reference sean-k-mooney, have you looked at the way placedock works? https://github.com/cdent/placedock
14:16:21 sean-k-mooney for stein + we coudl look at swappng to uwsgi in kolla i guess as an addtional mitigation
14:16:40 sean-k-mooney cdent: i have ran it once
14:17:01 sean-k-mooney i was tring to figure out could i use it with the osc-placement fuctional test without runnign devstack
14:17:05 cdent the set up there is designed to make it easy to have some other thing in the front (like a load balancer or k8s ingress thing)
14:17:22 cdent if the application fails to start, it quits
14:17:51 sean-k-mooney cdent: so in the kolla world we are running placement under mod_wsgi then putting haproxy in front of it ...
14:18:23 cdent very wasteful
14:19:02 sean-k-mooney yep but it "works" going to uwsgi would be a good thing in general for kolla i think
14:19:18 sean-k-mooney but back to the short term fix e.g. by end of this week/day
14:19:42 sean-k-mooney cdent: mdbooth are we going with the flag, excetion catching or decorator change
14:20:15 cdent sean-k-mooney: i think we're waiting to see what mdbooth's change looks like?
14:20:38 mdbooth cdent: I think regardless of my change, we want to go for the flag in the first instance
14:20:44 openstackgerrit Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.openstack.org/552924
14:20:57 mdbooth Just because it's simple and an obvious improvement
14:21:13 sean-k-mooney mdbooth: ok ill add a flag instead of catching the exception
14:21:15 mdbooth Then we can consider the finer points of wsgi, and whether a refactor is worth it later
14:21:50 openstackgerrit Balazs Gibizer proposed openstack/nova master: Consider allocations invovling child providers during allocation cleanup https://review.openstack.org/606050
14:23:45 dansmith jaypipes: could you look at this for me? It's been a while since I wrote it and my context is fading, so I'd like to get it reviewed: https://review.openstack.org/#/c/611665
14:25:59 cdent sean-k-mooney, mdbooth looks like zzzeek just did : https://review.openstack.org/#/c/613040/
14:26:39 sean-k-mooney cdent: that is in oslo db.
14:26:55 sean-k-mooney the intent was to add the flag to the callee code not the lib code
14:27:18 cdent this allows the callee to check for already started before configuring
14:27:19 sean-k-mooney that said i can use that but then its not backportable easilly
14:27:30 cdent right, I'm not suggesting you use it _now_
14:27:36 cdent just that it is available in the future
14:28:02 cdent and the change of exception is handy
14:31:06 sean-k-mooney acttuly i can use hasattter to see if it exits so i can contionally use it. ill submit a patch soon
14:31:29 sean-k-mooney cdent: enjoy your non compute thing :)
14:38:45 jaypipes dansmith: done
14:39:36 dansmith jaypipes: ah thanks, will fix those typos
14:42:47 openstackgerrit Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665
14:42:48 openstackgerrit Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663
14:42:48 openstackgerrit Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785
14:42:49 openstackgerrit Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934
14:42:49 openstackgerrit Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658
14:42:50 openstackgerrit Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829
14:42:57 openstackgerrit Matt Riedemann proposed openstack/python-novaclient stable/rocky: Fix up userdata argument to rebuild. https://review.openstack.org/613057
14:46:16 openstackgerrit Matthew Booth proposed openstack/nova master: Allow placement_context_manager to be replaced on reconfiguration https://review.openstack.org/613058
14:46:22 mdbooth sean-k-mooney: ^^^
14:46:50 mdbooth sean-k-mooney: Not quite as clean as I'd hoped because python syntax doesn't allow @db_api.placement_context_manager().writer
14:48:31 efried mdbooth: Not having looked at the patch at all, why do you need () ?
14:48:52 efried oh, I think I get it.
14:49:48 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Deprecate the unused instance-name https://review.openstack.org/602520
14:50:59 bauzas efried: the placement modeling for https://review.openstack.org/#/c/602474/2/specs/stein/approved/vgpu-stein.rst@103 is already made by the reshaper change https://review.openstack.org/#/c/599208/
14:51:27 efried bauzas: I thought that might be the case.
14:51:40 bauzas melwitt: once you're up, not sure I understand your concern about upgrade on https://review.openstack.org/#/c/602474 since I already commented this in the upgrade section
14:51:57 sean-k-mooney mdbooth: i was assuming you would have made it @db_api.writer but ya ill take a look when my browser stops crashing form the giat log i tried to open
14:52:01 bauzas efried: I just wanted to keep minimalistic changes to the alrady approved spec
14:52:01 efried bauzas: Is that described in the reshaper spec?
14:52:12 bauzas efried: no, that's direct code
14:52:17 efried bauzas: I think I'm trying to say it should be described in *some* spec *somewhere*.
14:52:36 bauzas efried: I could amend https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html if you wish
14:52:41 efried I don't disagree we should minimize changes to a spec reapproval in theory, but this seems like something worth including.
14:52:49 efried bauzas: That would be okay too.
14:53:10 bauzas what I reallly want is possible quick approval
14:53:18 efried bauzas: swhy I didn't downvote :)

Earlier   Later