| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-24 | |||
| 14:09:08 | mdbooth | sean-k-mooney: ack | |
| 14:09:37 | mdbooth | cdent: We have a downstream failure because we're starting placement while still running db_sync on its db | |
| 14:09:39 | cdent | but I also don't feel like we (the three of us) have all the info about how mod-wsgi works | |
| 14:10:28 | mdbooth | When we restart it, we get a failure because placement_context_manager is already configured. | |
| 14:11:07 | mdbooth | So obviously we shouldn't be doing that, but it highlights that *any* restart of placement like this will fail for the same reason. | |
| 14:11:21 | cdent | what kind of 'restart' is being done? | |
| 14:11:26 | sean-k-mooney | mdbooth: well we are not exactly restarting it. the application is crashing somehow and reloading | |
| 14:11:46 | mdbooth | cdent: We're not restarting apache. I did ask about that. | |
| 14:11:53 | sean-k-mooney | mdbooth: if we were to restart the container then we definetly would have had a clean env | |
| 14:12:05 | mdbooth | sean-k-mooney: ack | |
| 14:12:18 | cdent | another option is to try touching the nova-placement-api or placement-api file | |
| 14:12:28 | openstackgerrit | Gaudenz Steinlin proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039 | |
| 14:12:34 | cdent | that _may_ cause the daemon process (if that is what you are using) to clean itself up | |
| 14:12:52 | cdent | that's the "normal" way to do code reload and process reload with daemon process based mod-wsgi | |
| 14:13:17 | sean-k-mooney | cdent: updating the time stamp on an "imutable" continer feel kind of hacky | |
| 14:13:25 | cdent | long term: use uwsgi in the container, and have a FEP in some other container | |
| 14:13:36 | cdent | sean-k-mooney: i agree, in the case, just start the container back up | |
| 14:14:01 | cdent | the rules about how containers operate seems to be being selected sort of randomly | |
| 14:14:12 | cdent | in normal container life: if it doesn't work, you kill it and try again | |
| 14:14:15 | sean-k-mooney | so part of the issue is i think httpd does not exit but the apllicaiton cannoth relaod properly so the container wont restart | |
| 14:14:30 | cdent | and you don't _ever_ run something as heavy as apache2 in a container | |
| 14:14:48 | sean-k-mooney | e.g. if the whole thing exploed when we hit the unrecoverable error then docker would just restat the container and we woudl be fine | |
| 14:15:21 | cdent | you might be able to achieve that by no using daemon mode with mod-wsgi | |
| 14:15:30 | cdent | but uwsgi would be easier :) ;) | |
| 14:15:31 | sean-k-mooney | this is happening on Rocky by the way so in osp13 for us downstream | |
| 14:16:16 | cdent | for future reference sean-k-mooney, have you looked at the way placedock works? https://github.com/cdent/placedock | |
| 14:16:21 | sean-k-mooney | for stein + we coudl look at swappng to uwsgi in kolla i guess as an addtional mitigation | |
| 14:16:40 | sean-k-mooney | cdent: i have ran it once | |
| 14:17:01 | sean-k-mooney | i was tring to figure out could i use it with the osc-placement fuctional test without runnign devstack | |
| 14:17:05 | cdent | the set up there is designed to make it easy to have some other thing in the front (like a load balancer or k8s ingress thing) | |
| 14:17:22 | cdent | if the application fails to start, it quits | |
| 14:17:51 | sean-k-mooney | cdent: so in the kolla world we are running placement under mod_wsgi then putting haproxy in front of it ... | |
| 14:18:23 | cdent | very wasteful | |
| 14:19:02 | sean-k-mooney | yep but it "works" going to uwsgi would be a good thing in general for kolla i think | |
| 14:19:18 | sean-k-mooney | but back to the short term fix e.g. by end of this week/day | |
| 14:19:42 | sean-k-mooney | cdent: mdbooth are we going with the flag, excetion catching or decorator change | |
| 14:20:15 | cdent | sean-k-mooney: i think we're waiting to see what mdbooth's change looks like? | |
| 14:20:38 | mdbooth | cdent: I think regardless of my change, we want to go for the flag in the first instance | |
| 14:20:44 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.openstack.org/552924 | |
| 14:20:57 | mdbooth | Just because it's simple and an obvious improvement | |
| 14:21:13 | sean-k-mooney | mdbooth: ok ill add a flag instead of catching the exception | |
| 14:21:15 | mdbooth | Then we can consider the finer points of wsgi, and whether a refactor is worth it later | |
| 14:21:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Consider allocations invovling child providers during allocation cleanup https://review.openstack.org/606050 | |
| 14:23:45 | dansmith | jaypipes: could you look at this for me? It's been a while since I wrote it and my context is fading, so I'd like to get it reviewed: https://review.openstack.org/#/c/611665 | |
| 14:25:59 | cdent | sean-k-mooney, mdbooth looks like zzzeek just did : https://review.openstack.org/#/c/613040/ | |
| 14:26:39 | sean-k-mooney | cdent: that is in oslo db. | |
| 14:26:55 | sean-k-mooney | the intent was to add the flag to the callee code not the lib code | |
| 14:27:18 | cdent | this allows the callee to check for already started before configuring | |
| 14:27:19 | sean-k-mooney | that said i can use that but then its not backportable easilly | |
| 14:27:30 | cdent | right, I'm not suggesting you use it _now_ | |
| 14:27:36 | cdent | just that it is available in the future | |
| 14:28:02 | cdent | and the change of exception is handy | |
| 14:31:06 | sean-k-mooney | acttuly i can use hasattter to see if it exits so i can contionally use it. ill submit a patch soon | |
| 14:31:29 | sean-k-mooney | cdent: enjoy your non compute thing :) | |
| 14:38:45 | jaypipes | dansmith: done | |
| 14:39:36 | dansmith | jaypipes: ah thanks, will fix those typos | |
| 14:42:47 | openstackgerrit | Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665 | |
| 14:42:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663 | |
| 14:42:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785 | |
| 14:42:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934 | |
| 14:42:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 14:42:50 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 14:42:57 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient stable/rocky: Fix up userdata argument to rebuild. https://review.openstack.org/613057 | |
| 14:46:16 | openstackgerrit | Matthew Booth proposed openstack/nova master: Allow placement_context_manager to be replaced on reconfiguration https://review.openstack.org/613058 | |
| 14:46:22 | mdbooth | sean-k-mooney: ^^^ | |
| 14:46:50 | mdbooth | sean-k-mooney: Not quite as clean as I'd hoped because python syntax doesn't allow @db_api.placement_context_manager().writer | |
| 14:48:31 | efried | mdbooth: Not having looked at the patch at all, why do you need () ? | |
| 14:48:52 | efried | oh, I think I get it. | |
| 14:49:48 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Deprecate the unused instance-name https://review.openstack.org/602520 | |
| 14:50:59 | bauzas | efried: the placement modeling for https://review.openstack.org/#/c/602474/2/specs/stein/approved/vgpu-stein.rst@103 is already made by the reshaper change https://review.openstack.org/#/c/599208/ | |
| 14:51:27 | efried | bauzas: I thought that might be the case. | |
| 14:51:40 | bauzas | melwitt: once you're up, not sure I understand your concern about upgrade on https://review.openstack.org/#/c/602474 since I already commented this in the upgrade section | |
| 14:51:57 | sean-k-mooney | mdbooth: i was assuming you would have made it @db_api.writer but ya ill take a look when my browser stops crashing form the giat log i tried to open | |
| 14:52:01 | bauzas | efried: I just wanted to keep minimalistic changes to the alrady approved spec | |
| 14:52:01 | efried | bauzas: Is that described in the reshaper spec? | |
| 14:52:12 | bauzas | efried: no, that's direct code | |
| 14:52:17 | efried | bauzas: I think I'm trying to say it should be described in *some* spec *somewhere*. | |
| 14:52:36 | bauzas | efried: I could amend https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html if you wish | |
| 14:52:41 | efried | I don't disagree we should minimize changes to a spec reapproval in theory, but this seems like something worth including. | |
| 14:52:49 | efried | bauzas: That would be okay too. | |
| 14:53:10 | bauzas | what I reallly want is possible quick approval | |
| 14:53:18 | efried | bauzas: swhy I didn't downvote :) | |
| 14:53:24 | sean-k-mooney | mdbooth: that will still not rebind the context on reconfigurtion | |
| 14:53:27 | bauzas | and then, if left comments, a possible follow-up | |
| 14:53:31 | efried | sure | |
| 14:54:10 | bauzas | efried: or I could amend https://review.openstack.org/#/c/602474 in a follow-up if you prefer | |
| 14:56:27 | sean-k-mooney | mdbooth: you will need to do somehtin like this https://stackoverflow.com/a/33507308 | |
| 14:56:55 | efried | bauzas: There was some question (discussion with mriedem) as to whether these vgpu reshaper patches should be associated with the reshaper bp or the vgpu bp. I'm starting to think it's more appropriate to do the latter. The reshaper bp enables the work, but we're not going to go back and tag every future reshape impl against that same bp. | |
| 14:57:27 | bauzas | honestly, it's just a gerrit tag | |
| 14:57:31 | bauzas | so I don't really care | |
| 14:57:34 | efried | That being the case, IMO the text in question ought to go into https://review.openstack.org/#/c/602474 (the vgpu spec). | |
| 14:57:35 | bauzas | provided I have reviews :) | |
| 14:57:41 | mdbooth | sean-k-mooney: Ah, you're right | |
| 14:58:03 | bauzas | efried: fair, I'll write a follow-up | |
| 14:58:08 | efried | It's more than a gerrit tag. It feeds into being able to claim completion of a blueprint, etc. | |
| 14:58:24 | bauzas | I understand this but meh | |
| 14:58:52 | bauzas | either way, looks like it's a priority | |
| 14:59:12 | sean-k-mooney | mdbooth: ill submit the version with the flag for review. ill see if i can create a simple decorator after once the simple fix is up | |
| 14:59:31 | efried | bauzas: I'm not a spec core, so I can't approve it either way. | |
| 14:59:35 | mdbooth | sean-k-mooney: In lighter news, putting an emoji in a gerrit comment causes a 500 :) | |