| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-20 | |||
| 14:31:58 | jaypipes | mriedem: I have an afternoon call to talk about why Nova AZs aren't AWS AZs. | |
| 14:32:00 | dansmith | jaypipes: chez pipes sounds like a slum | |
| 14:33:16 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Add datapath type information to OVS vif objects https://review.openstack.org/474892 | |
| 14:34:02 | mriedem | btw this is a good example of why you can't run the scheduler with debug logging enabled | |
| 14:34:03 | mriedem | in prod | |
| 14:34:40 | dansmith | but you also can't run it without debug in prod | |
| 14:34:56 | mriedem | this reminds me of a bug rlrossit found | |
| 14:35:31 | mriedem | all the logging that happens in here https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L166 | |
| 14:36:06 | mriedem | when we turned on debug it killed the scheduler in our pre-prod cloud | |
| 14:37:22 | mriedem | sdague: so you've got my attention | |
| 14:37:41 | sdague | mriedem: ok, so the challenges / questions | |
| 14:38:00 | sdague | 1) do we need something like the request_log middleware? (assume answer is yes) | |
| 14:38:34 | sdague | 2) do we force people to do a paste pipeline update or do we hide this in existing middleware to help with upgrade? | |
| 14:39:08 | jaypipes | dansmith: it is. | |
| 14:39:20 | dansmith | jaypipes: :P | |
| 14:39:36 | sdague | 3) do we want to bring back in request timing (eventlet has it, this does not yet)? | |
| 14:40:56 | mriedem | what is request timing? | |
| 14:41:35 | mriedem | if the answer to getting the request id in the logs under uwsgi is #1, then yes for sure | |
| 14:41:45 | sdague | http://logs.openstack.org/02/485602/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0c29b8a/logs/screen-c-api.txt.gz#_Jul_20_13_20_35_279507 | |
| 14:41:47 | mriedem | even w/o global request id, we need the request id for basic debug | |
| 14:42:10 | mriedem | is that "time: 0.5609260" ? | |
| 14:42:12 | sdague | that's what a current eventlet wsgi log line looks like | |
| 14:42:13 | sdague | yeh | |
| 14:42:22 | mriedem | i don't think i ever use that | |
| 14:42:49 | mriedem | it would be nice to know for profiling i suppose | |
| 14:42:54 | sdague | http://logs.openstack.org/02/485602/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0c29b8a/logs/screen-n-api.txt.gz#_Jul_20_13_20_03_249571 is what the request_log patch makes the nova lines look like | |
| 14:42:59 | mriedem | if you have some requests taking 30 seconds or something | |
| 14:43:02 | sdague | so, 3 is pretty minor | |
| 14:43:16 | sdague | the real issue is #2, in how we get this in the hands of people | |
| 14:43:43 | sdague | because right now, the WIP patch requires this - https://review.openstack.org/#/c/485602/1/etc/nova/api-paste.ini | |
| 14:44:00 | mriedem | well, there is something to consider there, | |
| 14:44:01 | mriedem | which is, | |
| 14:44:03 | mriedem | wait for it, | |
| 14:44:18 | mriedem | this isn't a problem if you're running nova-api under eventlet, yes? | |
| 14:44:23 | sdague | correct | |
| 14:44:36 | mriedem | so if/when you move nova-api to uwsgi or apache, | |
| 14:44:42 | mriedem | you're going to be changing some config anyway, yes? | |
| 14:45:06 | sdague | maybe, though up until this point we've never told people they have to change this stuff | |
| 14:45:30 | mriedem | this stuff == api-paste.ini? | |
| 14:45:32 | sdague | yes | |
| 14:45:49 | mriedem | because changing nova-api to run under uwsgi is going to require changes to someone's config management / deploy tooling | |
| 14:45:52 | mriedem | just like it did in devstack | |
| 14:46:09 | sdague | sure | |
| 14:46:10 | mriedem | so it's not like this is some trivial change that doesn't require manual intervention to start running nova-api this way | |
| 14:46:31 | sdague | sure, so you are feeling that upgrade note and the paste change is fine? | |
| 14:46:48 | mriedem | so i don't think it's the end of the world to say, if you're going to start running nova-api under uwsgi AND you want the request IDs logged for debug (which we assume you do if you're not crazy), then you have to also do this other thing | |
| 14:46:59 | mriedem | with 8 days to FF, i'm feeling fine about that yes | |
| 14:47:09 | mriedem | wait what day is it? | |
| 14:47:11 | mriedem | 7 days to FF | |
| 14:47:24 | sdague | ok, I'll proceed forward based on that | |
| 14:47:57 | mriedem | if you wanted to tinker with sliding something into existing middleware that is smart enough to know if it's running under eventlet or not and auto enable, that'd be cool but probably queens at this point | |
| 14:47:58 | sdague | the functional parts of the patch should be ready for review - https://review.openstack.org/#/c/485602/1/nova/api/openstack/requestlog.py | |
| 14:48:21 | sdague | yeh, I'll put that as stretch goal | |
| 14:48:55 | mriedem | is there any existing oslo.middleware stuff that all/most services use that we could do the auto enable thing in there so everyone else gets it for free? | |
| 14:49:00 | mriedem | and doesn't have to copy the solution around? | |
| 14:49:11 | mriedem | noting that today is release freeze for oslo lis | |
| 14:49:13 | mriedem | *libs | |
| 14:49:22 | sdague | I don't think that there is | |
| 14:49:29 | mriedem | https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/request_id.py ? | |
| 14:50:08 | sdague | maybe, not everyone uses that | |
| 14:50:26 | sdague | but, regardless, I don't think we could get it in and through that process | |
| 14:50:33 | mriedem | sure, | |
| 14:50:34 | sdague | in the time alloted | |
| 14:50:42 | sdague | one of the challenges about fixed time boxes | |
| 14:50:46 | dansmith | ...with 146 things in the check queue | |
| 14:50:48 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Remove check_detach https://review.openstack.org/446671 | |
| 14:50:48 | mriedem | so maybe that's the other thing for queens is moving this into oslo.middleware so others can just pick it up from there | |
| 14:50:49 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Implement new attach Cinder flow https://review.openstack.org/330285 | |
| 14:52:14 | jangutter | Newbie question: https://review.openstack.org/#/c/485382/ failed a check in the gate (last time it failed a different check). Does this kinda thing happen often when upping requirements.txt? | |
| 14:59:43 | sdague | mriedem: yeh, it's kind of going to depend on the way wsgi bits are done, which is not standardized | |
| 15:00:02 | sdague | honestly, the ideal might be if oslo.service actually abstracted this stuff away | |
| 15:00:13 | sdague | but... that's got other challenges | |
| 15:05:39 | dansmith | melwitt: remind me why those need to be remotable? https://review.openstack.org/#/c/410945/58/nova/objects/quotas.py@62 | |
| 15:07:32 | dansmith | like, I thought we had removed all of the quota calls by this point that need to be done from compute or non-db-having services | |
| 15:09:31 | gibi | mriedem: I cannot joint to the nova meeting today. Could you cover the notification subteam part? | |
| 15:10:11 | mdbooth | melwitt: Addressed your comments on this patch with a test: https://review.openstack.org/#/c/479801/ | |
| 15:10:30 | mdbooth | Jenkins -1 is bogus, btw. Rechecked it, but following patch has passed. | |
| 15:11:13 | mdbooth | melwitt: Incidentally following patch is https://review.openstack.org/#/c/485601/ which is a minor cleanup to your quotas patch | |
| 15:11:38 | mdbooth | s/minor/trivial/ | |
| 15:22:17 | mriedem | gibi: sure | |
| 15:23:00 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Implement new attach Cinder flow https://review.openstack.org/330285 | |
| 15:27:07 | mriedem | sdague: are you going to recap the plan to the ML? | |
| 15:27:20 | gibi | mriedem: thanks | |
| 15:27:28 | openstackgerrit | Merged openstack/nova master: placement: alloc candidates only shared resources https://review.openstack.org/484900 | |
| 15:28:13 | mriedem | jaypipes: do we need anything in here to get released in os-traits today? https://review.openstack.org/#/q/os-traits+status:open | |
| 15:28:16 | mriedem | because today is the big day | |
| 15:28:26 | openstackgerrit | Merged openstack/python-novaclient master: Adjust test_resize_down_revert to account for counting quotas https://review.openstack.org/484152 | |
| 15:28:44 | mriedem | maybe the reqs update | |
| 15:28:44 | sdague | mriedem: I can | |
| 15:28:59 | sdague | I'm going to figure out test failures first | |
| 15:28:59 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Read datapath_type from VIF object https://review.openstack.org/474914 | |
| 15:29:00 | jaypipes | mriedem: nope | |
| 15:29:11 | mriedem | ok i'll get the req update in and then release it | |
| 15:29:17 | jaypipes | mriedem: danke | |
| 15:33:14 | mriedem | sdague: another thing to consider, report a bug for this request id logging thing and then we can tag it pike-rc-potential for tracking | |
| 15:33:18 | mriedem | i'm starting to tag things for the rc | |
| 15:35:11 | jaypipes | sean-k-mooney: ralonsoh's https://review.openstack.org/#/c/474914/ is good to go. feel free to +2 if agree. | |
| 15:35:19 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/485634 | |
| 15:35:41 | ralonsoh | jaypipes, sean-k-moone: thanks! | |
| 15:35:53 | mriedem | jaypipes: sean-k-mooney: ralonsoh: does ^ need to be in an os-vif release for pike? | |