| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-20 | |||
| 13:44:37 | openstack | Launchpad bug 1705487 in OpenStack Compute (nova) "placement logs an ERROR when PUT /allocation result in an invalid inventory" [Undecided,New] | |
| 13:55:15 | ftersin | mdbooth: hi. If you have a minute, could you look at ScaleIO review (https://review.openstack.org/#/c/407440/)? I addressed your (and mriedem) comments there, and now ready for new ones. | |
| 14:05:50 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: serial console log (completed) https://review.openstack.org/450636 | |
| 14:08:06 | mriedem | andreykurilin: melwitt: we need to get https://review.openstack.org/#/c/484152/ in to unblock novaclient tests | |
| 14:08:17 | mriedem | now that the counting instance quotas change merged in nova | |
| 14:08:23 | mriedem | or sdague ^ | |
| 14:09:24 | openstackgerrit | Merged openstack/nova master: [placement] cover deleting standard trait https://review.openstack.org/484153 | |
| 14:09:33 | andreykurilin | mriedem: done | |
| 14:09:43 | mriedem | gibi: just wanted to say thanks for kicking the tires on the scheduler + placement stuff, you're finding some nice hairy bugs | |
| 14:09:45 | mriedem | andreykurilin: thanks | |
| 14:11:46 | openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660 | |
| 14:18:45 | openstackgerrit | Kaitlin Farr proposed openstack/nova master: Remove deprecated keymgr code https://review.openstack.org/439855 | |
| 14:19:49 | gibi | mriedem: my pleasure :) | |
| 14:21:02 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 14:21:56 | sdague | mriedem: lookin | |
| 14:22:09 | sdague | mriedem: ah, andreykurilin already snagged it | |
| 14:22:21 | sdague | mriedem: you got a few moments to think about path forward on request logging? | |
| 14:23:24 | mriedem | sure, although it's over my head | |
| 14:24:17 | sdague | ok, well do you get my concern that we're not logging it through the python subsystem any more? | |
| 14:24:39 | mriedem | yeah it's definitely a problem, and as noted i'm seeing weird formatting | |
| 14:24:48 | mriedem | like parts of the log message are chopped off | |
| 14:25:20 | sdague | mriedem: you see my follow ups? | |
| 14:25:28 | mriedem | reading those now | |
| 14:25:50 | sdague | ok, do that, then we can chat, so I don't repeat myself :) | |
| 14:28:03 | mriedem | sdague: wait, say that again | |
| 14:29:22 | mriedem | sdague: oh man ok so that debug log message starts here then http://logs.openstack.org/65/483565/4/check/gate-tempest-dsvm-py35-ubuntu-xenial/9921636/logs/screen-n-sch.txt.gz#_Jul_19_20_17_18_800467 | |
| 14:29:24 | openstackgerrit | Alex Szarka proposed openstack/nova master: Reduce code complexity - manager.py https://review.openstack.org/359868 | |
| 14:29:27 | mriedem | and takes 3 full chunks | |
| 14:29:39 | mriedem | make that 4 chunks | |
| 14:29:39 | sdague | mriedem: yep | |
| 14:29:58 | mriedem | yikes, maybe we shouldn't log full instances on a cron in the scheduler | |
| 14:30:03 | jaypipes | mriedem: fyi, in and out this morning... more plumbing debacles at chez pipes. | |
| 14:30:19 | mriedem | that's only 3 instances, | |
| 14:30:37 | mriedem | think if you have 10K computes sending the update_instance_info periodic to the scheduler every minute, which is the default | |
| 14:30:42 | mriedem | with 1 million VMs | |
| 14:31:09 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: Handle missing volume vmdk during detach https://review.openstack.org/484675 | |
| 14:31:15 | mriedem | jaypipes: ok. btw, i started my morning with an internal email from a guy doing scale testing asking about issues with booting 3000+ VMs at once | |
| 14:31:28 | mriedem | and multiple scheduler workers, in mitaka | |
| 14:31:38 | jaypipes | mriedem: lovely. | |
| 14:31:58 | jaypipes | mriedem: I have an afternoon call to talk about why Nova AZs aren't AWS AZs. | |
| 14:32:00 | dansmith | jaypipes: chez pipes sounds like a slum | |
| 14:33:16 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Add datapath type information to OVS vif objects https://review.openstack.org/474892 | |
| 14:34:02 | mriedem | btw this is a good example of why you can't run the scheduler with debug logging enabled | |
| 14:34:03 | mriedem | in prod | |
| 14:34:40 | dansmith | but you also can't run it without debug in prod | |
| 14:34:56 | mriedem | this reminds me of a bug rlrossit found | |
| 14:35:31 | mriedem | all the logging that happens in here https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L166 | |
| 14:36:06 | mriedem | when we turned on debug it killed the scheduler in our pre-prod cloud | |
| 14:37:22 | mriedem | sdague: so you've got my attention | |
| 14:37:41 | sdague | mriedem: ok, so the challenges / questions | |
| 14:38:00 | sdague | 1) do we need something like the request_log middleware? (assume answer is yes) | |
| 14:38:34 | sdague | 2) do we force people to do a paste pipeline update or do we hide this in existing middleware to help with upgrade? | |
| 14:39:08 | jaypipes | dansmith: it is. | |
| 14:39:20 | dansmith | jaypipes: :P | |
| 14:39:36 | sdague | 3) do we want to bring back in request timing (eventlet has it, this does not yet)? | |
| 14:40:56 | mriedem | what is request timing? | |
| 14:41:35 | mriedem | if the answer to getting the request id in the logs under uwsgi is #1, then yes for sure | |
| 14:41:45 | sdague | http://logs.openstack.org/02/485602/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0c29b8a/logs/screen-c-api.txt.gz#_Jul_20_13_20_35_279507 | |
| 14:41:47 | mriedem | even w/o global request id, we need the request id for basic debug | |
| 14:42:10 | mriedem | is that "time: 0.5609260" ? | |
| 14:42:12 | sdague | that's what a current eventlet wsgi log line looks like | |
| 14:42:13 | sdague | yeh | |
| 14:42:22 | mriedem | i don't think i ever use that | |
| 14:42:49 | mriedem | it would be nice to know for profiling i suppose | |
| 14:42:54 | sdague | http://logs.openstack.org/02/485602/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0c29b8a/logs/screen-n-api.txt.gz#_Jul_20_13_20_03_249571 is what the request_log patch makes the nova lines look like | |
| 14:42:59 | mriedem | if you have some requests taking 30 seconds or something | |
| 14:43:02 | sdague | so, 3 is pretty minor | |
| 14:43:16 | sdague | the real issue is #2, in how we get this in the hands of people | |
| 14:43:43 | sdague | because right now, the WIP patch requires this - https://review.openstack.org/#/c/485602/1/etc/nova/api-paste.ini | |
| 14:44:00 | mriedem | well, there is something to consider there, | |
| 14:44:01 | mriedem | which is, | |
| 14:44:03 | mriedem | wait for it, | |
| 14:44:18 | mriedem | this isn't a problem if you're running nova-api under eventlet, yes? | |
| 14:44:23 | sdague | correct | |
| 14:44:36 | mriedem | so if/when you move nova-api to uwsgi or apache, | |
| 14:44:42 | mriedem | you're going to be changing some config anyway, yes? | |
| 14:45:06 | sdague | maybe, though up until this point we've never told people they have to change this stuff | |
| 14:45:30 | mriedem | this stuff == api-paste.ini? | |
| 14:45:32 | sdague | yes | |
| 14:45:49 | mriedem | because changing nova-api to run under uwsgi is going to require changes to someone's config management / deploy tooling | |
| 14:45:52 | mriedem | just like it did in devstack | |
| 14:46:09 | sdague | sure | |
| 14:46:10 | mriedem | so it's not like this is some trivial change that doesn't require manual intervention to start running nova-api this way | |
| 14:46:31 | sdague | sure, so you are feeling that upgrade note and the paste change is fine? | |
| 14:46:48 | mriedem | so i don't think it's the end of the world to say, if you're going to start running nova-api under uwsgi AND you want the request IDs logged for debug (which we assume you do if you're not crazy), then you have to also do this other thing | |
| 14:46:59 | mriedem | with 8 days to FF, i'm feeling fine about that yes | |
| 14:47:09 | mriedem | wait what day is it? | |
| 14:47:11 | mriedem | 7 days to FF | |
| 14:47:24 | sdague | ok, I'll proceed forward based on that | |
| 14:47:57 | mriedem | if you wanted to tinker with sliding something into existing middleware that is smart enough to know if it's running under eventlet or not and auto enable, that'd be cool but probably queens at this point | |
| 14:47:58 | sdague | the functional parts of the patch should be ready for review - https://review.openstack.org/#/c/485602/1/nova/api/openstack/requestlog.py | |
| 14:48:21 | sdague | yeh, I'll put that as stretch goal | |
| 14:48:55 | mriedem | is there any existing oslo.middleware stuff that all/most services use that we could do the auto enable thing in there so everyone else gets it for free? | |
| 14:49:00 | mriedem | and doesn't have to copy the solution around? | |
| 14:49:11 | mriedem | noting that today is release freeze for oslo lis | |
| 14:49:13 | mriedem | *libs | |
| 14:49:22 | sdague | I don't think that there is | |
| 14:49:29 | mriedem | https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/request_id.py ? | |
| 14:50:08 | sdague | maybe, not everyone uses that | |
| 14:50:26 | sdague | but, regardless, I don't think we could get it in and through that process | |