| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-20 | |||
| 13:33:02 | gibi | bauzas: hm, it might not coming from the claiming | |
| 13:33:51 | gibi | bauzas: ahh I found it, it is a "PUT /placement/allocations/485c7480-939c-4b88-8c00-0f346dc6a924" | |
| 13:33:54 | gibi | bauzas: so yes | |
| 13:34:09 | gibi | bauzas: then I will file a bug for this as well | |
| 13:34:42 | gibi | bauzas: should this log be on info or debug level? | |
| 13:34:58 | bauzas | debug IMHO | |
| 13:35:11 | bauzas | because it just means that's a race condition | |
| 13:35:56 | gibi | bauzas: OK, thanks | |
| 13:37:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/485634 | |
| 13:39:30 | openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660 | |
| 13:44:34 | gibi | bauzas: here is the bug report for the error log https://bugs.launchpad.net/nova/+bug/1705487 | |
| 13:44:37 | openstack | Launchpad bug 1705487 in OpenStack Compute (nova) "placement logs an ERROR when PUT /allocation result in an invalid inventory" [Undecided,New] | |
| 13:55:15 | ftersin | mdbooth: hi. If you have a minute, could you look at ScaleIO review (https://review.openstack.org/#/c/407440/)? I addressed your (and mriedem) comments there, and now ready for new ones. | |
| 14:05:50 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: serial console log (completed) https://review.openstack.org/450636 | |
| 14:08:06 | mriedem | andreykurilin: melwitt: we need to get https://review.openstack.org/#/c/484152/ in to unblock novaclient tests | |
| 14:08:17 | mriedem | now that the counting instance quotas change merged in nova | |
| 14:08:23 | mriedem | or sdague ^ | |
| 14:09:24 | openstackgerrit | Merged openstack/nova master: [placement] cover deleting standard trait https://review.openstack.org/484153 | |
| 14:09:33 | andreykurilin | mriedem: done | |
| 14:09:43 | mriedem | gibi: just wanted to say thanks for kicking the tires on the scheduler + placement stuff, you're finding some nice hairy bugs | |
| 14:09:45 | mriedem | andreykurilin: thanks | |
| 14:11:46 | openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660 | |
| 14:18:45 | openstackgerrit | Kaitlin Farr proposed openstack/nova master: Remove deprecated keymgr code https://review.openstack.org/439855 | |
| 14:19:49 | gibi | mriedem: my pleasure :) | |
| 14:21:02 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 14:21:56 | sdague | mriedem: lookin | |
| 14:22:09 | sdague | mriedem: ah, andreykurilin already snagged it | |
| 14:22:21 | sdague | mriedem: you got a few moments to think about path forward on request logging? | |
| 14:23:24 | mriedem | sure, although it's over my head | |
| 14:24:17 | sdague | ok, well do you get my concern that we're not logging it through the python subsystem any more? | |
| 14:24:39 | mriedem | yeah it's definitely a problem, and as noted i'm seeing weird formatting | |
| 14:24:48 | mriedem | like parts of the log message are chopped off | |
| 14:25:20 | sdague | mriedem: you see my follow ups? | |
| 14:25:28 | mriedem | reading those now | |
| 14:25:50 | sdague | ok, do that, then we can chat, so I don't repeat myself :) | |
| 14:28:03 | mriedem | sdague: wait, say that again | |
| 14:29:22 | mriedem | sdague: oh man ok so that debug log message starts here then http://logs.openstack.org/65/483565/4/check/gate-tempest-dsvm-py35-ubuntu-xenial/9921636/logs/screen-n-sch.txt.gz#_Jul_19_20_17_18_800467 | |
| 14:29:24 | openstackgerrit | Alex Szarka proposed openstack/nova master: Reduce code complexity - manager.py https://review.openstack.org/359868 | |
| 14:29:27 | mriedem | and takes 3 full chunks | |
| 14:29:39 | mriedem | make that 4 chunks | |
| 14:29:39 | sdague | mriedem: yep | |
| 14:29:58 | mriedem | yikes, maybe we shouldn't log full instances on a cron in the scheduler | |
| 14:30:03 | jaypipes | mriedem: fyi, in and out this morning... more plumbing debacles at chez pipes. | |
| 14:30:19 | mriedem | that's only 3 instances, | |
| 14:30:37 | mriedem | think if you have 10K computes sending the update_instance_info periodic to the scheduler every minute, which is the default | |
| 14:30:42 | mriedem | with 1 million VMs | |
| 14:31:09 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: Handle missing volume vmdk during detach https://review.openstack.org/484675 | |
| 14:31:15 | mriedem | jaypipes: ok. btw, i started my morning with an internal email from a guy doing scale testing asking about issues with booting 3000+ VMs at once | |
| 14:31:28 | mriedem | and multiple scheduler workers, in mitaka | |
| 14:31:38 | jaypipes | mriedem: lovely. | |
| 14:31:58 | jaypipes | mriedem: I have an afternoon call to talk about why Nova AZs aren't AWS AZs. | |
| 14:32:00 | dansmith | jaypipes: chez pipes sounds like a slum | |
| 14:33:16 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Add datapath type information to OVS vif objects https://review.openstack.org/474892 | |
| 14:34:02 | mriedem | btw this is a good example of why you can't run the scheduler with debug logging enabled | |
| 14:34:03 | mriedem | in prod | |
| 14:34:40 | dansmith | but you also can't run it without debug in prod | |
| 14:34:56 | mriedem | this reminds me of a bug rlrossit found | |
| 14:35:31 | mriedem | all the logging that happens in here https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L166 | |
| 14:36:06 | mriedem | when we turned on debug it killed the scheduler in our pre-prod cloud | |
| 14:37:22 | mriedem | sdague: so you've got my attention | |
| 14:37:41 | sdague | mriedem: ok, so the challenges / questions | |
| 14:38:00 | sdague | 1) do we need something like the request_log middleware? (assume answer is yes) | |
| 14:38:34 | sdague | 2) do we force people to do a paste pipeline update or do we hide this in existing middleware to help with upgrade? | |
| 14:39:08 | jaypipes | dansmith: it is. | |
| 14:39:20 | dansmith | jaypipes: :P | |
| 14:39:36 | sdague | 3) do we want to bring back in request timing (eventlet has it, this does not yet)? | |
| 14:40:56 | mriedem | what is request timing? | |
| 14:41:35 | mriedem | if the answer to getting the request id in the logs under uwsgi is #1, then yes for sure | |
| 14:41:45 | sdague | http://logs.openstack.org/02/485602/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0c29b8a/logs/screen-c-api.txt.gz#_Jul_20_13_20_35_279507 | |
| 14:41:47 | mriedem | even w/o global request id, we need the request id for basic debug | |
| 14:42:10 | mriedem | is that "time: 0.5609260" ? | |
| 14:42:12 | sdague | that's what a current eventlet wsgi log line looks like | |
| 14:42:13 | sdague | yeh | |
| 14:42:22 | mriedem | i don't think i ever use that | |
| 14:42:49 | mriedem | it would be nice to know for profiling i suppose | |
| 14:42:54 | sdague | http://logs.openstack.org/02/485602/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0c29b8a/logs/screen-n-api.txt.gz#_Jul_20_13_20_03_249571 is what the request_log patch makes the nova lines look like | |
| 14:42:59 | mriedem | if you have some requests taking 30 seconds or something | |
| 14:43:02 | sdague | so, 3 is pretty minor | |
| 14:43:16 | sdague | the real issue is #2, in how we get this in the hands of people | |
| 14:43:43 | sdague | because right now, the WIP patch requires this - https://review.openstack.org/#/c/485602/1/etc/nova/api-paste.ini | |
| 14:44:00 | mriedem | well, there is something to consider there, | |
| 14:44:01 | mriedem | which is, | |
| 14:44:03 | mriedem | wait for it, | |
| 14:44:18 | mriedem | this isn't a problem if you're running nova-api under eventlet, yes? | |
| 14:44:23 | sdague | correct | |
| 14:44:36 | mriedem | so if/when you move nova-api to uwsgi or apache, | |
| 14:44:42 | mriedem | you're going to be changing some config anyway, yes? | |
| 14:45:06 | sdague | maybe, though up until this point we've never told people they have to change this stuff | |
| 14:45:30 | mriedem | this stuff == api-paste.ini? | |
| 14:45:32 | sdague | yes | |
| 14:45:49 | mriedem | because changing nova-api to run under uwsgi is going to require changes to someone's config management / deploy tooling | |
| 14:45:52 | mriedem | just like it did in devstack | |
| 14:46:09 | sdague | sure | |
| 14:46:10 | mriedem | so it's not like this is some trivial change that doesn't require manual intervention to start running nova-api this way | |
| 14:46:31 | sdague | sure, so you are feeling that upgrade note and the paste change is fine? | |
| 14:46:48 | mriedem | so i don't think it's the end of the world to say, if you're going to start running nova-api under uwsgi AND you want the request IDs logged for debug (which we assume you do if you're not crazy), then you have to also do this other thing | |
| 14:46:59 | mriedem | with 8 days to FF, i'm feeling fine about that yes | |
| 14:47:09 | mriedem | wait what day is it? | |
| 14:47:11 | mriedem | 7 days to FF | |
| 14:47:24 | sdague | ok, I'll proceed forward based on that | |