| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-20 | |||
| 13:25:18 | gibi | bauzas: so you want to make a bit more finegrained fix | |
| 13:25:26 | dansmith | after the refactor, | |
| 13:25:36 | bauzas | gibi: https://review.openstack.org/#/c/485585/3/nova/tests/unit/scheduler/test_scheduler.py@135 tbc | |
| 13:25:39 | gibi | bauzas: where {} is returned only when we got empty allocations back | |
| 13:25:53 | dansmith | mdbooth: you can apply it to a smaller scope to avoid the race with the rpc call/notification stuff, | |
| 13:25:58 | mdbooth | dansmith: I thought you wanted me to split out the individual function changes. | |
| 13:26:02 | dansmith | and after that you can remove the state exclusion rght? | |
| 13:26:22 | bauzas | gibi: yup, that | |
| 13:26:24 | dansmith | mdbooth: well, I do, if possible | |
| 13:26:34 | gibi | bauzas: OK, I will rework the patch | |
| 13:26:36 | gibi | bauzas: thanks | |
| 13:26:37 | mdbooth | Those 2 really are the same issue. | |
| 13:26:50 | openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660 | |
| 13:26:56 | dansmith | mdbooth: how? | |
| 13:27:00 | bauzas | gibi: thanks for working on it :) | |
| 13:27:22 | mdbooth | Because the exclusion is the current mechanism by which we reduce the scope of errors_out_migration | |
| 13:28:18 | mdbooth | If you do it lexically, the exclusion no longer has a user. | |
| 13:28:23 | dansmith | mdbooth: but your commit message says there's no reason for that code at all right? | |
| 13:28:35 | mdbooth | The exclusion code? | |
| 13:28:35 | gibi | bauzas: I have another placement question if you have a minute | |
| 13:28:57 | dansmith | mdbooth: the state exclusion yeah | |
| 13:29:00 | mdbooth | Its purpose is undocumented, and I've only guessed what it is | |
| 13:29:07 | bauzas | gibi: sure | |
| 13:29:15 | mdbooth | But I'm assuming that the person who put it there did it deliberately | |
| 13:29:39 | gibi | bauzas: when the scheduler claims the resources it is possible that there is a conflict | |
| 13:29:46 | mdbooth | It appears to be to prevent the migration being put in an error state after certain points in the task | |
| 13:29:55 | gibi | bauzas: I see that placement return http 409 | |
| 13:30:00 | gibi | bauzas: and that seems correct to me | |
| 13:30:15 | gibi | bauzas: I mean when there is really a conflict | |
| 13:30:22 | bauzas | gibi: yup, correct | |
| 13:30:24 | dansmith | mdbooth: right but I don't see how removing that (which you think needs to happen) is atomically related to the other scope change | |
| 13:30:36 | dansmith | mdbooth: I gotta join this terrible 6:30am call now, so do whatever you think is right I guess | |
| 13:30:47 | gibi | bauzas: but at the same time placement logs a ERROR log about the invalid inventory and that seems not that nice as it suggest a software error for me | |
| 13:31:07 | mdbooth | dansmith: Enjoy :) | |
| 13:31:22 | bauzas | gibi: when PUT /allocation ? | |
| 13:31:39 | bauzas | gibi: if so, yup, it seems weirdo to pass a error level log | |
| 13:32:18 | gibi | bauzas: here is the stack trace https://pastebin.com/cJP97xQr | |
| 13:32:51 | openstackgerrit | Andrey Volkov proposed openstack/nova master: PoC: Select PCI devices with distinct tag values https://review.openstack.org/448008 | |
| 13:33:02 | gibi | bauzas: hm, it might not coming from the claiming | |
| 13:33:51 | gibi | bauzas: ahh I found it, it is a "PUT /placement/allocations/485c7480-939c-4b88-8c00-0f346dc6a924" | |
| 13:33:54 | gibi | bauzas: so yes | |
| 13:34:09 | gibi | bauzas: then I will file a bug for this as well | |
| 13:34:42 | gibi | bauzas: should this log be on info or debug level? | |
| 13:34:58 | bauzas | debug IMHO | |
| 13:35:11 | bauzas | because it just means that's a race condition | |
| 13:35:56 | gibi | bauzas: OK, thanks | |
| 13:37:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/485634 | |
| 13:39:30 | openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660 | |
| 13:44:34 | gibi | bauzas: here is the bug report for the error log https://bugs.launchpad.net/nova/+bug/1705487 | |
| 13:44:37 | openstack | Launchpad bug 1705487 in OpenStack Compute (nova) "placement logs an ERROR when PUT /allocation result in an invalid inventory" [Undecided,New] | |
| 13:55:15 | ftersin | mdbooth: hi. If you have a minute, could you look at ScaleIO review (https://review.openstack.org/#/c/407440/)? I addressed your (and mriedem) comments there, and now ready for new ones. | |
| 14:05:50 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: serial console log (completed) https://review.openstack.org/450636 | |
| 14:08:06 | mriedem | andreykurilin: melwitt: we need to get https://review.openstack.org/#/c/484152/ in to unblock novaclient tests | |
| 14:08:17 | mriedem | now that the counting instance quotas change merged in nova | |
| 14:08:23 | mriedem | or sdague ^ | |
| 14:09:24 | openstackgerrit | Merged openstack/nova master: [placement] cover deleting standard trait https://review.openstack.org/484153 | |
| 14:09:33 | andreykurilin | mriedem: done | |
| 14:09:43 | mriedem | gibi: just wanted to say thanks for kicking the tires on the scheduler + placement stuff, you're finding some nice hairy bugs | |
| 14:09:45 | mriedem | andreykurilin: thanks | |
| 14:11:46 | openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660 | |
| 14:18:45 | openstackgerrit | Kaitlin Farr proposed openstack/nova master: Remove deprecated keymgr code https://review.openstack.org/439855 | |
| 14:19:49 | gibi | mriedem: my pleasure :) | |
| 14:21:02 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 14:21:56 | sdague | mriedem: lookin | |
| 14:22:09 | sdague | mriedem: ah, andreykurilin already snagged it | |
| 14:22:21 | sdague | mriedem: you got a few moments to think about path forward on request logging? | |
| 14:23:24 | mriedem | sure, although it's over my head | |
| 14:24:17 | sdague | ok, well do you get my concern that we're not logging it through the python subsystem any more? | |
| 14:24:39 | mriedem | yeah it's definitely a problem, and as noted i'm seeing weird formatting | |
| 14:24:48 | mriedem | like parts of the log message are chopped off | |
| 14:25:20 | sdague | mriedem: you see my follow ups? | |
| 14:25:28 | mriedem | reading those now | |
| 14:25:50 | sdague | ok, do that, then we can chat, so I don't repeat myself :) | |
| 14:28:03 | mriedem | sdague: wait, say that again | |
| 14:29:22 | mriedem | sdague: oh man ok so that debug log message starts here then http://logs.openstack.org/65/483565/4/check/gate-tempest-dsvm-py35-ubuntu-xenial/9921636/logs/screen-n-sch.txt.gz#_Jul_19_20_17_18_800467 | |
| 14:29:24 | openstackgerrit | Alex Szarka proposed openstack/nova master: Reduce code complexity - manager.py https://review.openstack.org/359868 | |
| 14:29:27 | mriedem | and takes 3 full chunks | |
| 14:29:39 | sdague | mriedem: yep | |
| 14:29:39 | mriedem | make that 4 chunks | |
| 14:29:58 | mriedem | yikes, maybe we shouldn't log full instances on a cron in the scheduler | |
| 14:30:03 | jaypipes | mriedem: fyi, in and out this morning... more plumbing debacles at chez pipes. | |
| 14:30:19 | mriedem | that's only 3 instances, | |
| 14:30:37 | mriedem | think if you have 10K computes sending the update_instance_info periodic to the scheduler every minute, which is the default | |
| 14:30:42 | mriedem | with 1 million VMs | |
| 14:31:09 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: Handle missing volume vmdk during detach https://review.openstack.org/484675 | |
| 14:31:15 | mriedem | jaypipes: ok. btw, i started my morning with an internal email from a guy doing scale testing asking about issues with booting 3000+ VMs at once | |
| 14:31:28 | mriedem | and multiple scheduler workers, in mitaka | |
| 14:31:38 | jaypipes | mriedem: lovely. | |
| 14:31:58 | jaypipes | mriedem: I have an afternoon call to talk about why Nova AZs aren't AWS AZs. | |
| 14:32:00 | dansmith | jaypipes: chez pipes sounds like a slum | |
| 14:33:16 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Add datapath type information to OVS vif objects https://review.openstack.org/474892 | |
| 14:34:02 | mriedem | btw this is a good example of why you can't run the scheduler with debug logging enabled | |
| 14:34:03 | mriedem | in prod | |
| 14:34:40 | dansmith | but you also can't run it without debug in prod | |
| 14:34:56 | mriedem | this reminds me of a bug rlrossit found | |
| 14:35:31 | mriedem | all the logging that happens in here https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L166 | |
| 14:36:06 | mriedem | when we turned on debug it killed the scheduler in our pre-prod cloud | |
| 14:37:22 | mriedem | sdague: so you've got my attention | |
| 14:37:41 | sdague | mriedem: ok, so the challenges / questions | |
| 14:38:00 | sdague | 1) do we need something like the request_log middleware? (assume answer is yes) | |