| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-18 | |||
| 14:52:49 | mriedem | because of the associated bug | |
| 14:54:41 | dims | mriedem : +1, though we should probably add an upgrade note to ask operators to make sure that option is set in everyone's nova.conf | |
| 14:57:24 | mriedem | dims: hmm, if we have to add an upgrade release note then i worry about backporting it | |
| 14:57:48 | mriedem | dims: honestly i'm not really sure why the set_default there is overriding what the operator puts in the config, | |
| 14:57:52 | mriedem | but that was reported yesterday | |
| 14:58:00 | mriedem | dims: if you could figure that out it would be helpful | |
| 14:59:31 | dims | mriedem : it should not be overriding what's in nova.conf, it just sets the default value if there's nothing in nova.conf | |
| 14:59:59 | dims | mriedem : http://git.openstack.org/cgit/openstack/oslo.config/tree/oslo_config/cfg.py#n2736 | |
| 15:01:18 | gibi | jaypipes: with cdent fix I still get the same KeyError as before, logs are here http://paste.openstack.org/show/615743/ | |
| 15:01:23 | keerthi | scheduling always happen only 10 at a time in the nova ? | |
| 15:01:38 | keerthi | shall we change this value ? | |
| 15:03:15 | keerthi | mriedem ? | |
| 15:04:11 | jaypipes | gibi: k, thanks for that log output, that's helpful. I'm wondering if alex_xu's patch here might help: https://review.openstack.org/#/c/480379/ | |
| 15:04:25 | gibi | jaypipes: I can test that as well | |
| 15:04:25 | jaypipes | gibi: sorry to bother you on this, but do you think you can apply that patch too? | |
| 15:04:34 | jaypipes | gibi: ty sir, much appreciated! | |
| 15:04:51 | jaypipes | gibi: in the meantime, I am working on adding functional test scenarios for your specific case. | |
| 15:05:12 | gibi | jaypipes: put me in the review if any I will check that out | |
| 15:05:19 | jaypipes | gibi: cheers | |
| 15:05:21 | alex_xu | mriedem: replied https://review.openstack.org/#/c/469800/, if we really think it is time to merge this patch first, I can change to +w | |
| 15:05:48 | jaypipes | gibi: köszönöm! | |
| 15:06:02 | mriedem | dims: yeah that's what i thought, but he was saying he had a value set in nova.conf and it would still use the default from code, and removing that line fixed it | |
| 15:06:18 | gibi | jaypipes: szivesen | |
| 15:06:24 | mriedem | dims: i don't have a devstack handy to recreate | |
| 15:06:37 | mriedem | keerthi: scheduling what? server create? | |
| 15:06:44 | keerthi | yes... | |
| 15:07:23 | dims | mriedem : barbican has a DSVM where this scenario works just fine. nova.conf has the BarbicanKeyManager and it gets picked up ok | |
| 15:07:24 | keerthi | when ever i create boot request...scheduling happens only for 10 at a time | |
| 15:08:33 | keerthi | rest of the requests need to wait in the queue and it starts proceeding next..is there way we can change this behaviour in nova ? | |
| 15:08:36 | dims | mriedem : oops, castellan has a dsvm test - see http://logs.openstack.org/19/476819/1/check/gate-castellan-dsvm-functional/c6e0d8c/logs/etc/nova/nova.conf.txt.gz | |
| 15:08:45 | keerthi | mriedem ? | |
| 15:09:26 | dims | mriedem : api log - http://logs.openstack.org/19/476819/1/check/gate-castellan-dsvm-functional/c6e0d8c/logs/screen-n-api.txt.gz#_Jun_23_08_05_22_681652 | |
| 15:09:31 | mriedem | keerthi: nothing comes to mind | |
| 15:10:30 | keerthi | jaypipes can you help me in solving my issue ? | |
| 15:10:31 | mriedem | dims: hmm, yeah, some of the options stuff was also changed in i think pike, so i'm wondering if it was a latent bug in newton | |
| 15:11:20 | jaypipes | keerthi: please see /topic. this isn't a support channel. better to post a question to the openstack mailing list please. | |
| 15:12:12 | mriedem | keerthi: you probably have max_concurrent_builds defaulting to 10 | |
| 15:12:14 | dims | mriedem : nothing has changed in set_default (oslo.config) or in nova keymgr in a while.... will search though | |
| 15:12:16 | mriedem | which is used in the ompute | |
| 15:12:17 | mriedem | *compute | |
| 15:12:19 | mriedem | not the scheduler | |
| 15:12:21 | mriedem | jaypipes: ^ | |
| 15:12:52 | keerthi | Thanks mriedem. i will look in to this | |
| 15:19:23 | mriedem | jaypipes: not-tags-any is tested in https://review.openstack.org/#/c/469800/ | |
| 15:19:35 | mriedem | https://review.openstack.org/#/c/469800/35/nova/tests/functional/wsgi/test_servers.py@237 | |
| 15:19:49 | mriedem | the 4 filters are tested in the same functional test, there just need to be more wrinkles it sounds like | |
| 15:20:19 | mriedem | and we can't do any of this in sql | |
| 15:20:25 | mriedem | which sucks, but it is what it is | |
| 15:21:17 | jaypipes | mriedem: ok, fair enough. will remove my -1. | |
| 15:21:41 | gibi | jaypipes: with alex_xu's https://review.openstack.org/#/c/480379/ I still get the same stack trace. logs are here http://paste.openstack.org/show/615747/ | |
| 15:22:00 | gibi | jaypipes: I will try to combine cdent's and alex_xu's patch together | |
| 15:22:00 | mriedem | thanks | |
| 15:22:11 | jaypipes | gibi: k | |
| 15:29:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Test changes with multiple cells https://review.openstack.org/467383 | |
| 15:29:48 | jaypipes | gibi: hmm, ok I have an idea... | |
| 15:30:33 | jaypipes | gibi: can you do me a favor? | |
| 15:30:37 | jaypipes | gibi: these two lines: https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L2393-L2394 | |
| 15:30:47 | gibi | jaypipes: sure | |
| 15:30:55 | jaypipes | gibi: can you change them to have the usage fields on the *right* side of the condition? | |
| 15:31:21 | jaypipes | gibi: in other words, they should be: inv.c.resource_provider_id == usage.c.resource_provider_id | |
| 15:31:29 | jaypipes | and the same for the resource_class_id columns. | |
| 15:31:34 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform libvirt.error notification https://review.openstack.org/484851 | |
| 15:31:35 | gibi | jaypipes: sure I can do that. Meanwhile the two combined patches resulted the same HTTP 500 | |
| 15:31:58 | gibi | jaypipes: do you need that change top of master or top of some bugfixes? | |
| 15:32:05 | gibi | jaypipes: or doesnt matter | |
| 15:32:10 | jaypipes | gibi: I'm wondering if SQLAlchemy is constructing the LEFT JOIN verbatim with that order (which is an incorrect join order) | |
| 15:32:25 | jaypipes | gibi: just make it locally on whatever code you have running. | |
| 15:32:31 | gibi | jaypipes: OK | |
| 15:33:35 | jaypipes | zzzeek: hey Mike, does SA rewrite outerjoins to be the "correct" join condition column order if there's a mistake in the order of the join condition? :) see https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L2393-L2394 | |
| 15:33:39 | mgiles | mriedem ildikov I took a pass as the grenade tests that stvnoyes was going to work on. See: https://review.openstack.org/#/c/484469/ | |
| 15:33:52 | jaypipes | zzzeek: it would be awesome if SA solved all my mistakes for me :P | |
| 15:35:11 | ildikov | mgiles: great, thank you! | |
| 15:36:22 | zzzeek | jaypipes: column order....i dont think so? you mean "A LEFT OUTER JOIN B" and not "B LEFT OUTER JOIN A" ? | |
| 15:37:49 | zzzeek | jaypipes: like you want the two conditions to be in some order to satisfy an index or something ? | |
| 15:39:47 | jaypipes | zzzeek: no, I was mostly joking with you :) I have the SQL correct in the code comment above there but have the column order wrong in the SQLalchemy join. | |
| 15:40:25 | zzzeek | jaypipes: the SQL is rendering differently from what you specify ? | |
| 15:40:51 | ildikov | stephenfin: happy to try to answer questions if you can take a look at the live_migrate patch :) | |
| 15:41:15 | jaypipes | zzzeek: not sure, but I suspect it would be. I mean, I'm asking SQLalchemy to do a LEFT JOIN using the wrong order of columns in the join condition. | |
| 15:41:47 | jaypipes | zzzeek: it's my mistake, not SA's :) I was just joking with you about having SA read my mind ;) | |
| 15:42:14 | zzzeek | jaypipes: i still don't understand what "wrong order of columns" means | |
| 15:42:59 | jaypipes | zzzeek: oh, I'm saying that I should be doing a LEFT JOIN b ON a.col = b.col, but I'm asking SA to do a LEFT JOIN b ON b.col = a.col | |
| 15:43:14 | zzzeek | jaypipes: OK you mean in the condition around an operator | |
| 15:43:18 | jaypipes | and maybe SA is doing b LEFT JOIN a ON b.col = a.col | |
| 15:43:24 | jaypipes | zzzeek: ya | |
| 15:43:25 | zzzeek | jaypipes: if you are doing "col <operator> col" it should preserve that order | |
| 15:43:40 | jaypipes | zzzeek: right, and that order is wrong :) | |
| 15:43:46 | zzzeek | jaypipes: only if you have "<some literal python thing> <operator> col" might it switch things because the __eq__ operator is on the right | |
| 15:43:48 | jaypipes | zzzeek: thus me saying it was my mistake ;) | |
| 15:44:19 | gibi | jaypipes: I'm getting the same stacktrace after reodering the condition. git diff is on the top of the logs: http://paste.openstack.org/show/615749/ | |
| 15:44:20 | zzzeek | jaypipes: OK so, if you have "col <operator> othercol" that left/right is maintained | |
| 15:44:43 | zzzeek | jaypipes: also, it....shouldnt matter? unless you're trying to hit an index on oracle | |
| 15:45:22 | zzzeek | jaypipes: == operator is commutative... | |
| 15:45:50 | jaypipes | gibi: :( ok, back to the drawing board. I really don't know why a KeyError is being raised there. the root provider ID should be in the summaries dict since _get_usages_by_rp_and_rc() should be returning a record for that rp | |
| 15:46:12 | gibi | jaypipes: is there any log I can turn on to help? | |
| 15:46:34 | gibi | jaypipes: or if you provide a patch with extra LOGs then I can apply that | |
| 15:46:35 | jaypipes | zzzeek: right, but I was thinking maybe SA saw the == operator column order and maybe made the expression b LEFT JOIN a instead of a LEFT JOIN b. | |
| 15:46:41 | jaypipes | zzzeek: apparently not, though | |
| 15:46:55 | jaypipes | gibi: I'll do the latter | |
| 15:47:25 | gibi | jaypipes: OK. I'm still around for an hour or so then I can continue tomorrow | |
| 15:47:32 | gibi | jaypipes: thank again for helping | |