| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-20 | |||
| 14:10:56 | mdbooth | I'm not talking about an issue | |
| 14:11:02 | mdbooth | I'm talking about *any* issue | |
| 14:11:24 | mdbooth | I'm talking about an imaginary, as yet undiscovered bug which has resulted in inconsistent state in the hypervisor | |
| 14:11:49 | mdbooth | One of the goals of reboot is to try to get the instance going again anyway | |
| 14:12:03 | mdbooth | It would be nice if it wasn't | |
| 14:12:12 | mdbooth | But to the best of my knowledge, it is | |
| 14:15:54 | sahid | ok so you don't want me to fix this because there is an imaginary/undiscovered bug | |
| 14:16:03 | mdbooth | Incidentally, as I mentioned in one of my review comments, I think this is a very useful tool for operators to have. However, I also think it's problematic and ironically a potential source of bugs to do it in reboot. | |
| 14:16:13 | sahid | interesting... i have 2 well real bugs to show you because of this code | |
| 14:16:34 | mdbooth | sahid: No, I don't want you to do this because (IIUC) operators rely on this behaviour. | |
| 14:16:47 | mdbooth | Ideally we would separate it from reboot. | |
| 14:17:00 | sahid | if they do, so we should explain to them they should not i guess | |
| 14:17:11 | sahid | rscue and rebuild are better | |
| 14:17:12 | mdbooth | However, if it's a deliberate design decision, and something that users expect to be there, we can't just take it away without thought or acknowledgement. | |
| 14:17:27 | mdbooth | No, rescue doesn't help at all, and rebuild is entirely different. | |
| 14:17:52 | sahid | because hard-reboot has been designed such issue ? | |
| 14:18:53 | mdbooth | sahid: I suggest a bp. You might also post on the operators mailing list asking for input. | |
| 14:19:05 | mdbooth | It would also need a release note, I guess. | |
| 14:19:24 | sdague | bauzas: are you sure that placement is loading the noauth middleware? | |
| 14:19:31 | bauzas | sdague: yup | |
| 14:19:37 | bauzas | sdague: I just verified | |
| 14:19:50 | sdague | bauzas: if you push a patch I can look | |
| 14:20:00 | bauzas | sdague: okay, I'll do | |
| 14:20:48 | bauzas | mmm, I think I found the issue | |
| 14:23:33 | sahid | mdbooth: well if that could help i can provide a release note or a bp to fix this issue even if i'm not bringing any feature here | |
| 14:24:10 | mdbooth | sahid: I just want you to acknowledge and address a change in operator-expected behaviour. | |
| 14:24:40 | sahid | mdbooth: you are saying that but i never hear about this point, can you share to me some pointer ? | |
| 14:24:56 | sahid | a comment in lunchpad, a gerrit review or a thread in the ml ? | |
| 14:24:57 | mdbooth | No, I really can't. I've never seen it documented. | |
| 14:25:11 | sdague | mdbooth: I don't remember why the domain xml is rewriten, git log -S might find some things interesting to figure out if the relevant commit messages have clues | |
| 14:37:05 | bauzas | sdague: okay, so found the problem | |
| 14:37:43 | bauzas | sdague: https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L43-L44 | |
| 14:37:52 | bauzas | sdague: I just need to fake a token | |
| 14:38:31 | bauzas | sdague: do you know if we have a fake token lib or helper ? :) | |
| 14:39:23 | sdague | bauzas: https://github.com/openstack/nova/blob/9acfef2c90186644120942558437b3fcc90222de/nova/api/openstack/auth.py#L32-L47 | |
| 14:39:29 | sdague | it looks like that bit wasn't copied over | |
| 14:40:03 | bauzas | sdague: yeah, possibly because gabbi provides a fake token | |
| 14:40:12 | sdague | bauzas: could be | |
| 14:40:23 | bauzas | we miss cdent here | |
| 14:40:27 | sdague | honestly, I have no idea about the gabbi magic | |
| 14:40:36 | bauzas | me too | |
| 14:40:55 | bauzas | but since cdent wrote that noauth middleware for the gabbi functests, I guess it's why | |
| 14:41:30 | bauzas | I could just fake a token honestly and leave a big note for saying that we should rather have a better noauth middleware | |
| 14:42:09 | leakypipes | sdague, bauzas: there's no magic in gabbi... at least, certainly no more magic than the existing functional API tests. | |
| 14:42:43 | bauzas | leakypipes: well the problem with gabbi is that we haven't yet a correct fixture | |
| 14:43:02 | sdague | yeh, the noauthmiddleware is actually very gabbi specific | |
| 14:43:21 | sdague | bauzas: you might be able to replumb the existing noauth bits in there instead | |
| 14:43:24 | bauzas | and the current APIFixture we have in-tree is also a gabbi child | |
| 14:43:47 | bauzas | sdague: maybe | |
| 14:44:04 | bauzas | sdague: I just don't want to have a long patch to be reviewed | |
| 14:44:53 | sdague | bauzas: well, a long working patch is better than a non working one | |
| 14:45:45 | bauzas | sdague: leakypipes: bingo https://github.com/openstack/nova/blob/master/nova/tests/functional/api/openstack/placement/gabbits/aggregate.yaml#L7 | |
| 14:45:54 | leakypipes | sdague, bauzas: guys... | |
| 14:46:19 | leakypipes | sdague, bauzas: what is it you are trying to do? | |
| 14:46:20 | bauzas | leakypipes: because of that, we could be missing the placement upcall from the scheduler for Ocata... | |
| 14:46:42 | bauzas | just because we wanted to have a very separate WSGI app than the existing one | |
| 14:46:49 | bauzas | and using gabbi too | |
| 14:46:53 | leakypipes | bauzas: I'm not following you. How does a functional test fixture that disables Keystone auth impact the "scheduler upcall"? | |
| 14:46:57 | sdague | leakypipes: there is a need to get placement running inside the functional tests that aren't gabbi so you can have the whole working service collection | |
| 14:47:10 | bauzas | leakypipes: see my scheduler change and the functional tests job | |
| 14:47:33 | bauzas | we're getting NoValidHosts because the placement service isn't running | |
| 14:47:44 | bauzas | so I'm working since this morning on adding a fixture | |
| 14:47:46 | leakypipes | bauzas: and how is that related to gabbi? | |
| 14:48:03 | bauzas | leakypipes: because we have an existing fixture that is only gabbi'd | |
| 14:48:20 | leakypipes | bauzas: I have no idea what that means. | |
| 14:48:20 | mriedem | so let's ignore gabbi :) | |
| 14:48:24 | mriedem | the point is, | |
| 14:48:30 | bauzas | and when we added the placement service in Newton, we haven't verified that we could be adding a fixture for the existing tests | |
| 14:48:38 | bauzas | anyway | |
| 14:48:40 | mriedem | we need to run the placement service with a fixture, like we do for other services | |
| 14:48:48 | mriedem | like n-cond, n-sch, n-cpu, etc in the functional tets | |
| 14:48:49 | mriedem | *tests | |
| 14:48:50 | bauzas | I don't want to diverge too moch | |
| 14:48:51 | sdague | leakypipes: there exists no infrastructure in nova functional tests to start a running placement service outside of gabbi tests | |
| 14:48:53 | mriedem | those rely on a noauth fixture | |
| 14:48:55 | mriedem | since we don't have keystone | |
| 14:48:59 | bauzas | given the very short deadline | |
| 14:49:09 | bauzas | so folks, I'll be providing a WIP by the next 10 mins | |
| 14:49:37 | mriedem | we have an api client fixture for n-api for the functional tests, seems we'd want something similar for placement, | |
| 14:49:56 | mriedem | or simply have a PlacementFixture which isn't a real service | |
| 14:49:57 | sdague | mriedem: it's a little different than that because of the way things glue together | |
| 14:50:06 | sdague | but, yeh, bauzas is working through it | |
| 14:50:08 | mriedem | because when placement is split out, we'll need to just stub it anyway | |
| 14:50:11 | mriedem | like neutron | |
| 14:50:16 | bauzas | sdague: mriedem: lemme upload it | |
| 14:51:20 | leakypipes | bauzas: you're saying you need one of these for the placement API service, yes? https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L638 | |
| 14:51:37 | bauzas | leakypipes: I'm just adding it now since this morning | |
| 14:51:46 | bauzas | leakypipes: lemme upload it in the next 10 mins | |
| 14:52:20 | leakypipes | bauzas: ok. I just get annoyed with the kvetching about gabbi and "a separate WSGI service" when really this isn't that hard of a problem to solve. | |
| 14:52:51 | bauzas | leakypipes: it took me 1 hour for finding a solution because the WSGI placement service is not an oslo.service | |
| 14:52:55 | leakypipes | bauzas: gabbi tests a REST API interface. it doesn't test integration between two services. | |
| 14:53:04 | bauzas | fortunately, sdague gave me a good solution | |
| 14:54:24 | sdague | leakypipes: which we all understand, it's that we also test more than just that, and it wasn't anticipated. And because this was plumbed differently it takes more effort to plumb something in. | |
| 14:58:48 | sdague | oh, man, so happy to have gotten the anchor issue fixed in os-api-ref | |
| 14:58:59 | sdague | links should be a ton more useful in sharing now | |
| 14:59:39 | sdague | http://developer.openstack.org/api-ref/compute/?expanded=pause-server-pause-action-detail#pause-server-pause-action - for instance | |
| 14:59:41 | melwitt | johnthetubaguy: that's an excellent question. I should have asked you to review earlier :) what mriedem said is right, it was separated based on resource type (fixed_ips, networks, floating_ips in Quotas, everything else in ProjectUserQuota). I'll take a closer look and think about it | |
| 15:01:01 | johnthetubaguy | melwitt: oh, I thought everything could be in both, depending on the resource type, its not as bad if we just get to kill a table later | |
| 15:01:45 | johnthetubaguy | melwitt: but reading more, I am leaning towards the direct swap being simpler | |
| 15:01:57 | melwitt | johnthetubaguy: I think you're probably right. just not totally sure off the top of my head. and yes, also true that at worst (if we don't change it now) that it would be a table kill later | |