| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-20 | |||
| 14:09:09 | mdbooth | 2. ... | |
| 14:09:37 | sahid | but we need to be clear on what we are talking about, the qemu process or the guest operating system ? | |
| 14:09:57 | mdbooth | s/qemu process/hypervisor/ | |
| 14:10:12 | mdbooth | The aspect of reboot I'm talking about relates to the hypervisor | |
| 14:10:17 | mdbooth | Not the guest OS | |
| 14:10:34 | sahid | that seems really sad so... we really want to fix an issue like that | |
| 14:10:40 | mdbooth | What issue? | |
| 14:10:49 | sahid | the one you are talking about, no ? | |
| 14:10:56 | mdbooth | I'm not talking about an issue | |
| 14:11:02 | mdbooth | I'm talking about *any* issue | |
| 14:11:24 | mdbooth | I'm talking about an imaginary, as yet undiscovered bug which has resulted in inconsistent state in the hypervisor | |
| 14:11:49 | mdbooth | One of the goals of reboot is to try to get the instance going again anyway | |
| 14:12:03 | mdbooth | It would be nice if it wasn't | |
| 14:12:12 | mdbooth | But to the best of my knowledge, it is | |
| 14:15:54 | sahid | ok so you don't want me to fix this because there is an imaginary/undiscovered bug | |
| 14:16:03 | mdbooth | Incidentally, as I mentioned in one of my review comments, I think this is a very useful tool for operators to have. However, I also think it's problematic and ironically a potential source of bugs to do it in reboot. | |
| 14:16:13 | sahid | interesting... i have 2 well real bugs to show you because of this code | |
| 14:16:34 | mdbooth | sahid: No, I don't want you to do this because (IIUC) operators rely on this behaviour. | |
| 14:16:47 | mdbooth | Ideally we would separate it from reboot. | |
| 14:17:00 | sahid | if they do, so we should explain to them they should not i guess | |
| 14:17:11 | sahid | rscue and rebuild are better | |
| 14:17:12 | mdbooth | However, if it's a deliberate design decision, and something that users expect to be there, we can't just take it away without thought or acknowledgement. | |
| 14:17:27 | mdbooth | No, rescue doesn't help at all, and rebuild is entirely different. | |
| 14:17:52 | sahid | because hard-reboot has been designed such issue ? | |
| 14:18:53 | mdbooth | sahid: I suggest a bp. You might also post on the operators mailing list asking for input. | |
| 14:19:05 | mdbooth | It would also need a release note, I guess. | |
| 14:19:24 | sdague | bauzas: are you sure that placement is loading the noauth middleware? | |
| 14:19:31 | bauzas | sdague: yup | |
| 14:19:37 | bauzas | sdague: I just verified | |
| 14:19:50 | sdague | bauzas: if you push a patch I can look | |
| 14:20:00 | bauzas | sdague: okay, I'll do | |
| 14:20:48 | bauzas | mmm, I think I found the issue | |
| 14:23:33 | sahid | mdbooth: well if that could help i can provide a release note or a bp to fix this issue even if i'm not bringing any feature here | |
| 14:24:10 | mdbooth | sahid: I just want you to acknowledge and address a change in operator-expected behaviour. | |
| 14:24:40 | sahid | mdbooth: you are saying that but i never hear about this point, can you share to me some pointer ? | |
| 14:24:56 | sahid | a comment in lunchpad, a gerrit review or a thread in the ml ? | |
| 14:24:57 | mdbooth | No, I really can't. I've never seen it documented. | |
| 14:25:11 | sdague | mdbooth: I don't remember why the domain xml is rewriten, git log -S might find some things interesting to figure out if the relevant commit messages have clues | |
| 14:37:05 | bauzas | sdague: okay, so found the problem | |
| 14:37:43 | bauzas | sdague: https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L43-L44 | |
| 14:37:52 | bauzas | sdague: I just need to fake a token | |
| 14:38:31 | bauzas | sdague: do you know if we have a fake token lib or helper ? :) | |
| 14:39:23 | sdague | bauzas: https://github.com/openstack/nova/blob/9acfef2c90186644120942558437b3fcc90222de/nova/api/openstack/auth.py#L32-L47 | |
| 14:39:29 | sdague | it looks like that bit wasn't copied over | |
| 14:40:03 | bauzas | sdague: yeah, possibly because gabbi provides a fake token | |
| 14:40:12 | sdague | bauzas: could be | |
| 14:40:23 | bauzas | we miss cdent here | |
| 14:40:27 | sdague | honestly, I have no idea about the gabbi magic | |
| 14:40:36 | bauzas | me too | |
| 14:40:55 | bauzas | but since cdent wrote that noauth middleware for the gabbi functests, I guess it's why | |
| 14:41:30 | bauzas | I could just fake a token honestly and leave a big note for saying that we should rather have a better noauth middleware | |
| 14:42:09 | leakypipes | sdague, bauzas: there's no magic in gabbi... at least, certainly no more magic than the existing functional API tests. | |
| 14:42:43 | bauzas | leakypipes: well the problem with gabbi is that we haven't yet a correct fixture | |
| 14:43:02 | sdague | yeh, the noauthmiddleware is actually very gabbi specific | |
| 14:43:21 | sdague | bauzas: you might be able to replumb the existing noauth bits in there instead | |
| 14:43:24 | bauzas | and the current APIFixture we have in-tree is also a gabbi child | |
| 14:43:47 | bauzas | sdague: maybe | |
| 14:44:04 | bauzas | sdague: I just don't want to have a long patch to be reviewed | |
| 14:44:53 | sdague | bauzas: well, a long working patch is better than a non working one | |
| 14:45:45 | bauzas | sdague: leakypipes: bingo https://github.com/openstack/nova/blob/master/nova/tests/functional/api/openstack/placement/gabbits/aggregate.yaml#L7 | |
| 14:45:54 | leakypipes | sdague, bauzas: guys... | |
| 14:46:19 | leakypipes | sdague, bauzas: what is it you are trying to do? | |
| 14:46:20 | bauzas | leakypipes: because of that, we could be missing the placement upcall from the scheduler for Ocata... | |
| 14:46:42 | bauzas | just because we wanted to have a very separate WSGI app than the existing one | |
| 14:46:49 | bauzas | and using gabbi too | |
| 14:46:53 | leakypipes | bauzas: I'm not following you. How does a functional test fixture that disables Keystone auth impact the "scheduler upcall"? | |
| 14:46:57 | sdague | leakypipes: there is a need to get placement running inside the functional tests that aren't gabbi so you can have the whole working service collection | |
| 14:47:10 | bauzas | leakypipes: see my scheduler change and the functional tests job | |
| 14:47:33 | bauzas | we're getting NoValidHosts because the placement service isn't running | |
| 14:47:44 | bauzas | so I'm working since this morning on adding a fixture | |
| 14:47:46 | leakypipes | bauzas: and how is that related to gabbi? | |
| 14:48:03 | bauzas | leakypipes: because we have an existing fixture that is only gabbi'd | |
| 14:48:20 | mriedem | so let's ignore gabbi :) | |
| 14:48:20 | leakypipes | bauzas: I have no idea what that means. | |
| 14:48:24 | mriedem | the point is, | |
| 14:48:30 | bauzas | and when we added the placement service in Newton, we haven't verified that we could be adding a fixture for the existing tests | |
| 14:48:38 | bauzas | anyway | |
| 14:48:40 | mriedem | we need to run the placement service with a fixture, like we do for other services | |
| 14:48:48 | mriedem | like n-cond, n-sch, n-cpu, etc in the functional tets | |
| 14:48:49 | mriedem | *tests | |
| 14:48:50 | bauzas | I don't want to diverge too moch | |
| 14:48:51 | sdague | leakypipes: there exists no infrastructure in nova functional tests to start a running placement service outside of gabbi tests | |
| 14:48:53 | mriedem | those rely on a noauth fixture | |
| 14:48:55 | mriedem | since we don't have keystone | |
| 14:48:59 | bauzas | given the very short deadline | |
| 14:49:09 | bauzas | so folks, I'll be providing a WIP by the next 10 mins | |
| 14:49:37 | mriedem | we have an api client fixture for n-api for the functional tests, seems we'd want something similar for placement, | |
| 14:49:56 | mriedem | or simply have a PlacementFixture which isn't a real service | |
| 14:49:57 | sdague | mriedem: it's a little different than that because of the way things glue together | |
| 14:50:06 | sdague | but, yeh, bauzas is working through it | |
| 14:50:08 | mriedem | because when placement is split out, we'll need to just stub it anyway | |
| 14:50:11 | mriedem | like neutron | |
| 14:50:16 | bauzas | sdague: mriedem: lemme upload it | |
| 14:51:20 | leakypipes | bauzas: you're saying you need one of these for the placement API service, yes? https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L638 | |
| 14:51:37 | bauzas | leakypipes: I'm just adding it now since this morning | |
| 14:51:46 | bauzas | leakypipes: lemme upload it in the next 10 mins | |
| 14:52:20 | leakypipes | bauzas: ok. I just get annoyed with the kvetching about gabbi and "a separate WSGI service" when really this isn't that hard of a problem to solve. | |
| 14:52:51 | bauzas | leakypipes: it took me 1 hour for finding a solution because the WSGI placement service is not an oslo.service | |
| 14:52:55 | leakypipes | bauzas: gabbi tests a REST API interface. it doesn't test integration between two services. | |
| 14:53:04 | bauzas | fortunately, sdague gave me a good solution | |