| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-20 | |||
| 13:34:32 | mfeoktistov | Hello! Dear cores, can you please look at https://review.openstack.org/#/c/312488/ | |
| 13:36:45 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova: Updated from global requirements https://review.openstack.org/422323 | |
| 13:37:33 | sdague | mriedem: cool | |
| 13:39:52 | sdague | mriedem: once I get some breakfast and this set of grenade results, I'll take a look | |
| 13:40:28 | mriedem | the d-g change isn't really ready yet, i need to make it conditional on branches, or put a script in devstack to run discover_hosts and then d-g would only run it if that script exists | |
| 13:40:37 | mriedem | https://review.openstack.org/#/c/423005/2/devstack-vm-gate.sh | |
| 13:41:23 | mriedem | if you have input on what's the preferred approach there that would be good | |
| 13:42:13 | leakypipes | tbachman: :) | |
| 13:43:45 | mdbooth | mriedem: NFS https://review.openstack.org/#/c/383859/ | |
| 13:43:59 | mriedem | blarg shared file systems | |
| 13:44:09 | mdbooth | I've actually been hammering on this over the last couple of days because it fixes another bug too | |
| 13:44:40 | mdbooth | If you suspend an instance with an attached volume, the volume can be unmounted while it's suspended | |
| 13:44:48 | mdbooth | So it fails when you resume it | |
| 13:44:55 | mdbooth | Confirmed this patch fixes that bug | |
| 13:45:53 | mdbooth | https://review.openstack.org/#/c/383859/ | |
| 13:46:24 | mdbooth | Except it's not a race, it's entirely deterministic in this case | |
| 13:47:20 | mriedem | ok | |
| 13:47:55 | mriedem | i'll be honest i'm going to have a very distracted day today, trying to wrap things up before flying out for vacation tomorrow | |
| 13:48:03 | BobBall | snikitin: Glad to hear it | |
| 13:48:14 | mdbooth | Ah, it's already in there | |
| 13:49:20 | openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova: Set access_policy for messaging's dispatcher https://review.openstack.org/408494 | |
| 13:50:40 | mdbooth | Unrelated, can anybody check my memory for me: https://review.openstack.org/#/c/376350/ . The behaviour of reboot where is tries to fix potential local hypervisor problems is intentional, right? | |
| 13:51:21 | mdbooth | i.e. recreating backing files and rewriting the domain xml. | |
| 13:51:53 | mdbooth | Both of these behaviours are undesirable, imho, but iiuc they're relied upon by operators. Is that right? | |
| 13:52:13 | mdbooth | sdague: ^^^ | |
| 13:52:20 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova: Updated from global requirements https://review.openstack.org/422323 | |
| 13:52:25 | bauzas | mriedem: FYI, the scheduler change is blocked because also another problem due to us not having yet a fixture for our functional tests | |
| 13:52:58 | bauzas | mriedem: sdague: there is a Gabbi fixture in our gabbi branch, but not for the functional tests | |
| 13:53:32 | bauzas | sdague: any reason why when calling the placement API using urllib2 and noauth2 middleware, it still returns 401 ? | |
| 13:55:19 | mdbooth | mriedem: I ran the latest patch version against the devstack nfs CI, btw. It passed. | |
| 13:55:24 | johnthetubaguy | melwitt_: mriedem: I have been reading that quota stuff just now, any reason we can't change ProjectUserQuota to have user_id column as nullable, and just kill the Quota table? (or add a nullable user_id into the Quota table, and kill ProjectUserQuota)? | |
| 13:56:05 | mriedem | johnthetubaguy: i don't know off the top of my head, i know the tables are used based on resource type | |
| 13:56:17 | mriedem | i don't know how much of that is still valid after the move | |
| 13:57:49 | johnthetubaguy | mriedem: no worries, I am going to keep reading | |
| 13:58:57 | mriedem | i would assume things stay the same in those tables when we move to the api db, they might change their function or necessity after the counting quotas stuff is in, but i'd probably lean toward making any changes to the data model in a later release | |
| 14:03:10 | sahid | mdbooth: about https://review.openstack.org/#/c/376350/ - i do not understand your points | |
| 14:03:40 | sahid | if there the domain XML can be caorrupted in someways to we probzbly have to fix the root cause | |
| 14:03:50 | sahid | same for the disks of instance | |
| 14:05:46 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova: Updated from global requirements https://review.openstack.org/422323 | |
| 14:05:55 | mdbooth | sahid: I've posted a hopefully clearer response on the review. | |
| 14:05:58 | sahid | if you can explaining or at least report a bug so i could fix both of these issues but like that it's just you want something broken to stay broken | |
| 14:06:11 | mdbooth | Specific bugs misses the point. | |
| 14:07:05 | mdbooth | sahid: I was also unaware of this apparently undocumented design decision, which is why I'm hoping to get some additional input. | |
| 14:07:43 | sahid | yes because that seems weird the hard-reboot should be helpfull when the guest operating system get stuck | |
| 14:08:04 | mdbooth | It's about the hypervisor rather than the guest OS, I think. | |
| 14:08:12 | mdbooth | Well, I guess it's about either. | |
| 14:08:24 | sahid | but i do not think it's how operators should repair an instance "broken" | |
| 14:09:03 | mdbooth | It's like: steps to take before defenestrating the instance: | |
| 14:09:07 | mdbooth | 1. reboot | |
| 14:09:09 | mdbooth | 2. ... | |
| 14:09:37 | sahid | but we need to be clear on what we are talking about, the qemu process or the guest operating system ? | |
| 14:09:57 | mdbooth | s/qemu process/hypervisor/ | |
| 14:10:12 | mdbooth | The aspect of reboot I'm talking about relates to the hypervisor | |
| 14:10:17 | mdbooth | Not the guest OS | |
| 14:10:34 | sahid | that seems really sad so... we really want to fix an issue like that | |
| 14:10:40 | mdbooth | What issue? | |
| 14:10:49 | sahid | the one you are talking about, no ? | |
| 14:10:56 | mdbooth | I'm not talking about an issue | |
| 14:11:02 | mdbooth | I'm talking about *any* issue | |
| 14:11:24 | mdbooth | I'm talking about an imaginary, as yet undiscovered bug which has resulted in inconsistent state in the hypervisor | |
| 14:11:49 | mdbooth | One of the goals of reboot is to try to get the instance going again anyway | |
| 14:12:03 | mdbooth | It would be nice if it wasn't | |
| 14:12:12 | mdbooth | But to the best of my knowledge, it is | |
| 14:15:54 | sahid | ok so you don't want me to fix this because there is an imaginary/undiscovered bug | |
| 14:16:03 | mdbooth | Incidentally, as I mentioned in one of my review comments, I think this is a very useful tool for operators to have. However, I also think it's problematic and ironically a potential source of bugs to do it in reboot. | |
| 14:16:13 | sahid | interesting... i have 2 well real bugs to show you because of this code | |
| 14:16:34 | mdbooth | sahid: No, I don't want you to do this because (IIUC) operators rely on this behaviour. | |
| 14:16:47 | mdbooth | Ideally we would separate it from reboot. | |
| 14:17:00 | sahid | if they do, so we should explain to them they should not i guess | |
| 14:17:11 | sahid | rscue and rebuild are better | |
| 14:17:12 | mdbooth | However, if it's a deliberate design decision, and something that users expect to be there, we can't just take it away without thought or acknowledgement. | |
| 14:17:27 | mdbooth | No, rescue doesn't help at all, and rebuild is entirely different. | |
| 14:17:52 | sahid | because hard-reboot has been designed such issue ? | |
| 14:18:53 | mdbooth | sahid: I suggest a bp. You might also post on the operators mailing list asking for input. | |
| 14:19:05 | mdbooth | It would also need a release note, I guess. | |
| 14:19:24 | sdague | bauzas: are you sure that placement is loading the noauth middleware? | |
| 14:19:31 | bauzas | sdague: yup | |
| 14:19:37 | bauzas | sdague: I just verified | |
| 14:19:50 | sdague | bauzas: if you push a patch I can look | |
| 14:20:00 | bauzas | sdague: okay, I'll do | |
| 14:20:48 | bauzas | mmm, I think I found the issue | |
| 14:23:33 | sahid | mdbooth: well if that could help i can provide a release note or a bp to fix this issue even if i'm not bringing any feature here | |
| 14:24:10 | mdbooth | sahid: I just want you to acknowledge and address a change in operator-expected behaviour. | |
| 14:24:40 | sahid | mdbooth: you are saying that but i never hear about this point, can you share to me some pointer ? | |
| 14:24:56 | sahid | a comment in lunchpad, a gerrit review or a thread in the ml ? | |
| 14:24:57 | mdbooth | No, I really can't. I've never seen it documented. | |
| 14:25:11 | sdague | mdbooth: I don't remember why the domain xml is rewriten, git log -S might find some things interesting to figure out if the relevant commit messages have clues | |
| 14:37:05 | bauzas | sdague: okay, so found the problem | |
| 14:37:43 | bauzas | sdague: https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L43-L44 | |
| 14:37:52 | bauzas | sdague: I just need to fake a token | |
| 14:38:31 | bauzas | sdague: do you know if we have a fake token lib or helper ? :) | |
| 14:39:23 | sdague | bauzas: https://github.com/openstack/nova/blob/9acfef2c90186644120942558437b3fcc90222de/nova/api/openstack/auth.py#L32-L47 | |
| 14:39:29 | sdague | it looks like that bit wasn't copied over | |
| 14:40:03 | bauzas | sdague: yeah, possibly because gabbi provides a fake token | |
| 14:40:12 | sdague | bauzas: could be | |
| 14:40:23 | bauzas | we miss cdent here | |
| 14:40:27 | sdague | honestly, I have no idea about the gabbi magic | |
| 14:40:36 | bauzas | me too | |
| 14:40:55 | bauzas | but since cdent wrote that noauth middleware for the gabbi functests, I guess it's why | |
| 14:41:30 | bauzas | I could just fake a token honestly and leave a big note for saying that we should rather have a better noauth middleware | |