| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-31 | |||
| 12:48:54 | bauzas | for evacuate I mean | |
| 12:48:56 | mriedem | and that can pass | |
| 12:49:03 | mriedem | and you can call conductor on the same host as instance.host | |
| 12:49:13 | bauzas | wait | |
| 12:49:25 | bauzas | ah, nevremind | |
| 12:49:29 | mriedem | and eventually either it fails with NoValidHost and the instance state is reset (best case scenario), or we bypass the scheduler and rpc fails, and your instance is stuck in 'rebuilding' state | |
| 12:49:36 | bauzas | the API check is verifying the *source* | |
| 12:49:40 | mriedem | correct | |
| 12:50:06 | mriedem | you will fail either way, but a straight 400 is better than weird undefined failures once we've cast to compute | |
| 12:50:09 | mriedem | s/compute/conductor/ | |
| 12:50:28 | bauzas | well, when I wrote the original Newton spec about force flags and so on, I made it clear that if people are using 'force', they have to be super-cautious | |
| 12:50:40 | bauzas | that's what we said to them | |
| 12:50:53 | mriedem | how many operators do you think have read that spec? | |
| 12:50:58 | bauzas | the real problem was that pre-Newton, we weren't clear whether we were enforcing rules | |
| 12:51:18 | bauzas | I think I translated that in the API docs | |
| 12:51:26 | bauzas | but I could be missing that | |
| 12:51:27 | mriedem | "Force an evacuation by not verifying the provided destination host by the scheduler." | |
| 12:51:27 | mriedem | this is the description of the force parameter in the api ref | |
| 12:51:37 | mriedem | ^ is not, "holy shit, don't do this" | |
| 12:51:52 | mriedem | we should put a warning in there probably | |
| 12:52:36 | mriedem | also, | |
| 12:52:37 | bauzas | mriedem: I didn't wanted to be pedantic when I said about the spec, I just try to explain that I saw there was by that time I wrote the spec, a pretty clear consensus that if operators are providing destinations, they *have to* make sure it's an acceptable one | |
| 12:53:05 | bauzas | because it's anti-cloud | |
| 12:53:15 | bauzas | you specify a destination, fair enough | |
| 12:53:20 | mriedem | with https://review.openstack.org/#/c/499399/ now, we should probably seriously consider splitting the rebuild_instance conductor method / rpc api into rebuild_instance and evacuate_instance | |
| 12:53:26 | bauzas | but then, make sure it's a good one | |
| 12:53:30 | mriedem | because the if/else logic in there is getting pretty hairy | |
| 12:53:34 | sdague | efried: yeh, I don't know | |
| 12:54:21 | efried | sdague Just something I noticed while I was in the neighborhood; and you're git blamed on that comment :) | |
| 12:54:31 | bauzas | mriedem: there is a side concern to me: you can specify a destination but we don't tell whether it's case-sensitive or not | |
| 12:54:56 | bauzas | mriedem: and somewhere, it breaks | |
| 12:55:32 | mriedem | you'd get NoValidHost i'd thikn | |
| 12:55:37 | mriedem | since we'd filter out all hosts | |
| 12:55:43 | mriedem | since the ComputeNode.host doesn't match | |
| 12:56:01 | bauzas | there is a bug | |
| 12:56:07 | bauzas | wait, finding it | |
| 12:56:18 | mriedem | lemme guess, case insensitivity in mysql? | |
| 12:57:15 | bauzas | https://bugs.launchpad.net/nova/+bug/1709260 | |
| 12:57:17 | openstack | Launchpad bug 1709260 in OpenStack Compute (nova) "Addition of host to host-aggregate should be case -sensitive" [Low,In progress] - Assigned to Rajesh Tailor (ratailor) | |
| 12:57:29 | bauzas | it seems that DNS is case-insentive | |
| 12:57:38 | bauzas | case-insensitive | |
| 12:57:54 | bauzas | so in theory, we should accept to migrate to foo or FOO | |
| 12:58:04 | bauzas | but yeah, I guess it's because mysql | |
| 12:58:42 | bauzas | ratailor: around ? | |
| 12:58:50 | ratailor | bauzas, yep | |
| 12:58:56 | bauzas | ratailor: I feel I badly triaged your bug | |
| 12:59:12 | bauzas | ratailor: since DNS is case-insensitive, hostnames should be too | |
| 12:59:45 | ratailor | bauzas, I reproduced it, and found that mysql doesn't support case-sensitivity by-default. So I had to change the collation on related tables. | |
| 12:59:50 | ratailor | bauzas, which seems to work. | |
| 13:00:26 | mriedem | you know what is case sensitive by default (i think)? | |
| 13:00:29 | mriedem | POSTGRESQL! | |
| 13:01:23 | ratailor | I was discussing it with alex_xu some days ago and he mentioned that mysql is case-insensitive by-default. | |
| 13:01:23 | mriedem | sort of https://stackoverflow.com/questions/21796446/postgres-case-sensitivity | |
| 13:01:24 | bauzas | ratailor: tbh, I just feel that we should allow HoStNaME1 as a possible value for a host to be added in an aggregate | |
| 13:01:25 | mriedem | depends on quotes | |
| 13:01:39 | bauzas | ratailor: and rather fix the filter | |
| 13:01:44 | mriedem | ratailor: yeah, the mysql case issue is a known one | |
| 13:01:48 | bauzas | yup | |
| 13:01:48 | mriedem | that breaks several things | |
| 13:02:05 | ratailor | mriedem, the host-aggregate metadata keys as well, | |
| 13:02:10 | bauzas | ratailor: I'll rephrase your bug report if you agree | |
| 13:02:12 | mriedem | ratailor: https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/lowercase-metadata-keys.html | |
| 13:02:23 | ratailor | bauzas, no problem. | |
| 13:02:52 | bauzas | mriedem: well, hostnames can be FQDNs | |
| 13:03:04 | bauzas | mriedem: if so, those have to be case-insensitive | |
| 13:04:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create allocations against forced dest host during evacuate https://review.openstack.org/499399 | |
| 13:05:32 | ratailor | bauzas, I don't understand you mentioning HoStNaME1 as possible value for host to add in aggregate. | |
| 13:05:34 | bauzas | mriedem: about your point above with providing the same host for the target and source, if we accept to verify that by the API, we also need to add this to the live-migration one | |
| 13:06:02 | ratailor | with flavor matching metadata as host-aggregate metadata comes, this | |
| 13:06:02 | ratailor | as "COMPUTE0.example.com". And after that if instance creation request | |
| 13:06:02 | ratailor | HostNotFound error, that host is successfully added to host-aggregate | |
| 13:06:02 | ratailor | "COMPUTE0.example.com" (in capital case), then instead of throwing | |
| 13:06:02 | ratailor | and user tries to add this host to host-aggregate but by-mistake types | |
| 13:06:02 | ratailor | bauzas, As of now, if hostname is set as "compute0.example.com" (in lower case) | |
| 13:06:04 | ratailor | host is not filtered by scheduler, since there is no host with hostname | |
| 13:06:06 | ratailor | COMPUTE0.example.com, as added in host-aggregate | |
| 13:09:11 | bauzas | ratailor: I got the problem | |
| 13:09:43 | ratailor | bauzas, cool, | |
| 13:09:46 | bauzas | ratailor: what I feel is that we somehow should still accept COMPUTE0 as a possible value for the host to be added in the aggregate | |
| 13:09:57 | bauzas | from an API perspective | |
| 13:10:08 | bauzas | if we want to follow the DNS RFC | |
| 13:10:51 | bauzas | ratailor: what we could do tho is to lowercase that string before amending the aggregate | |
| 13:11:39 | bauzas | and that wouldn't trample https://specs.openstack.org/openstack/nova-specs/specs/newton/approved/lowercase-metadata-keys.html | |
| 13:11:52 | ratailor | bauzas, But that can happen other way round as well, hostname can be COMPUTE0 and user tries to add compute0 which should fail, as there is no hostname with compute0. | |
| 13:12:56 | bauzas | then the filter has to be fixed too | |
| 13:13:19 | bauzas | because it fails due to the filtter, right? | |
| 13:13:26 | ratailor | bauzas, I think, that's separate bug, which is only concerned about metadata keys. Is it somehow related to hostname. | |
| 13:14:29 | ratailor | bauzas, I think, the issue is with mysql only, | |
| 13:14:31 | ratailor | bauzas, https://github.com/openstack/nova/commit/402b3abf990d08d2af8331079d36a92d84d84b80 | |
| 13:15:27 | ratailor | bauzas, similar problem was there, and he has changed collation type for mysql backend. | |
| 13:18:59 | ratailor | bauzas, you want to discuss anything else, it EOD here. Can we continue tomorrow, if anything is left ? | |
| 13:20:13 | bauzas | ratailor: for sure | |
| 13:20:30 | bauzas | ratailor: have a good evening | |
| 13:20:40 | ratailor | bauzas, same to you! | |
| 13:23:50 | gibi | mriedem: I reverted https://review.openstack.org/#/c/491808 top of https://review.openstack.org/#/c/498482 and I still see that the recovered source compute tries to delete the instance | |
| 13:25:18 | gibi | mriedem: so I think setting the migration to failed is the solution we need. Do you suggest to add an extra check to the destroy_evacuated_instances about that instance.host != CONF.host as well? | |
| 13:26:49 | gibi | hm, the bug disappeared from launchpad https://bugs.launchpad.net/nova/+bug/1713783 | |
| 13:26:51 | openstack | gibi: Error: malone bug 1713783 not found | |
| 13:29:49 | openstackgerrit | Bob Ball proposed openstack/nova master: XenAPI: Unit tests must mock os_xenapi calls https://review.openstack.org/499573 | |
| 13:30:02 | gibi | mriedem: but you were able to just update that bug. that is weird | |