| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-02 | |||
| 15:34:50 | mriedem | also makes n-cell-region instance_update_at_top blow up | |
| 15:34:58 | mriedem | b/c we're logging the full instance | |
| 15:35:02 | mriedem | and something in there is unicode | |
| 15:35:20 | melwitt | the lp bug says the test uses a unicode display name for the instance | |
| 15:35:35 | melwitt | if that's still true then that would do it. I don't get why it doesn't always happen though | |
| 15:40:47 | mriedem | melwitt: yeah test_update_server_name_in_stop_state does create a server, stop it, then update the server name to unicode and then delete it | |
| 15:41:09 | melwitt | mriedem: sdague | |
| 15:41:11 | mriedem | i don't know why it wouldn't blow up everytime though | |
| 15:42:08 | mriedem | melwitt: also, when this fails http://logs.openstack.org/03/427903/7/check/gate-tempest-dsvm-cells-ubuntu-xenial/87da64a/console.html#_2017-05-02_08_22_13_375688 | |
| 15:42:24 | mriedem | it's while waiting for the server to be active, before the name is updated to be unicode | |
| 15:42:34 | melwitt | mriedem: sdague's comment on the lp bug mentions how devstack setting LC_ALL=C for grepping causes this, I wonder if there's some reason we couldn't use en_US.utf8 instead | |
| 15:43:45 | sdague | oh, that would totally explain why it's a gate only thing | |
| 15:43:58 | sdague | though, I thought the gate was C.utf8 | |
| 15:44:29 | Swanson | I see n-crt failing on my CI since yesterday. There something up with it? | |
| 15:44:43 | mriedem | Swanson: n-crt was deleted yesterday | |
| 15:44:53 | mriedem | Swanson: it's been deprecated since newton | |
| 15:45:02 | Swanson | mriedem, Whee! Easy fix. | |
| 15:45:20 | melwitt | sdague: looks like a mix in devstack? https://github.com/openstack-dev/devstack/search?utf8=%E2%9C%93&q=lc_all&type= | |
| 15:45:25 | Swanson | mriedem, Would explain my difficulty actually finding it. | |
| 15:46:02 | sdague | melwitt: yeh | |
| 15:46:26 | sdague | melwitt: if you want to propose change to en_US.utf8 everywhere, I'm good with that | |
| 15:46:37 | sdague | especially if it fixes this | |
| 15:48:10 | melwitt | sdague: yup, let's try that. C is ascii only so probably should be using utf8 anyhow. not 100% sure if it will fix this though | |
| 15:48:27 | sdague | yeh, I feel like I had that patch up before for other reasons | |
| 15:48:32 | sdague | but we never decided to land it | |
| 15:48:39 | sdague | but if it fixes this thing, win! | |
| 15:49:12 | melwitt | ++ | |
| 15:52:11 | mriedem | seems we should still not be logging something that's going to blow up here yeah? | |
| 15:54:50 | melwitt | mriedem: I'm not sure. I had been thinking the failure is environment specific, i.e. if the environment doesn't support utf8 | |
| 15:55:35 | mriedem | melwitt: we put lots of things in six.text_type though for the same reason | |
| 15:55:41 | mriedem | regardless of env | |
| 15:55:42 | melwitt | but I'm still not clear on why it doesn't happen all the time and what we could do the prevent it blowing up in a non utf8 env? | |
| 15:55:51 | melwitt | oh | |
| 15:57:33 | sdague | mriedem: it's one of those things that is more fundamental | |
| 15:57:56 | sdague | I feel like I went through a weird set of poking on this the last time | |
| 15:57:58 | mriedem | The String field in ovo has a stringify method which is used for __repr__, seems that should always be doing six.text_type | |
| 15:58:03 | melwitt | so six.text_type returns a unicode string. if the env doesn't support unicode, then how does it help? | |
| 15:58:10 | sdague | right, like melwitt said | |
| 15:58:33 | sdague | stuff gets super borky when you are in a non utf8 env | |
| 15:58:42 | melwitt | it seems like it would just do the same thing, but I dunno | |
| 15:58:48 | clarkb | C.utf8 is the locale I think people should be using based on what is built into libcs | |
| 15:59:20 | sdague | clarkb: well, that has a different issue in that "nobody" uses that except the dib images for infra | |
| 15:59:47 | clarkb | sdague: because its relatively new but aiui its what things are switching to? | |
| 15:59:56 | sdague | clarkb: are they? | |
| 16:00:01 | clarkb | ya fedora was aiui | |
| 16:00:10 | clarkb | which should mean the rest of the red hat world is too? | |
| 16:00:10 | sdague | ubuntu? | |
| 16:00:17 | mriedem | melwitt: maybe you're right, since it looks like the base __repr__ in ovo does encode it if on py27 https://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/base.py#L318 | |
| 16:00:26 | clarkb | ubuntu already had it but being desktop first I'm not sure if they will hcnage the default | |
| 16:00:42 | sdague | anyway, lets see if the old tried and true en_US.utf8 works | |
| 16:00:46 | clarkb | the idea behind C.utf8 is have the "sanity" of C (sorting etc) while supporting utf8 too | |
| 16:02:22 | melwitt | fwiw I get the error when I try the same example C.utf8 $ LC_ALL=C.utf8 python -c "a = u'\u00CD\u00F1st\u00E1\u00F1c\u00E9'; print a" | |
| 16:04:12 | clarkb | melwitt: what error? (that works for me locally) | |
| 16:04:35 | clarkb | I get "Íñstáñcé" | |
| 16:04:39 | jaypipes | cdent: yay, your request for additional functional tests identified an issue with the WHERE condition on shared providers... the difference of where a single parentheses is makes a world of difference :) | |
| 16:04:54 | melwitt | clarkb: hm, interesting. the error from the bug "UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)" | |
| 16:05:35 | clarkb | melwitt: and if you change it to print a.encode('utf8') I assume that works? | |
| 16:06:24 | clarkb | does locale -a show C.utf8 as being present locally? | |
| 16:06:25 | mriedem | clarkb: it does | |
| 16:06:41 | melwitt | clarkb: yeah, with the encode it works. I wonder if it depends on which version of python? | |
| 16:06:52 | mriedem | melwitt: apparently it does https://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/base.py#L318 | |
| 16:07:03 | mriedem | the problem is in ^ we do the %s replace before trying to encode it | |
| 16:07:07 | mriedem | and we blow up already | |
| 16:07:11 | stvnoyes | johnthetubaguy: Hi john, I updated one of the multiattach reviews to address your comments- https://review.openstack.org/#/c/456877/ pls take a look when you get a chance. also related review here - https://review.openstack.org/#/c/456851/ thx | |
| 16:07:26 | clarkb | mriedem: melwitt I think it either depends on locale locales (see locale -a) or the build of python and how it detects the locale? | |
| 16:07:35 | clarkb | I have python 2.7.13 | |
| 16:08:04 | melwitt | okay, I have C.utf8 in locale -a | |
| 16:08:16 | melwitt | er, C.UTF-8 | |
| 16:08:36 | melwitt | so it works when I use that | |
| 16:08:51 | melwitt | with the caps and the hyphen | |
| 16:12:24 | sfinucan | melwitt: You tried 'PYTHONIOENCODING=UTF-8 python ...'? | |
| 16:13:12 | melwitt | sfinucan: no, just setting LC_ALL=C.UTF-8 etc | |
| 16:14:18 | sfinucan | melwitt: I think that should fix the issue. Someone reported a similar things for pwclient (part of Patchwork) a while back https://lists.ozlabs.org/pipermail/patchwork/2017-April/004214.html | |
| 16:14:52 | sfinucan | the other "workarounds" are hacky as can be. Better just set the envvar | |
| 16:18:31 | sdague | clarkb: there is no C.UTF-8 on base centos 7 | |
| 16:19:14 | melwitt | sdague, clarkb: trying C.UTF-8 https://review.openstack.org/#/c/461840 | |
| 16:19:29 | melwitt | sdague: but what you just said means it's not going to work for centos 7 | |
| 16:19:57 | sdague | melwitt: yeh, lets just use en_US.utf8, we know that works | |
| 16:19:58 | clarkb | sdague: is ther C.utf8? | |
| 16:20:02 | melwitt | sdague: k | |
| 16:20:03 | sdague | clarkb: nope | |
| 16:20:05 | clarkb | sdague: thats what it is for me on suse at least | |
| 16:20:15 | sdague | [sdague@centos7 ~]$ locale -a | grep '^C' | |
| 16:20:15 | sdague | C | |
| 16:21:10 | sdague | clarkb: which is actually even worse if distros are calling it different things | |
| 16:21:32 | clarkb | ya the different name thing is new to me and annoying | |
| 16:22:26 | sdague | anyway, en_US.utf8 is a thing, it's been a thing for a long time, we already force that local for pip installs because people have utf8 in their Readme files which are auto included in setup.py and break in this same way if we don't | |
| 16:25:23 | edleafe | dansmith: regarding the removal of 'can_host' (https://review.openstack.org/#/c/461494/) if I remove the column from the model but don't provide a migration script to remove it, the functional db test fails with "Models and migration scripts aren't in sync" | |
| 16:25:32 | dansmith | edleafe: yup | |
| 16:25:38 | dansmith | edleafe: there's an exclusion list for that I think | |
| 16:26:40 | mriedem | there is, | |
| 16:26:44 | edleafe | dansmith: ugh. ok, I'll search for that | |
| 16:26:45 | dansmith | edleafe: example prior art: 5e9df4baf97fd1f495b81d3b8b612e24f344b325 | |
| 16:26:45 | mriedem | because we don't drop columns | |
| 16:26:55 | clarkb | sdague: ya the biggest issue is going to be if your software ends up in $location where someone has decided that don't need en_US (because they don't english or murica or whatever) | |
| 16:27:20 | clarkb | sdague: and aiui C/C.utf8 are supposed to be workaround for such things. By not being tied to a region/human language | |
| 16:28:59 | melwitt | sdague: should I do a check experimental on it to get more jobs? I can't remember what's in there for experimental and not finding it yet | |
| 16:29:14 | sdague | clarkb: yeh, well it's an edge condition, but so are a thousand other things. And honestly for repeatable development I'd rather just go easy | |
| 16:29:29 | sdague | melwitt: what was the failing job? | |
| 16:29:59 | sdague | I'm not sure if whatever that is in the devstack experimental list | |