| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-24 | |||
| 22:39:20 | efried | and getting too snobby for its britches | |
| 22:39:30 | aspiers | :) | |
| 22:39:32 | aspiers | happens to all the best places | |
| 22:41:20 | efried | if you like it they will come | |
| 22:41:40 | efried | case in point, the Trocadero | |
| 22:41:57 | efried | Carnaby Street | |
| 22:42:03 | efried | Camden Town | |
| 22:42:23 | aspiers | :) | |
| 22:43:38 | aspiers | it's a bit like the Ballmer Peak | |
| 22:44:10 | aspiers | places are shit for ages, then suddenly get cool, then everyone realises and floods the trendy cool place and ruins it again | |
| 22:45:06 | efried | 1630-beer: Disabled compute service request filter (mriedem) | |
| 22:45:06 | efried | 1615-1630: Compute capabilities traits placement request filter (mriedem) | |
| 22:45:06 | efried | Friday | |
| 22:45:06 | efried | Thursday 1330-1415: Let's plan the next steps of the bandwidth support feature (gibi) | |
| 22:45:06 | efried | aspiers, mriedem, gibi: I swapped both the traits filter topics with the bandwidth continuation. So now we have | |
| 22:45:14 | efried | um, jaypipes, did you want to be around for the latter? | |
| 22:45:21 | aspiers | awesome thanks | |
| 22:49:32 | mriedem | efried: ok, although i didn't think the bw spec was that controversial | |
| 22:49:35 | mriedem | just needs to be updated | |
| 22:50:56 | efried | mriedem: Sorry, lack of context, I just realized that aspiers responded to me and not the list - he wanted to be around for the traits filter topics (which also increases our chances of convincing him to own them :) but is unavailable Thursday. | |
| 22:51:39 | efried | mriedem, gibi: And definitely if we don't need as much (or any) time to talk about the bandwidth stuff, let me know so I can shuffle some more stuff in. | |
| 22:52:24 | efried | mriedem: I got the impression that gibi wanted to have a kind of live design session for some of the issues. | |
| 22:52:35 | mriedem | i'll let gibi answer that, i think the biggest hangups in the bw migration spec was (1) do we use a microversion (sounds like yeah probably) and (2) upgrade issues which i think he understands now | |
| 22:55:21 | efried | neat, I'll fup with gibi | |
| 22:56:55 | mriedem | dansmith: re https://review.opendev.org/#/c/649197/2/nova/compute/rpcapi.py@381 i think that's as easy as just resetting the compute.rpcapi._ROUTER global on ComputeManager.reset() or am i missing some nasty detail? | |
| 22:57:11 | dansmith | mriedem: nope, I think that'll do it | |
| 22:57:52 | mriedem | what do you think about just throwing reset_globals() from https://review.opendev.org/#/c/649197/2/nova/tests/unit/compute/test_rpcapi.py@37 into the rpcapi module itself? then i can hit it from multiple locations | |
| 23:01:14 | dansmith | sec, phone | |
| 23:03:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Always pass HostAPI to get_availability_zones https://review.opendev.org/655583 | |
| 23:04:45 | mriedem | heh "Problme: Even though splited, still long and boring. Need fully review." | |
| 23:04:54 | mriedem | L224 in the ptg etherpad | |
| 23:06:14 | mriedem | efried: L223 is likely not something we really need for the ptg https://etherpad.openstack.org/p/nova-ptg-train it's just a request for review | |
| 23:06:25 | mriedem | which i've done once before, just - it's a lot of code and i forgot about it | |
| 23:07:05 | eandersson | What is this fun error | |
| 23:07:06 | eandersson | > ValueError: unmatched '{' in format | |
| 23:07:10 | eandersson | When running cell mapping | |
| 23:07:26 | efried | mriedem: ack | |
| 23:07:27 | mriedem | are you using templates for the transport_url or db connection? | |
| 23:07:40 | eandersson | no template | |
| 23:08:17 | eandersson | my best guess is a special char in the password is causing some fun bug | |
| 23:08:21 | mriedem | do you have a stacktrace you can paste? | |
| 23:08:24 | efried | mriedem: a little background, I've scheduled that there because of the stx association, and dtroyer will still be in the room. If we don't need to discuss it, we can take a coffee break at that time :) | |
| 23:08:25 | mriedem | maybe | |
| 23:08:34 | mriedem | i suppose you don't want to paste the trace b/c of said password... | |
| 23:09:27 | mriedem | efried: ok, whatever, i think it's just a request for review | |
| 23:09:35 | efried | ack | |
| 23:09:45 | mriedem | melwitt and i have both reviewed it before | |
| 23:10:02 | efried | well cool, if y'all merge it before the PTG I'll definitely strike it :) | |
| 23:10:12 | mriedem | it's like 800 LOC though | |
| 23:10:15 | mriedem | heh | |
| 23:10:22 | efried | and in merge conflict | |
| 23:10:30 | mriedem | if it were a blueprint (which it arguably could be) then it could sit in a runway | |
| 23:10:35 | mriedem | the bottom patch isn't | |
| 23:10:51 | ccstone | (I'm also working with eandersson) He already gave away the special character which is { :) | |
| 23:10:55 | mriedem | then the reviews would floweth like fine wine | |
| 23:11:04 | efried | sure they would | |
| 23:11:36 | efried | lemme see what a rebase would look like. | |
| 23:12:02 | mriedem | ccstone: which release of rocky? | |
| 23:12:05 | mriedem | latest patch? | |
| 23:12:26 | mriedem | 18.2.0? | |
| 23:12:42 | mriedem | i was thinking maybe https://review.opendev.org/#/c/633086/ but that might not hit you if you're not using templated urls | |
| 23:14:16 | mriedem | ccstone: can you provide a bug or trace without the password in it? i don't know which command is being referenced | |
| 23:14:27 | ccstone | Yeah, we're on 18.2.0, looks like we do have that patch. | |
| 23:15:57 | eandersson | http://paste.openstack.org/show/749729/ | |
| 23:16:09 | openstackgerrit | Eric Fried proposed openstack/nova master: Clean up orphan instances https://review.opendev.org/627765 | |
| 23:16:09 | openstackgerrit | Eric Fried proposed openstack/nova master: Clean up orphan instances virt driver https://review.opendev.org/648912 | |
| 23:16:17 | efried | mriedem, melwitt: fyi, rebase was easy ^ | |
| 23:18:08 | mriedem | eandersson: ccstone: please report a bug, i want to say we talked about special characters in passwords when adding that code and there were some assertions made, but i can't remember now, would definitely want dansmith to look | |
| 23:18:10 | mriedem | but he's busy | |
| 23:20:31 | dansmith | mriedem: sure, that seems okay I think (re reset_globals) | |
| 23:21:12 | mriedem | eandersson: ccstone: looks like we'll catch and trace that ValueError but things don't fail right? we return the URL as-is | |
| 23:21:35 | dansmith | mriedem: eandersson ccstone: passwords should be url-encoded | |
| 23:21:42 | ccstone | mriedem: Will do, thanks! Our immediate fix was to change the rabbit password, but we're still getting the error, I assume because something has already put the transport_url in the database. Is there a nova-manage command to update that? | |
| 23:22:16 | mriedem | ccstone: nova-manage cell_v2 update_cell https://docs.openstack.org/nova/rocky/cli/nova-manage.html#nova-cells-v2 | |
| 23:22:17 | dansmith | I do remember something about one place where we couldn't urlencode it ourselves for you but I forget why | |
| 23:22:21 | dansmith | but it was non-trivial silliness | |
| 23:22:34 | eandersson | Yea - we didn't realize that it wasn't reading it from the config. | |
| 23:22:41 | eandersson | (anymore at least) | |
| 23:22:49 | ccstone | I think the issue was that it wasn't URL encoded when we ran simple_cell_setup (also we did that step on Newton code) | |
| 23:22:55 | ccstone | Or part of that issue at least | |
| 23:23:09 | mriedem | fwiw i remember f'ing around with this several times https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/cmd/manage.py#L1006 | |
| 23:23:13 | mriedem | for formatting the db url | |
| 23:23:16 | mriedem | using sqlalchemy utilities | |
| #openstack-nova - 2019-04-25 | |||
| 00:00:29 | ccstone | Thanks mriedem, we were able to correct the transport_url in the DB using that and that's what finally made the error stop. | |
| 00:01:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova.compute.rpcapi.ComputeAPI.router a singleton https://review.opendev.org/649197 | |
| 00:01:11 | mriedem | ccstone: "using that" => nova-manage cell_v2 update_cell? | |
| 00:08:05 | cooper6581 | mriedem: (with ccstone and eandersson being trained). You are right that it should have just returned the URL as-is, but I'm assuming CONF.transport_url ended up not being None | |
| 00:08:07 | cooper6581 | https://ghosthub.corp.blizzard.net/openstack/nova/blob/import/rocky/nova/objects/cell_mapping.py#L145 | |
| 00:08:16 | cooper6581 | oops, wrong link | |
| 00:08:39 | ccstone | mriedem: Yeah - we had updated nova.conf with a new rabbit password and were still getting the "unmatched '{'" error using nova-manage. Updating the transport_url in the DB via update_cell and the link you provided was what ultimately fixed it for us. Seems like simple_cell_setup allowed us to provide a bad transport_url without URL encoding. | |
| 00:09:06 | mriedem | ok | |
| 00:09:12 | cooper6581 | https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L145 | |
| 00:11:17 | mriedem | i guess if there is anything bug worthy to follow up with this to help someone else from falling in the same trap that would be good to report, maybe it could just be docs | |
| 00:18:50 | mriedem | ha this actually ran tests https://review.opendev.org/#/c/655222/ | |
| 00:18:56 | mriedem | was sure that would explode in devstack setup | |
| 00:19:12 | mriedem | dansmith: ^ | |
| 00:19:28 | mriedem | looks like the only thing that shit the bed was cold migration testing since tempest doesn't have a toggle for that | |
| 00:32:30 | mriedem | i see why it's failing there, i'm saying resize to same host is ok b/c i only have 1 compute per cell, but you can't cold migrate to the same host with the libvirt driver, that will be fixed with cross-cell resize ftw | |
| 00:46:49 | openstackgerrit | Merged openstack/os-vif master: Refactor functional base test classes https://review.opendev.org/643101 | |
| 01:03:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add nova-multi-cell job https://review.opendev.org/655222 | |