| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-24 | |||
| 23:20:31 | dansmith | mriedem: sure, that seems okay I think (re reset_globals) | |
| 23:21:12 | mriedem | eandersson: ccstone: looks like we'll catch and trace that ValueError but things don't fail right? we return the URL as-is | |
| 23:21:35 | dansmith | mriedem: eandersson ccstone: passwords should be url-encoded | |
| 23:21:42 | ccstone | mriedem: Will do, thanks! Our immediate fix was to change the rabbit password, but we're still getting the error, I assume because something has already put the transport_url in the database. Is there a nova-manage command to update that? | |
| 23:22:16 | mriedem | ccstone: nova-manage cell_v2 update_cell https://docs.openstack.org/nova/rocky/cli/nova-manage.html#nova-cells-v2 | |
| 23:22:17 | dansmith | I do remember something about one place where we couldn't urlencode it ourselves for you but I forget why | |
| 23:22:21 | dansmith | but it was non-trivial silliness | |
| 23:22:34 | eandersson | Yea - we didn't realize that it wasn't reading it from the config. | |
| 23:22:41 | eandersson | (anymore at least) | |
| 23:22:49 | ccstone | I think the issue was that it wasn't URL encoded when we ran simple_cell_setup (also we did that step on Newton code) | |
| 23:22:55 | ccstone | Or part of that issue at least | |
| 23:23:09 | mriedem | fwiw i remember f'ing around with this several times https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/cmd/manage.py#L1006 | |
| 23:23:13 | mriedem | for formatting the db url | |
| 23:23:16 | mriedem | using sqlalchemy utilities | |
| #openstack-nova - 2019-04-25 | |||
| 00:00:29 | ccstone | Thanks mriedem, we were able to correct the transport_url in the DB using that and that's what finally made the error stop. | |
| 00:01:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova.compute.rpcapi.ComputeAPI.router a singleton https://review.opendev.org/649197 | |
| 00:01:11 | mriedem | ccstone: "using that" => nova-manage cell_v2 update_cell? | |
| 00:08:05 | cooper6581 | mriedem: (with ccstone and eandersson being trained). You are right that it should have just returned the URL as-is, but I'm assuming CONF.transport_url ended up not being None | |
| 00:08:07 | cooper6581 | https://ghosthub.corp.blizzard.net/openstack/nova/blob/import/rocky/nova/objects/cell_mapping.py#L145 | |
| 00:08:16 | cooper6581 | oops, wrong link | |
| 00:08:39 | ccstone | mriedem: Yeah - we had updated nova.conf with a new rabbit password and were still getting the "unmatched '{'" error using nova-manage. Updating the transport_url in the DB via update_cell and the link you provided was what ultimately fixed it for us. Seems like simple_cell_setup allowed us to provide a bad transport_url without URL encoding. | |
| 00:09:06 | mriedem | ok | |
| 00:09:12 | cooper6581 | https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L145 | |
| 00:11:17 | mriedem | i guess if there is anything bug worthy to follow up with this to help someone else from falling in the same trap that would be good to report, maybe it could just be docs | |
| 00:18:50 | mriedem | ha this actually ran tests https://review.opendev.org/#/c/655222/ | |
| 00:18:56 | mriedem | was sure that would explode in devstack setup | |
| 00:19:12 | mriedem | dansmith: ^ | |
| 00:19:28 | mriedem | looks like the only thing that shit the bed was cold migration testing since tempest doesn't have a toggle for that | |
| 00:32:30 | mriedem | i see why it's failing there, i'm saying resize to same host is ok b/c i only have 1 compute per cell, but you can't cold migrate to the same host with the libvirt driver, that will be fixed with cross-cell resize ftw | |
| 00:46:49 | openstackgerrit | Merged openstack/os-vif master: Refactor functional base test classes https://review.opendev.org/643101 | |
| 01:03:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add nova-multi-cell job https://review.opendev.org/655222 | |
| 01:09:15 | mriedem | efried: just fyi i've removed the cross-cell-resize series from the runway slot (time expired) and added https://blueprints.launchpad.net/nova/+spec/add-host-and-hypervisor-hostname-flag-to-create-server | |
| 01:09:17 | mriedem | boxiang: ^ | |
| 01:09:30 | mriedem | https://etherpad.openstack.org/p/nova-runways-train | |
| 01:52:50 | melwitt | lyarwood, mriedem: ceph dashboard fix attempt https://review.opendev.org/655591 | |
| 03:08:40 | openstackgerrit | Merged openstack/nova stable/stein: Fix {min|max}_version in ironic Adapter setup https://review.opendev.org/655429 | |
| 03:08:46 | openstackgerrit | Merged openstack/nova stable/rocky: Fix regression in glance client call https://review.opendev.org/655167 | |
| 07:44:21 | kashyap | efried: Hehe, I learnt a new expression, "one-armed-wallpaperer" (or "one-armed paper hanger", I see). | |
| 08:23:10 | kashyap | efried: sean-k-mooney: Just looked at at the video models thing: yes, it makes sense to add the me, FWIW. However, I'd appreciate a one-line summary of each of the three models being added in the BP | |
| 08:23:44 | kashyap | efried: sean-k-mooney E.g. "The 'none' video model type disables the automatic addition of a video device to domains with 'graphics' specified in their XML. This can be useful with GPU mediated devices which can serve as the only rendering devices within the guest." | |
| 08:23:57 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655332 | |
| 09:00:19 | gibi | efried, mriedem: regarding the length of the PTG slot :1330-1415: Let's plan the next steps of the bandwidth support feature (gibi) | |
| 09:01:04 | gibi | efried, mriedem: I think we can speed things up, so we can be done in ~15-20 mins | |
| 09:25:16 | aspiers | check jobs seem pretty flakey at the moment | |
| 09:25:35 | aspiers | it's not normal to need 2 rechecks, right? | |
| 09:30:31 | aspiers | kashyap: any idea? https://review.opendev.org/#/c/655268/2 keeps getting V-1 for no good reason AFAICS | |
| 09:31:01 | kashyap | aspiers: Sometimes is completely "normal" to require more than 2 rechecks :-( | |
| 09:31:07 | aspiers | :-( | |
| 09:31:35 | aspiers | This is an awesome technique for dealing with troublesome CI https://cfp.all-systems-go.io/ASG2018/talk/207/ | |
| 09:31:51 | aspiers | I think until OpenStack has something like that, flakey gates will always remain common | |
| 09:31:54 | kashyap | I don't have any bright ideas (besides "recheck and pray to the JuJu at the bottom of the sea"). I've experienced the same in the past. | |
| 09:32:45 | aspiers | Using ML means the "flakes" can be clustered automatically (unsupervised) which then means it's crystal clear which flakes cause the most failures and require the most human attention to fix | |
| 09:33:07 | aspiers | That way we could get to the long tail of weird corner cases quicker | |
| 09:33:39 | aspiers | e.g. one flake might cause 30% of failures, and another 0.3% | |
| 09:33:56 | aspiers | but until you gather stats on that, it's hard to know where to put effort | |
| 09:34:27 | aspiers | elastic recheck aims for that of course, but it requires manual maintenance | |
| 09:35:38 | kashyap | aspiers: Yeah, suggest it on the 'openstack-discuss' list, if you have a minute to write the email | |
| 10:03:27 | aspiers | kashyap: might be better to find mtreinish and the infra folks in Denver next week first | |
| 10:03:36 | aspiers | I've already reached out to the cockpit project | |
| 10:03:41 | aspiers | no reply yet | |
| 10:04:32 | aspiers | kashyap: when you get a moment, reviews on https://review.opendev.org/#/c/655268/ and https://review.opendev.org/#/c/633855/ would be much appreciated ;-) | |
| 10:05:08 | kashyap | aspiers: Will do. Currently doing something I promised efried that I'll address (in my own spec, though) today. Trying to keep the word. :D | |
| 10:05:35 | aspiers | sure, good luck with it :) | |
| 10:26:06 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.opendev.org/555081 | |
| 10:33:18 | openstackgerrit | Adam Spiers proposed openstack/os-resource-classes master: Add MEM_ENCRYPTION_CONTEXT resource class https://review.opendev.org/655665 | |
| 10:46:38 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.opendev.org/555081 | |
| 10:46:49 | stephenfin | sean-k-mooney: You're mostly happy with ^ at this point, I assume? | |
| 10:47:16 | sean-k-mooney | ya mostly | |
| 10:47:23 | stephenfin | I've added more words and fixed the typos to assuage efried's concerns (I hope) | |
| 10:47:36 | sean-k-mooney | i was going to suggest using a different threath the HW_CPU_HYPERTHREADING | |
| 10:47:43 | sean-k-mooney | but its already in os traits | |
| 10:47:50 | sean-k-mooney | e.g. HW_CPU_SMT | |
| 10:48:00 | stephenfin | sean-k-mooney: We could just _not_ use that? | |
| 10:48:18 | sean-k-mooney | then we have another dead trait in plamcent | |
| 10:48:20 | stephenfin | Or even remove it if no-one else is using it yet | |
| 10:48:24 | sean-k-mooney | *os-traits | |
| 10:48:55 | sean-k-mooney | stephenfin: we cant remove traits from os-traits | |
| 10:49:06 | sean-k-mooney | its an add only model | |
| 10:49:23 | stephenfin | if no one is using it anywhere though? | |
| 10:49:26 | sean-k-mooney | to simplfy db updates | |
| 10:49:27 | stephenfin | and never has | |
| 10:49:32 | stephenfin | Oh, there's a DB involved | |
| 10:49:33 | sean-k-mooney | even then | |
| 10:49:36 | sean-k-mooney | ya | |
| 10:49:42 | aspiers | stephenfin: you're repeating an earlier discussion ;-) | |
| 10:49:48 | aspiers | this is now mentioned in http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html | |
| 10:49:56 | aspiers | I asked the exact same question a few days ago | |
| 10:50:01 | sean-k-mooney | yep | |
| 10:50:03 | stephenfin | Fair. I'd also be ok with a dead trait, tbh, but I also don't care that much | |
| 10:50:16 | sean-k-mooney | im fine with hte hyperthreaing one | |
| 10:50:25 | aspiers | I think dead traits can be hidden to the point that they're only visible in the os-traits code where hardly anyone looks | |
| 10:50:29 | sean-k-mooney | people use it as a generic term even if it is a threadmark | |
| 10:50:34 | aspiers | well I suppose they're visible via introspection too | |
| 10:50:40 | stephenfin | fwiw, I tried to use SMT in the documentation e.g. https://docs.openstack.org/nova/pike/admin/cpu-topologies.html | |
| 10:51:03 | sean-k-mooney | yes i know and you gave a talk on the topic | |
| 10:51:36 | stephenfin | indeed :) | |
| 10:51:59 | sean-k-mooney | so ya i was in the middel of reading v27 | |
| 10:52:08 | sean-k-mooney | ill skim through v28 | |
| 10:52:34 | stephenfin | Yeah, spotted an issue with my description of that trait, actually :) | |
| 10:52:50 | stephenfin | I must admit, I really don't like Gerrit for doc reviews :-\ | |