| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-22 | |||
| 15:24:28 | alex_xu | I'm ok with both, just want to figure out what is the plan for T release since we have serveal patch will do the same thing | |
| 15:24:37 | sean-k-mooney | i could be miss rememebring that | |
| 15:24:50 | belmoreira | stephenfin, melwitt it doesn't work with workarounds/enable_consoleauth = True (with everything running stein) | |
| 15:25:32 | aspiers | sean-k-mooney, alex_xu: what are the pros / cons? | |
| 15:25:39 | melwitt | belmoreira: I guess... that does make sense after the TTL has expired for already existing tokens, if you have not deployed proxies per cell yet | |
| 15:25:51 | melwitt | because new tokens are *not* stored in nova-consoleauth | |
| 15:26:03 | melwitt | *new token auths | |
| 15:26:16 | aspiers | both code paths run in the scheduler, right? | |
| 15:26:18 | efried | aspiers, alex_xu: Perhaps after we've merged stephenfin's patch to consolidate all that request spec mess, we could refactor the various request filters to do that. (I would still want it modular so it's not a monolithic mass of conversions.) But for now, that from_request_spec/from_extra_specs code is a bloody mess and it would be pretty hard to work the logic into the right places. So for now I say leave it in the request filte | |
| 15:26:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Get instances outside of update_available_resource lock https://review.opendev.org/677790 | |
| 15:26:46 | alex_xu | efried: ok, got it | |
| 15:26:51 | stephenfin | efried: Are you referring to https://review.opendev.org/#/c/674894/ ? | |
| 15:26:57 | aspiers | efried: OK thanks | |
| 15:27:28 | aspiers | alex_xu: so that means https://review.opendev.org/#/c/664420/42 is ready for re-review | |
| 15:27:39 | efried | stephenfin: yes, that's the one. | |
| 15:27:42 | belmoreira | melwitt right... is there any easy way to enable tokens in nova-consoleauth again? (I'm in the middle of a crisis here) | |
| 15:27:52 | alex_xu | aspiers: yea | |
| 15:28:02 | aspiers | alex_xu: oops sorry I mean https://review.opendev.org/#/c/664420/ previous link was an old PS | |
| 15:28:08 | stephenfin | I think I could move that to the top of the list of approved patches if you want it now? I don't think it has a dependency on any of my other functional changes | |
| 15:28:12 | efried | stephenfin: I wonder, is there a way it could be moved out/earlier?... jinx | |
| 15:28:19 | stephenfin | coke! | |
| 15:28:41 | aspiers | alex_xu, stephenfin: also I fixed the tiny bug in https://review.opendev.org/#/c/638680 | |
| 15:28:49 | efried | stephenfin: as discussed above, we're not blocking on it, we're going to continue to do the logic in the request filter for now. So don't worry about it unless it's easy. | |
| 15:29:04 | stephenfin | Sure. Let me try | |
| 15:32:18 | melwitt | belmoreira: hm, looking at the code, it should still work if you have workarounds/enable_consoleauth = True in stein. I see in nova/compute/api.py it still stores the token console auth in nova-consoleauth in addition to in the cell database. so I can't see why it wouldn't work yet | |
| 15:33:55 | belmoreira | melwitt from the consoleauth I only get: Checking Token: ***, False | |
| 15:34:50 | belmoreira | melwitt from nova-novncproxy: https://www.codepile.net/pile/O1P8ee1q | |
| 15:34:59 | melwitt | belmoreira: did you set workarounds/enable_consoleauth = True on hosts where nova-api and nova-novncproxy runs? | |
| 15:35:54 | stephenfin | belmoreira: Maybe it's just me, but I don't see anything in that codepile link | |
| 15:38:27 | belmoreira | melwitt workarounds/enable_consoleauth it's also for the nova-api? I'm only running it in the vnc nodes | |
| 15:38:55 | melwitt | belmoreira: you need it for nova-api to tell it to store token auth in nova-consoleauth service | |
| 15:39:21 | melwitt | and you need it for nonvcproxy to tell it to look in nova-consoleauth to find token auths when validating | |
| 15:39:24 | melwitt | so you need both | |
| 15:39:48 | belmoreira | that will be easy to change. Let me do it | |
| 15:41:04 | openstackgerrit | Merged openstack/nova master: Track libvirt host/domain capabilities for multiple machine types https://review.opendev.org/673151 | |
| 15:41:13 | openstackgerrit | Merged openstack/nova master: Extract new base class for provider usage functional tests https://review.opendev.org/676964 | |
| 15:53:12 | efried | mriedem: clarkb suggested trying to repro the innodb thing with a post that copies database logs. | |
| 15:53:12 | efried | I'm trying to hack up a patch to do that, but there's at least three things about it that I don't understand. | |
| 15:56:31 | openstackgerrit | Adam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | |
| 15:56:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: claims: Remove useless caching https://review.opendev.org/671791 | |
| 15:56:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795 | |
| 15:56:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797 | |
| 15:56:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798 | |
| 15:56:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799 | |
| 15:56:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Rework 'hardware.numa_usage_from_instances' https://review.opendev.org/672565 | |
| 15:56:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | |
| 15:56:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894 | |
| 15:56:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.opendev.org/537414 | |
| 15:56:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add '[compute] cpu_dedicated_set' option https://review.opendev.org/671792 | |
| 15:56:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 15:56:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | |
| 15:56:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | |
| 15:56:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 15:56:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 15:56:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | |
| 15:56:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338 | |
| 15:56:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 15:57:02 | stephenfin | efried: Yeah, trivial to rebase | |
| 15:57:07 | efried | nice, thanks stephenfin | |
| 15:57:23 | openstackgerrit | Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | |
| 15:57:57 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add |
|
| 15:58:12 | openstackgerrit | Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | |
| 15:58:26 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 16:00:24 | efried | mriedem: Any idea what this [1] would look like if I wanted to copy "the database logs" ? | |
| 16:00:24 | efried | [1] https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/playbooks/infra-puppet-apply/post.yaml | |
| 16:06:00 | openstackgerrit | melanie witt proposed openstack/nova master: Handle websockify v0.9.0 in console proxy https://review.opendev.org/677856 | |
| 16:06:02 | mriedem | efried: i know devstack based jobs copy the mysqldb logs | |
| 16:06:37 | mriedem | well they used to anyway | |
| 16:08:20 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 16:08:21 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 16:08:29 | mriedem | efried: i'd guess it's a matter of adding a post-run play to collect the db logs https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-lvm/post.yaml | |
| 16:08:31 | mriedem | like ^ | |
| 16:08:58 | mriedem | which i think runs after some other tempest/devstack post task copies things to /logs | |
| 16:09:17 | mriedem | where the sqlite lots are stored, idk - or does this only fail in mysql? | |
| 16:09:52 | mriedem | devstack-gate for legacy jobs would save off the mysql logs here https://github.com/openstack/devstack-gate/blob/9459f44cb5f604b84ca6b5642b89d7aeceae9f35/functions.sh#L667 | |
| 16:11:02 | mriedem | so we need a play that copies those to devstack_log_dir | |
| 16:11:05 | mriedem | $devstack_log_dir | |
| 16:12:12 | mriedem | ok i see something in devstack-base | |
| 16:12:32 | mriedem | https://github.com/openstack/devstack/blob/master/.zuul.yaml#L243 | |
| 16:12:44 | mriedem | though that must not be working b/c i'm not seeing mysql logs in tempest job output | |
| 16:12:53 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 16:12:57 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 16:14:15 | efried | mriedem: trying there --^ | |
| 16:14:24 | efried | if you can see immediate problems, please lmk. | |
| 16:14:50 | belmoreira | melwitt I get the same result. Doesn't work | |
| 16:15:15 | melwitt | belmoreira: and you tried a fresh vnc console for an instance? it won't work for the already existing ones | |
| 16:15:30 | belmoreira | melwitt fresh | |
| 16:16:48 | sean-k-mooney | stephenfin: before i forget yes the comment you left in the toplogy api patch make sense to me | |
| 16:16:59 | stephenfin | (y) | |
| 16:18:03 | belmoreira | melwitt can you see the trace? https://www.codepile.net/pile/pVkj75Ll | |
| 16:19:13 | mriedem | efried: looks like by default mysql doesn't enable logging | |
| 16:19:21 | mriedem | https://dev.mysql.com/doc/refman/5.7/en/server-logs.html | |
| 16:19:31 | mriedem | so that's probably why we don't get them in the devstack job | |
| 16:20:01 | belmoreira | melwitt but is going to the nova-consoleauth because nova-consoleauth also fails to verify the token | |
| 16:20:01 | melwitt | belmoreira: yeah. actually, that's showing it not falling back to nova-consoleauth. it's failing the database-based validation (expected). but with CONF.workarounds.enable_consoleauth = True it should fall back. checking the code again... | |
| 16:20:13 | efried | mriedem: iiuc this bug happens on unit tests, not devstacks; is there something we need to do to enable logging there? | |
| 16:20:48 | mriedem | efried: it's not devstack, but the unit test jobs setup mysql and postgresql for opportunistic db testing for the schema migrations | |
| 16:21:00 | melwitt | belmoreira: you see it going to nova-consoleauth too? I'm missing it in the trace | |
| 16:21:03 | mriedem | that's what nova/tools/test-setup.sh is for | |
| 16:21:22 | mriedem | efried: so likely just means we need to configure mysql for logging in that script | |