| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-22 | |||
| 15:26:57 | aspiers | efried: OK thanks | |
| 15:27:28 | aspiers | alex_xu: so that means https://review.opendev.org/#/c/664420/42 is ready for re-review | |
| 15:27:39 | efried | stephenfin: yes, that's the one. | |
| 15:27:42 | belmoreira | melwitt right... is there any easy way to enable tokens in nova-consoleauth again? (I'm in the middle of a crisis here) | |
| 15:27:52 | alex_xu | aspiers: yea | |
| 15:28:02 | aspiers | alex_xu: oops sorry I mean https://review.opendev.org/#/c/664420/ previous link was an old PS | |
| 15:28:08 | stephenfin | I think I could move that to the top of the list of approved patches if you want it now? I don't think it has a dependency on any of my other functional changes | |
| 15:28:12 | efried | stephenfin: I wonder, is there a way it could be moved out/earlier?... jinx | |
| 15:28:19 | stephenfin | coke! | |
| 15:28:41 | aspiers | alex_xu, stephenfin: also I fixed the tiny bug in https://review.opendev.org/#/c/638680 | |
| 15:28:49 | efried | stephenfin: as discussed above, we're not blocking on it, we're going to continue to do the logic in the request filter for now. So don't worry about it unless it's easy. | |
| 15:29:04 | stephenfin | Sure. Let me try | |
| 15:32:18 | melwitt | belmoreira: hm, looking at the code, it should still work if you have workarounds/enable_consoleauth = True in stein. I see in nova/compute/api.py it still stores the token console auth in nova-consoleauth in addition to in the cell database. so I can't see why it wouldn't work yet | |
| 15:33:55 | belmoreira | melwitt from the consoleauth I only get: Checking Token: ***, False | |
| 15:34:50 | belmoreira | melwitt from nova-novncproxy: https://www.codepile.net/pile/O1P8ee1q | |
| 15:34:59 | melwitt | belmoreira: did you set workarounds/enable_consoleauth = True on hosts where nova-api and nova-novncproxy runs? | |
| 15:35:54 | stephenfin | belmoreira: Maybe it's just me, but I don't see anything in that codepile link | |
| 15:38:27 | belmoreira | melwitt workarounds/enable_consoleauth it's also for the nova-api? I'm only running it in the vnc nodes | |
| 15:38:55 | melwitt | belmoreira: you need it for nova-api to tell it to store token auth in nova-consoleauth service | |
| 15:39:21 | melwitt | and you need it for nonvcproxy to tell it to look in nova-consoleauth to find token auths when validating | |
| 15:39:24 | melwitt | so you need both | |
| 15:39:48 | belmoreira | that will be easy to change. Let me do it | |
| 15:41:04 | openstackgerrit | Merged openstack/nova master: Track libvirt host/domain capabilities for multiple machine types https://review.opendev.org/673151 | |
| 15:41:13 | openstackgerrit | Merged openstack/nova master: Extract new base class for provider usage functional tests https://review.opendev.org/676964 | |
| 15:53:12 | efried | I'm trying to hack up a patch to do that, but there's at least three things about it that I don't understand. | |
| 15:53:12 | efried | mriedem: clarkb suggested trying to repro the innodb thing with a post that copies database logs. | |
| 15:56:31 | openstackgerrit | Adam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | |
| 15:56:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: claims: Remove useless caching https://review.opendev.org/671791 | |
| 15:56:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797 | |
| 15:56:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795 | |
| 15:56:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799 | |
| 15:56:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798 | |
| 15:56:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | |
| 15:56:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Rework 'hardware.numa_usage_from_instances' https://review.opendev.org/672565 | |
| 15:56:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.opendev.org/537414 | |
| 15:56:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894 | |
| 15:56:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 15:56:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add '[compute] cpu_dedicated_set' option https://review.opendev.org/671792 | |
| 15:56:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | |
| 15:56:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | |
| 15:56:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 15:56:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 15:56:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338 | |
| 15:56:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | |
| 15:56:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 15:57:02 | stephenfin | efried: Yeah, trivial to rebase | |
| 15:57:07 | efried | nice, thanks stephenfin | |
| 15:57:23 | openstackgerrit | Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | |
| 15:57:57 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add |
|
| 15:58:12 | openstackgerrit | Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | |
| 15:58:26 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 16:00:24 | efried | [1] https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/playbooks/infra-puppet-apply/post.yaml | |
| 16:00:24 | efried | mriedem: Any idea what this [1] would look like if I wanted to copy "the database logs" ? | |
| 16:06:00 | openstackgerrit | melanie witt proposed openstack/nova master: Handle websockify v0.9.0 in console proxy https://review.opendev.org/677856 | |
| 16:06:02 | mriedem | efried: i know devstack based jobs copy the mysqldb logs | |
| 16:06:37 | mriedem | well they used to anyway | |
| 16:08:20 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 16:08:21 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 16:08:29 | mriedem | efried: i'd guess it's a matter of adding a post-run play to collect the db logs https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-lvm/post.yaml | |
| 16:08:31 | mriedem | like ^ | |
| 16:08:58 | mriedem | which i think runs after some other tempest/devstack post task copies things to /logs | |
| 16:09:17 | mriedem | where the sqlite lots are stored, idk - or does this only fail in mysql? | |
| 16:09:52 | mriedem | devstack-gate for legacy jobs would save off the mysql logs here https://github.com/openstack/devstack-gate/blob/9459f44cb5f604b84ca6b5642b89d7aeceae9f35/functions.sh#L667 | |
| 16:11:02 | mriedem | so we need a play that copies those to devstack_log_dir | |
| 16:11:05 | mriedem | $devstack_log_dir | |
| 16:12:12 | mriedem | ok i see something in devstack-base | |
| 16:12:32 | mriedem | https://github.com/openstack/devstack/blob/master/.zuul.yaml#L243 | |
| 16:12:44 | mriedem | though that must not be working b/c i'm not seeing mysql logs in tempest job output | |
| 16:12:53 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 16:12:57 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 16:14:15 | efried | mriedem: trying there --^ | |
| 16:14:24 | efried | if you can see immediate problems, please lmk. | |
| 16:14:50 | belmoreira | melwitt I get the same result. Doesn't work | |
| 16:15:15 | melwitt | belmoreira: and you tried a fresh vnc console for an instance? it won't work for the already existing ones | |
| 16:15:30 | belmoreira | melwitt fresh | |
| 16:16:48 | sean-k-mooney | stephenfin: before i forget yes the comment you left in the toplogy api patch make sense to me | |
| 16:16:59 | stephenfin | (y) | |
| 16:18:03 | belmoreira | melwitt can you see the trace? https://www.codepile.net/pile/pVkj75Ll | |
| 16:19:13 | mriedem | efried: looks like by default mysql doesn't enable logging | |
| 16:19:21 | mriedem | https://dev.mysql.com/doc/refman/5.7/en/server-logs.html | |
| 16:19:31 | mriedem | so that's probably why we don't get them in the devstack job | |
| 16:20:01 | melwitt | belmoreira: yeah. actually, that's showing it not falling back to nova-consoleauth. it's failing the database-based validation (expected). but with CONF.workarounds.enable_consoleauth = True it should fall back. checking the code again... | |
| 16:20:01 | belmoreira | melwitt but is going to the nova-consoleauth because nova-consoleauth also fails to verify the token | |
| 16:20:13 | efried | mriedem: iiuc this bug happens on unit tests, not devstacks; is there something we need to do to enable logging there? | |
| 16:20:48 | mriedem | efried: it's not devstack, but the unit test jobs setup mysql and postgresql for opportunistic db testing for the schema migrations | |
| 16:21:00 | melwitt | belmoreira: you see it going to nova-consoleauth too? I'm missing it in the trace | |
| 16:21:03 | mriedem | that's what nova/tools/test-setup.sh is for | |
| 16:21:22 | mriedem | efried: so likely just means we need to configure mysql for logging in that script | |
| 16:22:38 | belmoreira | melwitt I see the consoleauth log: Checking Token: a86d2dcb-a8e3-4fa6-b707-92ae1243c96f, False | |
| 16:23:47 | melwitt | oh, I see yeah. sorry. I got it backward, with enable_consoleauth = True it will try nova-consoleauth first and then fall back to the database | |
| 16:24:25 | belmoreira | and actually I see the token being sent to the nova-consoleauth: Received Token: *** | |
| 16:24:31 | efried | mriedem: sorry if I'm getting lost here; are you telling me my patch right now won't actually have anything to copy unless we do something to that test-setup.sh? | |
| 16:24:48 | mriedem | efried: we can see, but i think so | |
| 16:24:50 | mriedem | from what i've been reading | |
| 16:24:53 | mriedem | and seeing in the devstack jobs | |
| 16:24:58 | mriedem | or not seeing really | |
| 16:25:17 | mriedem | devstack jobs will attempt to copy /var/log/mysql.log/.err but i don't see those in devstack job logs that get published, | |
| 16:25:24 | mriedem | and the mysql docs say logging is not enabled by deafult | |
| 16:25:27 | mriedem | *default | |
| 16:25:35 | mriedem | so that's probably why i'm not seeing anything published | |