Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-22
15:38:27 belmoreira melwitt workarounds/enable_consoleauth it's also for the nova-api? I'm only running it in the vnc nodes
15:38:55 melwitt belmoreira: you need it for nova-api to tell it to store token auth in nova-consoleauth service
15:39:21 melwitt and you need it for nonvcproxy to tell it to look in nova-consoleauth to find token auths when validating
15:39:24 melwitt so you need both
15:39:48 belmoreira that will be easy to change. Let me do it
15:41:04 openstackgerrit Merged openstack/nova master: Track libvirt host/domain capabilities for multiple machine types https://review.opendev.org/673151
15:41:13 openstackgerrit Merged openstack/nova master: Extract new base class for provider usage functional tests https://review.opendev.org/676964
15:53:12 efried mriedem: clarkb suggested trying to repro the innodb thing with a post that copies database logs.
15:53:12 efried I'm trying to hack up a patch to do that, but there's at least three things about it that I don't understand.
15:56:31 openstackgerrit Adam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported https://review.opendev.org/638680
15:56:42 openstackgerrit Stephen Finucane proposed openstack/nova master: claims: Remove useless caching https://review.opendev.org/671791
15:56:43 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795
15:56:43 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797
15:56:44 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798
15:56:44 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799
15:56:45 openstackgerrit Stephen Finucane proposed openstack/nova master: Rework 'hardware.numa_usage_from_instances' https://review.opendev.org/672565
15:56:45 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336
15:56:46 openstackgerrit Stephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894
15:56:46 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.opendev.org/537414
15:56:47 openstackgerrit Stephen Finucane proposed openstack/nova master: Add '[compute] cpu_dedicated_set' option https://review.opendev.org/671792
15:56:47 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793
15:56:48 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693
15:56:48 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395
15:56:49 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800
15:56:49 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103
15:56:50 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571
15:56:50 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338
15:56:51 openstackgerrit Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801
15:57:02 stephenfin efried: Yeah, trivial to rebase
15:57:07 efried nice, thanks stephenfin
15:57:23 openstackgerrit Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334
15:57:57 openstackgerrit Adam Spiers proposed openstack/nova master: Add and to config.py https://review.opendev.org/636318
15:58:12 openstackgerrit Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565
15:58:26 openstackgerrit Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616
16:00:24 efried mriedem: Any idea what this [1] would look like if I wanted to copy "the database logs" ?
16:00:24 efried [1] https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/playbooks/infra-puppet-apply/post.yaml
16:06:00 openstackgerrit melanie witt proposed openstack/nova master: Handle websockify v0.9.0 in console proxy https://review.opendev.org/677856
16:06:02 mriedem efried: i know devstack based jobs copy the mysqldb logs
16:06:37 mriedem well they used to anyway
16:08:20 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051
16:08:21 openstack bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251
16:08:29 mriedem efried: i'd guess it's a matter of adding a post-run play to collect the db logs https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-lvm/post.yaml
16:08:31 mriedem like ^
16:08:58 mriedem which i think runs after some other tempest/devstack post task copies things to /logs
16:09:17 mriedem where the sqlite lots are stored, idk - or does this only fail in mysql?
16:09:52 mriedem devstack-gate for legacy jobs would save off the mysql logs here https://github.com/openstack/devstack-gate/blob/9459f44cb5f604b84ca6b5642b89d7aeceae9f35/functions.sh#L667
16:11:02 mriedem so we need a play that copies those to devstack_log_dir
16:11:05 mriedem $devstack_log_dir
16:12:12 mriedem ok i see something in devstack-base
16:12:32 mriedem https://github.com/openstack/devstack/blob/master/.zuul.yaml#L243
16:12:44 mriedem though that must not be working b/c i'm not seeing mysql logs in tempest job output
16:12:53 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051
16:12:57 openstack bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251
16:14:15 efried mriedem: trying there --^
16:14:24 efried if you can see immediate problems, please lmk.
16:14:50 belmoreira melwitt I get the same result. Doesn't work
16:15:15 melwitt belmoreira: and you tried a fresh vnc console for an instance? it won't work for the already existing ones
16:15:30 belmoreira melwitt fresh
16:16:48 sean-k-mooney stephenfin: before i forget yes the comment you left in the toplogy api patch make sense to me
16:16:59 stephenfin (y)
16:18:03 belmoreira melwitt can you see the trace? https://www.codepile.net/pile/pVkj75Ll
16:19:13 mriedem efried: looks like by default mysql doesn't enable logging
16:19:21 mriedem https://dev.mysql.com/doc/refman/5.7/en/server-logs.html
16:19:31 mriedem so that's probably why we don't get them in the devstack job
16:20:01 belmoreira melwitt but is going to the nova-consoleauth because nova-consoleauth also fails to verify the token
16:20:01 melwitt belmoreira: yeah. actually, that's showing it not falling back to nova-consoleauth. it's failing the database-based validation (expected). but with CONF.workarounds.enable_consoleauth = True it should fall back. checking the code again...
16:20:13 efried mriedem: iiuc this bug happens on unit tests, not devstacks; is there something we need to do to enable logging there?
16:20:48 mriedem efried: it's not devstack, but the unit test jobs setup mysql and postgresql for opportunistic db testing for the schema migrations
16:21:00 melwitt belmoreira: you see it going to nova-consoleauth too? I'm missing it in the trace
16:21:03 mriedem that's what nova/tools/test-setup.sh is for
16:21:22 mriedem efried: so likely just means we need to configure mysql for logging in that script
16:22:38 belmoreira melwitt I see the consoleauth log: Checking Token: a86d2dcb-a8e3-4fa6-b707-92ae1243c96f, False
16:23:47 melwitt oh, I see yeah. sorry. I got it backward, with enable_consoleauth = True it will try nova-consoleauth first and then fall back to the database
16:24:25 belmoreira and actually I see the token being sent to the nova-consoleauth: Received Token: ***
16:24:31 efried mriedem: sorry if I'm getting lost here; are you telling me my patch right now won't actually have anything to copy unless we do something to that test-setup.sh?
16:24:48 mriedem efried: we can see, but i think so
16:24:50 mriedem from what i've been reading
16:24:53 mriedem and seeing in the devstack jobs
16:24:58 mriedem or not seeing really
16:25:17 mriedem devstack jobs will attempt to copy /var/log/mysql.log/.err but i don't see those in devstack job logs that get published,
16:25:24 mriedem and the mysql docs say logging is not enabled by deafult
16:25:27 mriedem *default
16:25:35 mriedem so that's probably why i'm not seeing anything published
16:25:43 belmoreira melwitt can you point me to this code?
16:25:50 melwitt belmoreira: yeah... so this implies the original get_vnc_console call isn't storing the auth in nova-consoleauth. but it should be as long as you've set enable_consoleauth = True on the nova-api configs...
16:25:53 melwitt yes. one sec
16:26:31 melwitt belmoreira: https://github.com/openstack/nova/blob/stable/stein/nova/compute/api.py#L3948-L3964
16:26:40 melwitt that's what stores the token auth in nova-consoleauth when enabled
16:27:00 melwitt and you already the other side working, the console proxy is looking there
16:27:11 melwitt so something is missing about the nova-api side
16:29:15 efried stephenfin: Would you please hit https://review.opendev.org/#/c/677856/ again before you leave?
16:29:25 belmoreira melwitt thanks a lot for your help
16:29:52 belmoreira melwitt what was really missing in my side was enable_consoleauth = True in the nova-api
16:30:00 stephenfin Yup, done
16:30:20 efried mriedem: would you mind making that edit, since you seem to have at least some idea how that would be done? I'm flying blind here.
16:30:20 efried Or did you want to wait until this one finishes and verify that the logs are absent?
16:30:27 belmoreira after that I see nova-consoleauth getting the token
16:30:30 efried thanks stephenfin
16:30:38 melwitt belmoreira: yeah, but you said you added that and restarted the nova-api's right? and it's still not working?
16:30:52 belmoreira melwiit but it was falling... and the log in the nonvc was confusing

Earlier   Later