| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-22 | |||
| 15:56:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795 | |
| 15:56:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797 | |
| 15:56:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798 | |
| 15:56:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799 | |
| 15:56:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Rework 'hardware.numa_usage_from_instances' https://review.opendev.org/672565 | |
| 15:56:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | |
| 15:56:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894 | |
| 15:56:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.opendev.org/537414 | |
| 15:56:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add '[compute] cpu_dedicated_set' option https://review.opendev.org/671792 | |
| 15:56:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 15:56:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | |
| 15:56:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | |
| 15:56:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 15:56:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 15:56:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | |
| 15:56:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338 | |
| 15:56:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 15:57:02 | stephenfin | efried: Yeah, trivial to rebase | |
| 15:57:07 | efried | nice, thanks stephenfin | |
| 15:57:23 | openstackgerrit | Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | |
| 15:57:57 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add |
|
| 15:58:12 | openstackgerrit | Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | |
| 15:58:26 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 16:00:24 | efried | mriedem: Any idea what this [1] would look like if I wanted to copy "the database logs" ? | |
| 16:00:24 | efried | [1] https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/playbooks/infra-puppet-apply/post.yaml | |
| 16:06:00 | openstackgerrit | melanie witt proposed openstack/nova master: Handle websockify v0.9.0 in console proxy https://review.opendev.org/677856 | |
| 16:06:02 | mriedem | efried: i know devstack based jobs copy the mysqldb logs | |
| 16:06:37 | mriedem | well they used to anyway | |
| 16:08:20 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 16:08:21 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 16:08:29 | mriedem | efried: i'd guess it's a matter of adding a post-run play to collect the db logs https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-lvm/post.yaml | |
| 16:08:31 | mriedem | like ^ | |
| 16:08:58 | mriedem | which i think runs after some other tempest/devstack post task copies things to /logs | |
| 16:09:17 | mriedem | where the sqlite lots are stored, idk - or does this only fail in mysql? | |
| 16:09:52 | mriedem | devstack-gate for legacy jobs would save off the mysql logs here https://github.com/openstack/devstack-gate/blob/9459f44cb5f604b84ca6b5642b89d7aeceae9f35/functions.sh#L667 | |
| 16:11:02 | mriedem | so we need a play that copies those to devstack_log_dir | |
| 16:11:05 | mriedem | $devstack_log_dir | |
| 16:12:12 | mriedem | ok i see something in devstack-base | |
| 16:12:32 | mriedem | https://github.com/openstack/devstack/blob/master/.zuul.yaml#L243 | |
| 16:12:44 | mriedem | though that must not be working b/c i'm not seeing mysql logs in tempest job output | |
| 16:12:53 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 16:12:57 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 16:14:15 | efried | mriedem: trying there --^ | |
| 16:14:24 | efried | if you can see immediate problems, please lmk. | |
| 16:14:50 | belmoreira | melwitt I get the same result. Doesn't work | |
| 16:15:15 | melwitt | belmoreira: and you tried a fresh vnc console for an instance? it won't work for the already existing ones | |
| 16:15:30 | belmoreira | melwitt fresh | |
| 16:16:48 | sean-k-mooney | stephenfin: before i forget yes the comment you left in the toplogy api patch make sense to me | |
| 16:16:59 | stephenfin | (y) | |
| 16:18:03 | belmoreira | melwitt can you see the trace? https://www.codepile.net/pile/pVkj75Ll | |
| 16:19:13 | mriedem | efried: looks like by default mysql doesn't enable logging | |
| 16:19:21 | mriedem | https://dev.mysql.com/doc/refman/5.7/en/server-logs.html | |
| 16:19:31 | mriedem | so that's probably why we don't get them in the devstack job | |
| 16:20:01 | belmoreira | melwitt but is going to the nova-consoleauth because nova-consoleauth also fails to verify the token | |
| 16:20:01 | melwitt | belmoreira: yeah. actually, that's showing it not falling back to nova-consoleauth. it's failing the database-based validation (expected). but with CONF.workarounds.enable_consoleauth = True it should fall back. checking the code again... | |
| 16:20:13 | efried | mriedem: iiuc this bug happens on unit tests, not devstacks; is there something we need to do to enable logging there? | |
| 16:20:48 | mriedem | efried: it's not devstack, but the unit test jobs setup mysql and postgresql for opportunistic db testing for the schema migrations | |
| 16:21:00 | melwitt | belmoreira: you see it going to nova-consoleauth too? I'm missing it in the trace | |
| 16:21:03 | mriedem | that's what nova/tools/test-setup.sh is for | |
| 16:21:22 | mriedem | efried: so likely just means we need to configure mysql for logging in that script | |
| 16:22:38 | belmoreira | melwitt I see the consoleauth log: Checking Token: a86d2dcb-a8e3-4fa6-b707-92ae1243c96f, False | |
| 16:23:47 | melwitt | oh, I see yeah. sorry. I got it backward, with enable_consoleauth = True it will try nova-consoleauth first and then fall back to the database | |
| 16:24:25 | belmoreira | and actually I see the token being sent to the nova-consoleauth: Received Token: *** | |
| 16:24:31 | efried | mriedem: sorry if I'm getting lost here; are you telling me my patch right now won't actually have anything to copy unless we do something to that test-setup.sh? | |
| 16:24:48 | mriedem | efried: we can see, but i think so | |
| 16:24:50 | mriedem | from what i've been reading | |
| 16:24:53 | mriedem | and seeing in the devstack jobs | |
| 16:24:58 | mriedem | or not seeing really | |
| 16:25:17 | mriedem | devstack jobs will attempt to copy /var/log/mysql.log/.err but i don't see those in devstack job logs that get published, | |
| 16:25:24 | mriedem | and the mysql docs say logging is not enabled by deafult | |
| 16:25:27 | mriedem | *default | |
| 16:25:35 | mriedem | so that's probably why i'm not seeing anything published | |
| 16:25:43 | belmoreira | melwitt can you point me to this code? | |
| 16:25:50 | melwitt | belmoreira: yeah... so this implies the original get_vnc_console call isn't storing the auth in nova-consoleauth. but it should be as long as you've set enable_consoleauth = True on the nova-api configs... | |
| 16:25:53 | melwitt | yes. one sec | |
| 16:26:31 | melwitt | belmoreira: https://github.com/openstack/nova/blob/stable/stein/nova/compute/api.py#L3948-L3964 | |
| 16:26:40 | melwitt | that's what stores the token auth in nova-consoleauth when enabled | |
| 16:27:00 | melwitt | and you already the other side working, the console proxy is looking there | |
| 16:27:11 | melwitt | so something is missing about the nova-api side | |
| 16:29:15 | efried | stephenfin: Would you please hit https://review.opendev.org/#/c/677856/ again before you leave? | |
| 16:29:25 | belmoreira | melwitt thanks a lot for your help | |
| 16:29:52 | belmoreira | melwitt what was really missing in my side was enable_consoleauth = True in the nova-api | |
| 16:30:00 | stephenfin | Yup, done | |
| 16:30:20 | efried | mriedem: would you mind making that edit, since you seem to have at least some idea how that would be done? I'm flying blind here. | |
| 16:30:20 | efried | Or did you want to wait until this one finishes and verify that the logs are absent? | |
| 16:30:27 | belmoreira | after that I see nova-consoleauth getting the token | |
| 16:30:30 | efried | thanks stephenfin | |
| 16:30:38 | melwitt | belmoreira: yeah, but you said you added that and restarted the nova-api's right? and it's still not working? | |
| 16:30:52 | belmoreira | melwiit but it was falling... and the log in the nonvc was confusing | |
| 16:31:26 | belmoreira | melwitt the issue was memcache for some reason needed a restart | |
| 16:31:50 | belmoreira | melwitt it's working \o/ Thanks a lot again | |
| 16:32:07 | melwitt | belmoreira: oh, weird. I don't understand why that would be needed but relieved it's working now | |
| 16:33:07 | belmoreira | I will add "enable_consoleauth = True" into my prod apis. This will give more time to better decide how to setup this per cell | |
| 16:35:28 | belmoreira | melwitt the main issue that I see for this new setup is because I have a lot of cells and we allow vnc console access from outside world. With this means that I will need to have an "ingress" to match the vnc url to the correct server. | |
| 16:36:11 | melwitt | belmoreira: good call. just please be sure to get the per cell setup worked out before Train because all of the workaround stuff gets removed at that time. so you will want the per cell setup working in Stein *before* you upgrade to Train | |
| 16:36:13 | belmoreira | melwitt that's why I'm using vnc as a central service | |
| 16:37:33 | belmoreira | melwitt thanks a lotn again. This gives me some time to properly plan. I didn't follow all this refactor done in the vnc | |
| 16:39:45 | melwitt | belmoreira: yeah... well, the key thing is the proxies need access to the cell databases. so if you can set that up in a way without having to give ingress to your cell controllers (if you made them separate maybe?) then there are some options. maybe talk to mnaser to see how he does it | |
| 16:40:34 | melwitt | belmoreira: you're not alone. this has been a troublesome one :( maybe there is yet more I need to add to the documentation | |
| 16:42:27 | mriedem | efried: "Or did you want to wait until this one finishes and verify that the logs are absent?" let's go with that | |