Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-22
16:06:37 mriedem well they used to anyway
16:08:20 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051
16:08:21 openstack bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251
16:08:29 mriedem efried: i'd guess it's a matter of adding a post-run play to collect the db logs https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-lvm/post.yaml
16:08:31 mriedem like ^
16:08:58 mriedem which i think runs after some other tempest/devstack post task copies things to /logs
16:09:17 mriedem where the sqlite lots are stored, idk - or does this only fail in mysql?
16:09:52 mriedem devstack-gate for legacy jobs would save off the mysql logs here https://github.com/openstack/devstack-gate/blob/9459f44cb5f604b84ca6b5642b89d7aeceae9f35/functions.sh#L667
16:11:02 mriedem so we need a play that copies those to devstack_log_dir
16:11:05 mriedem $devstack_log_dir
16:12:12 mriedem ok i see something in devstack-base
16:12:32 mriedem https://github.com/openstack/devstack/blob/master/.zuul.yaml#L243
16:12:44 mriedem though that must not be working b/c i'm not seeing mysql logs in tempest job output
16:12:53 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051
16:12:57 openstack bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251
16:14:15 efried mriedem: trying there --^
16:14:24 efried if you can see immediate problems, please lmk.
16:14:50 belmoreira melwitt I get the same result. Doesn't work
16:15:15 melwitt belmoreira: and you tried a fresh vnc console for an instance? it won't work for the already existing ones
16:15:30 belmoreira melwitt fresh
16:16:48 sean-k-mooney stephenfin: before i forget yes the comment you left in the toplogy api patch make sense to me
16:16:59 stephenfin (y)
16:18:03 belmoreira melwitt can you see the trace? https://www.codepile.net/pile/pVkj75Ll
16:19:13 mriedem efried: looks like by default mysql doesn't enable logging
16:19:21 mriedem https://dev.mysql.com/doc/refman/5.7/en/server-logs.html
16:19:31 mriedem so that's probably why we don't get them in the devstack job
16:20:01 belmoreira melwitt but is going to the nova-consoleauth because nova-consoleauth also fails to verify the token
16:20:01 melwitt belmoreira: yeah. actually, that's showing it not falling back to nova-consoleauth. it's failing the database-based validation (expected). but with CONF.workarounds.enable_consoleauth = True it should fall back. checking the code again...
16:20:13 efried mriedem: iiuc this bug happens on unit tests, not devstacks; is there something we need to do to enable logging there?
16:20:48 mriedem efried: it's not devstack, but the unit test jobs setup mysql and postgresql for opportunistic db testing for the schema migrations
16:21:00 melwitt belmoreira: you see it going to nova-consoleauth too? I'm missing it in the trace
16:21:03 mriedem that's what nova/tools/test-setup.sh is for
16:21:22 mriedem efried: so likely just means we need to configure mysql for logging in that script
16:22:38 belmoreira melwitt I see the consoleauth log: Checking Token: a86d2dcb-a8e3-4fa6-b707-92ae1243c96f, False
16:23:47 melwitt oh, I see yeah. sorry. I got it backward, with enable_consoleauth = True it will try nova-consoleauth first and then fall back to the database
16:24:25 belmoreira and actually I see the token being sent to the nova-consoleauth: Received Token: ***
16:24:31 efried mriedem: sorry if I'm getting lost here; are you telling me my patch right now won't actually have anything to copy unless we do something to that test-setup.sh?
16:24:48 mriedem efried: we can see, but i think so
16:24:50 mriedem from what i've been reading
16:24:53 mriedem and seeing in the devstack jobs
16:24:58 mriedem or not seeing really
16:25:17 mriedem devstack jobs will attempt to copy /var/log/mysql.log/.err but i don't see those in devstack job logs that get published,
16:25:24 mriedem and the mysql docs say logging is not enabled by deafult
16:25:27 mriedem *default
16:25:35 mriedem so that's probably why i'm not seeing anything published
16:25:43 belmoreira melwitt can you point me to this code?
16:25:50 melwitt belmoreira: yeah... so this implies the original get_vnc_console call isn't storing the auth in nova-consoleauth. but it should be as long as you've set enable_consoleauth = True on the nova-api configs...
16:25:53 melwitt yes. one sec
16:26:31 melwitt belmoreira: https://github.com/openstack/nova/blob/stable/stein/nova/compute/api.py#L3948-L3964
16:26:40 melwitt that's what stores the token auth in nova-consoleauth when enabled
16:27:00 melwitt and you already the other side working, the console proxy is looking there
16:27:11 melwitt so something is missing about the nova-api side
16:29:15 efried stephenfin: Would you please hit https://review.opendev.org/#/c/677856/ again before you leave?
16:29:25 belmoreira melwitt thanks a lot for your help
16:29:52 belmoreira melwitt what was really missing in my side was enable_consoleauth = True in the nova-api
16:30:00 stephenfin Yup, done
16:30:20 efried mriedem: would you mind making that edit, since you seem to have at least some idea how that would be done? I'm flying blind here.
16:30:20 efried Or did you want to wait until this one finishes and verify that the logs are absent?
16:30:27 belmoreira after that I see nova-consoleauth getting the token
16:30:30 efried thanks stephenfin
16:30:38 melwitt belmoreira: yeah, but you said you added that and restarted the nova-api's right? and it's still not working?
16:30:52 belmoreira melwiit but it was falling... and the log in the nonvc was confusing
16:31:26 belmoreira melwitt the issue was memcache for some reason needed a restart
16:31:50 belmoreira melwitt it's working \o/ Thanks a lot again
16:32:07 melwitt belmoreira: oh, weird. I don't understand why that would be needed but relieved it's working now
16:33:07 belmoreira I will add "enable_consoleauth = True" into my prod apis. This will give more time to better decide how to setup this per cell
16:35:28 belmoreira melwitt the main issue that I see for this new setup is because I have a lot of cells and we allow vnc console access from outside world. With this means that I will need to have an "ingress" to match the vnc url to the correct server.
16:36:11 melwitt belmoreira: good call. just please be sure to get the per cell setup worked out before Train because all of the workaround stuff gets removed at that time. so you will want the per cell setup working in Stein *before* you upgrade to Train
16:36:13 belmoreira melwitt that's why I'm using vnc as a central service
16:37:33 belmoreira melwitt thanks a lotn again. This gives me some time to properly plan. I didn't follow all this refactor done in the vnc
16:39:45 melwitt belmoreira: yeah... well, the key thing is the proxies need access to the cell databases. so if you can set that up in a way without having to give ingress to your cell controllers (if you made them separate maybe?) then there are some options. maybe talk to mnaser to see how he does it
16:40:34 melwitt belmoreira: you're not alone. this has been a troublesome one :( maybe there is yet more I need to add to the documentation
16:42:27 mriedem efried: "Or did you want to wait until this one finishes and verify that the logs are absent?" let's go with that
16:43:36 efried ack, thx
16:43:51 mriedem it's 11:43 and i feel like i haven't really settled into being productive today
16:44:06 efried it's like being PTL all over again, eh?
16:44:19 efried Maybe doing PTL stuff felt productive to you
16:44:38 mriedem i wasn't doing "ptl stuff" most days
16:45:04 sean-k-mooney do we still need n-cauth? just noticed its enabled in some of the tempest jobs
16:45:33 mriedem sean-k-mooney: we should'nt no
16:45:46 mriedem hell i thought stephenfin had patches to remove it in train
16:45:58 sean-k-mooney ok ya he did
16:46:00 mnaser im pretty sure i saw it removed
16:46:21 sean-k-mooney oh never mind i was looking at a stable run
16:46:29 mriedem zoinks
16:48:30 sean-k-mooney i was looking at a stable/rocky log master is correct
16:53:26 openstackgerrit Matt Riedemann proposed openstack/nova master: docs: pretty up return code table or heal_allocations https://review.opendev.org/678063
16:55:26 openstackgerrit Eric Fried proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899
16:55:26 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for node.list https://review.opendev.org/656027
16:55:27 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for validating instance and node https://review.opendev.org/656028
16:55:27 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690
16:55:28 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for add/remove instance info from node https://review.opendev.org/659691
16:55:28 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for getting network metadata from node https://review.opendev.org/670213
16:57:50 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: pretty up return code table for sync_aggregates https://review.opendev.org/678064
17:00:27 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980
17:01:22 mriedem lyarwood: if you're back, can you hit these stein backports when you get a chance? https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/stein+topic:bug/1839560 - regressoin since rocky for baremetal nodes so would be good to get those in
17:02:48 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497
17:09:08 stephenfin dansmith, mriedem: When a compute node record is deleted, can it ever come back?
17:09:16 dansmith wellllll
17:09:22 dansmith about that...

Earlier   Later