Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-22
16:29:15 efried stephenfin: Would you please hit https://review.opendev.org/#/c/677856/ again before you leave?
16:29:25 belmoreira melwitt thanks a lot for your help
16:29:52 belmoreira melwitt what was really missing in my side was enable_consoleauth = True in the nova-api
16:30:00 stephenfin Yup, done
16:30:20 efried Or did you want to wait until this one finishes and verify that the logs are absent?
16:30:20 efried mriedem: would you mind making that edit, since you seem to have at least some idea how that would be done? I'm flying blind here.
16:30:27 belmoreira after that I see nova-consoleauth getting the token
16:30:30 efried thanks stephenfin
16:30:38 melwitt belmoreira: yeah, but you said you added that and restarted the nova-api's right? and it's still not working?
16:30:52 belmoreira melwiit but it was falling... and the log in the nonvc was confusing
16:31:26 belmoreira melwitt the issue was memcache for some reason needed a restart
16:31:50 belmoreira melwitt it's working \o/ Thanks a lot again
16:32:07 melwitt belmoreira: oh, weird. I don't understand why that would be needed but relieved it's working now
16:33:07 belmoreira I will add "enable_consoleauth = True" into my prod apis. This will give more time to better decide how to setup this per cell
16:35:28 belmoreira melwitt the main issue that I see for this new setup is because I have a lot of cells and we allow vnc console access from outside world. With this means that I will need to have an "ingress" to match the vnc url to the correct server.
16:36:11 melwitt belmoreira: good call. just please be sure to get the per cell setup worked out before Train because all of the workaround stuff gets removed at that time. so you will want the per cell setup working in Stein *before* you upgrade to Train
16:36:13 belmoreira melwitt that's why I'm using vnc as a central service
16:37:33 belmoreira melwitt thanks a lotn again. This gives me some time to properly plan. I didn't follow all this refactor done in the vnc
16:39:45 melwitt belmoreira: yeah... well, the key thing is the proxies need access to the cell databases. so if you can set that up in a way without having to give ingress to your cell controllers (if you made them separate maybe?) then there are some options. maybe talk to mnaser to see how he does it
16:40:34 melwitt belmoreira: you're not alone. this has been a troublesome one :( maybe there is yet more I need to add to the documentation
16:42:27 mriedem efried: "Or did you want to wait until this one finishes and verify that the logs are absent?" let's go with that
16:43:36 efried ack, thx
16:43:51 mriedem it's 11:43 and i feel like i haven't really settled into being productive today
16:44:06 efried it's like being PTL all over again, eh?
16:44:19 efried Maybe doing PTL stuff felt productive to you
16:44:38 mriedem i wasn't doing "ptl stuff" most days
16:45:04 sean-k-mooney do we still need n-cauth? just noticed its enabled in some of the tempest jobs
16:45:33 mriedem sean-k-mooney: we should'nt no
16:45:46 mriedem hell i thought stephenfin had patches to remove it in train
16:45:58 sean-k-mooney ok ya he did
16:46:00 mnaser im pretty sure i saw it removed
16:46:21 sean-k-mooney oh never mind i was looking at a stable run
16:46:29 mriedem zoinks
16:48:30 sean-k-mooney i was looking at a stable/rocky log master is correct
16:53:26 openstackgerrit Matt Riedemann proposed openstack/nova master: docs: pretty up return code table or heal_allocations https://review.opendev.org/678063
16:55:26 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for node.list https://review.opendev.org/656027
16:55:26 openstackgerrit Eric Fried proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899
16:55:27 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690
16:55:27 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for validating instance and node https://review.opendev.org/656028
16:55:28 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for getting network metadata from node https://review.opendev.org/670213
16:55:28 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK for add/remove instance info from node https://review.opendev.org/659691
16:57:50 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: pretty up return code table for sync_aggregates https://review.opendev.org/678064
17:00:27 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980
17:01:22 mriedem lyarwood: if you're back, can you hit these stein backports when you get a chance? https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/stein+topic:bug/1839560 - regressoin since rocky for baremetal nodes so would be good to get those in
17:02:48 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497
17:09:08 stephenfin dansmith, mriedem: When a compute node record is deleted, can it ever come back?
17:09:16 dansmith wellllll
17:09:22 dansmith about that...
17:10:20 dansmith stephenfin: last week? no.. this week? yes
17:10:21 dansmith https://review.opendev.org/#/c/675496/
17:11:30 stephenfin dansmith: We should probably remove the deleted check from this online migration, in that case? https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L504
17:11:36 melwitt does anyone know how/where notifications are logged?
17:11:52 dansmith stephenfin: no
17:12:22 dansmith stephenfin: you don't want to count deleted nodes for anything until/unless they're undeleted
17:12:42 dansmith and we'll do all the migratory behavior when we resurrect and then subsequently save it
17:12:47 sean-k-mooney melwitt: i guess its proably depned on what is sending them. i would expect to see some notifiaciotn logs in n-cpu
17:12:51 stephenfin doesn't that mean we need to keep that online migration around forever then?
17:13:17 sean-k-mooney e.g. if the compute service is emmiting vm lifecycle notifications
17:13:47 dansmith stephenfin: no, because we do the migration activity upon resurrection
17:13:49 melwitt sean-k-mooney: yeah, I don't know where or how they are logged. there's a CONF.notifications.default_level for the log level but I don't see it used anywhere in nova code
17:13:50 melwitt and I can't find what logs the notifications or what format they're in
17:14:02 dansmith stephenfin: if we ever want to remove *that* behavior we'll need a blocker type migration
17:14:12 stephenfin gotcha
17:14:49 sean-k-mooney melwitt: its unlikely but the only other place that would make sense would be the conductor
17:15:11 sean-k-mooney but it feel inefficnet to need to send them form the conductor so i doubt we would do that
17:16:35 sean-k-mooney melwitt: are you trying to find the ones used by nova join
17:16:46 melwitt I'm trying to find the ones anywhere in nova
17:16:56 melwitt I don't see how or where they are logged
17:20:15 melwitt looks like the only use of CONF.notifications.default_level was removed here https://github.com/openstack/nova/commit/9f48aee9b0ea68f7c8eba6a1f3d076e4194d804d#diff-2f33aae68e09a2fe816870ac389cfe95
17:20:38 melwitt I'm guessing that decorator somehow logged things?
17:20:42 melwitt I dunno
17:21:57 sean-k-mooney melwitt: maybe this https://github.com/openstack/nova/blob/master/nova/rpc.py#L420
17:23:03 melwitt mriedem: do you know if we intentionally stopped logging notifications in rocky? commit linked above ^
17:23:19 melwitt sean-k-mooney: that looks like some kind of error condition. not general log of any notification
17:23:36 dansmith did we ever log them?
17:23:52 dansmith they'd be spread all over the place if we did
17:24:12 melwitt based on the existence of the CONF.notifications.default_level option we did
17:24:13 melwitt yeah
17:24:26 melwitt but the option is no longer used anywhere after that commit
17:24:42 melwitt so I wonder if that decorator used to result in logging somehow. I don't know how it worked
17:25:25 dansmith isn't that level something about which notifcations we emit, not the log level?
17:26:13 dansmith "Default notification level for outgoing notifications."
17:26:18 melwitt I dunno, sounds like a log level https://github.com/openstack/nova/blob/master/nova/conf/notifications.py#L51-L57
17:27:05 melwitt but yeah, I guess it could have meant that notifications used to be categorized and you could prune down which to emit?
17:27:23 dansmith right, like only emit started/stopped type things and not progress-related ones
17:27:32 dansmith or "I checked on a thing" ones
17:27:36 dansmith I could definitely be wrong
17:27:44 melwitt maybe we never logged notifications. guh, I don't know
17:27:44 sean-k-mooney melwitt: we have some failutre logs here https://github.com/openstack/nova/blob/master/nova/notifications/base.py#L134-L139
17:27:54 dansmith I didn't really think we did, tbh
17:28:06 melwitt sean-k-mooney: yeah, I know we have failure logs
17:40:57 mriedem melwitt: we do'nt log notificatoins
17:41:29 mriedem like dan said, that's not a log level
17:41:36 mriedem each event has a level associated with it
17:42:02 mriedem why? what customer is saying they had notifications logged?
17:42:12 mriedem note - you can configure oslo.messaging to just log notifications
17:42:17 mriedem that's not a nova thing, that's an oslo thing
17:42:25 melwitt thanks. fyi, the default_level option is no longer used anywhere so that might be a bug? but unrelated to what I was looking for
17:43:01 mriedem https://docs.openstack.org/oslo.messaging/latest/configuration/opts.html#oslo_messaging_notifications.driver
17:43:09 mriedem [oslo_messaging_notifications]/driver=log

Earlier   Later