Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-22
17:13:17 sean-k-mooney e.g. if the compute service is emmiting vm lifecycle notifications
17:13:47 dansmith stephenfin: no, because we do the migration activity upon resurrection
17:13:49 melwitt sean-k-mooney: yeah, I don't know where or how they are logged. there's a CONF.notifications.default_level for the log level but I don't see it used anywhere in nova code
17:13:50 melwitt and I can't find what logs the notifications or what format they're in
17:14:02 dansmith stephenfin: if we ever want to remove *that* behavior we'll need a blocker type migration
17:14:12 stephenfin gotcha
17:14:49 sean-k-mooney melwitt: its unlikely but the only other place that would make sense would be the conductor
17:15:11 sean-k-mooney but it feel inefficnet to need to send them form the conductor so i doubt we would do that
17:16:35 sean-k-mooney melwitt: are you trying to find the ones used by nova join
17:16:46 melwitt I'm trying to find the ones anywhere in nova
17:16:56 melwitt I don't see how or where they are logged
17:20:15 melwitt looks like the only use of CONF.notifications.default_level was removed here https://github.com/openstack/nova/commit/9f48aee9b0ea68f7c8eba6a1f3d076e4194d804d#diff-2f33aae68e09a2fe816870ac389cfe95
17:20:38 melwitt I'm guessing that decorator somehow logged things?
17:20:42 melwitt I dunno
17:21:57 sean-k-mooney melwitt: maybe this https://github.com/openstack/nova/blob/master/nova/rpc.py#L420
17:23:03 melwitt mriedem: do you know if we intentionally stopped logging notifications in rocky? commit linked above ^
17:23:19 melwitt sean-k-mooney: that looks like some kind of error condition. not general log of any notification
17:23:36 dansmith did we ever log them?
17:23:52 dansmith they'd be spread all over the place if we did
17:24:12 melwitt based on the existence of the CONF.notifications.default_level option we did
17:24:13 melwitt yeah
17:24:26 melwitt but the option is no longer used anywhere after that commit
17:24:42 melwitt so I wonder if that decorator used to result in logging somehow. I don't know how it worked
17:25:25 dansmith isn't that level something about which notifcations we emit, not the log level?
17:26:13 dansmith "Default notification level for outgoing notifications."
17:26:18 melwitt I dunno, sounds like a log level https://github.com/openstack/nova/blob/master/nova/conf/notifications.py#L51-L57
17:27:05 melwitt but yeah, I guess it could have meant that notifications used to be categorized and you could prune down which to emit?
17:27:23 dansmith right, like only emit started/stopped type things and not progress-related ones
17:27:32 dansmith or "I checked on a thing" ones
17:27:36 dansmith I could definitely be wrong
17:27:44 melwitt maybe we never logged notifications. guh, I don't know
17:27:44 sean-k-mooney melwitt: we have some failutre logs here https://github.com/openstack/nova/blob/master/nova/notifications/base.py#L134-L139
17:27:54 dansmith I didn't really think we did, tbh
17:28:06 melwitt sean-k-mooney: yeah, I know we have failure logs
17:40:57 mriedem melwitt: we do'nt log notificatoins
17:41:29 mriedem like dan said, that's not a log level
17:41:36 mriedem each event has a level associated with it
17:42:02 mriedem why? what customer is saying they had notifications logged?
17:42:12 mriedem note - you can configure oslo.messaging to just log notifications
17:42:17 mriedem that's not a nova thing, that's an oslo thing
17:42:25 melwitt thanks. fyi, the default_level option is no longer used anywhere so that might be a bug? but unrelated to what I was looking for
17:43:01 mriedem https://docs.openstack.org/oslo.messaging/latest/configuration/opts.html#oslo_messaging_notifications.driver
17:43:09 mriedem [oslo_messaging_notifications]/driver=log
17:43:15 melwitt mriedem: no, we're trying to debug something and trying to verify that nova is sending notifications. and were looking for a clue about how to tell if it is
17:43:17 melwitt thanks
17:44:15 mriedem btw, gibi's last summit talk about versioned notifications (which i think you're in the video) has some nice stuff about configuring and testing notificaoins out to see them
17:44:42 mriedem https://www.openstack.org/videos/summits/denver-2019/nova-versioned-notifications-the-result-of-a-3-year-journey
17:45:11 mriedem oh let me guess - something something heat isn't getting notifications
17:45:43 mriedem maybe the customer flipped on versioned notifications only and heat only listens for unversioned
17:45:48 melwitt novajoin, the vendordata service
17:45:49 melwitt so yeah, pretty much
17:46:15 melwitt yeah, we checked and both are set to use unversioned so it's not something obvious
17:47:48 melwitt haha, indeed I attended that talk. I'll look at that again
17:51:04 melwitt looks like something is afoot with the driver setting, it's set to 'noop' which seems bad :(
17:52:50 melwitt good thing you pointed that out, I had forgotten about it
17:52:57 melwitt [oslo_messaging_notifications]/driver
17:59:23 efried sean-k-mooney: were you one who was vocal about what "privsep stage 3" should look like?
18:03:37 efried nm, I see you're already on the cyborg privsep review, which is where this was going.
18:16:25 melwitt looks like there was a recent change to tripleo to disable notification drivers when telemetry is disabled, so this deploy got caught up in that
18:17:29 melwitt thanks again for the hint about the [oslo_messaging_notifications]/driver setting. mriedem++
18:23:15 mriedem worked?
18:24:05 melwitt they haven't tried yet with driver=messagingv2 but I'm sure this is the issue
18:24:59 mriedem ah yeah
18:25:59 melwitt but novajoin is in the mix here too and needs the notifications to work
18:25:59 melwitt there was an assumption that only telemetry needed the notifications and since it's disabled, it set driver=noop to avoid needless notification traffic,
18:28:18 mriedem silly tripleo
18:28:30 mriedem i thought heat also consumed nova notifications, or maybe that's indirectly via ceilometer?
18:28:57 mriedem but yeah lots of projects listen to nova notifications besides those, you've also got designate, watcher, blazar, etc - though maybe tripleo doesn't support those
18:29:09 melwitt yeah, this is the undercloud and that confuses me, but maybe that's why
18:29:32 melwitt I think notifications are always on in the overcloud
18:29:59 melwitt in order for other things to work, I assume we have some other service that needs them
18:30:02 mriedem oh yeah, forgot about the cloud turducken
18:30:09 melwitt yeah
18:30:36 mriedem well i suppose it always feels good to come back and say, "this isn't a nova problem, it's a tripleo problem"
18:31:03 melwitt heh, indeed
18:31:13 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.opendev.org/537414
18:31:14 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793
18:31:14 openstackgerrit Stephen Finucane proposed openstack/nova master: Add '[compute] cpu_dedicated_set' option https://review.opendev.org/671792
18:31:15 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395
18:31:15 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693
18:31:16 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103
18:31:16 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800
18:31:17 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338
18:31:17 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571
18:31:18 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729
18:31:18 openstackgerrit Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801
18:31:19 openstackgerrit Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895
18:38:53 openstackgerrit Eric Fried proposed openstack/nova master: Add docstring to check_availability_zone function https://review.opendev.org/677858
18:40:55 mriedem anyone else have an opinion about nova migration-list with --migration-type or just --type in https://review.opendev.org/#/c/675117/ ?
18:59:12 efried mriedem: sans click, I would say --migration-type just because --type is so generic.
19:02:45 artom mriedem, make it --kind-of-migration and watch the comments pour in
19:04:19 mriedem --migration-flavour?
19:05:51 artom Points for the UK spelling :D
19:06:13 artom (aka the *correct* spelling. Yours truly, a Canadian)
19:06:48 artom (except when I feel like being Quebecois or Russian - yay absence of national identity!)
19:14:15 efried mriedem: I left helpful comments on https://review.opendev.org/#/c/678051/
19:20:07 mriedem artom: how would you do it in old russ?
19:20:38 artom Gather 'round a bottle of vodka (or two) and let the solution emerge by itself
19:22:42 mriedem efried: you might need:
19:22:44 mriedem become: yes

Earlier   Later