| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-22 | |||
| 16:43:36 | efried | ack, thx | |
| 16:43:51 | mriedem | it's 11:43 and i feel like i haven't really settled into being productive today | |
| 16:44:06 | efried | it's like being PTL all over again, eh? | |
| 16:44:19 | efried | Maybe doing PTL stuff felt productive to you | |
| 16:44:38 | mriedem | i wasn't doing "ptl stuff" most days | |
| 16:45:04 | sean-k-mooney | do we still need n-cauth? just noticed its enabled in some of the tempest jobs | |
| 16:45:33 | mriedem | sean-k-mooney: we should'nt no | |
| 16:45:46 | mriedem | hell i thought stephenfin had patches to remove it in train | |
| 16:45:58 | sean-k-mooney | ok ya he did | |
| 16:46:00 | mnaser | im pretty sure i saw it removed | |
| 16:46:21 | sean-k-mooney | oh never mind i was looking at a stable run | |
| 16:46:29 | mriedem | zoinks | |
| 16:48:30 | sean-k-mooney | i was looking at a stable/rocky log master is correct | |
| 16:53:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: docs: pretty up return code table or heal_allocations https://review.opendev.org/678063 | |
| 16:55:26 | openstackgerrit | Eric Fried proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899 | |
| 16:55:26 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for node.list https://review.opendev.org/656027 | |
| 16:55:27 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for validating instance and node https://review.opendev.org/656028 | |
| 16:55:27 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690 | |
| 16:55:28 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for add/remove instance info from node https://review.opendev.org/659691 | |
| 16:55:28 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for getting network metadata from node https://review.opendev.org/670213 | |
| 16:57:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: pretty up return code table for sync_aggregates https://review.opendev.org/678064 | |
| 17:00:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | |
| 17:01:22 | mriedem | lyarwood: if you're back, can you hit these stein backports when you get a chance? https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/stein+topic:bug/1839560 - regressoin since rocky for baremetal nodes so would be good to get those in | |
| 17:02:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | |
| 17:09:08 | stephenfin | dansmith, mriedem: When a compute node record is deleted, can it ever come back? | |
| 17:09:16 | dansmith | wellllll | |
| 17:09:22 | dansmith | about that... | |
| 17:10:20 | dansmith | stephenfin: last week? no.. this week? yes | |
| 17:10:21 | dansmith | https://review.opendev.org/#/c/675496/ | |
| 17:11:30 | stephenfin | dansmith: We should probably remove the deleted check from this online migration, in that case? https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L504 | |
| 17:11:36 | melwitt | does anyone know how/where notifications are logged? | |
| 17:11:52 | dansmith | stephenfin: no | |
| 17:12:22 | dansmith | stephenfin: you don't want to count deleted nodes for anything until/unless they're undeleted | |
| 17:12:42 | dansmith | and we'll do all the migratory behavior when we resurrect and then subsequently save it | |
| 17:12:47 | sean-k-mooney | melwitt: i guess its proably depned on what is sending them. i would expect to see some notifiaciotn logs in n-cpu | |
| 17:12:51 | stephenfin | doesn't that mean we need to keep that online migration around forever then? | |
| 17:13:17 | sean-k-mooney | e.g. if the compute service is emmiting vm lifecycle notifications | |
| 17:13:47 | dansmith | stephenfin: no, because we do the migration activity upon resurrection | |
| 17:13:49 | melwitt | sean-k-mooney: yeah, I don't know where or how they are logged. there's a CONF.notifications.default_level for the log level but I don't see it used anywhere in nova code | |
| 17:13:50 | melwitt | and I can't find what logs the notifications or what format they're in | |
| 17:14:02 | dansmith | stephenfin: if we ever want to remove *that* behavior we'll need a blocker type migration | |
| 17:14:12 | stephenfin | gotcha | |
| 17:14:49 | sean-k-mooney | melwitt: its unlikely but the only other place that would make sense would be the conductor | |
| 17:15:11 | sean-k-mooney | but it feel inefficnet to need to send them form the conductor so i doubt we would do that | |
| 17:16:35 | sean-k-mooney | melwitt: are you trying to find the ones used by nova join | |
| 17:16:46 | melwitt | I'm trying to find the ones anywhere in nova | |
| 17:16:56 | melwitt | I don't see how or where they are logged | |
| 17:20:15 | melwitt | looks like the only use of CONF.notifications.default_level was removed here https://github.com/openstack/nova/commit/9f48aee9b0ea68f7c8eba6a1f3d076e4194d804d#diff-2f33aae68e09a2fe816870ac389cfe95 | |
| 17:20:38 | melwitt | I'm guessing that decorator somehow logged things? | |
| 17:20:42 | melwitt | I dunno | |
| 17:21:57 | sean-k-mooney | melwitt: maybe this https://github.com/openstack/nova/blob/master/nova/rpc.py#L420 | |
| 17:23:03 | melwitt | mriedem: do you know if we intentionally stopped logging notifications in rocky? commit linked above ^ | |
| 17:23:19 | melwitt | sean-k-mooney: that looks like some kind of error condition. not general log of any notification | |
| 17:23:36 | dansmith | did we ever log them? | |
| 17:23:52 | dansmith | they'd be spread all over the place if we did | |
| 17:24:12 | melwitt | based on the existence of the CONF.notifications.default_level option we did | |
| 17:24:13 | melwitt | yeah | |
| 17:24:26 | melwitt | but the option is no longer used anywhere after that commit | |
| 17:24:42 | melwitt | so I wonder if that decorator used to result in logging somehow. I don't know how it worked | |
| 17:25:25 | dansmith | isn't that level something about which notifcations we emit, not the log level? | |
| 17:26:13 | dansmith | "Default notification level for outgoing notifications." | |
| 17:26:18 | melwitt | I dunno, sounds like a log level https://github.com/openstack/nova/blob/master/nova/conf/notifications.py#L51-L57 | |
| 17:27:05 | melwitt | but yeah, I guess it could have meant that notifications used to be categorized and you could prune down which to emit? | |
| 17:27:23 | dansmith | right, like only emit started/stopped type things and not progress-related ones | |
| 17:27:32 | dansmith | or "I checked on a thing" ones | |
| 17:27:36 | dansmith | I could definitely be wrong | |
| 17:27:44 | sean-k-mooney | melwitt: we have some failutre logs here https://github.com/openstack/nova/blob/master/nova/notifications/base.py#L134-L139 | |
| 17:27:44 | melwitt | maybe we never logged notifications. guh, I don't know | |
| 17:27:54 | dansmith | I didn't really think we did, tbh | |
| 17:28:06 | melwitt | sean-k-mooney: yeah, I know we have failure logs | |
| 17:40:57 | mriedem | melwitt: we do'nt log notificatoins | |
| 17:41:29 | mriedem | like dan said, that's not a log level | |
| 17:41:36 | mriedem | each event has a level associated with it | |
| 17:42:02 | mriedem | why? what customer is saying they had notifications logged? | |
| 17:42:12 | mriedem | note - you can configure oslo.messaging to just log notifications | |
| 17:42:17 | mriedem | that's not a nova thing, that's an oslo thing | |
| 17:42:25 | melwitt | thanks. fyi, the default_level option is no longer used anywhere so that might be a bug? but unrelated to what I was looking for | |
| 17:43:01 | mriedem | https://docs.openstack.org/oslo.messaging/latest/configuration/opts.html#oslo_messaging_notifications.driver | |
| 17:43:09 | mriedem | [oslo_messaging_notifications]/driver=log | |
| 17:43:15 | melwitt | mriedem: no, we're trying to debug something and trying to verify that nova is sending notifications. and were looking for a clue about how to tell if it is | |
| 17:43:17 | melwitt | thanks | |
| 17:44:15 | mriedem | btw, gibi's last summit talk about versioned notifications (which i think you're in the video) has some nice stuff about configuring and testing notificaoins out to see them | |
| 17:44:42 | mriedem | https://www.openstack.org/videos/summits/denver-2019/nova-versioned-notifications-the-result-of-a-3-year-journey | |
| 17:45:11 | mriedem | oh let me guess - something something heat isn't getting notifications | |
| 17:45:43 | mriedem | maybe the customer flipped on versioned notifications only and heat only listens for unversioned | |
| 17:45:48 | melwitt | novajoin, the vendordata service | |
| 17:45:49 | melwitt | so yeah, pretty much | |
| 17:46:15 | melwitt | yeah, we checked and both are set to use unversioned so it's not something obvious | |
| 17:47:48 | melwitt | haha, indeed I attended that talk. I'll look at that again | |
| 17:51:04 | melwitt | looks like something is afoot with the driver setting, it's set to 'noop' which seems bad :( | |
| 17:52:50 | melwitt | good thing you pointed that out, I had forgotten about it | |
| 17:52:57 | melwitt | [oslo_messaging_notifications]/driver | |
| 17:59:23 | efried | sean-k-mooney: were you one who was vocal about what "privsep stage 3" should look like? | |
| 18:03:37 | efried | nm, I see you're already on the cyborg privsep review, which is where this was going. | |
| 18:16:25 | melwitt | looks like there was a recent change to tripleo to disable notification drivers when telemetry is disabled, so this deploy got caught up in that | |
| 18:17:29 | melwitt | thanks again for the hint about the [oslo_messaging_notifications]/driver setting. mriedem++ | |
| 18:23:15 | mriedem | worked? | |
| 18:24:05 | melwitt | they haven't tried yet with driver=messagingv2 but I'm sure this is the issue | |
| 18:24:59 | mriedem | ah yeah | |
| 18:25:59 | melwitt | there was an assumption that only telemetry needed the notifications and since it's disabled, it set driver=noop to avoid needless notification traffic, | |