| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-28 | |||
| 15:50:18 | sean-k-mooney | Tahvok: one think you could try is deploying a seperate rabitmq instance for nova. that would help you isolate the issue. that said i know you may not want to do that on a running cloud | |
| 15:50:30 | mriedem | Tahvok: what versions of oslo.db and oslo.messaging are you using? | |
| 15:50:44 | mriedem | "AttributeError: '_TransactionContextManager' object has no attribute 'async_'"" suggests you're using an old oslo.db | |
| 15:50:48 | sean-k-mooney | mriedem: Tahvok mentioned it was an ocata cloud this morning | |
| 15:51:05 | mriedem | that doesn't tell me what i'd need to konw | |
| 15:51:12 | mriedem | you can be using min or max versions of oslo.db from ocata | |
| 15:51:13 | Tahvok | mriedem: sec, looking | |
| 15:51:15 | mriedem | or something completely different | |
| 15:51:36 | stephenfin | mriedem: diliprenkila had the same issue, unless you're mixing them up | |
| 15:51:37 | dansmith | mriedem: are you confusing Tahvok and diliprenkila ? | |
| 15:51:45 | stephenfin | dansmith: Yeah :) | |
| 15:52:07 | mriedem | yeah sorry | |
| 15:52:20 | stephenfin | mriedem: I think we've an issue there though. The patch I linked was released in 4.40.0, but we pin on a lower version https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7 | |
| 15:52:36 | stephenfin | I assume we should be handling the older version of bumping our minimum | |
| 15:52:56 | Tahvok | All versions of nova-conductor container: http://paste.openstack.org/show/728972/ | |
| 15:52:57 | stephenfin | ...and I also assume we don't see this in tests because we don't run a lower-constraints functional test | |
| 15:53:24 | sean-k-mooney | stephenfin: well is it on master or stable. we cant bump stable miniums | |
| 15:53:56 | stephenfin | sean-k-mooney: Hmm, lemme check | |
| 15:54:31 | stephenfin | sean-k-mooney: Master (git branch --contains 2d532963fa2e013e16cc403f2674a4488c4170ab) | |
| 15:55:18 | Tahvok | sean-k-mooney: I think deploying a separate cloud would not replicate the problem (I simply don't have 160 compute hosts sitting around) | |
| 15:55:19 | sean-k-mooney | stephenfin: i ment the issue you thing bumping the min verion would fix | |
| 15:55:35 | stephenfin | diliprenkila: Looks like that's a bug. Want to open one with that log and I'll fix that quickly? | |
| 15:55:47 | stephenfin | sean-k-mooney: Yeah, that's what I'm referring to | |
| 15:56:12 | sean-k-mooney | Tahvok: i was not suggesting deploying a seperate cloud. just 1 more rabbitmq node and pointing your existing nodes to use it. | |
| 15:56:29 | mriedem | stephenfin: sorry, is diliprenkila on master? | |
| 15:56:32 | diliprenkila | <mriedem> diliprenkila oslo.db==4.40.0 ,oslo.messaging==8.1.0 on the compute nodes and oslo.db==4.38.0, oslo.messaging==6.4.1 on the nova controller node | |
| 15:57:18 | sean-k-mooney | diliprenkila: so you are running an older controler then compute node? | |
| 15:57:20 | openstackgerrit | Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170 | |
| 15:57:54 | diliprenkila | ||
| 15:57:55 | openstack | Launchpad bug 1788833 in OpenStack Compute (nova) "Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_" [Undecided,New] | |
| 15:57:59 | Tahvok | sean-k-mooney: yeah, doing that on production env is a bit risky, at least we would need lots of time to prepare such change.. I was hoping there is some better way to check what conductor was replying to | |
| 15:58:14 | jaypipes | dansmith, kosamara: pls review https://review.openstack.org/597170 | |
| 15:58:29 | stephenfin | sean-k-mooney: This is the issue https://github.com/openstack/nova/commit/2d532963fa2e013e16cc403f2674a4488c4170ab#diff-8fec546e4c39f78d233f8e21dadaa3ff | |
| 15:58:51 | sean-k-mooney | Tahvok: am you could maybe dump the contence of the message queues from rabbitmq but other then that im not sure | |
| 15:59:01 | stephenfin | sean-k-mooney: We want that change but only when oslo.db>=4.40.0 | |
| 15:59:06 | diliprenkila | <sean-k-mooney> Yes i am using old ones, but i did installed all nova packages from ubuntu cloud archive rocky | |
| 15:59:28 | dansmith | Tahvok: you probably need to sniff the bus or something like that | |
| 15:59:46 | dansmith | Tahvok: or talk to oslo.messaging people about how that "unknown" thing can even happen | |
| 15:59:53 | Tahvok | I was thinking maybe simply stopping all conductor services, and then looking which queue was accumulating the most? | |
| 15:59:56 | stephenfin | diliprenkila: You're using master nova on the broken node though, I imagine | |
| 16:00:08 | stephenfin | Because the code that's broken isn't on stable/rocky, from what I can see | |
| 16:00:17 | diliprenkila | <stephenfin> Yes i am using master | |
| 16:00:18 | sean-k-mooney | stephenfin: chaning from async to async_ | |
| 16:00:55 | efried | kosamara: Are you still around? | |
| 16:01:10 | dansmith | Tahvok: the only time conductor initiates a message (not a reply) is during a build or resize type operation, that I can think of | |
| 16:01:15 | sean-k-mooney | stephenfin: it looks like that is in RC3. we cant bump miniums for rocky rc3 at this point | |
| 16:01:29 | stephenfin | sean-k-mooney: sec. We don't need to | |
| 16:01:34 | dansmith | Tahvok: sniffing the bus, you may find that one node is going crazy, spewing messages or something | |
| 16:02:01 | stephenfin | sean-k-mooney: But I don't think it is. 'git branch --contains 2d532963fa2e013e16cc403f2674a4488c4170ab' doesn't show it | |
| 16:02:27 | Tahvok | dansmith: we actually have some compute nodes with errors like: [instance: 631c2697-1cdf-4d97-8ae9-006cc5ed6e35] Instance not resizing, skipping migration. | |
| 16:02:43 | Tahvok | But this happen every sync operation (once a minute) | |
| 16:02:55 | Tahvok | Which is a separate issue we are trying to investigate.. | |
| 16:03:00 | dansmith | Tahvok: I mean during the active part of a migration, in response to an api call | |
| 16:03:25 | Tahvok | dansmith: then it's not the case.. As nothing is migrating right now | |
| 16:03:33 | dansmith | Tahvok: right | |
| 16:03:35 | sean-k-mooney | stephenfin: if it was backported the commit id would have changed but the chaing id would still be in the git log. ill check | |
| 16:04:41 | sean-k-mooney | stephenfin: ya its in rc3 | |
| 16:05:11 | sean-k-mooney | stephenfin: actully one second | |
| 16:05:18 | openstack | Launchpad bug 1788833 in OpenStack Compute (nova) "Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_" [Undecided,New] | |
| 16:05:18 | mriedem | stephenfin: did i summarize this correctly? https://bugs.launchpad.net/nova/+bug/1788833/comments/1 | |
| 16:05:47 | stephenfin | mriedem: Yup | |
| 16:06:07 | Tahvok | Just realized I didn't paste the messages I'm actually seeing that conductor is handling: http://paste.openstack.org/show/nqYc4ghBYpLAFvinYYAp/ | |
| 16:06:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't use '_TransactionContextManager._async' https://review.openstack.org/597173 | |
| 16:06:48 | sean-k-mooney | stephenfin: ya its in rc3 | |
| 16:06:50 | stephenfin | mriedem, sean-k-mooney: And there's the fix (tl;dr: this can has some kicking left in it) | |
| 16:07:32 | Tahvok | dansmith: all of them are actually replies as you can see | |
| 16:07:34 | stephenfin | sean-k-mooney: How'd you figure that out? | |
| 16:08:03 | dansmith | Tahvok: that doesn't really tell us anything | |
| 16:08:06 | sean-k-mooney | git fetch --tags && git checkout 18.0.0.0rc3 | |
| 16:08:17 | sean-k-mooney | then git log and search for change id | |
| 16:08:45 | Tahvok | dansmith: doesn't a reply message indicate it's a sync from compute nodes? | |
| 16:08:56 | stephenfin | sean-k-mooney: Ah, 'git branch -a --contains 2d532963fa2e013e16cc403f2674a4488c4170ab' | |
| 16:09:09 | stephenfin | The '-a' is important. I don't have stable/rocky locally yet | |
| 16:09:12 | sean-k-mooney | stephenfin: also if you look at https://github.com/openstack/nova/commit/2d532963fa2e013e16cc403f2674a4488c4170ab#diff-8fec546e4c39f78d233f8e21dadaa3ff it shows what branches have it | |
| 16:09:43 | dansmith | Tahvok: I don't think those lines are telling you that they're replies, they're telling you what queue will be used for the reply | |
| 16:11:01 | sean-k-mooney | stephenfin: mriedem in anycase https://review.openstack.org/#/c/597173/1 will need to get applied to stable/rocky | |
| 16:11:25 | mriedem | yes i know | |
| 16:11:27 | mriedem | i left a comment in there | |
| 16:11:45 | mriedem | we can't do an rc4 so i guess this is just going to be a known broken issue for anyone not using oslo.db 4.40 which is at least in upper-constraints for stable/rocky | |
| 16:12:03 | mriedem | and i'd think/hope most deployments should be using what's in upper-constraints for dependent libraries since those are the versions we test against | |
| 16:13:52 | melwitt | . | |
| 16:14:20 | sean-k-mooney | mriedem: yeah thats unfortunate but at least as you said upper constratins allows 4.4 | |
| 16:14:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't use '_TransactionContextManager._async' https://review.openstack.org/597173 | |
| 16:14:44 | sean-k-mooney | *4.40 | |
| 16:16:04 | efried | stephenfin: Getting late for you, you want me to propose the fup? | |
| 16:17:00 | melwitt | dansmith: cool, will look | |
| 16:18:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Revert "Don't use '_TransactionContextManager._async'" https://review.openstack.org/597174 | |
| 16:18:07 | stephenfin | efried: Cheers but it was pretty easy :) | |
| 16:18:48 | efried | stephenfin: Cool. Can you put Related-Bug in the commit? | |
| 16:19:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Revert "Don't use '_TransactionContextManager._async'" https://review.openstack.org/597174 | |
| 16:19:16 | stephenfin | efried: done | |
| 16:19:18 | efried | Thanks. +2 | |
| 16:32:05 | openstackgerrit | Merged openstack/nova master: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/540258 | |
| 16:39:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Further simplification of test_numa_servers https://review.openstack.org/596832 | |
| 16:39:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Validate huge pages https://review.openstack.org/399653 | |
| 16:46:24 | openstackgerrit | Merged openstack/nova master: Record cell success/failure/timeout in CrossCellLister https://review.openstack.org/594265 | |
| 17:09:38 | sean-k-mooney | oh by the way i finally got around to rebasing https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1759420 if people have time to review | |
| 17:17:06 | Tahvok | Is it safe to reduce the number of conductor workers | |
| 17:17:13 | Tahvok | ? | |
| 17:17:37 | Tahvok | I'm only afraid if I would lose some data.. | |