| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-27 | |||
| 19:41:12 | dansmith | "from the controller that lost the aggregates" doesn't mean anything to me | |
| 19:41:19 | dansmith | do you mean from a compute node? | |
| 19:41:41 | greatgatsby | oops, sorry, yes, the compute | |
| 19:42:01 | dansmith | there is some aggregate-manipulation that happens in the provider tree stuff now that I'm not familiar enough to speak about without some study | |
| 19:42:08 | dansmith | so must be something related to that I guess | |
| 19:42:27 | dansmith | best option would be to chat with gibi in his morning (central EU) | |
| 19:43:40 | dansmith | is the uuid in the PUT /resource_providers/uuid/aggregate the uuid of that same compute node? | |
| 19:43:50 | greatgatsby | yes, exactly | |
| 19:44:19 | greatgatsby | it's the only PUT to the resources providers and only to that specific node | |
| 19:45:28 | dansmith | I really didn't think that any of the computes would be adjusting their own aggregate membership, so that seems weird to me | |
| 19:45:40 | dansmith | is there something in the compute log that matches that timestamp? | |
| 19:47:51 | greatgatsby | there probably is, but I haven't found it. I've been grepping through logs all day. I just check the nova_api aggregates table in the database, the creation time is still months old, so they didn't get recreated. Updated is NULL though. | |
| #openstack-nova - 2023-09-28 | |||
| 07:45:32 | gibi | greatgatsby, dansmith: based on my serach the only code that writes aggregates to placement is the report client's set_aggregates_for_provider() https://github.com/openstack/nova/blob/87d4807848bb9546c1fca972da2eb2eda13eb08d/nova/scheduler/client/report.py#L1122 | |
| 07:46:02 | gibi | we call that during nova aggregate manipulation but you said that is not happening | |
| 07:46:22 | gibi | so I ignore those places now | |
| 07:47:03 | gibi | we call that from update_from_provider_tree() as well that is the codepath dansmith refers to above | |
| 07:49:25 | gibi | that will basically flush the nova-compute's internal caches (the provider tree) back to placement | |
| 07:50:12 | gibi | so we need to track down what updates aggregates in that cache | |
| 07:52:29 | sahid | o/ | |
| 07:53:04 | sahid | any issue with our ci, it looks like zull did not pass completely for https://review.opendev.org/c/openstack/nova/+/896512 | |
| 07:53:07 | sahid | ? | |
| 08:29:26 | gibi | greatgatsby, dansmith: there is _refresh_associations in https://github.com/openstack/nova/blob/87d4807848bb9546c1fca972da2eb2eda13eb08d/nova/scheduler/client/report.py#L817-L818 that updates the placement aggregates in the caches | |
| 08:29:57 | gibi | greatgatsby: there is a DEBUG log associated with it | |
| 08:30:05 | gibi | "Refreshing aggregate associations for resource provider" | |
| 08:30:25 | gibi | do you have nova-compute debug logs from that time period? do you see this logged? | |
| 08:38:27 | sahid | let's try a recheck... | |
| 09:47:02 | zigo | Thanks guys for all the work. | |
| 09:47:18 | zigo | stephenfin: Especially you. Now I have to finish the work of backporting all the SQLA 2.x patches ... :P | |
| 12:08:15 | sahid | all is green | |
| 16:50:04 | greatgatsby | gibi: thanks so much for the info! I'm in and out of the office today, so I'll try to take a look. | |
| 17:19:59 | opendevreview | Merged openstack/placement master: Changed /tmp/migrate-db.rc to /root/migrate-db.rc https://review.opendev.org/c/openstack/placement/+/882436 | |
| #openstack-nova - 2023-09-29 | |||
| 08:42:28 | gibi | greatgatsby: what you are seeing feels like a bug somewhere in nova but at the moment I have no idea where so without some reproduction I cannot fix it | |
| 09:19:32 | bauzas | gibi: I just looked at yesterday's convo, can you summarize it ? (sorry yesterday was off) | |
| 09:31:01 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add missing Xena version in release notes https://review.opendev.org/c/openstack/osc-placement/+/896920 | |
| 09:47:49 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Fix overline length https://review.opendev.org/c/openstack/osc-placement/+/896922 | |
| 09:48:41 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add missing Xena version in release notes https://review.opendev.org/c/openstack/osc-placement/+/896920 | |
| 10:08:21 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add xena to toctree https://review.opendev.org/c/openstack/osc-placement/+/896923 | |
| 10:09:29 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add missing Xena version in release notes https://review.opendev.org/c/openstack/osc-placement/+/896924 | |
| 11:19:56 | opendevreview | Merged openstack/osc-placement master: Update master for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808458 | |
| 11:19:57 | opendevreview | Merged openstack/osc-placement stable/zed: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869753 | |
| 11:21:27 | opendevreview | Merged openstack/osc-placement master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/osc-placement/+/894069 | |
| 11:51:30 | opendevreview | Merged openstack/osc-placement stable/yoga: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869754 | |
| 12:10:22 | opendevreview | Merged openstack/osc-placement stable/xena: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869768 | |
| 12:35:27 | greatgatsby | gibi: really appreciate the help. I grepped through the logs and do not see the "Refreshing aggregate..." string anywhere in there for the full day on which the sync drift occurred | |
| 12:42:14 | greatgatsby | gibi: sorry, ignore that last message, I need to look into something here. Again, I really appreciate the help. We're monitoring this very closely, so hopefully I can keep provide more info if/when it occurs again. | |
| 14:04:11 | opendevreview | Merged openstack/nova stable/2023.1: Add debug logging when Instance raises OrphanedObjectError https://review.opendev.org/c/openstack/nova/+/888160 | |
| 15:53:56 | auniyal | Hi dansmith, can you please review this https://review.opendev.org/c/openstack/nova/+/895162 sometime, this is related to device tagging bug. | |
| 15:54:37 | auniyal | I was trying to add a reproducer specifically here- https://review.opendev.org/c/openstack/nova/+/895162/6/nova/tests/functional/libvirt/test_pci_sriov_servers.py#4252 | |
| 15:54:44 | auniyal | but the test is passing for me | |
| 15:55:13 | auniyal | I must be missing something, | |
| #openstack-nova - 2023-09-30 | |||
| 08:53:38 | gibi | greatgatsby: do you have DEBUG log enabled for nova-compute? | |
| 08:55:03 | gibi | greatgatsby: as of summary. I still trying to figure out where nova-compute updated an internal caches about the placement aggregates (as that is later flushed back to placemnet). The DEBUG log above would be can place where nova-compute updates the internal cache. But honeslty these are just stabs in the blind | |
| 09:06:43 | gibi | s/can/one/ | |
| 12:16:02 | greatgatsby | gibi: unfortunately it seems we didn't have debug logging enabled at the time the agg unsync occurred. I've enabled it and I'll search for the "Refreshing... " message if/when this happens again. Again, thanks for the help. | |
| #openstack-nova - 2023-10-01 | |||
| 14:41:48 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2024.1 Caracal https://review.opendev.org/c/openstack/nova/+/896305 | |
| 14:42:01 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2024.1 Caracal https://review.opendev.org/c/openstack/nova/+/896305 | |
| 18:10:27 | opendevreview | sean mooney proposed openstack/nova master: remove py2 only syntax https://review.opendev.org/c/openstack/nova/+/896984 | |
| 18:10:28 | opendevreview | sean mooney proposed openstack/nova master: remove use of % format strings https://review.opendev.org/c/openstack/nova/+/896985 | |
| 18:10:28 | opendevreview | sean mooney proposed openstack/nova master: pyupgrade py 3.0 -> 3.6 https://review.opendev.org/c/openstack/nova/+/896986 | |
| 18:10:29 | opendevreview | sean mooney proposed openstack/nova master: add precommit pyupgrade hook and tox env https://review.opendev.org/c/openstack/nova/+/896987 | |
| 18:10:29 | opendevreview | sean mooney proposed openstack/nova master: ignore pyupgrade series in git blame https://review.opendev.org/c/openstack/nova/+/896988 | |
| #openstack-nova - 2023-10-02 | |||
| 06:58:22 | bauzas | good morning Nova | |
| 07:26:15 | gibi | o/ | |
| 07:45:45 | dvo-plv | Hello, I would like to remind about to take a look on the reproposed spec file ? https://review.opendev.org/c/openstack/nova-specs/+/895924 | |
| 07:45:46 | dvo-plv | and check dependent patch for it https://review.opendev.org/c/openstack/nova/+/876075 | |
| 07:53:35 | bauzas | yup, specs reviews is on my agenda this week | |
| 09:08:17 | opendevreview | sean mooney proposed openstack/nova master: remove py2 only syntax https://review.opendev.org/c/openstack/nova/+/896984 | |
| 09:08:18 | opendevreview | sean mooney proposed openstack/nova master: remove use of % format strings https://review.opendev.org/c/openstack/nova/+/896985 | |
| 09:08:18 | opendevreview | sean mooney proposed openstack/nova master: pyupgrade py 3.0 -> 3.6 https://review.opendev.org/c/openstack/nova/+/896986 | |
| 09:08:19 | opendevreview | sean mooney proposed openstack/nova master: add precommit pyupgrade hook and tox env https://review.opendev.org/c/openstack/nova/+/896987 | |
| 09:08:19 | opendevreview | sean mooney proposed openstack/nova master: ignore pyupgrade series in git blame https://review.opendev.org/c/openstack/nova/+/896988 | |
| 09:09:45 | opendevreview | sean mooney proposed openstack/nova master: ignore pyupgrade series in git blame https://review.opendev.org/c/openstack/nova/+/896988 | |
| 09:10:22 | opendevreview | sean mooney proposed openstack/nova master: ignore pyupgrade series in git blame https://review.opendev.org/c/openstack/nova/+/896988 | |
| 09:26:37 | dvo-plv | bauzas, good to hear,thank you | |
| 10:20:34 | opendevreview | Stephen Finucane proposed openstack/nova master: doc: Remove crud from conf.py file https://review.opendev.org/c/openstack/nova/+/896224 | |
| 10:20:35 | opendevreview | Stephen Finucane proposed openstack/nova master: pre-commit: Update plugin versions https://review.opendev.org/c/openstack/nova/+/896226 | |
| 10:20:35 | opendevreview | Stephen Finucane proposed openstack/nova master: tox: Use pre-commit for pep8 target https://review.opendev.org/c/openstack/nova/+/896227 | |
| 10:20:36 | opendevreview | Stephen Finucane proposed openstack/nova master: pre-commit: Add mypy https://review.opendev.org/c/openstack/nova/+/896228 | |
| 10:20:36 | opendevreview | Stephen Finucane proposed openstack/nova master: Blacken codebase https://review.opendev.org/c/openstack/nova/+/896229 | |
| 10:20:37 | opendevreview | Stephen Finucane proposed openstack/nova master: pre-commit: Integrate black https://review.opendev.org/c/openstack/nova/+/896230 | |
| 10:20:37 | opendevreview | Stephen Finucane proposed openstack/nova master: Ignore black change https://review.opendev.org/c/openstack/nova/+/896231 | |
| 10:20:38 | opendevreview | Stephen Finucane proposed openstack/nova master: pre-commit: Use native hacking pre-commit hook https://review.opendev.org/c/openstack/nova/+/897005 | |
| 12:38:34 | opendevreview | sean mooney proposed openstack/nova master: remove deprecated vmware virt driver https://review.opendev.org/c/openstack/nova/+/897017 | |
| 12:55:30 | opendevreview | Merged openstack/os-vif stable/2023.2: Update TOX_CONSTRAINTS_FILE for stable/2023.2 https://review.opendev.org/c/openstack/os-vif/+/894062 | |
| 14:56:47 | stephenfin | bauzas: I'm not sure I understand your comment on https://review.opendev.org/c/openstack/nova/+/896226/? Does my reply make sense? | |
| 14:57:41 | bauzas | stephenfin: tbc, I'm not a big fan of pre-commit hooks | |
| 14:57:59 | bauzas | they slow our development process and avoid new contributors to easily push some code asking for help | |
| 14:58:15 | stephenfin | but you don't need to use them | |
| 14:58:17 | bauzas | in general, I do atomic commits and then I squash | |
| 14:58:39 | bauzas | the more pre-commit hooks we add, the slower a git commit happens | |
| 14:58:45 | stephenfin | only if you enable them | |
| 14:58:54 | stephenfin | which I'm guessing you don't? | |
| 14:58:56 | bauzas | sure, but you're adding them by default | |
| 14:59:13 | bauzas | which relates to my other concern | |
| 14:59:14 | stephenfin | I am? | |
| 14:59:23 | bauzas | and yeah, I know I can disable hooks | |
| 14:59:37 | stephenfin | hooks aren't enabled by default. It's a manual step to install them | |
| 14:59:45 | bauzas | that's why I'm proposing an half-way solution, which is to use pre-push hooks | |
| 15:00:07 | bauzas | it's an automatic step on my env | |
| 15:00:28 | stephenfin | before we talk about that, back up: where is this being added by default? | |