| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-27 | |||
| 14:53:23 | opendevreview | Sylvain Bauza proposed openstack/nova-specs master: explain why ironic-shards was reverted in the spec https://review.opendev.org/c/openstack/nova-specs/+/896679 | |
| 14:58:05 | opendevreview | Stephen Finucane proposed openstack/placement master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/placement/+/886229 | |
| 14:59:45 | opendevreview | Dan Smith proposed openstack/nova-specs master: Add libvirt-dev-alias spec https://review.opendev.org/c/openstack/nova-specs/+/892806 | |
| 15:01:54 | opendevreview | Stephen Finucane proposed openstack/placement master: db: Wrap raw SQL query in sqlalchemy.text https://review.opendev.org/c/openstack/placement/+/896650 | |
| 15:01:59 | opendevreview | Stephen Finucane proposed openstack/placement master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/placement/+/886229 | |
| 18:51:35 | mnaser | https://bugs.launchpad.net/nova/+bug/2037585 -- does deleting neutron ports in a specific order sound like a reasonable fix here? | |
| 18:51:46 | mnaser | if so i can push that proposal | |
| 19:30:59 | greatgatsby | triggers this to happen. | |
| 19:30:59 | greatgatsby | Hi. I was asking before about our compute aggregates become out of sync with the resource provider aggregates. We had it happen again last night, and I've found a PUT /resource_providers/<uuid>/aggregates in the logs at the time we detected them out of sync. What service or state might trigger this update on its own? I've been grepping through logs most of the day, but still haven't found what | |
| 19:31:12 | greatgatsby | thanks *a lot* for any help | |
| 19:31:52 | JayF | greatgatsby: just curious, are the resources in question Ironic nodes? | |
| 19:33:35 | greatgatsby | thanks for the reply. No ironic, we pxe boot and use ansible for config | |
| 19:33:59 | JayF | ack; sorry I won't be much help then as that's my specialty; good luck! | |
| 19:34:10 | greatgatsby | thanks :-) | |
| 19:34:58 | dansmith | greatgatsby: nova-api will do that in response to creating or deleting aggregates on the nova side | |
| 19:35:06 | dansmith | nothing else should be doing that, AFAIK | |
| 19:35:20 | dansmith | can you trace the origin with a req-id or source ip? | |
| 19:36:32 | greatgatsby | source ip in one case came from the controller that lost the resource provider aggs. The nova aggs have never changed (or at least, we haven't changed them, and they stay consistent) | |
| 19:36:59 | greatgatsby | so, for example, compute 1 will still have the correct linux agg on the nova side, but it disappears on the resource provider side | |
| 19:37:06 | greatgatsby | this happened late last night | |
| 19:38:55 | greatgatsby | in one PUT message I see "nova-compute keystoneauth1/4.5.0 python-requests...." | |
| 19:41:12 | dansmith | "from the controller that lost the aggregates" doesn't mean anything to me | |
| 19:41:19 | dansmith | do you mean from a compute node? | |
| 19:41:41 | greatgatsby | oops, sorry, yes, the compute | |
| 19:42:01 | dansmith | there is some aggregate-manipulation that happens in the provider tree stuff now that I'm not familiar enough to speak about without some study | |
| 19:42:08 | dansmith | so must be something related to that I guess | |
| 19:42:27 | dansmith | best option would be to chat with gibi in his morning (central EU) | |
| 19:43:40 | dansmith | is the uuid in the PUT /resource_providers/uuid/aggregate the uuid of that same compute node? | |
| 19:43:50 | greatgatsby | yes, exactly | |
| 19:44:19 | greatgatsby | it's the only PUT to the resources providers and only to that specific node | |
| 19:45:28 | dansmith | I really didn't think that any of the computes would be adjusting their own aggregate membership, so that seems weird to me | |
| 19:45:40 | dansmith | is there something in the compute log that matches that timestamp? | |
| 19:47:51 | greatgatsby | there probably is, but I haven't found it. I've been grepping through logs all day. I just check the nova_api aggregates table in the database, the creation time is still months old, so they didn't get recreated. Updated is NULL though. | |
| #openstack-nova - 2023-09-28 | |||
| 07:45:32 | gibi | greatgatsby, dansmith: based on my serach the only code that writes aggregates to placement is the report client's set_aggregates_for_provider() https://github.com/openstack/nova/blob/87d4807848bb9546c1fca972da2eb2eda13eb08d/nova/scheduler/client/report.py#L1122 | |
| 07:46:02 | gibi | we call that during nova aggregate manipulation but you said that is not happening | |
| 07:46:22 | gibi | so I ignore those places now | |
| 07:47:03 | gibi | we call that from update_from_provider_tree() as well that is the codepath dansmith refers to above | |
| 07:49:25 | gibi | that will basically flush the nova-compute's internal caches (the provider tree) back to placement | |
| 07:50:12 | gibi | so we need to track down what updates aggregates in that cache | |
| 07:52:29 | sahid | o/ | |
| 07:53:04 | sahid | any issue with our ci, it looks like zull did not pass completely for https://review.opendev.org/c/openstack/nova/+/896512 | |
| 07:53:07 | sahid | ? | |
| 08:29:26 | gibi | greatgatsby, dansmith: there is _refresh_associations in https://github.com/openstack/nova/blob/87d4807848bb9546c1fca972da2eb2eda13eb08d/nova/scheduler/client/report.py#L817-L818 that updates the placement aggregates in the caches | |
| 08:29:57 | gibi | greatgatsby: there is a DEBUG log associated with it | |
| 08:30:05 | gibi | "Refreshing aggregate associations for resource provider" | |
| 08:30:25 | gibi | do you have nova-compute debug logs from that time period? do you see this logged? | |
| 08:38:27 | sahid | let's try a recheck... | |
| 09:47:02 | zigo | Thanks guys for all the work. | |
| 09:47:18 | zigo | stephenfin: Especially you. Now I have to finish the work of backporting all the SQLA 2.x patches ... :P | |
| 12:08:15 | sahid | all is green | |
| 16:50:04 | greatgatsby | gibi: thanks so much for the info! I'm in and out of the office today, so I'll try to take a look. | |
| 17:19:59 | opendevreview | Merged openstack/placement master: Changed /tmp/migrate-db.rc to /root/migrate-db.rc https://review.opendev.org/c/openstack/placement/+/882436 | |
| #openstack-nova - 2023-09-29 | |||
| 08:42:28 | gibi | greatgatsby: what you are seeing feels like a bug somewhere in nova but at the moment I have no idea where so without some reproduction I cannot fix it | |
| 09:19:32 | bauzas | gibi: I just looked at yesterday's convo, can you summarize it ? (sorry yesterday was off) | |
| 09:31:01 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add missing Xena version in release notes https://review.opendev.org/c/openstack/osc-placement/+/896920 | |
| 09:47:49 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Fix overline length https://review.opendev.org/c/openstack/osc-placement/+/896922 | |
| 09:48:41 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add missing Xena version in release notes https://review.opendev.org/c/openstack/osc-placement/+/896920 | |
| 10:08:21 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add xena to toctree https://review.opendev.org/c/openstack/osc-placement/+/896923 | |
| 10:09:29 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add missing Xena version in release notes https://review.opendev.org/c/openstack/osc-placement/+/896924 | |
| 11:19:56 | opendevreview | Merged openstack/osc-placement master: Update master for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808458 | |
| 11:19:57 | opendevreview | Merged openstack/osc-placement stable/zed: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869753 | |
| 11:21:27 | opendevreview | Merged openstack/osc-placement master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/osc-placement/+/894069 | |
| 11:51:30 | opendevreview | Merged openstack/osc-placement stable/yoga: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869754 | |
| 12:10:22 | opendevreview | Merged openstack/osc-placement stable/xena: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869768 | |
| 12:35:27 | greatgatsby | gibi: really appreciate the help. I grepped through the logs and do not see the "Refreshing aggregate..." string anywhere in there for the full day on which the sync drift occurred | |
| 12:42:14 | greatgatsby | gibi: sorry, ignore that last message, I need to look into something here. Again, I really appreciate the help. We're monitoring this very closely, so hopefully I can keep provide more info if/when it occurs again. | |
| 14:04:11 | opendevreview | Merged openstack/nova stable/2023.1: Add debug logging when Instance raises OrphanedObjectError https://review.opendev.org/c/openstack/nova/+/888160 | |
| 15:53:56 | auniyal | Hi dansmith, can you please review this https://review.opendev.org/c/openstack/nova/+/895162 sometime, this is related to device tagging bug. | |
| 15:54:37 | auniyal | I was trying to add a reproducer specifically here- https://review.opendev.org/c/openstack/nova/+/895162/6/nova/tests/functional/libvirt/test_pci_sriov_servers.py#4252 | |
| 15:54:44 | auniyal | but the test is passing for me | |
| 15:55:13 | auniyal | I must be missing something, | |
| #openstack-nova - 2023-09-30 | |||
| 08:53:38 | gibi | greatgatsby: do you have DEBUG log enabled for nova-compute? | |
| 08:55:03 | gibi | greatgatsby: as of summary. I still trying to figure out where nova-compute updated an internal caches about the placement aggregates (as that is later flushed back to placemnet). The DEBUG log above would be can place where nova-compute updates the internal cache. But honeslty these are just stabs in the blind | |
| 09:06:43 | gibi | s/can/one/ | |
| 12:16:02 | greatgatsby | gibi: unfortunately it seems we didn't have debug logging enabled at the time the agg unsync occurred. I've enabled it and I'll search for the "Refreshing... " message if/when this happens again. Again, thanks for the help. | |
| #openstack-nova - 2023-10-01 | |||
| 14:41:48 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2024.1 Caracal https://review.opendev.org/c/openstack/nova/+/896305 | |
| 14:42:01 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2024.1 Caracal https://review.opendev.org/c/openstack/nova/+/896305 | |
| 18:10:27 | opendevreview | sean mooney proposed openstack/nova master: remove py2 only syntax https://review.opendev.org/c/openstack/nova/+/896984 | |
| 18:10:28 | opendevreview | sean mooney proposed openstack/nova master: pyupgrade py 3.0 -> 3.6 https://review.opendev.org/c/openstack/nova/+/896986 | |
| 18:10:28 | opendevreview | sean mooney proposed openstack/nova master: remove use of % format strings https://review.opendev.org/c/openstack/nova/+/896985 | |
| 18:10:29 | opendevreview | sean mooney proposed openstack/nova master: ignore pyupgrade series in git blame https://review.opendev.org/c/openstack/nova/+/896988 | |
| 18:10:29 | opendevreview | sean mooney proposed openstack/nova master: add precommit pyupgrade hook and tox env https://review.opendev.org/c/openstack/nova/+/896987 | |
| #openstack-nova - 2023-10-02 | |||
| 06:58:22 | bauzas | good morning Nova | |
| 07:26:15 | gibi | o/ | |
| 07:45:45 | dvo-plv | Hello, I would like to remind about to take a look on the reproposed spec file ? https://review.opendev.org/c/openstack/nova-specs/+/895924 | |
| 07:45:46 | dvo-plv | and check dependent patch for it https://review.opendev.org/c/openstack/nova/+/876075 | |
| 07:53:35 | bauzas | yup, specs reviews is on my agenda this week | |
| 09:08:17 | opendevreview | sean mooney proposed openstack/nova master: remove py2 only syntax https://review.opendev.org/c/openstack/nova/+/896984 | |
| 09:08:18 | opendevreview | sean mooney proposed openstack/nova master: pyupgrade py 3.0 -> 3.6 https://review.opendev.org/c/openstack/nova/+/896986 | |
| 09:08:18 | opendevreview | sean mooney proposed openstack/nova master: remove use of % format strings https://review.opendev.org/c/openstack/nova/+/896985 | |
| 09:08:19 | opendevreview | sean mooney proposed openstack/nova master: ignore pyupgrade series in git blame https://review.opendev.org/c/openstack/nova/+/896988 | |
| 09:08:19 | opendevreview | sean mooney proposed openstack/nova master: add precommit pyupgrade hook and tox env https://review.opendev.org/c/openstack/nova/+/896987 | |
| 09:09:45 | opendevreview | sean mooney proposed openstack/nova master: ignore pyupgrade series in git blame https://review.opendev.org/c/openstack/nova/+/896988 | |
| 09:10:22 | opendevreview | sean mooney proposed openstack/nova master: ignore pyupgrade series in git blame https://review.opendev.org/c/openstack/nova/+/896988 | |
| 09:26:37 | dvo-plv | bauzas, good to hear,thank you | |
| 10:20:34 | opendevreview | Stephen Finucane proposed openstack/nova master: doc: Remove crud from conf.py file https://review.opendev.org/c/openstack/nova/+/896224 | |
| 10:20:35 | opendevreview | Stephen Finucane proposed openstack/nova master: tox: Use pre-commit for pep8 target https://review.opendev.org/c/openstack/nova/+/896227 | |
| 10:20:35 | opendevreview | Stephen Finucane proposed openstack/nova master: pre-commit: Update plugin versions https://review.opendev.org/c/openstack/nova/+/896226 | |
| 10:20:36 | opendevreview | Stephen Finucane proposed openstack/nova master: Blacken codebase https://review.opendev.org/c/openstack/nova/+/896229 | |
| 10:20:36 | opendevreview | Stephen Finucane proposed openstack/nova master: pre-commit: Add mypy https://review.opendev.org/c/openstack/nova/+/896228 | |