| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-26 | |||
| 17:09:34 | gmann | bauzas: I mean if all things mentioned in the spec are not implemented then we should not mark as Implemented or we update the spec in case of half implementation - https://review.opendev.org/c/openstack/nova-specs/+/862833 | |
| 17:10:08 | gmann | I remember we have updated the specs in such cases to match the spec documentation and actual implemetation | |
| 20:16:13 | opendevreview | Ghanshyam proposed openstack/nova-specs master: Remove Hyper-V driver https://review.opendev.org/c/openstack/nova-specs/+/896592 | |
| #openstack-nova - 2023-09-27 | |||
| 01:57:38 | opendevreview | Natz cheng proposed openstack/os-vif master: cleanup py27 support https://review.opendev.org/c/openstack/os-vif/+/896598 | |
| 07:08:59 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: AggregateMultitenancyIsolation to support unlimited tenant https://review.opendev.org/c/openstack/nova/+/896512 | |
| 08:47:22 | dvo-plv | Hello, All | |
| 08:48:00 | dvo-plv | Maybe you will have a chance to take a look on the reproposed spec file ? https://review.opendev.org/c/openstack/nova-specs/+/895924 | |
| 08:48:32 | dvo-plv | and check dependent patch for it https://review.opendev.org/c/openstack/nova/+/876075 | |
| 09:28:30 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: AggregateMultitenancyIsolation to support unlimited tenant https://review.opendev.org/c/openstack/nova/+/896512 | |
| 10:09:33 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds device tagging functional tests https://review.opendev.org/c/openstack/nova/+/895162 | |
| 10:20:40 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds device tagging functional tests https://review.opendev.org/c/openstack/nova/+/895162 | |
| 11:02:38 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds device tagging functional tests https://review.opendev.org/c/openstack/nova/+/895162 | |
| 11:02:38 | opendevreview | Amit Uniyal proposed openstack/nova master: Device tags: don't pass pf_interface=True to get_mac_by_pci_address https://review.opendev.org/c/openstack/nova/+/670593 | |
| 11:02:39 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Device tagging: expose target PCI address, not source https://review.opendev.org/c/openstack/nova/+/672127 | |
| 12:32:26 | stephenfin | bauzas: Can you look at https://review.opendev.org/c/openstack/placement/+/886229 and https://review.opendev.org/c/openstack/placement/+/888281 when you get a chance? (SQLAlchemy 2.x stuff) | |
| 12:32:44 | bauzas | stephenfin: yeah I'll try | |
| 12:37:10 | opendevreview | Stephen Finucane proposed openstack/placement master: db: Workaround SQLAlchemy 2.x bug https://review.opendev.org/c/openstack/placement/+/888281 | |
| 12:37:11 | opendevreview | Stephen Finucane proposed openstack/placement master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/placement/+/886229 | |
| 12:37:11 | opendevreview | Stephen Finucane proposed openstack/placement master: db: Wrap raw SQL query in sqlalchemy.text https://review.opendev.org/c/openstack/placement/+/896650 | |
| 12:58:50 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Fix the libvirt.images_type value in KVM docs https://review.opendev.org/c/openstack/nova/+/896651 | |
| 13:00:01 | bauzas | stephenfin: ack, thanks for the quick respins, will check | |
| 14:33:44 | sahid | o/ I have proposed a change for the filter AggregateMultiTenancyFilter, I have reused what we do in placement | |
| 14:33:50 | sahid | https://review.opendev.org/c/openstack/nova/+/896512 | |
| 14:34:15 | sahid | can you just tell whether that looks to be in the right direction for you guys? | |
| 14:35:08 | bauzas | I'll try | |
| 14:36:48 | sahid | thank you | |
| 14:53:23 | opendevreview | Sylvain Bauza proposed openstack/nova-specs master: explain why ironic-shards was reverted in the spec https://review.opendev.org/c/openstack/nova-specs/+/896679 | |
| 14:58:05 | opendevreview | Stephen Finucane proposed openstack/placement master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/placement/+/886229 | |
| 14:59:45 | opendevreview | Dan Smith proposed openstack/nova-specs master: Add libvirt-dev-alias spec https://review.opendev.org/c/openstack/nova-specs/+/892806 | |
| 15:01:54 | opendevreview | Stephen Finucane proposed openstack/placement master: db: Wrap raw SQL query in sqlalchemy.text https://review.opendev.org/c/openstack/placement/+/896650 | |
| 15:01:59 | opendevreview | Stephen Finucane proposed openstack/placement master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/placement/+/886229 | |
| 18:51:35 | mnaser | https://bugs.launchpad.net/nova/+bug/2037585 -- does deleting neutron ports in a specific order sound like a reasonable fix here? | |
| 18:51:46 | mnaser | if so i can push that proposal | |
| 19:30:59 | greatgatsby | Hi. I was asking before about our compute aggregates become out of sync with the resource provider aggregates. We had it happen again last night, and I've found a PUT /resource_providers/<uuid>/aggregates in the logs at the time we detected them out of sync. What service or state might trigger this update on its own? I've been grepping through logs most of the day, but still haven't found what | |
| 19:30:59 | greatgatsby | triggers this to happen. | |
| 19:31:12 | greatgatsby | thanks *a lot* for any help | |
| 19:31:52 | JayF | greatgatsby: just curious, are the resources in question Ironic nodes? | |
| 19:33:35 | greatgatsby | thanks for the reply. No ironic, we pxe boot and use ansible for config | |
| 19:33:59 | JayF | ack; sorry I won't be much help then as that's my specialty; good luck! | |
| 19:34:10 | greatgatsby | thanks :-) | |
| 19:34:58 | dansmith | greatgatsby: nova-api will do that in response to creating or deleting aggregates on the nova side | |
| 19:35:06 | dansmith | nothing else should be doing that, AFAIK | |
| 19:35:20 | dansmith | can you trace the origin with a req-id or source ip? | |
| 19:36:32 | greatgatsby | source ip in one case came from the controller that lost the resource provider aggs. The nova aggs have never changed (or at least, we haven't changed them, and they stay consistent) | |
| 19:36:59 | greatgatsby | so, for example, compute 1 will still have the correct linux agg on the nova side, but it disappears on the resource provider side | |
| 19:37:06 | greatgatsby | this happened late last night | |
| 19:38:55 | greatgatsby | in one PUT message I see "nova-compute keystoneauth1/4.5.0 python-requests...." | |
| 19:41:12 | dansmith | "from the controller that lost the aggregates" doesn't mean anything to me | |
| 19:41:19 | dansmith | do you mean from a compute node? | |
| 19:41:41 | greatgatsby | oops, sorry, yes, the compute | |
| 19:42:01 | dansmith | there is some aggregate-manipulation that happens in the provider tree stuff now that I'm not familiar enough to speak about without some study | |
| 19:42:08 | dansmith | so must be something related to that I guess | |
| 19:42:27 | dansmith | best option would be to chat with gibi in his morning (central EU) | |
| 19:43:40 | dansmith | is the uuid in the PUT /resource_providers/uuid/aggregate the uuid of that same compute node? | |
| 19:43:50 | greatgatsby | yes, exactly | |
| 19:44:19 | greatgatsby | it's the only PUT to the resources providers and only to that specific node | |
| 19:45:28 | dansmith | I really didn't think that any of the computes would be adjusting their own aggregate membership, so that seems weird to me | |
| 19:45:40 | dansmith | is there something in the compute log that matches that timestamp? | |
| 19:47:51 | greatgatsby | there probably is, but I haven't found it. I've been grepping through logs all day. I just check the nova_api aggregates table in the database, the creation time is still months old, so they didn't get recreated. Updated is NULL though. | |
| #openstack-nova - 2023-09-28 | |||
| 07:45:32 | gibi | greatgatsby, dansmith: based on my serach the only code that writes aggregates to placement is the report client's set_aggregates_for_provider() https://github.com/openstack/nova/blob/87d4807848bb9546c1fca972da2eb2eda13eb08d/nova/scheduler/client/report.py#L1122 | |
| 07:46:02 | gibi | we call that during nova aggregate manipulation but you said that is not happening | |
| 07:46:22 | gibi | so I ignore those places now | |
| 07:47:03 | gibi | we call that from update_from_provider_tree() as well that is the codepath dansmith refers to above | |
| 07:49:25 | gibi | that will basically flush the nova-compute's internal caches (the provider tree) back to placement | |
| 07:50:12 | gibi | so we need to track down what updates aggregates in that cache | |
| 07:52:29 | sahid | o/ | |
| 07:53:04 | sahid | any issue with our ci, it looks like zull did not pass completely for https://review.opendev.org/c/openstack/nova/+/896512 | |
| 07:53:07 | sahid | ? | |
| 08:29:26 | gibi | greatgatsby, dansmith: there is _refresh_associations in https://github.com/openstack/nova/blob/87d4807848bb9546c1fca972da2eb2eda13eb08d/nova/scheduler/client/report.py#L817-L818 that updates the placement aggregates in the caches | |
| 08:29:57 | gibi | greatgatsby: there is a DEBUG log associated with it | |
| 08:30:05 | gibi | "Refreshing aggregate associations for resource provider" | |
| 08:30:25 | gibi | do you have nova-compute debug logs from that time period? do you see this logged? | |
| 08:38:27 | sahid | let's try a recheck... | |
| 09:47:02 | zigo | Thanks guys for all the work. | |
| 09:47:18 | zigo | stephenfin: Especially you. Now I have to finish the work of backporting all the SQLA 2.x patches ... :P | |
| 12:08:15 | sahid | all is green | |
| 16:50:04 | greatgatsby | gibi: thanks so much for the info! I'm in and out of the office today, so I'll try to take a look. | |
| 17:19:59 | opendevreview | Merged openstack/placement master: Changed /tmp/migrate-db.rc to /root/migrate-db.rc https://review.opendev.org/c/openstack/placement/+/882436 | |
| #openstack-nova - 2023-09-29 | |||
| 08:42:28 | gibi | greatgatsby: what you are seeing feels like a bug somewhere in nova but at the moment I have no idea where so without some reproduction I cannot fix it | |
| 09:19:32 | bauzas | gibi: I just looked at yesterday's convo, can you summarize it ? (sorry yesterday was off) | |
| 09:31:01 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add missing Xena version in release notes https://review.opendev.org/c/openstack/osc-placement/+/896920 | |
| 09:47:49 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Fix overline length https://review.opendev.org/c/openstack/osc-placement/+/896922 | |
| 09:48:41 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add missing Xena version in release notes https://review.opendev.org/c/openstack/osc-placement/+/896920 | |
| 10:08:21 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add xena to toctree https://review.opendev.org/c/openstack/osc-placement/+/896923 | |
| 10:09:29 | opendevreview | Franciszek Przewoźny proposed openstack/osc-placement master: Add missing Xena version in release notes https://review.opendev.org/c/openstack/osc-placement/+/896924 | |
| 11:19:56 | opendevreview | Merged openstack/osc-placement master: Update master for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808458 | |
| 11:19:57 | opendevreview | Merged openstack/osc-placement stable/zed: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869753 | |
| 11:21:27 | opendevreview | Merged openstack/osc-placement master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/osc-placement/+/894069 | |
| 11:51:30 | opendevreview | Merged openstack/osc-placement stable/yoga: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869754 | |
| 12:10:22 | opendevreview | Merged openstack/osc-placement stable/xena: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869768 | |
| 12:35:27 | greatgatsby | gibi: really appreciate the help. I grepped through the logs and do not see the "Refreshing aggregate..." string anywhere in there for the full day on which the sync drift occurred | |
| 12:42:14 | greatgatsby | gibi: sorry, ignore that last message, I need to look into something here. Again, I really appreciate the help. We're monitoring this very closely, so hopefully I can keep provide more info if/when it occurs again. | |
| 14:04:11 | opendevreview | Merged openstack/nova stable/2023.1: Add debug logging when Instance raises OrphanedObjectError https://review.opendev.org/c/openstack/nova/+/888160 | |
| 15:53:56 | auniyal | Hi dansmith, can you please review this https://review.opendev.org/c/openstack/nova/+/895162 sometime, this is related to device tagging bug. | |
| 15:54:37 | auniyal | I was trying to add a reproducer specifically here- https://review.opendev.org/c/openstack/nova/+/895162/6/nova/tests/functional/libvirt/test_pci_sriov_servers.py#4252 | |
| 15:54:44 | auniyal | but the test is passing for me | |
| 15:55:13 | auniyal | I must be missing something, | |
| #openstack-nova - 2023-09-30 | |||
| 08:53:38 | gibi | greatgatsby: do you have DEBUG log enabled for nova-compute? | |
| 08:55:03 | gibi | greatgatsby: as of summary. I still trying to figure out where nova-compute updated an internal caches about the placement aggregates (as that is later flushed back to placemnet). The DEBUG log above would be can place where nova-compute updates the internal cache. But honeslty these are just stabs in the blind | |
| 09:06:43 | gibi | s/can/one/ | |