| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-30 | |||
| 01:27:49 | sean-k-mooney | i.e. ssh into the vm and add somethign to the tpm | |
| 01:28:02 | sean-k-mooney | then live migrate it or hard reboot it and show its still there | |
| 01:28:10 | melwitt | that would be good if we can. I was thinking about that yesterday with regard to the reboot bug/launchpad | |
| 01:28:11 | sean-k-mooney | i dotn actuly know how to do that | |
| 01:28:28 | sean-k-mooney | i assume there is a tool for that but its probaly not in busy box | |
| 01:28:31 | melwitt | I don't either. will have to do some research | |
| 01:29:03 | sean-k-mooney | i dont knwo enought about tpms to know if there is a /sys interface that we can jsut echo too or something adn read back with cat | |
| 01:29:20 | sean-k-mooney | that woudl be the best option but future us problem | |
| 01:29:31 | melwitt | yeah. I hope it could be that easy | |
| 01:30:29 | gmaan | sean-k-mooney: for your tomorrow. did you get chance to look into manager role tests, stephenfin is +2 on that now, waiting for your review https://review.opendev.org/q/topic:%22bp/policy-manager-role-default%22+status:open | |
| 01:31:34 | sean-k-mooney | partly but that actully what im hoping to do in the morning. so all going well ill add +2w later tomorrow | |
| 01:31:46 | gmaan | thanks | |
| 01:31:48 | sean-k-mooney | i was happy with the production code i just did not fuly look at the test code before | |
| 01:32:57 | gmaan | k, in summary for tests coverage: 1. I moved existing tests to use member token 2. added a few new tests | |
| 01:34:22 | sean-k-mooney | cool night all o/ | |
| 01:34:29 | gmaan | GN | |
| 01:35:58 | melwitt | gnight o/ | |
| 03:03:43 | gmaan | stephenfin: finished the openapi series, some comments for follow up and some for fixing. | |
| 09:45:45 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Address issues with images APIs https://review.opendev.org/c/openstack/nova/+/956102 | |
| 09:45:46 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Separate volume, snapshot and volume attachments https://review.opendev.org/c/openstack/nova/+/952347 | |
| 09:45:46 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Use valid UUIDs for cinder resources https://review.opendev.org/c/openstack/nova/+/952935 | |
| 09:45:47 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Only apply "soft" additionalProperties validation to requests https://review.opendev.org/c/openstack/nova/+/952936 | |
| 09:45:47 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for volumes APIs https://review.opendev.org/c/openstack/nova/+/952348 | |
| 09:45:48 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for snapshots APIs https://review.opendev.org/c/openstack/nova/+/952349 | |
| 09:45:48 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for volume attachments APIs https://review.opendev.org/c/openstack/nova/+/952350 | |
| 09:45:49 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for floating IP APIs https://review.opendev.org/c/openstack/nova/+/952972 | |
| 09:45:49 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for security group APIs https://review.opendev.org/c/openstack/nova/+/952973 | |
| 09:45:51 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for services APIs https://review.opendev.org/c/openstack/nova/+/953196 | |
| 09:45:51 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server usage audit log APIs https://review.opendev.org/c/openstack/nova/+/953209 | |
| 09:45:53 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for tenant network APIs https://review.opendev.org/c/openstack/nova/+/956088 | |
| 09:45:53 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for simple tenant usage APIs https://review.opendev.org/c/openstack/nova/+/956096 | |
| 09:45:55 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for limits API https://review.opendev.org/c/openstack/nova/+/956139 | |
| 09:45:55 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for quota class sets API https://review.opendev.org/c/openstack/nova/+/956140 | |
| 09:45:57 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for quota sets API https://review.opendev.org/c/openstack/nova/+/956141 | |
| 09:45:57 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for migrations API https://review.opendev.org/c/openstack/nova/+/956142 | |
| 09:45:59 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server migrations API https://review.opendev.org/c/openstack/nova/+/956143 | |
| 09:45:59 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server tags API https://review.opendev.org/c/openstack/nova/+/956144 | |
| 09:46:01 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server metadata APIs https://review.opendev.org/c/openstack/nova/+/956145 | |
| 09:47:07 | opendevreview | Ian Watson proposed openstack/nova master: Changing how placement updates are triggered in RT https://review.opendev.org/c/openstack/nova/+/955687 | |
| 09:48:56 | opendevreview | Ian Watson proposed openstack/nova master: Changing how placement updates are triggered in RT https://review.opendev.org/c/openstack/nova/+/955687 | |
| 10:00:17 | opendevreview | Ian Watson proposed openstack/nova master: Add migrations_supported methed for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 10:12:16 | opendevreview | Ian Watson proposed openstack/nova master: Add migrations_supported methed for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 10:30:26 | opendevreview | Ian Watson proposed openstack/nova master: Add migrations_supported methed for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 10:58:12 | opendevreview | Ian Watson proposed openstack/nova master: Add migrations_supported methed for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 11:00:48 | opendevreview | Ian Watson proposed openstack/nova master: Add migrations_supported methed for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 11:02:00 | opendevreview | Ian Watson proposed openstack/nova master: Add migrations_supported methed for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 11:02:53 | opendevreview | Ian Watson proposed openstack/nova master: Add migrations_supported methed for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 11:04:29 | opendevreview | Ian Watson proposed openstack/nova master: Add migrations_supported methed for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 11:04:59 | opendevreview | Ian Watson proposed openstack/nova master: Add migration conditional for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 11:06:32 | opendevreview | Ian Watson proposed openstack/nova master: Add migration conditional for resource tracker https://review.opendev.org/c/openstack/nova/+/955690 | |
| 11:49:21 | opendevreview | Merged openstack/nova master: [test]Speed up fs retry tests by mocking sleep https://review.opendev.org/c/openstack/nova/+/955904 | |
| 11:50:56 | opendevreview | Merged openstack/nova master: [test]Speed up ironic console test by decreasing timeout https://review.opendev.org/c/openstack/nova/+/955905 | |
| 11:51:26 | opendevreview | Merged openstack/nova master: [test]Speed up RBD test by decreasing retry interval https://review.opendev.org/c/openstack/nova/+/955906 | |
| 11:57:13 | opendevreview | Merged openstack/nova master: [test]Speed up qemu announce test by mocking sleep https://review.opendev.org/c/openstack/nova/+/955907 | |
| 11:59:16 | stephenfin | gibi: Picking this up from last week: it appears the device detach libvirt event stuff is broken on every release since Caracal and it always times out https://github.com/kubernetes/cloud-provider-openstack/pull/2742#issuecomment-3135940269 | |
| 11:59:32 | stephenfin | I'm still looking so jfyi for now | |
| 11:59:59 | sean-k-mooney | i mean its not optional and i dont think we have see that in devstack in our jobs | |
| 12:00:11 | sean-k-mooney | is this only happing in that ci env | |
| 12:00:16 | sean-k-mooney | or did you hit it locally | |
| 12:00:39 | stephenfin | no, I can reproduce in CI and across multiple PSI instances | |
| 12:01:03 | stephenfin | s/multiple PSI instances/multiple local VMs/ (for the non-RHers) | |
| 12:01:30 | sean-k-mooney | what exactly doe this look like in the nova logs | |
| 12:01:46 | sean-k-mooney | i dont see that in your comment above | |
| 12:03:04 | sean-k-mooney | oh its in drivers | |
| 12:03:08 | stephenfin | sean-k-mooney: it manifests as a 20 second "stall" while detaching volumes https://paste.opendev.org/show/b6K2tt0kBpOO2liORP7k/ | |
| 12:03:08 | sean-k-mooney | https://github.com/kubernetes/cloud-provider-openstack/pull/2742#issuecomment-3114089612 | |
| 12:03:48 | sean-k-mooney | its blocking the comptue agent for 20 seconds | |
| 12:04:03 | sean-k-mooney | or there was jsut nothing else happeing at that time on that host | |
| 12:04:41 | stephenfin | sean-k-mooney, gibi: It's also happening upstream: our CI logs are full of those warnings | |
| 12:05:05 | sean-k-mooney | stephenfin: im going to check the ovs-hybrid plug job | |
| 12:05:17 | sean-k-mooney | i want to see if this is a ubuntu bug or if it show up in debian | |
| 12:05:20 | stephenfin | For example, the tempest-integrated-compute job from a patch in melwitt's vTPM series https://review.opendev.org/c/openstack/nova/+/942501 | |
| 12:05:21 | sean-k-mooney | or rhel | |
| 12:05:27 | stephenfin | https://d6e5655f0b1ee2f8df74-e35c415f7cc7f78f33337c81de121e72.ssl.cf2.rackcdn.com/openstack/52b57c77f805400888762a87e08196f1/controller/logs/screen-n-cpu.txt | |
| 12:05:35 | sean-k-mooney | ya but that also using ubuntu | |
| 12:05:50 | sean-k-mooney | i want to check a non ubuntu job to see if tis a bug in ther elibvirt or if its a nova issue | |
| 12:07:00 | stephenfin | I see 27 hits grepping for 'Waiting for libvirt event', which is over 9 minutes of wasted time (though not really since I think tempest runs in parallel?) | |
| 12:07:04 | stephenfin | sean-k-mooney: ack | |
| 12:07:32 | sean-k-mooney | oh i tured off cinder in that job | |
| 12:07:53 | sean-k-mooney | let me see if i can check the ceph job i just proposed moving to debian | |
| 12:10:35 | stephenfin | sean-k-mooney: do you know how DevStack works on CentOS Stream currently? I can trivially reproduce this locally if needed too | |
| 12:10:42 | sean-k-mooney | Waiting for libvirt event about the detach of device vdb with device alias ua-88d8994a-f5be-42d8-9a2f-5e030b1f4ce8 from instance 621ab0d3-61a0-437f-8d31-e9c3d7ae3eae is timed out. | |
| 12:11:04 | sean-k-mooney | stephenfin: it does | |
| 12:11:09 | sean-k-mooney | so its happenign on debian too | |
| 12:11:37 | sean-k-mooney | with ceph in this case you were using lvm right | |
| 12:11:52 | sean-k-mooney | the backend should not really matter | |
| 12:12:02 | stephenfin | yes (pretty much a "stock" DevStack configuration) | |
| 12:12:04 | sean-k-mooney | but that is both local block and network type covered | |
| 12:12:10 | sean-k-mooney | ack | |
| 12:12:19 | sean-k-mooney | let me see if i still have a c9s devstack somewhere | |
| 12:14:32 | sean-k-mooney | that woudl be no but i can check our watcher ci job | |
| 12:14:46 | sean-k-mooney | actully no | |
| 12:15:04 | sean-k-mooney | we have master jobs that trigger with our new installer on cenetos butthsoe are disbaling cinder right now | |
| 12:15:14 | sean-k-mooney | i think neutorn and maybe cinder have centos josb | |
| 12:15:34 | sean-k-mooney | we might have an experminatal one too | |
| 12:16:07 | stephenfin | I'll just create one locally | |
| 12:16:38 | stephenfin | However, I see a few related changed that I'm going to try reverting out of caracal first | |
| 12:22:52 | gibi | I did the last round of fixes in that codepath so I remember it was working at that time to wait for the events. If the event does not come we still have code to check if the actual detach happened and move on. But as you noted it might be excessive waiting time | |
| 12:23:53 | gibi | if this is visible in upstream CI then please file an upstream bug with the links and I will try to look at it | |
| 12:27:14 | gibi | In https://d6e5655f0b1ee2f8df74-e35c415f7cc7f78f33337c81de121e72.ssl.cf2.rackcdn.com/openstack/52b57c77f805400888762a87e08196f1/controller/logs/screen-n-cpu.txt I see cases when it is on 20 sec wait and we move one | |