| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-05-14 | |||
| 16:25:17 | andrewbonney | One query on bugs if I could too after others | |
| 16:25:40 | sean-k-mooney | andrewbonney: sure. JayF do you want to summerise | |
| 16:25:51 | bauzas | JayF: I don't see anything controversial with reapproving https://blueprints.launchpad.net/nova/+spec/ironic-guest-metadata as specless | |
| 16:26:17 | JayF | thank! | |
| 16:26:20 | JayF | *thanks | |
| 16:26:21 | bauzas | most of my concerns come with testing this, but this is not related to the blueprint itself | |
| 16:26:50 | bauzas | anyone having concerns with reapproving ^ as specless ? | |
| 16:27:16 | bauzas | speak now or forever hold your peace | |
| 16:27:35 | bauzas | okay, I'll do the paperwork stamp | |
| 16:27:50 | bauzas | #agreed https://blueprints.launchpad.net/nova/+spec/ironic-guest-metadata reapproved as specless for Dalmatian | |
| 16:28:15 | bauzas | next one, who's this ? sean-k-mooney I think ? | |
| 16:28:18 | sean-k-mooney | so hopefully my item will be jsut as quick | |
| 16:28:20 | sean-k-mooney | https://blueprints.launchpad.net/nova/+spec/eventlet-removal-part-1 | |
| 16:28:25 | sean-k-mooney | can we appove that as specless | |
| 16:28:47 | sean-k-mooney | i mostly have it implmeneted | |
| 16:29:05 | dansmith | the critical bits are the implementation details, | |
| 16:29:07 | sean-k-mooney | there is some feedback form stephen on the implmentaion but i think it will be ready for review next week | |
| 16:29:22 | dansmith | so I think it's reasonable to call this cleanup/maintenance and focus on the actual code, IMHO | |
| 16:29:35 | dansmith | i.e. no API or RPC or DB changes, just internals | |
| 16:29:37 | bauzas | yeah I think we said it's just for tracking | |
| 16:29:50 | bauzas | tracking progress and adding it to our review eyes | |
| 16:29:56 | sean-k-mooney | cool well the patches are avaiable for review here https://review.opendev.org/q/topic:%22eventlet-removal-part-1%22 | |
| 16:30:12 | bauzas | I'm fine with approving it then as specless and I think we even agreed on that at the PTG | |
| 16:30:21 | sean-k-mooney | yep | |
| 16:30:27 | bauzas | okay, | |
| 16:30:44 | bauzas | #agreed https://blueprints.launchpad.net/nova/+spec/eventlet-removal-part-1 accepted as specless blueprint as agreed during the PTG | |
| 16:30:53 | sean-k-mooney | i belive andrewbonney had a question around a bug? | |
| 16:31:26 | sean-k-mooney | assuming there was nothign else on the adgenda | |
| 16:31:40 | andrewbonney | Thanks. Just a procedure thing really. I have a few reports since March and just wanted to check if I should be following up here generally or in meetings for feedback? | |
| 16:31:53 | andrewbonney | I appreciate you get a lot so just trying to do the right thing | |
| 16:32:36 | sean-k-mooney | as in you filed a number of bug reporst since march? | |
| 16:32:47 | andrewbonney | Yeah that's right, https://bugs.launchpad.net/nova/+bugs?field.bug_reporter=andrewbonney | |
| 16:32:49 | sean-k-mooney | and you want to knwo if they have been looked at or how to progress them | |
| 16:33:38 | andrewbonney | Sure. For maybe one I could propose a patch, but generally they need review from those more familiar with the codebase | |
| 16:33:44 | bauzas | andrewbonney: we agreed on previous PTG that we can't really triage the new bugs every week as we lack volunteers and time for that | |
| 16:33:52 | andrewbonney | Sure | |
| 16:34:09 | sean-k-mooney | looking at the first one https://bugs.launchpad.net/nova/+bug/2058725 sound familar and im trying to recall if it was intentional | |
| 16:34:18 | bauzas | unfortunately, we then let the reporters to communicate to us any bug in particular they could consider as critical | |
| 16:34:55 | bauzas | andrewbonney: for the other bugs that are less important, our channel is always open for discussion and you shouldn't be afraid of pinging ourselves | |
| 16:35:00 | gibi | andrewbonney: if you could try to push a test case that reproduces the issue in unit or functional test env would greatly increas the chance that we get to it | |
| 16:35:25 | bauzas | some people usually bug ourselves in this chat, asking either for reviews, help or categorisation | |
| 16:35:33 | opendevreview | Stephen Finucane proposed openstack/nova-specs master: Follow-up for openapi spec https://review.opendev.org/c/openstack/nova-specs/+/919603 | |
| 16:36:10 | andrewbonney | Thanks, just don't want to annoy | |
| 16:37:05 | sean-k-mooney | its fine. https://bugs.launchpad.net/nova/+bug/2065403 sound like you are not running it in a loop until completion | |
| 16:37:44 | sean-k-mooney | although the interaction with -max-count problmeatic | |
| 16:37:57 | andrewbonney | I think the default behaviour does run in a loop, that's where the problem lies. With max-count it doesn't run in a loop as far as I can see | |
| 16:38:42 | bauzas | if people agree, we can close the meeting now and let people to discuss about andrewbonney's bug reports just after | |
| 16:38:49 | sean-k-mooney | sure | |
| 16:39:30 | bauzas | cool | |
| 16:39:32 | bauzas | thanks all then | |
| 16:39:38 | bauzas | #endmeeting | |
| 16:39:38 | opendevmeet | Meeting ended Tue May 14 16:39:38 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:39:38 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-05-14-16.01.html | |
| 16:39:38 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-05-14-16.01.txt | |
| 16:39:38 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-05-14-16.01.log.html | |
| 16:40:34 | opendevreview | Merged openstack/nova master: Upload glance image with --file in ceph job https://review.opendev.org/c/openstack/nova/+/918689 | |
| 16:41:11 | fwiesel | Thanks | |
| 16:41:48 | sean-k-mooney | hum 29147 instnace in batches of 50. i wonder if we are missing a marker/stable order | |
| 16:45:32 | andrewbonney | I was wondering why populate_instance_compute_id relied on exception handling rather than checking for 'node' presence in the initial DB query, but there may be good reasons | |
| 16:46:47 | andrewbonney | The fact there's no DB cursor means it hits the same un-migratable records on each iteration | |
| 16:47:39 | sean-k-mooney | well node here https://opendev.org/openstack/nova/blame/commit/7096423b343ffce9622fd078fc2b3a87fd3386f7/nova/objects/instance.py#L1366 | |
| 16:47:43 | sean-k-mooney | is not coming form the instnace | |
| 16:48:41 | sean-k-mooney | that is looking up the compute node record | |
| 16:49:04 | sean-k-mooney | instnace.node i think is the compute node name | |
| 16:49:50 | dansmith | yeah I don't understand the question.. we're not "relying" on exception handling for anything in that code | |
| 16:50:02 | dansmith | ... and we do loop ... | |
| 16:51:09 | sean-k-mooney | my incliantion is to say that of the 30k instances a large proporation fo those recored are for deleted instnaces? | |
| 16:51:12 | sean-k-mooney | thats a guess | |
| 16:51:20 | andrewbonney | Yes that's right | |
| 16:51:35 | andrewbonney | I think this is only an issue where lots of records are not running instances | |
| 16:52:15 | andrewbonney | Do you end up in a state where the method exits because the first 50 records returned by the DB don't qualify for migration, which triggers the end of the loop | |
| 16:52:19 | andrewbonney | *So | |
| 16:52:25 | sean-k-mooney | so a workaround in that case would be to run nova-manage archive-deleted-rows | |
| 16:52:27 | dansmith | sorry I missed something, what is the issue ? | |
| 16:53:06 | sean-k-mooney | when they run the commadn its printing "50 rows matched query populate_instance_compute_id, 6 migrated" | |
| 16:53:21 | dansmith | oh I see above, you're expecting to be able to run just --max-count $number_of_active_instances ? | |
| 16:53:21 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/2065403 | |
| 16:53:43 | dansmith | yeah I mean you have to run those things untll they're done | |
| 16:54:03 | dansmith | sometimes we can skip records that are deleted and other times we can't really | |
| 16:54:08 | andrewbonney | We ran without max-count initially as part of a standard upgrade. It completed, but hasn't migrated everything because once the first 50 records are unsuitable it exits | |
| 16:54:49 | andrewbonney | I only realised our first deployment was in this state because I noticed the migration taking ~15 mins plus on a second deployment, because it keeps trying to migrate unsuitable records | |
| 16:55:00 | dansmith | ah, I see because you have a bunch of (hopefully deleted) instances that point to missing compute nodes? | |
| 16:55:07 | andrewbonney | Yeah that's it | |
| 16:55:11 | dansmith | gotcha | |
| 16:55:37 | andrewbonney | Happy to work around it for us, but I doubt we'll be the only ones to hit it | |
| 16:55:59 | sean-k-mooney | so just looking at our docs https://docs.openstack.org/nova/latest/cli/nova-manage.html#db-online-data-migrations | |
| 16:56:11 | sean-k-mooney | we state that if you dont pass --max-count | |
| 16:56:11 | dansmith | yeah, I'm not sure the best way to handle that in the context of how that manage command works | |
| 16:56:21 | sean-k-mooney | we will loop internanlly in batchs of 50 | |
| 16:56:36 | dansmith | yeah, but we don't have marker stuff there to prevent us from looping over the same 50 over and over | |
| 16:56:42 | sean-k-mooney | but if you do pass --max-count you have to loop externally and look at the repsoce code | |
| 16:56:54 | sean-k-mooney | well we dont actully need that | |
| 16:57:13 | dansmith | so I mean, we could just ignore deleted instances, but that means we may have a problem adding a constraint later where this will come up again | |
| 16:57:14 | sean-k-mooney | since we only get instnace that dont have the compute_id set | |
| 16:57:30 | sean-k-mooney | but the issue is we are not settign a sentinel for those that dont match | |
| 16:57:33 | dansmith | right but the problem is that if the 50 are not migratable, we'll never mark them as identified and keey finding them | |
| 16:57:38 | dansmith | yeah | |
| 16:57:49 | sean-k-mooney | so we need to set it to all 0 or something like that | |
| 16:57:53 | dansmith | I mean, that's one way to go, but it's not really reasonable I think | |
| 16:57:58 | dansmith | because that makes it not an FK | |