Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-05-14
16:46:47 andrewbonney The fact there's no DB cursor means it hits the same un-migratable records on each iteration
16:47:39 sean-k-mooney well node here https://opendev.org/openstack/nova/blame/commit/7096423b343ffce9622fd078fc2b3a87fd3386f7/nova/objects/instance.py#L1366
16:47:43 sean-k-mooney is not coming form the instnace
16:48:41 sean-k-mooney that is looking up the compute node record
16:49:04 sean-k-mooney instnace.node i think is the compute node name
16:49:50 dansmith yeah I don't understand the question.. we're not "relying" on exception handling for anything in that code
16:50:02 dansmith ... and we do loop ...
16:51:09 sean-k-mooney my incliantion is to say that of the 30k instances a large proporation fo those recored are for deleted instnaces?
16:51:12 sean-k-mooney thats a guess
16:51:20 andrewbonney Yes that's right
16:51:35 andrewbonney I think this is only an issue where lots of records are not running instances
16:52:15 andrewbonney Do you end up in a state where the method exits because the first 50 records returned by the DB don't qualify for migration, which triggers the end of the loop
16:52:19 andrewbonney *So
16:52:25 sean-k-mooney so a workaround in that case would be to run nova-manage archive-deleted-rows
16:52:27 dansmith sorry I missed something, what is the issue ?
16:53:06 sean-k-mooney when they run the commadn its printing "50 rows matched query populate_instance_compute_id, 6 migrated"
16:53:21 dansmith oh I see above, you're expecting to be able to run just --max-count $number_of_active_instances ?
16:53:21 sean-k-mooney https://bugs.launchpad.net/nova/+bug/2065403
16:53:43 dansmith yeah I mean you have to run those things untll they're done
16:54:03 dansmith sometimes we can skip records that are deleted and other times we can't really
16:54:08 andrewbonney We ran without max-count initially as part of a standard upgrade. It completed, but hasn't migrated everything because once the first 50 records are unsuitable it exits
16:54:49 andrewbonney I only realised our first deployment was in this state because I noticed the migration taking ~15 mins plus on a second deployment, because it keeps trying to migrate unsuitable records
16:55:00 dansmith ah, I see because you have a bunch of (hopefully deleted) instances that point to missing compute nodes?
16:55:07 andrewbonney Yeah that's it
16:55:11 dansmith gotcha
16:55:37 andrewbonney Happy to work around it for us, but I doubt we'll be the only ones to hit it
16:55:59 sean-k-mooney so just looking at our docs https://docs.openstack.org/nova/latest/cli/nova-manage.html#db-online-data-migrations
16:56:11 sean-k-mooney we state that if you dont pass --max-count
16:56:11 dansmith yeah, I'm not sure the best way to handle that in the context of how that manage command works
16:56:21 sean-k-mooney we will loop internanlly in batchs of 50
16:56:36 dansmith yeah, but we don't have marker stuff there to prevent us from looping over the same 50 over and over
16:56:42 sean-k-mooney but if you do pass --max-count you have to loop externally and look at the repsoce code
16:56:54 sean-k-mooney well we dont actully need that
16:57:13 dansmith so I mean, we could just ignore deleted instances, but that means we may have a problem adding a constraint later where this will come up again
16:57:14 sean-k-mooney since we only get instnace that dont have the compute_id set
16:57:30 sean-k-mooney but the issue is we are not settign a sentinel for those that dont match
16:57:33 dansmith right but the problem is that if the 50 are not migratable, we'll never mark them as identified and keey finding them
16:57:38 dansmith yeah
16:57:49 sean-k-mooney so we need to set it to all 0 or something like that
16:57:53 dansmith I mean, that's one way to go, but it's not really reasonable I think
16:57:58 dansmith because that makes it not an FK
16:58:18 sean-k-mooney oh right ya it would fail the FK check
16:58:47 sean-k-mooney can we return the instance uuid as a marker if we add an orderby
16:58:57 dansmith I mean, it's not marked as an FK right now, because we suck at that, but still, it's "wrong" I think
16:59:12 dansmith right, so we could add in marker stuff there, but that's kindof a bigger deal really
16:59:24 dansmith changes the interface of all of these migration routines
16:59:38 sean-k-mooney well it woudl need a new cli arg for a start and it woudl chnage the resonce so not backporable
17:00:09 dansmith one thing we could do is if we get to the bottom of this loop and couldn't migrate any of the things we found, just raise an exception and say "yo dawg, you need to archive your db"
17:00:28 dansmith it's not a CLI thing so much as the interface for these migration functions
17:00:34 dansmith they all return all,hit right now
17:01:50 dansmith I need to be reviewing the monster EE spec, so let's just table this and I
17:01:56 dansmith will do some thinking on it later
17:03:21 andrewbonney Thanks for the time
#openstack-nova - 2024-05-15
03:08:09 stan is there any way to connect the vm instance CPU and memory usage?
08:28:07 opendevreview Rajesh Tailor proposed openstack/nova master: Fix typos and punctuation issues https://review.opendev.org/c/openstack/nova/+/878779
14:19:03 opendevreview Dan Smith proposed openstack/nova master: Enable OCaaS for several nova jobs https://review.opendev.org/c/openstack/nova/+/919738
14:19:04 opendevreview Dan Smith proposed openstack/nova master: Stop using split UEC image (mostly) https://review.opendev.org/c/openstack/nova/+/919739
14:25:40 sean-k-mooney dansmith: OCasS is openstack client as a service right? creating a single cleint instnace and reusing it to make queies
14:25:48 dansmith yes
14:26:07 sean-k-mooney when did that merge cause i rememebr that beign a thing in the works for years
14:26:19 dansmith saves 5m of wasted wall clock time alone on a simple devstack run, not to mention the cpu and disk
14:26:37 dansmith sean-k-mooney: yesterday, I just freshened it up and fixed some stuff and we shoved it in
14:26:48 sean-k-mooney ok cool
14:28:15 sean-k-mooney ill review both those once the ci runs
14:28:39 sean-k-mooney dansmith: can you hit this https://review.opendev.org/c/openstack/nova/+/915562
14:28:53 sean-k-mooney there is a placement one too
14:29:09 sean-k-mooney here https://review.opendev.org/c/openstack/placement/+/915568
14:30:52 dansmith gladly
15:08:26 opendevreview mike_mp@zzzcomputing.com proposed openstack/nova master: do not use str(url) to stringify a URL for subsequent use https://review.opendev.org/c/openstack/nova/+/913910
15:08:30 chris218 Hi I'm reading about CPU topologies and pinning in the docs, from the looks of it if I configure few CPUs as dedicated I won't be able to use them as shared if I run out CPUs specified as shared?
15:10:08 gibi chris218: you understand it correctly. You need to statically split your physical CPUs between shared, dedicated, reserved for the os
15:11:45 chris218 So there isn't a way to tell nova to treat all the CPUs as shared or dedicated depending on what the users will request?
15:20:33 opendevreview Alexey Stupnikov proposed openstack/nova master: [doc] Improve description for nova-manage db purge https://review.opendev.org/c/openstack/nova/+/919746
15:24:48 opendevreview Stephen Finucane proposed openstack/nova master: do not use str(url) to stringify a URL for subsequent use https://review.opendev.org/c/openstack/nova/+/913910
15:25:32 stephenfin melwitt: Fixed the test ^
15:26:57 melwitt stephenfin: ack, will look
15:30:16 stephenfin bauzas: The OpenAPI spec has 2 +2s but I think folks are waiting on you to review it again. Could you do that or indicate if we can merge as-is? https://review.opendev.org/c/openstack/nova-specs/+/909448
15:30:33 bauzas stephenfin: I'll try for sure
16:51:18 opendevreview Merged openstack/nova master: Remove SQLAlchemy tips jobs https://review.opendev.org/c/openstack/nova/+/915562
16:51:24 opendevreview Merged openstack/placement master: Remove SQLAlchemy tips jobs https://review.opendev.org/c/openstack/placement/+/915568
17:29:22 sean-k-mooney dansmith: is there a reason we are not setting openstack-cli-server: true in the base tempest job?
17:29:39 sean-k-mooney well or base devstack job
17:29:41 dansmith sean-k-mooney: just wanted to be gradual
17:29:50 dansmith oh you mean in the other repo or ours?
17:30:02 sean-k-mooney in the ohter repo
17:30:09 sean-k-mooney but if you want to test it in ours first
17:30:13 dansmith yeah, I don't think the owners are quite ready for that
17:30:13 sean-k-mooney then im ok with that
17:30:25 dansmith I think we should lead the way here and show that it's stable and then we can argue for it
17:30:28 sean-k-mooney ok just confiming
17:31:22 sean-k-mooney i was checkign the local.conf in tempest-integrated-compute and realised it was not set
17:31:35 sean-k-mooney then i realsed you only enbaled it in some of the jobs and i was trying to figure out why
17:37:28 clarkb dansmith: nova-next's devstack runtime was just under 11 minutes with your change. Thats pretty impressive
17:38:39 dansmith clarkb: yeah, I saw a super fast nova-live migration too
17:38:53 JayF clarkb: fyi I have https://review.opendev.org/c/openstack/ironic/+/919620 up as well, harder to tell w/ironic b/c our times vary so much per cloud
17:38:55 sean-k-mooney 39 mins
17:39:03 JayF with a ty to Dan for enabling that in Ironic
17:39:11 sean-k-mooney i -1 the patch because your current patch is only setting it on the conotler nodes not the compute nodes
17:39:21 sean-k-mooney but over all it looks good
17:40:47 clarkb JayF: the devstack log file logs a bunch of devstack specific runtime info at the end

Earlier   Later