Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-24
16:54:44 artom So there's a window between doing the claim and setting the status when the resource tracker periodic can kick in and miss the migration
16:55:07 artom And free the incoming instance's resources because it's still on the source according to the DB
16:55:10 mriedem compared to the cold migrate / resize of https://www.health.harvard.edu/blog/common-anticholinergic-drugs-like-benadryl-linked-increased-dementia-risk-201501287667
16:55:12 mriedem oops
16:55:12 mriedem ha
16:55:18 artom Dementia indeed
16:55:22 mriedem 'pre-migrating'
16:55:40 artom I guess now we know why pre-migrating exists :)
16:55:58 mriedem https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L4408-L4412
16:56:18 artom Yep, saw that
16:56:19 sean-k-mooney pre-migration is the default status no?
16:56:26 sean-k-mooney at least for cold migraitons
16:56:37 artom sean-k-mooney, it's one giant mess
16:56:37 mriedem artom: ok so i guess just remove 'accepted' from that db api query and see if ya.wang can reproduce with his test script right?
16:57:02 artom mriedem, as an initial smoke test, yeah, but I'm not sure that's the correct permanent solution
16:57:39 mriedem why? if you haven't realized yet, the migration status stuff and the rt is all very tightly coupled
16:57:55 artom Well...
16:57:56 sean-k-mooney mriedem: i like that comment :) not the reason why its needed but i like the phrasing
16:58:04 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Rework the PCI passthrough guides https://review.opendev.org/635243
16:58:04 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Remove 'adv-config', 'system-admin' subdocs https://review.opendev.org/684402
16:58:05 artom The migration status is technically part of our API, no?
16:58:26 artom And... even it isn't - as you said, it's all tightly coupled, I'd be afraid to break something else
16:58:55 artom Ideally we would implement your TODO
16:59:08 artom Though I suppose one doens't hurt the other
16:59:11 sean-k-mooney if we removed accepted we would need to add it to migration_get_in_progress_by_instance right
16:59:16 mriedem artom: "accepted" is only set while a live migration is in-progress
16:59:22 mriedem and that api is for getting in-progress migrations
16:59:34 mriedem the only reason it didn't matter before was because the RT filtered out anything that wasn't evacuate/resize/cold migrate
16:59:59 artom mriedem, that's a good point
17:00:15 mriedem artom: specifically this v
17:00:16 mriedem https://review.opendev.org/#/c/634606/86/nova/compute/resource_tracker.py
17:00:21 mriedem _is_trackable_migration
17:00:41 mriedem so accepted is definitely an in-progress live migration and something we track now
17:00:42 artom Yeah, I remember that
17:00:50 artom Yep, fair enough
17:01:02 artom If we removed that method entirely we don't need to filter out accepted anymore
17:01:14 sean-k-mooney should we be adding accepted here too https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L4438
17:01:48 sean-k-mooney it would be kind of nice if we had 1 list
17:01:56 sean-k-mooney that was shared between those two
17:02:38 mriedem sean-k-mooney: idk, pre-migrating isn't in that list either
17:03:06 sean-k-mooney its not clear that pre-migration means its in progress
17:03:30 ozzzo is this not the correct channel to ask for help with Nova?
17:03:33 artom sean-k-mooney, so, I tracked it up the rabbit hole, that one's only used in the server migrations API
17:03:35 sean-k-mooney you can end up in pre-migration if you do an evac to the same host
17:03:41 mriedem a lot of that crap was added for the /servers/{server_id}/migration apis added awhile back so you could list in-progress live migrations specifically and cancel/force-complete them
17:03:53 mriedem so i wouldn't be surprised if a lot of this is biased to live migratoin only
17:04:03 mriedem artom: yup ^
17:04:43 artom ozzzo, if you're looking for help running/deploying Nova, #openstack in the place
17:04:53 artom ozzzo, this channel is for development (see /topic)
17:05:00 ozzzo no, nobody ever answers a question in #openstack
17:05:05 ozzzo I tried that a few weeks ago
17:05:30 ozzzo I've been idling in there and I see noobs come in asking questions every day, and nobody ever answers
17:05:46 sean-k-mooney ozzzo: if we are not too busy we sometime help if its not a specifc deployment issue
17:05:51 ozzzo they don't even recommend another channel, just silence
17:05:52 sean-k-mooney ozzzo: what is the question?
17:06:02 mriedem ozzzo: sounds like an opportunity for you to be the change in that channel :)
17:06:05 ozzzo after rebuilding controllers in my dev cluster, is is possible to re-register the existing HV, or do I need to rebuild them?
17:06:33 dansmith ozzzo: see topic
17:07:07 sean-k-mooney if you start and stop the compute agents it shoudl recreate the compute service recorred but it wont rebuild any instance info
17:07:36 sean-k-mooney for a dev environment its normally simpler to redeploy as its only a hand full of nodes
17:08:07 sean-k-mooney ozzzo: if you are using devstack you can often just do a git pull and restart the service instead of unstacking and stacking again
17:08:27 ozzzo we
17:08:32 ozzzo re using ansible-openstack
17:08:35 ozzzo I tried stop/starting services, and even rebooting, but they don't re-register
17:08:50 dansmith ozzzo: maybe you and sean-k-mooney can go to #openstack to discuss further
17:09:15 sean-k-mooney i was going to suggest the openstack ansibel channel
17:09:18 dansmith or that
17:09:26 ozzzo Failed to create resource provider <hostname>
17:09:44 ozzzo ok I'll try asking there, thank you for the advice
17:09:52 kashyap Occasionally having a user question discussed here is not really some blasphemy.
17:09:58 sean-k-mooney ya i think i know what the issue. is. the compute service uuid has change but you are using the same hostnames
17:10:26 sean-k-mooney so the placmet resouce provide exists with the hostname and old uuid
17:10:40 sean-k-mooney and when nova tries to create it again it gets that failure
17:11:27 sean-k-mooney thats a known failure mode. you need to just delete the old resouce providers
17:11:39 dansmith sean-k-mooney: he just asked in #openstack
17:11:44 dansmith sean-k-mooney: maybe you could answer him there
17:11:56 dansmith you know, generate some traffic over there, help wake people up
17:11:58 sean-k-mooney ok ya im not there but i can
17:12:33 sean-k-mooney also im on PTO for the next 3 days ill only be online for another 40 mins or so
17:12:58 sean-k-mooney am is there anything i should look at before i finish up today?
17:13:45 ozzzo where can I see the old resources; are they openstack services?
17:20:25 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Make nova.compute.rpcapi.ComputeAPI.router a singleton https://review.opendev.org/684405
17:20:26 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Func test for migrate reschedule with pinned compute rpc https://review.opendev.org/684406
17:20:26 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Handle legacy request spec dict in ComputeTaskManager._cold_migrate https://review.opendev.org/684407
17:23:30 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional tests for [cinder]/cross_az_attach=False https://review.opendev.org/684141
17:39:11 openstackgerrit Artom Lifshitz proposed openstack/nova master: Stop filtering out 'accepted' for in-progress migrations https://review.opendev.org/684409
18:14:22 KeithMnemonic mriedem,sean-k-mooney: What was not fixed is perception. The 3 items needed are not something we do by default in our release
18:14:47 KeithMnemonic so for the customer since they do they after install, it was their view it was not fixed
18:15:01 mriedem ack
18:20:25 KeithMnemonic and ideally I think their "pie in the sky" is for the amount of storage/usage on their backend to be used in anything that shows consumption or checks free space
18:24:18 openstackgerrit Merged openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948
18:24:41 mriedem KeithMnemonic: let me guess, they are using ceph?
18:25:02 mriedem and confused about nova saying they have $num_computes x total ceph pool size of storage?
18:25:37 mriedem KeithMnemonic: or you mean to use cinder for all storage rather than local disk on the compute hosts, e.g. volume-backed everywhere?
18:26:08 mriedem the latter has details in https://review.opendev.org/#/c/511965/
18:26:26 mriedem the former https://review.opendev.org/#/c/650188/
18:26:39 KeithMnemonic second, they needed large root volumes, so until they applied the 3 steps, they did not understand why a backend with 50 TB was complaining about no space ;-)
18:27:39 mriedem tbc i think the only thing you really need in rocky to properly track disk usage for volume-backed servers in nova is to disable the DiskFilter
18:28:11 mriedem that removes the root disk filtering and claims during scheduling/build and what the resource tracker reports out of the hypervisors API
18:28:25 mriedem the zero_disk_flavor policy thing is more tied to a security issue than anything

Earlier   Later