| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-24 | |||
| 16:51:59 | artom | mriedem, so, I think I figured out the NUMA LM thing | |
| 16:52:56 | artom | We filter out 'accepted' migrations in https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L4422 | |
| 16:53:08 | artom | (That's called from the update resources periodic task | |
| 16:53:10 | artom | ) | |
| 16:54:10 | artom | But we only set the 'preparing' status here: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7020 | |
| 16:54:23 | mriedem | and we set accepted in the live migration task | |
| 16:54:23 | artom | (It's created with 'accepted' by the conductor) | |
| 16:54:44 | artom | So there's a window between doing the claim and setting the status when the resource tracker periodic can kick in and miss the migration | |
| 16:55:07 | artom | And free the incoming instance's resources because it's still on the source according to the DB | |
| 16:55:10 | mriedem | compared to the cold migrate / resize of https://www.health.harvard.edu/blog/common-anticholinergic-drugs-like-benadryl-linked-increased-dementia-risk-201501287667 | |
| 16:55:12 | mriedem | ha | |
| 16:55:12 | mriedem | oops | |
| 16:55:18 | artom | Dementia indeed | |
| 16:55:22 | mriedem | 'pre-migrating' | |
| 16:55:40 | artom | I guess now we know why pre-migrating exists :) | |
| 16:55:58 | mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L4408-L4412 | |
| 16:56:18 | artom | Yep, saw that | |
| 16:56:19 | sean-k-mooney | pre-migration is the default status no? | |
| 16:56:26 | sean-k-mooney | at least for cold migraitons | |
| 16:56:37 | mriedem | artom: ok so i guess just remove 'accepted' from that db api query and see if ya.wang can reproduce with his test script right? | |
| 16:56:37 | artom | sean-k-mooney, it's one giant mess | |
| 16:57:02 | artom | mriedem, as an initial smoke test, yeah, but I'm not sure that's the correct permanent solution | |
| 16:57:39 | mriedem | why? if you haven't realized yet, the migration status stuff and the rt is all very tightly coupled | |
| 16:57:55 | artom | Well... | |
| 16:57:56 | sean-k-mooney | mriedem: i like that comment :) not the reason why its needed but i like the phrasing | |
| 16:58:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Remove 'adv-config', 'system-admin' subdocs https://review.opendev.org/684402 | |
| 16:58:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Rework the PCI passthrough guides https://review.opendev.org/635243 | |
| 16:58:05 | artom | The migration status is technically part of our API, no? | |
| 16:58:26 | artom | And... even it isn't - as you said, it's all tightly coupled, I'd be afraid to break something else | |
| 16:58:55 | artom | Ideally we would implement your TODO | |
| 16:59:08 | artom | Though I suppose one doens't hurt the other | |
| 16:59:11 | sean-k-mooney | if we removed accepted we would need to add it to migration_get_in_progress_by_instance right | |
| 16:59:16 | mriedem | artom: "accepted" is only set while a live migration is in-progress | |
| 16:59:22 | mriedem | and that api is for getting in-progress migrations | |
| 16:59:34 | mriedem | the only reason it didn't matter before was because the RT filtered out anything that wasn't evacuate/resize/cold migrate | |
| 16:59:59 | artom | mriedem, that's a good point | |
| 17:00:15 | mriedem | artom: specifically this v | |
| 17:00:16 | mriedem | https://review.opendev.org/#/c/634606/86/nova/compute/resource_tracker.py | |
| 17:00:21 | mriedem | _is_trackable_migration | |
| 17:00:41 | mriedem | so accepted is definitely an in-progress live migration and something we track now | |
| 17:00:42 | artom | Yeah, I remember that | |
| 17:00:50 | artom | Yep, fair enough | |
| 17:01:02 | artom | If we removed that method entirely we don't need to filter out accepted anymore | |
| 17:01:14 | sean-k-mooney | should we be adding accepted here too https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L4438 | |
| 17:01:48 | sean-k-mooney | it would be kind of nice if we had 1 list | |
| 17:01:56 | sean-k-mooney | that was shared between those two | |
| 17:02:38 | mriedem | sean-k-mooney: idk, pre-migrating isn't in that list either | |
| 17:03:06 | sean-k-mooney | its not clear that pre-migration means its in progress | |
| 17:03:30 | ozzzo | is this not the correct channel to ask for help with Nova? | |
| 17:03:33 | artom | sean-k-mooney, so, I tracked it up the rabbit hole, that one's only used in the server migrations API | |
| 17:03:35 | sean-k-mooney | you can end up in pre-migration if you do an evac to the same host | |
| 17:03:41 | mriedem | a lot of that crap was added for the /servers/{server_id}/migration apis added awhile back so you could list in-progress live migrations specifically and cancel/force-complete them | |
| 17:03:53 | mriedem | so i wouldn't be surprised if a lot of this is biased to live migratoin only | |
| 17:04:03 | mriedem | artom: yup ^ | |
| 17:04:43 | artom | ozzzo, if you're looking for help running/deploying Nova, #openstack in the place | |
| 17:04:53 | artom | ozzzo, this channel is for development (see /topic) | |
| 17:05:00 | ozzzo | no, nobody ever answers a question in #openstack | |
| 17:05:05 | ozzzo | I tried that a few weeks ago | |
| 17:05:30 | ozzzo | I've been idling in there and I see noobs come in asking questions every day, and nobody ever answers | |
| 17:05:46 | sean-k-mooney | ozzzo: if we are not too busy we sometime help if its not a specifc deployment issue | |
| 17:05:51 | ozzzo | they don't even recommend another channel, just silence | |
| 17:05:52 | sean-k-mooney | ozzzo: what is the question? | |
| 17:06:02 | mriedem | ozzzo: sounds like an opportunity for you to be the change in that channel :) | |
| 17:06:05 | ozzzo | after rebuilding controllers in my dev cluster, is is possible to re-register the existing HV, or do I need to rebuild them? | |
| 17:06:33 | dansmith | ozzzo: see topic | |
| 17:07:07 | sean-k-mooney | if you start and stop the compute agents it shoudl recreate the compute service recorred but it wont rebuild any instance info | |
| 17:07:36 | sean-k-mooney | for a dev environment its normally simpler to redeploy as its only a hand full of nodes | |
| 17:08:07 | sean-k-mooney | ozzzo: if you are using devstack you can often just do a git pull and restart the service instead of unstacking and stacking again | |
| 17:08:27 | ozzzo | we | |
| 17:08:32 | ozzzo | re using ansible-openstack | |
| 17:08:35 | ozzzo | I tried stop/starting services, and even rebooting, but they don't re-register | |
| 17:08:50 | dansmith | ozzzo: maybe you and sean-k-mooney can go to #openstack to discuss further | |
| 17:09:15 | sean-k-mooney | i was going to suggest the openstack ansibel channel | |
| 17:09:18 | dansmith | or that | |
| 17:09:26 | ozzzo | Failed to create resource provider <hostname> | |
| 17:09:44 | ozzzo | ok I'll try asking there, thank you for the advice | |
| 17:09:52 | kashyap | Occasionally having a user question discussed here is not really some blasphemy. | |
| 17:09:58 | sean-k-mooney | ya i think i know what the issue. is. the compute service uuid has change but you are using the same hostnames | |
| 17:10:26 | sean-k-mooney | so the placmet resouce provide exists with the hostname and old uuid | |
| 17:10:40 | sean-k-mooney | and when nova tries to create it again it gets that failure | |
| 17:11:27 | sean-k-mooney | thats a known failure mode. you need to just delete the old resouce providers | |
| 17:11:39 | dansmith | sean-k-mooney: he just asked in #openstack | |
| 17:11:44 | dansmith | sean-k-mooney: maybe you could answer him there | |
| 17:11:56 | dansmith | you know, generate some traffic over there, help wake people up | |
| 17:11:58 | sean-k-mooney | ok ya im not there but i can | |
| 17:12:33 | sean-k-mooney | also im on PTO for the next 3 days ill only be online for another 40 mins or so | |
| 17:12:58 | sean-k-mooney | am is there anything i should look at before i finish up today? | |
| 17:13:45 | ozzzo | where can I see the old resources; are they openstack services? | |
| 17:20:25 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Make nova.compute.rpcapi.ComputeAPI.router a singleton https://review.opendev.org/684405 | |
| 17:20:26 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Handle legacy request spec dict in ComputeTaskManager._cold_migrate https://review.opendev.org/684407 | |
| 17:20:26 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Func test for migrate reschedule with pinned compute rpc https://review.opendev.org/684406 | |
| 17:23:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional tests for [cinder]/cross_az_attach=False https://review.opendev.org/684141 | |
| 17:39:11 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Stop filtering out 'accepted' for in-progress migrations https://review.opendev.org/684409 | |
| 18:14:22 | KeithMnemonic | mriedem,sean-k-mooney: What was not fixed is perception. The 3 items needed are not something we do by default in our release | |
| 18:14:47 | KeithMnemonic | so for the customer since they do they after install, it was their view it was not fixed | |
| 18:15:01 | mriedem | ack | |
| 18:20:25 | KeithMnemonic | and ideally I think their "pie in the sky" is for the amount of storage/usage on their backend to be used in anything that shows consumption or checks free space | |
| 18:24:18 | openstackgerrit | Merged openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948 | |
| 18:24:41 | mriedem | KeithMnemonic: let me guess, they are using ceph? | |
| 18:25:02 | mriedem | and confused about nova saying they have $num_computes x total ceph pool size of storage? | |