Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-20
11:55:24 lyarwood after 5:30
11:55:36 lyarwood I'm just burnt out today
11:55:48 sean-k-mooney then have a lazy friday
11:56:34 stephenfin bauzas: You about? Want to rubberstamp this for me, please? https://review.opendev.org/#/c/713611/
12:23:08 lyarwood gibi: https://review.opendev.org/#/c/714057/ if you have bandwidth can you check this fup to the nova-live-migration fixes yesterday.
12:26:10 lyarwood elod: https://review.opendev.org/#/q/topic:bug/1867380+(status:open+OR+status:merged) - would you mind going through the rest of this series so we can unblock stable/pike
12:29:16 elod lyarwood: just commented one
12:30:36 elod lyarwood: but otherwise it would be good to push through these soon :) and thanks for the fixes!
12:30:41 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713839
12:30:41 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713840
12:30:44 lyarwood elod: yup thanks, addressed above ^
12:31:15 elod lyarwood: cool, thanks!
12:40:52 artom stephenfin, yep, thanks - lemme coffee up and read that
12:45:59 gibi lyarwood: done. Thanks
13:31:15 luyao dansmith, artom : are you about?
13:31:33 artom lyarwood, I am, dansmith probably isn't, given the time zone
13:32:19 artom Err, luyao ^^
13:33:06 luyao artom: I replied on https://review.opendev.org/#/c/687856, could you look it again
13:35:04 artom luyao, is there a specific question you'd like answered though? Reading the conversations quickly, it looks like the ball is in your court to update the patch
13:35:44 openstackgerrit Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494
13:41:34 luyao artom: I don't update my patch since dansmith have some concern on moving cleanup before rpc call and some other details I don't know which solustion is better.
13:42:27 luyao artom: I saw you mentioned functional test, I run your live migration functional test on top of my patch, they runs well
13:58:07 artom luyao, yeah, the fact that the functional test is passing is reassuring (and thanks for the review - I have it fixed locally, will push a new version soon)
14:13:28 luyao artom: If you still have questions or concerns about my reply on vpmem live migration , please leave comments there. Or you can ping me in IRC, I'll be online for about two hours I can response promply during this time.
14:13:42 artom luyao, ok, thanks
14:26:46 openstackgerrit Merged openstack/nova master: libvirt: Correctly resize encrypted LUKSv1 volumes https://review.opendev.org/706900
14:34:07 openstackgerrit Artom Lifshitz proposed openstack/nova master: Increase code reuse in test_numa_servers https://review.opendev.org/714125
14:34:31 artom stephenfin, ^^ quick side-fix for you
14:34:48 artom (as I work on NUMA LM func tests)
14:44:00 luyao dansmith: Are you about?
14:44:18 dansmith luyao: yes, but I don't really want to argue about moving that rpc call
14:45:22 dansmith I've read your comments, I understand that it may not matter right now, but what I said is still my opinion: that changing the order of critical points of synchronization is not something to take lightly
14:45:31 dansmith so, my opinion is still the same
14:51:25 luyao dansmith: it will not break anything, so I don't understand what's the risk
14:52:39 luyao dansmith: I try other solution which alex_xu help clarifing, what's you comments about those solusion
14:54:10 alex_xu maybe we should move the instance's state update to later, but that across the src and dest node, in the end, we have to deal with the upgrade issue.
14:57:24 belmoreira Hi, I need some help...
14:57:41 belmoreira nova-compute (dedicated to ironic) market all ironic nodes (compute_nodes table) and as consequence removed all the resource_providers/inventories/allocations for those nodes
14:59:09 belmoreira restarting nova-compute and removing the soft delete of the compute_nodes, the resource provider/inventory is created again
14:59:31 belmoreira but not the allocations... I remember that this was removed from nova some time ago
14:59:54 belmoreira any suggestion how I can recreate thousands of allocations?
15:00:35 dansmith belmoreira: I assume heal instance allocations, just a sec
15:00:51 belmoreira I remember that mnaser and bauzas did some kind of tool
15:01:10 belmoreira dansmith I using stein
15:01:15 dansmith in nova-manage, I think there's an instance heal
15:01:18 dansmith oh
15:02:17 bauzas placement-audit you mean ?
15:02:50 belmoreira placement-audit doesn't recreate allocations if they are missing?
15:03:05 dansmith heal_allocations is the subcmd
15:03:25 dansmith belmoreira: that was in stein
15:03:34 mnaser yeah heal allocations is the thing you need to add missing entries, placement audit is what you need to remove extra entries
15:04:54 belmoreira thanks all, let me have a look
15:28:23 mriedem i think https://review.opendev.org/#/c/678100/ is still relevant to ^ for the compute side delete of allocations there
15:28:26 mriedem but it needs love
15:33:05 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Avoid circular reference during serialization https://review.opendev.org/714148
15:39:19 belmoreira thanks dansmith, mnaser, bauzas. 4305 allocations recreated! In Stein it goes through all deployment. Needed to patch it to do it per cell (this will be a great addition to master).
15:39:28 belmoreira For some reason nova deleted all the ironic resource providers/inventories/allocations.
15:39:44 belmoreira I'm investigating what trigger this... no idea for now... I will let you know when I have more info
15:40:30 dansmith I thought it does do it per cell in master, but I'd have to go back and look
15:42:56 belmoreira it iterates per cell but is not possible to select the cell
15:43:44 dansmith ah okay
15:51:38 mnaser belmoreira: changed hostname or compute node by any chance ?
15:55:32 belmoreira mnaser maybe... I'm configuring "conductor_groups" for ironic. To avoid this precise issue I configured only one conductor group to keep the same nova-compute... and move from there. Still looking... for now I don't understand what went wrong
15:55:58 mnaser belmoreira: maybe check the compute nodes table
15:56:42 belmoreira mnaser the compute_nodes table had all the resources deleted
15:59:12 belmoreira mnaser to recover I made the entries available again (deleted=0, deleted_at=NULL)
15:59:36 belmoreira new entries are not created because the uuid conflict
15:59:49 mnaser belmoreira: pretty sure you might look at the compute logs and see that it was deleting them
16:00:07 mnaser also I think I ran into that and mriedem fixed it at the time
16:01:14 belmoreira yeah, I need to go deep into the logs... for now I'm still adding the nodes into the right placement_aggregate for them to be available
16:36:19 mriedem see that link above - if you're doing rebalancing of nodes when the old compute sees the nodes are no longer reported there it deletes them along with the providers and their allocations
16:36:51 belmoreira thanks mriedem
16:36:59 mriedem https://review.opendev.org/#/c/678100/3/nova/compute/manager.py@9208
16:37:09 mriedem that cascading delete thing was added back when the computes still self-healed the allocations in the RT
16:37:15 mriedem but that self-heal was removed a few releases ago
16:37:21 mriedem so now we just delete your stuff on re-balance
16:37:26 mriedem you're welcome
16:38:54 mriedem belmoreira: i think this is the todo you're looking for for heal_allocations yes? https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L2119
16:39:36 mriedem that's a very easy change https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L2173
16:41:33 belmoreira mriedem yes, it's easy to do... I did't under stress and just ran it :)
16:42:05 belmoreira we have a new member in the team, this is a good low hanging fruit for him
17:16:01 openstackgerrit Merged openstack/nova master: Introduce scope_types in os-flavor-access https://review.opendev.org/713559
18:11:03 openstackgerrit melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478
18:11:05 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
18:32:17 openstackgerrit melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478
18:32:18 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
18:34:56 openstackgerrit melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116
18:34:57 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
18:49:24 openstackgerrit Artom Lifshitz proposed openstack/nova master: func tests: move _run_periodics() into own mixin https://review.opendev.org/705545
18:49:25 openstackgerrit Artom Lifshitz proposed openstack/nova master: tests: work around malformed serial XML https://review.opendev.org/705546
18:49:25 openstackgerrit Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595
18:49:26 openstackgerrit Artom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146 https://review.opendev.org/687404
18:49:27 openstack bug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms)
19:03:40 artom dansmith, ^^ if you have the time and energy this Friday afternoon. Next week is fine too :) I'm going to go interact with my spawns for a bit
19:30:24 openstackgerrit Artom Lifshitz proposed openstack/nova master: func tests: move _run_periodics() into own mixin https://review.opendev.org/705545
19:30:25 openstackgerrit Artom Lifshitz proposed openstack/nova master: tests: work around malformed serial XML https://review.opendev.org/705546
19:30:25 openstackgerrit Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595
19:30:26 openstackgerrit Artom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146 https://review.opendev.org/687404
19:30:28 openstack bug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms)
19:50:03 artom dansmith, so, you'd rather the class inheriting from the Mixin sets self.computes?

Earlier   Later