Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-19
14:27:29 mriedem to see that they are going in order
14:27:38 mriedem since it was a clean cherry-pick i just fixed it using the gerrit ui
14:27:45 tosky yep, noticed
14:28:01 mriedem alex_xu: this functional test race failure is spiking http://status.openstack.org/elastic-recheck/#1856902
14:28:09 mriedem i'm inclined to skip that bug until someone can post a fix
14:28:18 mriedem stephenfin: ^ any thoughts on that?
14:36:12 shilpasd mriedem:hi
14:36:34 mriedem oh is see the bug
14:37:21 shilpasd mriedem: for https://review.opendev.org/#/c/694462/3/nova/db/sqlalchemy/api.py@2010, i have made changes suggested, but its returning list of tuples
14:37:47 mriedem stephenfin: i think the issue is the delete server helper in the libvirt functional tests is no longer waiting for the server to be gone
14:37:56 mriedem probably due to the big refactoring going on
14:38:59 mriedem shilpasd: ok, i guess you'd have to debug it
14:39:26 mriedem stephenfin: or this was just always racy and we're hitting fast nodes
14:42:49 shilpasd mriedem: can you please check 'http://paste.openstack.org/show/78779' lines 40 and 41
14:43:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix race in test_create_servers_with_vpmem https://review.opendev.org/699997
14:43:21 mriedem stephenfin: melwitt: ^
14:43:21 shilpasd mriedem: i need to iterate again to have list of uuids
14:43:46 mriedem that paste doesn't work
14:45:54 shilpasd mriedem: means its wrong approach?
14:46:09 mriedem i mean your paste url is broken
14:47:18 shilpasd http://paste.openstack.org/show/787793/
14:47:28 shilpasd corrected one
14:48:24 mriedem i don't know if filter(and_(...)) works the same as filter_by chained together, i generally have to play with sqlalchemy orm stuff when doing things like this
14:48:28 mriedem which is what i meant by debugging
14:49:38 shilpasd okay, will do that and check
14:50:02 shilpasd tried filter_by but how 'in_' parameter works with filter_by
14:50:20 shilpasd models.BlockDeviceMapping.instance_uuid.in_(instance_uuids)
14:57:04 mriedem shilpasd: if i knew the answer off the top of my head i'd tell you
14:57:08 mriedem you're going to have to play with it
14:57:11 mriedem as would i
14:59:47 shilpasd mriedem: yes, will do it
15:04:50 mriedem shilpasd: this works for me http://paste.openstack.org/show/787794/
15:07:17 shilpasd mriedem: thanks will add, this is more better that what i had
15:07:22 shilpasd will check this
15:22:38 shilpasd mriedem: it failing if we have list items more than 1, will look further how to resolve this
15:23:23 shilpasd mriedem: and regarding deletion of TC 'https://review.opendev.org/#/c/694462/3/nova/tests/unit/db/test_db_api.py@993'
15:24:06 shilpasd i have removed since observed that deleted instance are not considered in 'os-simple-tenant-usage'
15:24:09 shilpasd so removed
15:24:44 tosky mriedem: re the devstack-plugin-nfs job backport: I think it should work flawlessly up to pike, but I'm a bit unsure what to do with ocata
15:25:11 tosky because devstack/ocata lacks most of the infrastructure for zuul v3 jobs (mainly roles)
15:26:06 tosky would it be fine to copy the legacy job in-tree? After cleaning up the legacy job also in cinder, I'd like to remove it from project-config
15:29:48 stephenfin mriedem: Skip it
15:30:20 stephenfin I'm in a meeting rn but I can work on switching that test to use events instead of polling status, since that should be less racey
15:31:09 mriedem stephenfin: i already have a fix posted https://review.opendev.org/699997
15:31:44 mriedem tosky: i would just remove it from queens, pike and ocata
15:32:01 mriedem it's a non-voting experimental queue job so it doesn't get noticed anyway, and those branches are in extended maintenance mode
15:32:27 tosky mriedem: oh, ok, thanks
15:32:43 mriedem shilpasd: you shouldn't remove db api test code just because of how one caller is going to use this
15:33:08 mriedem shilpasd: you have to stop thinking about the db api layer for this as something solely for the simple tenant usage stuff, i.e. not using the active_by_window_joined method for testing it
15:33:08 stephenfin I wonder why that suddenly changed. None of my changes touched that file :/
15:33:13 stephenfin Except for 7ae1a1091326292b02d879ae5410d5e7e78d3997 but that's unrelated
15:33:21 mriedem stephenfin: maybe just hitting faster nodes
15:33:27 stephenfin Could be
15:33:30 mriedem when it fails it's on some rax nodes
15:33:38 mriedem fewer noisy neighbors right now
15:33:44 stephenfin +W'd anyway
15:33:46 stephenfin nice find
15:41:02 stephenfin mriedem_afk: Should I figure out some way to break up https://review.opendev.org/#/c/696514/ to get it over the line?
16:33:23 openstackgerrit Shilpa Devharakar proposed openstack/nova master: Handle new is_volume_backend join column query https://review.opendev.org/694462
16:33:23 openstackgerrit Shilpa Devharakar proposed openstack/nova master: Instance object changes for the new 'is_volume_backed' expected_attr https://review.opendev.org/694463
16:33:24 openstackgerrit Shilpa Devharakar proposed openstack/nova master: Ignore root_gb if instance is booted from volume https://review.opendev.org/612626
16:35:12 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Block rebuild when NUMA topology changed https://review.opendev.org/698530
16:35:13 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Disable NUMATopologyFilter on rebuild https://review.opendev.org/698532
16:35:13 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/700021
16:35:35 stephenfin artom: Respun that to address your comments. Could you take another look? ^
16:36:12 stephenfin lyarwood: You'll need to hit that too, if that's okay? I backported an additional change (https://review.opendev.org/700021) to avoid a merge conflict
16:36:57 artom stephenfin, thank you! Lemme finish a thing, then I'll do my due diligence and +1
16:37:14 stephenfin once mriedem_afk is back, ima bug him to hit the series too since I'm not sure what other stable core can review that stuff
16:40:14 aarents lyarwood: if you may have a look on that https://review.opendev.org/#/c/696084/ Kashyap also, it will be helpfull !
16:45:20 artom stephenfin, I think you got the wrong changeID on https://review.opendev.org/#/c/698530/3
16:45:55 stephenfin gdi
16:47:54 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Block rebuild when NUMA topology changed https://review.opendev.org/698530
16:47:54 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Disable NUMATopologyFilter on rebuild https://review.opendev.org/698532
17:45:21 openstackgerrit Mykola Yakovliev proposed openstack/nova master: Fix boot_roles in InstanceSystemMetadata https://review.opendev.org/698040
18:33:54 openstackgerrit Merged openstack/nova master: Fix race in test_create_servers_with_vpmem https://review.opendev.org/699997
19:22:27 gmann stephenfin ping
20:09:22 openstackgerrit Matt Riedemann proposed openstack/nova master: Ensure source service is up before resizing/migrating https://review.opendev.org/700062
20:25:18 openstackgerrit Matt Riedemann proposed openstack/nova master: Ensure source service is up before resizing/migrating https://review.opendev.org/700062
20:55:29 mriedem artom: stephenfin: since you summoned me https://review.opendev.org/#/c/698530/
21:02:13 mriedem and https://review.opendev.org/#/c/698532/
21:02:35 mriedem do we want to wait for https://review.opendev.org/#/c/698260/ to get cleaned up and backported on top of the train backports? because if so, someone is likely going to have to clean up https://review.opendev.org/#/c/698260/ for sean
21:14:28 openstackgerrit Matt Riedemann proposed openstack/nova master: FUP for in-place numa rebuild https://review.opendev.org/698260
21:14:44 mriedem done ^ let's get that approved on master and stacked on top of the backports
21:36:42 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: pike-only: remove broken non-voting ceph jobs https://review.opendev.org/700072
21:45:15 TheJulia Are there any known issues with nova-compute on the master branch right now?
21:45:37 TheJulia like... ones that might show up with grenade running
21:46:09 mriedem that's pretty broad
21:46:32 mriedem you have a grenade job that rebalances ironic nodes between 2 computes right?
21:46:51 mriedem there is a known race issue with that https://review.opendev.org/#/c/694802/
21:47:05 mriedem https://bugs.launchpad.net/nova/+bug/1841481
21:47:05 openstack Launchpad bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
21:49:23 TheJulia mriedem: single node stack where nova-compute seems to die after being upgraded and started up for a little while
21:50:09 TheJulia mriedem: but, I'm only asking broadly because I've only glanced at the logs enough to have a feeling it is nova-compute. I've not gone back and gone through everything comparing out the smoke test times yet
21:50:28 TheJulia hence, why I asked if there is anything known before I go dig for a few hours
21:50:55 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: ocata-only: drop non-voting ceph job https://review.opendev.org/700077
21:51:02 mriedem then not that i know of
21:51:29 TheJulia mriedem: for context, two grenade jobs, one does a full upgrade, the other does a partial upgrade for historical reasons and that breaks most cycles leading to me wanting to drink.
21:51:38 TheJulia Okay, yippie!
21:51:39 TheJulia :)
22:26:57 TheJulia hmm, in rescheduling for post upgrade smoke testing

Earlier   Later