| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-19 | |||
| 14:27:38 | mriedem | since it was a clean cherry-pick i just fixed it using the gerrit ui | |
| 14:27:45 | tosky | yep, noticed | |
| 14:28:01 | mriedem | alex_xu: this functional test race failure is spiking http://status.openstack.org/elastic-recheck/#1856902 | |
| 14:28:09 | mriedem | i'm inclined to skip that bug until someone can post a fix | |
| 14:28:18 | mriedem | stephenfin: ^ any thoughts on that? | |
| 14:36:12 | shilpasd | mriedem:hi | |
| 14:36:34 | mriedem | oh is see the bug | |
| 14:37:21 | shilpasd | mriedem: for https://review.opendev.org/#/c/694462/3/nova/db/sqlalchemy/api.py@2010, i have made changes suggested, but its returning list of tuples | |
| 14:37:47 | mriedem | stephenfin: i think the issue is the delete server helper in the libvirt functional tests is no longer waiting for the server to be gone | |
| 14:37:56 | mriedem | probably due to the big refactoring going on | |
| 14:38:59 | mriedem | shilpasd: ok, i guess you'd have to debug it | |
| 14:39:26 | mriedem | stephenfin: or this was just always racy and we're hitting fast nodes | |
| 14:42:49 | shilpasd | mriedem: can you please check 'http://paste.openstack.org/show/78779' lines 40 and 41 | |
| 14:43:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix race in test_create_servers_with_vpmem https://review.opendev.org/699997 | |
| 14:43:21 | shilpasd | mriedem: i need to iterate again to have list of uuids | |
| 14:43:21 | mriedem | stephenfin: melwitt: ^ | |
| 14:43:46 | mriedem | that paste doesn't work | |
| 14:45:54 | shilpasd | mriedem: means its wrong approach? | |
| 14:46:09 | mriedem | i mean your paste url is broken | |
| 14:47:18 | shilpasd | http://paste.openstack.org/show/787793/ | |
| 14:47:28 | shilpasd | corrected one | |
| 14:48:24 | mriedem | i don't know if filter(and_(...)) works the same as filter_by chained together, i generally have to play with sqlalchemy orm stuff when doing things like this | |
| 14:48:28 | mriedem | which is what i meant by debugging | |
| 14:49:38 | shilpasd | okay, will do that and check | |
| 14:50:02 | shilpasd | tried filter_by but how 'in_' parameter works with filter_by | |
| 14:50:20 | shilpasd | models.BlockDeviceMapping.instance_uuid.in_(instance_uuids) | |
| 14:57:04 | mriedem | shilpasd: if i knew the answer off the top of my head i'd tell you | |
| 14:57:08 | mriedem | you're going to have to play with it | |
| 14:57:11 | mriedem | as would i | |
| 14:59:47 | shilpasd | mriedem: yes, will do it | |
| 15:04:50 | mriedem | shilpasd: this works for me http://paste.openstack.org/show/787794/ | |
| 15:07:17 | shilpasd | mriedem: thanks will add, this is more better that what i had | |
| 15:07:22 | shilpasd | will check this | |
| 15:22:38 | shilpasd | mriedem: it failing if we have list items more than 1, will look further how to resolve this | |
| 15:23:23 | shilpasd | mriedem: and regarding deletion of TC 'https://review.opendev.org/#/c/694462/3/nova/tests/unit/db/test_db_api.py@993' | |
| 15:24:06 | shilpasd | i have removed since observed that deleted instance are not considered in 'os-simple-tenant-usage' | |
| 15:24:09 | shilpasd | so removed | |
| 15:24:44 | tosky | mriedem: re the devstack-plugin-nfs job backport: I think it should work flawlessly up to pike, but I'm a bit unsure what to do with ocata | |
| 15:25:11 | tosky | because devstack/ocata lacks most of the infrastructure for zuul v3 jobs (mainly roles) | |
| 15:26:06 | tosky | would it be fine to copy the legacy job in-tree? After cleaning up the legacy job also in cinder, I'd like to remove it from project-config | |
| 15:29:48 | stephenfin | mriedem: Skip it | |
| 15:30:20 | stephenfin | I'm in a meeting rn but I can work on switching that test to use events instead of polling status, since that should be less racey | |
| 15:31:09 | mriedem | stephenfin: i already have a fix posted https://review.opendev.org/699997 | |
| 15:31:44 | mriedem | tosky: i would just remove it from queens, pike and ocata | |
| 15:32:01 | mriedem | it's a non-voting experimental queue job so it doesn't get noticed anyway, and those branches are in extended maintenance mode | |
| 15:32:27 | tosky | mriedem: oh, ok, thanks | |
| 15:32:43 | mriedem | shilpasd: you shouldn't remove db api test code just because of how one caller is going to use this | |
| 15:33:08 | stephenfin | I wonder why that suddenly changed. None of my changes touched that file :/ | |
| 15:33:08 | mriedem | shilpasd: you have to stop thinking about the db api layer for this as something solely for the simple tenant usage stuff, i.e. not using the active_by_window_joined method for testing it | |
| 15:33:13 | stephenfin | Except for 7ae1a1091326292b02d879ae5410d5e7e78d3997 but that's unrelated | |
| 15:33:21 | mriedem | stephenfin: maybe just hitting faster nodes | |
| 15:33:27 | stephenfin | Could be | |
| 15:33:30 | mriedem | when it fails it's on some rax nodes | |
| 15:33:38 | mriedem | fewer noisy neighbors right now | |
| 15:33:44 | stephenfin | +W'd anyway | |
| 15:33:46 | stephenfin | nice find | |
| 15:41:02 | stephenfin | mriedem_afk: Should I figure out some way to break up https://review.opendev.org/#/c/696514/ to get it over the line? | |
| 16:33:23 | openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Instance object changes for the new 'is_volume_backed' expected_attr https://review.opendev.org/694463 | |
| 16:33:23 | openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Handle new is_volume_backend join column query https://review.opendev.org/694462 | |
| 16:33:24 | openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Ignore root_gb if instance is booted from volume https://review.opendev.org/612626 | |
| 16:35:12 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Block rebuild when NUMA topology changed https://review.opendev.org/698530 | |
| 16:35:13 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/700021 | |
| 16:35:13 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Disable NUMATopologyFilter on rebuild https://review.opendev.org/698532 | |
| 16:35:35 | stephenfin | artom: Respun that to address your comments. Could you take another look? ^ | |
| 16:36:12 | stephenfin | lyarwood: You'll need to hit that too, if that's okay? I backported an additional change (https://review.opendev.org/700021) to avoid a merge conflict | |
| 16:36:57 | artom | stephenfin, thank you! Lemme finish a thing, then I'll do my due diligence and +1 | |
| 16:37:14 | stephenfin | once mriedem_afk is back, ima bug him to hit the series too since I'm not sure what other stable core can review that stuff | |
| 16:40:14 | aarents | lyarwood: if you may have a look on that https://review.opendev.org/#/c/696084/ Kashyap also, it will be helpfull ! | |
| 16:45:20 | artom | stephenfin, I think you got the wrong changeID on https://review.opendev.org/#/c/698530/3 | |
| 16:45:55 | stephenfin | gdi | |
| 16:47:54 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Disable NUMATopologyFilter on rebuild https://review.opendev.org/698532 | |
| 16:47:54 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Block rebuild when NUMA topology changed https://review.opendev.org/698530 | |
| 17:45:21 | openstackgerrit | Mykola Yakovliev proposed openstack/nova master: Fix boot_roles in InstanceSystemMetadata https://review.opendev.org/698040 | |
| 18:33:54 | openstackgerrit | Merged openstack/nova master: Fix race in test_create_servers_with_vpmem https://review.opendev.org/699997 | |
| 19:22:27 | gmann | stephenfin ping | |
| 20:09:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Ensure source service is up before resizing/migrating https://review.opendev.org/700062 | |
| 20:25:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Ensure source service is up before resizing/migrating https://review.opendev.org/700062 | |
| 20:55:29 | mriedem | artom: stephenfin: since you summoned me https://review.opendev.org/#/c/698530/ | |
| 21:02:13 | mriedem | and https://review.opendev.org/#/c/698532/ | |
| 21:02:35 | mriedem | do we want to wait for https://review.opendev.org/#/c/698260/ to get cleaned up and backported on top of the train backports? because if so, someone is likely going to have to clean up https://review.opendev.org/#/c/698260/ for sean | |
| 21:14:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: FUP for in-place numa rebuild https://review.opendev.org/698260 | |
| 21:14:44 | mriedem | done ^ let's get that approved on master and stacked on top of the backports | |
| 21:36:42 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: pike-only: remove broken non-voting ceph jobs https://review.opendev.org/700072 | |
| 21:45:15 | TheJulia | Are there any known issues with nova-compute on the master branch right now? | |
| 21:45:37 | TheJulia | like... ones that might show up with grenade running | |
| 21:46:09 | mriedem | that's pretty broad | |
| 21:46:32 | mriedem | you have a grenade job that rebalances ironic nodes between 2 computes right? | |
| 21:46:51 | mriedem | there is a known race issue with that https://review.opendev.org/#/c/694802/ | |
| 21:47:05 | openstack | Launchpad bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 21:47:05 | mriedem | https://bugs.launchpad.net/nova/+bug/1841481 | |
| 21:49:23 | TheJulia | mriedem: single node stack where nova-compute seems to die after being upgraded and started up for a little while | |
| 21:50:09 | TheJulia | mriedem: but, I'm only asking broadly because I've only glanced at the logs enough to have a feeling it is nova-compute. I've not gone back and gone through everything comparing out the smoke test times yet | |
| 21:50:28 | TheJulia | hence, why I asked if there is anything known before I go dig for a few hours | |
| 21:50:55 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: ocata-only: drop non-voting ceph job https://review.opendev.org/700077 | |
| 21:51:02 | mriedem | then not that i know of | |
| 21:51:29 | TheJulia | mriedem: for context, two grenade jobs, one does a full upgrade, the other does a partial upgrade for historical reasons and that breaks most cycles leading to me wanting to drink. | |
| 21:51:38 | TheJulia | Okay, yippie! | |
| 21:51:39 | TheJulia | :) | |
| 22:26:57 | TheJulia | hmm, in rescheduling for post upgrade smoke testing | |
| #openstack-nova - 2019-12-20 | |||
| 01:12:27 | openstackgerrit | Merged openstack/nova stable/stein: Zuul v3: use devstack-plugin-nfs-tempest-full https://review.opendev.org/699924 | |