Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-30
13:44:56 mriedem the source resource provider will have allocations held by the migration record
13:45:04 bauzas but when confirming the resize, I get an exception
13:45:22 mriedem that's separate from the leaked allocation issue
13:46:03 bauzas hum, so, once I delete the service, I'll just delete the instance?
13:46:04 mriedem for one thing, if you restart the source compute service it will create a new compute node with a new uuid and a new resource provider, so the migration record won't be able to find allocations on that one
13:46:20 mriedem i guess i'd have to know what the error is when you go to confirm the resize
13:46:35 bauzas secf
13:47:36 bauzas mriedem: http://paste.openstack.org/show/780354/
13:48:22 bauzas when confirming the resize, I get an exception saying that it doesn't get the source compute UUID
13:49:20 mriedem does the test stop/start a new compute service after deleting the original source compute service?
13:49:30 mriedem starting a new compute service is going to create a new compute node record
13:49:50 mriedem if you didn't fully stop the service it's going to have residual cached information in the ResourceTracker.compute_nodes dict
13:50:23 mriedem which is likely a bug in your test, though someone could totally forget to stop the compute service before deleting it, we have warnings in the API about that
13:51:35 openstackgerrit Merged openstack/os-vif stable/train: Update TOX/UPPER_CONSTRAINTS_FILE for stable/train https://review.opendev.org/683489
13:51:52 bauzas mriedem: okay, I can restart the service before confirming, then
13:51:53 bauzas thanks
13:52:13 mriedem well, to do things the proper way, you should be stopping the source compute service, then deleting it, and then starting a new one
13:52:29 mriedem https://docs.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service
13:53:22 mriedem so what is happening i think is that because you're confirming with a stale ResourceTracker.compute_nodes entry, it's trying to update a record that was deleted when you deleted the compute service
13:53:26 mriedem w/o stopping it
13:54:50 mriedem one could argue that drop_move_claim should handle that since we don't prevent you from deleting the compute service in the api w/o stopping it first
13:54:56 mriedem but it's a different bug
13:55:49 bauzas okay, I'll quickly write it
13:56:32 mriedem b/c of this https://opendev.org/openstack/nova/src/branch/master/nova/compute/resource_tracker.py#L714 is why we don't create a new compute node record when you start the service after deleting it
13:56:39 mriedem and we're using the old now-deleted version
13:57:03 mriedem https://review.opendev.org/#/c/684849/ is a similar issue
13:57:13 mriedem i've found a few problems in the RT code lately b/c of that caching
13:57:30 mriedem https://review.opendev.org/#/q/I9fa1d509a3de405d6246fb8670612c65c10cc93b
13:58:49 mriedem feel free to hit the queens backport for ^ before you get a bz downstream for it anyway
14:02:22 mriedem need another stable core to approve this backport for train rc2 https://review.opendev.org/#/c/685387/
14:02:49 bauzas mriedem: can do
14:06:53 mriedem bauzas: btw i might have misled you with a comment on your placement audit patch about recreating an evacuate scenario with leaked allocations, if i said there is a migration-based allocatoin on the source provider during evacuate, that was wrong
14:07:04 mriedem we only do migration-based allocation swap-a-roo for cold and live migrate
14:07:07 mriedem *and resize
14:07:37 bauzas mriedem: hmmm
14:07:44 mriedem there is an allocation on the source provider, it's just not held by a migration record, it's held by the instance consumer
14:08:04 bauzas mriedem: that's what I found
14:08:15 bauzas mriedem: I already have a test for evacuate which works
14:08:59 bauzas mriedem: https://review.opendev.org/#/c/670112/5/nova/tests/functional/test_nova_manage.py@1470 (but I need to rewrite this test to use the existing helpers thanks to gibi :) )
15:14:31 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Use nova-net explicitly in '_IntegratedTestBase' subclasses https://review.opendev.org/684326
15:14:32 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Use nova-net explicitly in 'ServersSampleBase' subclasses https://review.opendev.org/684330
15:14:32 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Use nova-net explicitly in 'ApiSampleTestBase' subclasses https://review.opendev.org/684328
15:14:33 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_floating_ip_pools' functional tests https://review.opendev.org/684333
15:14:33 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_availability_zone' functional tests https://review.opendev.org/684332
15:14:34 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_server_tags' functional tests https://review.opendev.org/684335
15:14:34 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_quota_sets' functional tests https://review.opendev.org/684334
15:14:35 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_hosts' functional tests https://review.opendev.org/684337
15:14:35 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_servers' functional tests https://review.opendev.org/684336
15:14:36 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_rescue' functional tests https://review.opendev.org/684339
15:14:36 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_networks_associate' functional tests https://review.opendev.org/684338
15:14:37 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_attach_interfaces' functional tests https://review.opendev.org/684341
15:14:37 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_hypervisors' functional tests https://review.opendev.org/684340
15:14:38 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Neutron fixture cleanup https://review.opendev.org/684343
15:14:38 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_simple_tenant_usage' functional tests https://review.opendev.org/684342
15:14:39 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345
15:14:39 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_floating_ips' functional tests https://review.opendev.org/684344
15:25:24 openstackgerrit Adam Spiers proposed openstack/nova master: Also enable iommu for virtio controllers in libvirt https://review.opendev.org/684825
15:25:56 openstackgerrit Adam Spiers proposed openstack/nova master: Also enable iommu for virtio controllers in libvirt https://review.opendev.org/684825
15:29:55 openstackgerrit Merged openstack/nova stable/train: Stop filtering out 'accepted' for in-progress migrations https://review.opendev.org/685387
15:34:27 openstackgerrit Eric Fried proposed openstack/nova master: make virtual pmem feature compatible with python3 https://review.opendev.org/685593
15:36:34 dansmith efried: looks like that should be an rc2 thing eh?
15:39:35 dansmith really, the guest config module should handle that
15:41:59 efried dansmith: Could make an argument for RC2, yeah.
15:42:08 efried since basically py3 is busted otherwise.
15:42:17 dansmith it's a regression and a broken feature we added in train, so it seems clear to me
15:42:48 efried dansmith: Since Alex was +2 and I just changed the test, would you feel good about fast approving it?
15:43:27 efried "the guest config module should handle" like by knowing which fields are supposed to be int and casting them before stringifying them?
15:43:31 dansmith I'm pulling it down to mess with it
15:43:54 dansmith efried: uhyeah.. right now it would stringify an object in that field if you gave it to it
15:43:56 dansmith which is dumb
15:44:02 dansmith obviously not for this fix though
15:44:26 efried yeah, seems like probably that would be a pervasive problem throughout
15:44:34 efried though I haven't looked
15:44:35 dansmith it is, from a quick look
15:45:07 efried It took me a hot minute to figure out why we weren't getting by on the fact that the ovo fields are IntegerField()
15:45:15 efried but it's because we're doing the conversion after that
15:45:22 efried s/conversion/division/
15:45:35 dansmith yeah, this is way after any object stuff
15:50:58 openstackgerrit Dan Smith proposed openstack/nova stable/train: make virtual pmem feature compatible with python3 https://review.opendev.org/685742
15:58:30 dansmith I guess if we
15:58:48 dansmith are going to merge things based purely on manual testing by the submitter, we should ask for them to do it in py2 and py3
15:58:56 dansmith (or not do that thing)
16:00:40 efried Well, it wasn't manual, it was 3pCI, but I never thought to check that it was running py3.
16:00:51 efried Are we planning to continue supporting py2 in ussuri?
16:01:24 efried the argument could be made that testing (3pCI et al) should *only* be py3
16:03:10 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_availability_zone' functional tests https://review.opendev.org/684332
16:03:10 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Use nova-net explicitly in functional tests https://review.opendev.org/684326
16:03:11 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_quota_sets' functional tests https://review.opendev.org/684334
16:03:11 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_floating_ip_pools' functional tests https://review.opendev.org/684333
16:03:12 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_hosts' functional tests https://review.opendev.org/684337
16:03:12 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_servers' functional tests https://review.opendev.org/684336
16:03:12 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_server_tags' functional tests https://review.opendev.org/684335
16:03:13 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_networks_associate' functional tests https://review.opendev.org/684338
16:03:14 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_hypervisors' functional tests https://review.opendev.org/684340
16:03:14 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_rescue' functional tests https://review.opendev.org/684339
16:03:15 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Neutron fixture cleanup https://review.opendev.org/684343
16:03:15 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_simple_tenant_usage' functional tests https://review.opendev.org/684342
16:03:15 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_attach_interfaces' functional tests https://review.opendev.org/684341
16:03:16 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Migrate 'test_floating_ips' functional tests https://review.opendev.org/684344
16:03:17 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345

Earlier   Later