Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-02
11:53:50 johnthetubaguy zioproto: for sure the api db is where it should be now
11:54:10 zioproto johnthetubaguy: so I confirm that doing in the main nova db ` update instance_types set deleted=1;` it fixes the problem
11:54:28 zioproto looks like when the flavor where migrated to the new db, they were not deleted on the main nova db
11:54:42 zioproto now I am looking into the host aggregates, I have quite the same problem
11:55:11 johnthetubaguy zioproto: sounds like you have a permissions error for the connection strings being used for the online db migrations, did you get any errors?
11:55:15 zioproto johnthetubaguy: also `openstack aggregate list` returns everything double
11:55:28 johnthetubaguy yeah, it deletes stuff in the same way
11:56:13 zioproto ok dansmith has a dump of my DB. I guess I have to wait it is morning in UTC-9
11:56:29 zioproto I had tons of errors, because of another bug we fixed
11:57:00 johnthetubaguy zioproto: if it helps I think these are the things we will be running with the newton online db migration: https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L786
11:58:02 johnthetubaguy zioproto: you can see it trying to delete the old flavors: https://github.com/openstack/nova/blob/stable/newton/nova/objects/flavor.py#L703
11:58:28 johnthetubaguy zioproto: in theory you should have seem some errors logged from this code: https://github.com/openstack/nova/blob/stable/newton/nova/objects/flavor.py#L722
11:58:34 johnthetubaguy zioproto: did you see any of those?
12:06:54 openstackgerrit Gao Yuan proposed openstack/nova master: Stop using mox stubs in nova.tests.unit.api.openstack.compute.test_serversV21 https://review.openstack.org/438873
12:06:54 openstackgerrit Gao Yuan proposed openstack/nova master: Remove mox from nova.tests.unit.api.openstack.compute.test_block_device_mapping https://review.openstack.org/440475
12:28:45 openstackgerrit Rico Lin proposed openstack/python-novaclient master: [Fix gate]Update test requirement https://review.openstack.org/440501
12:36:04 zioproto johnthetubaguy: no, not any of those
12:36:08 zioproto johnthetubaguy: http://paste.openstack.org/show/601128/
12:36:38 zioproto is it correct to run `nova-manage db online_data_migrations` in mitaka before starting and then again in newton when the migration is completed ?
12:37:46 johnthetubaguy zioproto: only if you run the mitaka version of the code when you have mitaka running, and the newton ones only when you are fully schema and code updated to newton
12:38:21 johnthetubaguy zioproto: thats looks like the error you get after it failed differently the first time
12:50:24 zioproto johnthetubaguy: I will update another staging cluster we have and I will give feedbacl
13:04:25 openstackgerrit Matthew Booth proposed openstack/nova master: Improve the readability of check_instance_state https://review.openstack.org/440536
13:14:08 johnthetubaguy zioproto: it would be great to get those logs for the first run. I am sure dansmith would be able to give you more advice on what is worth collecting once he is online
13:16:46 mriedem dansmith: +2 on melwitt's counting quotas spec which was amended for pike to note the allocations user/project tracking for quota when a cell is down https://review.openstack.org/#/c/431749/
13:17:03 mriedem dansmith: also noted in there, in ocata we just required that placement existed for comptues to starrt but we didn't require a minimum microversion,
13:17:17 mriedem i think in pike we have to require that placement 1.4 (ocata) is available otherwise the scheduler won't work since we don't fallback
13:17:25 mriedem so something to remember to put into nova-status
13:24:36 mriedem edleafe: cdent: fyi https://bugs.launchpad.net/nova/+bug/1669433
13:24:36 openstack Launchpad bug 1669433 in OpenStack Compute (nova) "nova-status needs to check that placement 1.4 is available for pike" [High,Confirmed]
13:24:57 cdent mriedem: noted, thanks
13:26:53 johnthetubaguy mriedem: I keep meaning to hit that spec
13:27:48 mriedem johnthetubaguy: it's just a re-proposal with the added stuff about storing user/project in the allocations table, what we talked about at the PTG
13:27:58 johnthetubaguy mriedem: ah, cool
13:28:08 andymccr owalsh-afk: following on from yesterday i think if you check the output of api_db version and it's 0 --> then do api_db sync first - this should work, testing it now though. Should avoid the need for an upgrade flag!
13:36:46 lyarwood johnthetubaguy: thanks for the review on the detach patch, I'll try to get another version posted before the meeting at 5 with the cinder team
13:37:04 mriedem melwitt: did you see rajesh's email about the ceph monitor ip change bug again? that reminded me of talking about it at the ptg. i was wondering if you were planning on making changes there.
13:50:47 sdague mriedem: when you get a mental breathing space, I actually need to talk through the options here - https://review.openstack.org/#/c/438049/6/nova/api/openstack/identity.py - because I'm torn in a few directions
14:07:56 mdbooth lyarwood johnthetubaguy: Just had a random thought. I think Nova is still going to have to track connect/disconnect for detach. Say you've got a shared connection containing N connected volumes. The user disconnects all N almost simultaneously. When the last one is disconnected, cinder can tell Nova in that disconnect call that there are none left. However, depending on the order Nova processes them, it might not be the last one. IOW
14:07:56 mdbooth it's a race. I think only Nova can reliably know when it's no longer in use on the host.
14:08:46 mdbooth And by Nova I mean os-brick obviously.
14:08:49 mdbooth But still Nova.
14:09:05 johnthetubaguy mdbooth: I thought the locking method, and re-fetching the list with the lock held, and held for the duration of the disconnect, would work around that
14:09:19 johnthetubaguy mdbooth: there could be a big hole in that though
14:11:46 openstackgerrit Gábor Antal proposed openstack/nova-specs master: Show persisted scheduler hints in server details https://review.openstack.org/440580
14:15:09 mriedem sdague: i've got to start getting maya ready to leave in about 5 minutes, which usually takes about 10 minutes because of the constant redirecting, do you want to talk about it later? i see you're in -dev talking quotas. alternatively, dump your brain in the ML and i'll read it when i'm in the office later and have time to weigh the options.
14:15:13 edleafe mriedem: ack on the nova-status bug
14:15:16 openstackgerrit Gábor Antal proposed openstack/nova-specs master: Show persisted scheduler hints in server details https://review.openstack.org/440580
14:15:21 mriedem sdague: plus if it's in the list then others can weigh in
14:17:37 mriedem sileht: i feel like manila had the exact same type of issue you're talking about in the ML with the broken telemetry job
14:17:45 mriedem and it had something to do with the order of their devstack plugin setup,
14:17:53 mriedem such that it was trying to list instances before the instances were in a cell
14:17:54 mriedem something like that
14:18:17 sileht We don't do anything special
14:18:20 mriedem sileht: so need to know when simple_cell_setup is called relative to when the openstack server list returns nothing
14:19:17 openstackgerrit Balazs Gibizer proposed openstack/nova master: fix missing instance.delete notification https://review.openstack.org/437222
14:19:35 mdbooth johnthetubaguy: Nova re-fetching the list with the lock held would work, but only if cinder also didn't update the list until Nova had confirmed the detach.
14:19:43 sileht mriedem, we call "openstack server list" after ./stack.sh says everything is ready
14:19:52 openstackgerrit Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297
14:19:59 sileht mriedem, and then we polls nova every second during 10 minutes
14:20:59 mdbooth johnthetubaguy: It's starting to sound to me like a nova-side implementation is going to be simpler.
14:21:19 mdbooth After all, it really is an implementation detail of the driver.
14:21:41 johnthetubaguy mdbooth: right, its assuming the attachments that are shared on the nova host are only modified by Nova (which we can enforce now, using service tokens)
14:22:00 mriedem sileht: ok simple_cell_setup is run well before that http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/devstacklog.txt.gz#_2017-03-02_09_08_19_738
14:22:04 johnthetubaguy mdbooth: generally thats always the case anyways, but it can be enforced too
14:22:11 sileht mriedem, yes
14:23:17 mriedem sileht: is it listing servers with a filter?
14:23:23 mriedem or just listing servers and checking for one result?
14:23:43 sileht mriedem, just GET /servers/detail HTTP/1.1
14:23:48 mriedem when did this start failing?
14:24:21 sileht I would says some days ago
14:25:16 sileht We didn't notice the exact time because we fall into an oslo.messaging from time to time and people recheck a lot this job
14:25:32 mriedem markus_z: looks like we have a novnc test failure in tempest http://logs.openstack.org/63/435563/3/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/20e15f9/console.html#_2017-03-01_20_30_52_369683
14:25:36 mriedem intermittent i mean
14:25:44 sileht since yesterday or the day before, we seen that the error is no more the oslo.messaging bug
14:25:55 sileht and it's always failing
14:25:58 openstackgerrit Gábor Antal proposed openstack/nova-specs master: Show persisted scheduler hints in server details https://review.openstack.org/440580
14:26:01 mriedem sileht: we should be able to identify when the failure started using logstash
14:26:18 sileht mriedem, i'm suspecting a policies bug, because when ceilometer polls nova with ceilometer service account and 'all_tenants=True', it see the instances
14:26:39 sileht instances owner are admin/admin
14:26:49 mriedem sileht: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20json%20path%20%24.servers%5B0%5D.status%20cannot%20match%20%7Bu'servers'%3A%20%5B%5D%7D%5C%22&from=7d
14:26:53 sileht and we do the 'openstack server list' as admin/admin
14:27:33 markus_z mriedem: I haven't seen that test before, I can take a look.
14:28:06 mriedem markus_z: pretty high failure rate
14:28:10 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20_validate_rfb_negotiation%5C%22&from=7d
14:28:12 mriedem sdague: ^
14:28:17 mriedem i don't think we're tracking that - mtreinish ^
14:28:53 mriedem sileht: looks like it started on 2/28
14:29:04 mriedem so should dig through related looking changes https://github.com/openstack/nova/commits/master
14:29:06 mriedem i have to run for a bit
14:29:52 mtreinish mriedem: I don't think we're tracking it yet
14:34:08 markus_z the first hits start on 02/21, the tempest test merged 02/03
14:34:46 openstackgerrit Balazs Gibizer proposed openstack/nova master: fix missing instance.delete notification https://review.openstack.org/437222
14:34:58 openstackgerrit Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297
14:36:15 markus_z That's the first one: http://logs.openstack.org/38/431038/3/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/5e1d485/console.html
14:39:29 openstackgerrit Balazs Gibizer proposed openstack/nova master: Improve assertJsonEqual error reporting https://review.openstack.org/428199
14:48:37 bhagyashris johnthetubaguy: Hi, i am waiting for your opinion regarding: which response code should i return if rotation==0?
14:49:14 johnthetubaguy bhagyashris: on that patch, I basically said there should be no API change, I believe?
14:49:44 johnthetubaguy bhagyashris: I suspect that wasn't very clear though, sorry.
14:50:00 johnthetubaguy bhagyashris: I would just upload what you have so far, or did you already do that?

Earlier   Later