| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-02 | |||
| 11:49:35 | vanch | Hi. Is it possible to create server with several volumes attached? block-device-mapping won't help. | |
| 11:49:36 | vanch | Seems like i'm affected by this bug https://bugs.launchpad.net/nova/+bug/1560965 and can't attach volumes to instances. | |
| 11:49:36 | openstack | Launchpad bug 1560965 in OpenStack Dashboard (Horizon) "libvirt selects wrong root device name" [Undecided,In progress] - Assigned to Matthew Taylor (matthew-taylor-f) | |
| 11:51:44 | zioproto | johnthetubaguy: after running the database migrations from from mitaka to newton if I make `openstack flavor list` I get the list with all flavors two times. So far I found flavor information only in the api DB in the flavors table. But also in the main nova db in the instance_types table. I have the feeling only 1 should exist. | |
| 11:53:26 | johnthetubaguy | zioproto: sounds wrong, the online db migration generally deletes the old ones during the move, dansmith might remember more about that flavor migration stuff | |
| 11:53:50 | johnthetubaguy | zioproto: for sure the api db is where it should be now | |
| 11:54:10 | zioproto | johnthetubaguy: so I confirm that doing in the main nova db ` update instance_types set deleted=1;` it fixes the problem | |
| 11:54:28 | zioproto | looks like when the flavor where migrated to the new db, they were not deleted on the main nova db | |
| 11:54:42 | zioproto | now I am looking into the host aggregates, I have quite the same problem | |
| 11:55:11 | johnthetubaguy | zioproto: sounds like you have a permissions error for the connection strings being used for the online db migrations, did you get any errors? | |
| 11:55:15 | zioproto | johnthetubaguy: also `openstack aggregate list` returns everything double | |
| 11:55:28 | johnthetubaguy | yeah, it deletes stuff in the same way | |
| 11:56:13 | zioproto | ok dansmith has a dump of my DB. I guess I have to wait it is morning in UTC-9 | |
| 11:56:29 | zioproto | I had tons of errors, because of another bug we fixed | |
| 11:57:00 | johnthetubaguy | zioproto: if it helps I think these are the things we will be running with the newton online db migration: https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L786 | |
| 11:58:02 | johnthetubaguy | zioproto: you can see it trying to delete the old flavors: https://github.com/openstack/nova/blob/stable/newton/nova/objects/flavor.py#L703 | |
| 11:58:28 | johnthetubaguy | zioproto: in theory you should have seem some errors logged from this code: https://github.com/openstack/nova/blob/stable/newton/nova/objects/flavor.py#L722 | |
| 11:58:34 | johnthetubaguy | zioproto: did you see any of those? | |
| 12:06:54 | openstackgerrit | Gao Yuan proposed openstack/nova master: Stop using mox stubs in nova.tests.unit.api.openstack.compute.test_serversV21 https://review.openstack.org/438873 | |
| 12:06:54 | openstackgerrit | Gao Yuan proposed openstack/nova master: Remove mox from nova.tests.unit.api.openstack.compute.test_block_device_mapping https://review.openstack.org/440475 | |
| 12:28:45 | openstackgerrit | Rico Lin proposed openstack/python-novaclient master: [Fix gate]Update test requirement https://review.openstack.org/440501 | |
| 12:36:04 | zioproto | johnthetubaguy: no, not any of those | |
| 12:36:08 | zioproto | johnthetubaguy: http://paste.openstack.org/show/601128/ | |
| 12:36:38 | zioproto | is it correct to run `nova-manage db online_data_migrations` in mitaka before starting and then again in newton when the migration is completed ? | |
| 12:37:46 | johnthetubaguy | zioproto: only if you run the mitaka version of the code when you have mitaka running, and the newton ones only when you are fully schema and code updated to newton | |
| 12:38:21 | johnthetubaguy | zioproto: thats looks like the error you get after it failed differently the first time | |
| 12:50:24 | zioproto | johnthetubaguy: I will update another staging cluster we have and I will give feedbacl | |
| 13:04:25 | openstackgerrit | Matthew Booth proposed openstack/nova master: Improve the readability of check_instance_state https://review.openstack.org/440536 | |
| 13:14:08 | johnthetubaguy | zioproto: it would be great to get those logs for the first run. I am sure dansmith would be able to give you more advice on what is worth collecting once he is online | |
| 13:16:46 | mriedem | dansmith: +2 on melwitt's counting quotas spec which was amended for pike to note the allocations user/project tracking for quota when a cell is down https://review.openstack.org/#/c/431749/ | |
| 13:17:03 | mriedem | dansmith: also noted in there, in ocata we just required that placement existed for comptues to starrt but we didn't require a minimum microversion, | |
| 13:17:17 | mriedem | i think in pike we have to require that placement 1.4 (ocata) is available otherwise the scheduler won't work since we don't fallback | |
| 13:17:25 | mriedem | so something to remember to put into nova-status | |
| 13:24:36 | mriedem | edleafe: cdent: fyi https://bugs.launchpad.net/nova/+bug/1669433 | |
| 13:24:36 | openstack | Launchpad bug 1669433 in OpenStack Compute (nova) "nova-status needs to check that placement 1.4 is available for pike" [High,Confirmed] | |
| 13:24:57 | cdent | mriedem: noted, thanks | |
| 13:26:53 | johnthetubaguy | mriedem: I keep meaning to hit that spec | |
| 13:27:48 | mriedem | johnthetubaguy: it's just a re-proposal with the added stuff about storing user/project in the allocations table, what we talked about at the PTG | |
| 13:27:58 | johnthetubaguy | mriedem: ah, cool | |
| 13:28:08 | andymccr | owalsh-afk: following on from yesterday i think if you check the output of api_db version and it's 0 --> then do api_db sync first - this should work, testing it now though. Should avoid the need for an upgrade flag! | |
| 13:36:46 | lyarwood | johnthetubaguy: thanks for the review on the detach patch, I'll try to get another version posted before the meeting at 5 with the cinder team | |
| 13:37:04 | mriedem | melwitt: did you see rajesh's email about the ceph monitor ip change bug again? that reminded me of talking about it at the ptg. i was wondering if you were planning on making changes there. | |
| 13:50:47 | sdague | mriedem: when you get a mental breathing space, I actually need to talk through the options here - https://review.openstack.org/#/c/438049/6/nova/api/openstack/identity.py - because I'm torn in a few directions | |
| 14:07:56 | mdbooth | lyarwood johnthetubaguy: Just had a random thought. I think Nova is still going to have to track connect/disconnect for detach. Say you've got a shared connection containing N connected volumes. The user disconnects all N almost simultaneously. When the last one is disconnected, cinder can tell Nova in that disconnect call that there are none left. However, depending on the order Nova processes them, it might not be the last one. IOW | |
| 14:07:56 | mdbooth | it's a race. I think only Nova can reliably know when it's no longer in use on the host. | |
| 14:08:46 | mdbooth | And by Nova I mean os-brick obviously. | |
| 14:08:49 | mdbooth | But still Nova. | |
| 14:09:05 | johnthetubaguy | mdbooth: I thought the locking method, and re-fetching the list with the lock held, and held for the duration of the disconnect, would work around that | |
| 14:09:19 | johnthetubaguy | mdbooth: there could be a big hole in that though | |
| 14:11:46 | openstackgerrit | Gábor Antal proposed openstack/nova-specs master: Show persisted scheduler hints in server details https://review.openstack.org/440580 | |
| 14:15:09 | mriedem | sdague: i've got to start getting maya ready to leave in about 5 minutes, which usually takes about 10 minutes because of the constant redirecting, do you want to talk about it later? i see you're in -dev talking quotas. alternatively, dump your brain in the ML and i'll read it when i'm in the office later and have time to weigh the options. | |
| 14:15:13 | edleafe | mriedem: ack on the nova-status bug | |
| 14:15:16 | openstackgerrit | Gábor Antal proposed openstack/nova-specs master: Show persisted scheduler hints in server details https://review.openstack.org/440580 | |
| 14:15:21 | mriedem | sdague: plus if it's in the list then others can weigh in | |
| 14:17:37 | mriedem | sileht: i feel like manila had the exact same type of issue you're talking about in the ML with the broken telemetry job | |
| 14:17:45 | mriedem | and it had something to do with the order of their devstack plugin setup, | |
| 14:17:53 | mriedem | such that it was trying to list instances before the instances were in a cell | |
| 14:17:54 | mriedem | something like that | |
| 14:18:17 | sileht | We don't do anything special | |
| 14:18:20 | mriedem | sileht: so need to know when simple_cell_setup is called relative to when the openstack server list returns nothing | |
| 14:19:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: fix missing instance.delete notification https://review.openstack.org/437222 | |
| 14:19:35 | mdbooth | johnthetubaguy: Nova re-fetching the list with the lock held would work, but only if cinder also didn't update the list until Nova had confirmed the detach. | |
| 14:19:43 | sileht | mriedem, we call "openstack server list" after ./stack.sh says everything is ready | |
| 14:19:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297 | |
| 14:19:59 | sileht | mriedem, and then we polls nova every second during 10 minutes | |
| 14:20:59 | mdbooth | johnthetubaguy: It's starting to sound to me like a nova-side implementation is going to be simpler. | |
| 14:21:19 | mdbooth | After all, it really is an implementation detail of the driver. | |
| 14:21:41 | johnthetubaguy | mdbooth: right, its assuming the attachments that are shared on the nova host are only modified by Nova (which we can enforce now, using service tokens) | |
| 14:22:00 | mriedem | sileht: ok simple_cell_setup is run well before that http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/devstacklog.txt.gz#_2017-03-02_09_08_19_738 | |
| 14:22:04 | johnthetubaguy | mdbooth: generally thats always the case anyways, but it can be enforced too | |
| 14:22:11 | sileht | mriedem, yes | |
| 14:23:17 | mriedem | sileht: is it listing servers with a filter? | |
| 14:23:23 | mriedem | or just listing servers and checking for one result? | |
| 14:23:43 | sileht | mriedem, just GET /servers/detail HTTP/1.1 | |
| 14:23:48 | mriedem | when did this start failing? | |
| 14:24:21 | sileht | I would says some days ago | |
| 14:25:16 | sileht | We didn't notice the exact time because we fall into an oslo.messaging from time to time and people recheck a lot this job | |
| 14:25:32 | mriedem | markus_z: looks like we have a novnc test failure in tempest http://logs.openstack.org/63/435563/3/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/20e15f9/console.html#_2017-03-01_20_30_52_369683 | |
| 14:25:36 | mriedem | intermittent i mean | |
| 14:25:44 | sileht | since yesterday or the day before, we seen that the error is no more the oslo.messaging bug | |
| 14:25:55 | sileht | and it's always failing | |
| 14:25:58 | openstackgerrit | Gábor Antal proposed openstack/nova-specs master: Show persisted scheduler hints in server details https://review.openstack.org/440580 | |
| 14:26:01 | mriedem | sileht: we should be able to identify when the failure started using logstash | |
| 14:26:18 | sileht | mriedem, i'm suspecting a policies bug, because when ceilometer polls nova with ceilometer service account and 'all_tenants=True', it see the instances | |
| 14:26:39 | sileht | instances owner are admin/admin | |
| 14:26:49 | mriedem | sileht: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20json%20path%20%24.servers%5B0%5D.status%20cannot%20match%20%7Bu'servers'%3A%20%5B%5D%7D%5C%22&from=7d | |
| 14:26:53 | sileht | and we do the 'openstack server list' as admin/admin | |
| 14:27:33 | markus_z | mriedem: I haven't seen that test before, I can take a look. | |
| 14:28:06 | mriedem | markus_z: pretty high failure rate | |
| 14:28:10 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20_validate_rfb_negotiation%5C%22&from=7d | |
| 14:28:12 | mriedem | sdague: ^ | |
| 14:28:17 | mriedem | i don't think we're tracking that - mtreinish ^ | |
| 14:28:53 | mriedem | sileht: looks like it started on 2/28 | |
| 14:29:04 | mriedem | so should dig through related looking changes https://github.com/openstack/nova/commits/master | |
| 14:29:06 | mriedem | i have to run for a bit | |
| 14:29:52 | mtreinish | mriedem: I don't think we're tracking it yet | |
| 14:34:08 | markus_z | the first hits start on 02/21, the tempest test merged 02/03 | |
| 14:34:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: fix missing instance.delete notification https://review.openstack.org/437222 | |
| 14:34:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297 | |
| 14:36:15 | markus_z | That's the first one: http://logs.openstack.org/38/431038/3/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/5e1d485/console.html | |