| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-10 | |||
| 16:30:42 | cdent | and the gabbi test fixture for function does too | |
| 16:31:25 | mriedem | tempest? you mean devstack? | |
| 16:31:30 | cdent | yeah, sorry | |
| 16:31:30 | mriedem | i don't see allocation_ratio set in nova.conf http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 16:31:58 | mriedem | http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-n-cpu.txt.gz#_Dec_15_15_20_27_560214 | |
| 16:32:00 | cdent | (I'm simultaneously working on some tempest stuff) | |
| 16:32:27 | melwitt | I've noticed the test_live_migration_actions functional test has been intermittently failing in the gate again http://logs.openstack.org/90/333990/22/check/openstack-tox-functional/24eeb2b/testr_results.html.gz | |
| 16:33:00 | mriedem | cdent: in this test run, this is the first inventory update after the compute node RP is created http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-n-cpu.txt.gz#_Dec_15_15_20_29_162934 | |
| 16:33:25 | melwitt | gibi ^ | |
| 16:33:54 | cdent | mriedem: presumably the set to 0.0 here shouldn't be? http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-n-cpu.txt.gz#_Dec_15_15_20_27_560214 | |
| 16:34:29 | mriedem | cdent: 0.0 is the default in the configs | |
| 16:34:44 | cdent | I get that. | |
| 16:35:01 | cdent | But one wouldn't expect that work, would one? | |
| 16:35:08 | cdent | unless 0 is meant to mean 1 | |
| 16:35:22 | cdent | and that's the conversation I seem to remember having, but can't find the thread of | |
| 16:35:52 | mriedem | i don't see how it works with the capacity checks on the placement side for updating inventory | |
| 16:36:00 | mriedem | for this ci run, this is the first inventory update http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-placement-api.txt.gz#_Dec_15_15_20_29_138893 | |
| 16:36:16 | mriedem | but doesn't tell us much | |
| 16:36:28 | gibi | melwitt: thanks for the heads up, I will look into it | |
| 16:36:56 | melwitt | thanks gibi | |
| 16:37:15 | cdent | mriedem: ah, it is the compute node object that does some jiggery pokery if it is doesn't like the values of the allocation ratios: see _from_db_object in class ComputeNode | |
| 16:37:44 | mriedem | yup i just found that | |
| 16:37:54 | cdent | some presumably somewhere in the first step that pokery is happening, but then getting ruined in the second | |
| 16:37:54 | mriedem | maciejjozefczyk: | |
| 16:38:41 | mriedem | maciejjozefczyk: https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L198 | |
| 16:40:51 | maciejjozefczyk | mriedem: mmmmm | |
| 16:41:49 | maciejjozefczyk | mriedem: and yes, I dont have it specified in config at all | |
| 16:43:29 | mriedem | i don't really understand why the inventory request wouldn't have the allocation_ratio values from the compute node object though | |
| 16:43:39 | mriedem | because in your patch, _update is called after the compute node record is created, | |
| 16:43:56 | mriedem | and ComputeNode.create() eventually calls _from_db_object to set the 'default' allocation ratios on the object itself | |
| 16:43:57 | cfriesen | just stumbled over something odd...if the os_type isn't set, then in some scenarios the ephemeral disks are formated vfat. if the ephemeral size is too big for vfat, this chokes. Should we default to something else in the code or just not format it at all if a default isn't specified in the config file? | |
| 16:44:03 | mriedem | which will be used in _normalize_inventory_from_cn_obj | |
| 16:44:51 | cfriesen | mdbooth: ^ | |
| 16:48:08 | mriedem | maciejjozefczyk: if you have a recreate of that failure, try logging the compute node record in _update() before calling reportclient.set_inventory_for_provider | |
| 16:57:04 | maciejjozefczyk | mriedem: ok | |
| 17:01:07 | diga | jaypipes: Hi | |
| 17:01:38 | diga | jaypipes: have you seen my mail on this bug - https://bugs.launchpad.net/nova/+bug/1719933 | |
| 17:01:40 | openstack | Launchpad bug 1719933 in OpenStack Compute (nova) "placement server needs to retry allocations, server-side" [Medium,Triaged] - Assigned to Jay Pipes (jaypipes) | |
| 17:03:57 | cdent | diga: twitter suggests that jay is rather ill today | |
| 17:04:13 | diga | cdent: ohh | |
| 17:05:50 | maciejjozefczyk | mriedem: Ok, I'll paste it tomorrow, need to go | |
| 17:05:55 | mdbooth | cfriesen: The behaviour of that formatting is lost in the depths of time. That sounds like a bona fide bug, though. | |
| 17:06:07 | mriedem | maciejjozefczyk: o/ | |
| 17:06:07 | diga | cdent: do you have sometime, can you help me ? | |
| 17:06:24 | cdent | diga: I might be able to yeah, what's up? | |
| 17:06:34 | mdbooth | cfriesen: Do other clouds format blank ephemeral disks for you? It seems like such a weird thing to do in Nova. | |
| 17:07:39 | maciejjozefczyk | first sight, strange, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L763 gives me https://pastebin.com/skR0X0Mm (at init) | |
| 17:07:46 | mdbooth | cfriesen: I don't think we can change that behaviour without at least a microversion bump, btw. Probably a cycle of ops discussion, a prominent release note, and a microversion bump. | |
| 17:07:55 | maciejjozefczyk | mriedem: and cpu_allocation_ratio is set as 16.0 | |
| 17:08:35 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver https://review.openstack.org/526094 | |
| 17:08:49 | mriedem | maciejjozefczyk: that's because it pulls the object from the db and runs it through ComputeNode._from_db_object which sets the 'default' allocation ratios (hardcoded in code) if not set in config | |
| 17:08:59 | mdbooth | cfriesen: When the ephemeral disk is too big, does the mkfs choke causing the spawn to fail? | |
| 17:09:14 | maciejjozefczyk | mriedem: I'll check it tomorrow why then it is changed to 0 | |
| 17:09:25 | maciejjozefczyk | mriedem: thats the real cause, yes? | |
| 17:09:51 | mriedem | yeah i don't know what is changing it to 0.0 before we hit placement | |
| 17:10:11 | maciejjozefczyk | mriedem: I'll find it for you ;) bb | |
| 17:10:18 | mriedem | godspeed | |
| 17:11:42 | diga | cdent: | |
| 17:11:49 | diga | cdent: - Here I got the error after adding scenarios - http://paste.openstack.org/show/641294/ | |
| 17:12:26 | diga | cdent: changes are made in git diff - http://paste.openstack.org/show/641293/ | |
| 17:12:28 | cdent | diga: do you have a work in progress that you post up to gerrit so there's code to look at? | |
| 17:13:10 | diga | cdent: I have submit the patch yet, Let me submit the patch then | |
| 17:13:20 | diga | s/have/haven't | |
| 17:14:15 | diga | cdent: yes, I am working on this as per jaypipes suggestions | |
| 17:14:43 | cdent | diga: note that DbDeadLock is already handled, what's not handled is ConcurrentUpdate | |
| 17:15:28 | diga | cdent: okay | |
| 17:15:46 | cdent | _set_allocations method already has the retry handing on it | |
| 17:16:00 | cdent | but it is only set up for handling db api exceptions, which ConcurrentUpdate is not | |
| 17:16:28 | mriedem | sdague: this is the novaclient change for the file injection deprecation and userdata + rebuild stuff https://review.openstack.org/#/c/528128/ - closes out that bp and unblocks the next novaclient change in the series for the next microversion; client release freeze is creeping up so i'd like to get some reviews on this stuff | |
| 17:16:40 | diga | okk, I will work on it then | |
| 17:17:10 | cdent | diga: let me know if/how I can help | |
| 17:17:51 | diga | cdent: currently main challenge in nova is to reproduce the issue, db part is reproduced, but how to reproduce ConcurrentUpdate | |
| 17:18:00 | diga | cdent: some pointers can be helpful | |
| 17:18:03 | mriedem | diga: i have a devstack patch that reproduces it... | |
| 17:18:46 | mriedem | https://review.openstack.org/#/c/507918/ | |
| 17:18:46 | cdent | diga: I think you can probably do something similar to what you've done in your existing test, but put the side effect on the method that increase the generation | |
| 17:19:01 | cdent | diga: _increment_provider_generation | |
| 17:19:41 | diga | cdent: okay | |
| 17:20:07 | diga | mriedem: I will take a look at it | |
| 17:20:47 | mriedem | it's probably not very helpful though for recreating a concurrent update failure in a unit test | |
| 17:21:00 | mriedem | it was just something i think i noticed while investigating failures in that devstack patch | |
| 17:21:14 | diga | mriedem: but it needs to rerun devstack with this change | |
| 17:21:40 | diga | mriedem: ok, got it | |
| 17:29:08 | stephenfin | artom: https://review.rdoproject.org/r/#/c/11283/ | |
| 17:34:37 | diga | mriedem: Thanks for sharing the link, it will be certainly helpful | |
| 17:34:54 | diga | cdent: thanks for your help | |
| 17:35:24 | diga | cdent: mriedem : will ping you if I need any help, will update you by tomorrow | |
| 17:38:25 | cfriesen | mdbooth: sorry, was off in a meeting. Yes, the initial spawn fails when the ephemeral disk is too big. I wonder if we could change the default based on size, to either use something else or just not format it instead of choking. | |
| 17:39:55 | mdbooth | cfriesen: I think it has to be considered part of the api. I think the only change we can make to it without a microversion bump is a minimal change to make it not fail. | |
| 17:40:21 | mdbooth | cfriesen: So we could, for eg, not format it at all, but only in the case that we know it would cause a failure to build. | |
| 17:41:52 | cfriesen | mdbooth: agreed. I'll open a bug and maybe propose a fix. | |
| 18:53:51 | openstackgerrit | melanie witt proposed openstack/nova master: Detach volume after deleting instance with no host https://review.openstack.org/340614 | |
| 18:55:18 | mriedem | stvnoyes: finally got that multiattach snapshot test to pass http://logs.openstack.org/86/531386/7/check/tempest-full/2f25c03/job-output.txt.gz#_2018-01-10_02_15_35_840008 | |
| 18:56:01 | stvnoyes | excellent. I've been working on libvirt 3.10. Finally got past the dependency issues working on getting stack up now | |
| 18:56:33 | mriedem | for the tempest patch, it's getting big, so i might need to think about splitting it up | |
| 18:56:40 | mriedem | mtreinish: how do you feel about this? https://review.openstack.org/#/c/266605/25/tempest/api/compute/volumes/test_attach_volume.py | |
| 18:56:53 | melwitt | is anyone willing to please review the local delete patch ^ if an instance with attached volumes etc goes into error state, when it's deleted the volumes and networks aren't disconnected from the instance and have to be manually disconnected in order to be used again | |
| 18:56:56 | mriedem | should i do the first patch with the config option and 1 test, then add the other tests in subsequent patches? | |
| 19:01:06 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver https://review.openstack.org/526094 | |
| 19:04:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406 | |