Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-10
16:34:44 cdent I get that.
16:35:01 cdent But one wouldn't expect that work, would one?
16:35:08 cdent unless 0 is meant to mean 1
16:35:22 cdent and that's the conversation I seem to remember having, but can't find the thread of
16:35:52 mriedem i don't see how it works with the capacity checks on the placement side for updating inventory
16:36:00 mriedem for this ci run, this is the first inventory update http://logs.openstack.org/24/520024/6/check/legacy-tempest-dsvm-neutron-full/255f8c4/logs/screen-placement-api.txt.gz#_Dec_15_15_20_29_138893
16:36:16 mriedem but doesn't tell us much
16:36:28 gibi melwitt: thanks for the heads up, I will look into it
16:36:56 melwitt thanks gibi
16:37:15 cdent mriedem: ah, it is the compute node object that does some jiggery pokery if it is doesn't like the values of the allocation ratios: see _from_db_object in class ComputeNode
16:37:44 mriedem yup i just found that
16:37:54 mriedem maciejjozefczyk:
16:37:54 cdent some presumably somewhere in the first step that pokery is happening, but then getting ruined in the second
16:38:41 mriedem maciejjozefczyk: https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L198
16:40:51 maciejjozefczyk mriedem: mmmmm
16:41:49 maciejjozefczyk mriedem: and yes, I dont have it specified in config at all
16:43:29 mriedem i don't really understand why the inventory request wouldn't have the allocation_ratio values from the compute node object though
16:43:39 mriedem because in your patch, _update is called after the compute node record is created,
16:43:56 mriedem and ComputeNode.create() eventually calls _from_db_object to set the 'default' allocation ratios on the object itself
16:43:57 cfriesen just stumbled over something odd...if the os_type isn't set, then in some scenarios the ephemeral disks are formated vfat. if the ephemeral size is too big for vfat, this chokes. Should we default to something else in the code or just not format it at all if a default isn't specified in the config file?
16:44:03 mriedem which will be used in _normalize_inventory_from_cn_obj
16:44:51 cfriesen mdbooth: ^
16:48:08 mriedem maciejjozefczyk: if you have a recreate of that failure, try logging the compute node record in _update() before calling reportclient.set_inventory_for_provider
16:57:04 maciejjozefczyk mriedem: ok
17:01:07 diga jaypipes: Hi
17:01:38 diga jaypipes: have you seen my mail on this bug - https://bugs.launchpad.net/nova/+bug/1719933
17:01:40 openstack Launchpad bug 1719933 in OpenStack Compute (nova) "placement server needs to retry allocations, server-side" [Medium,Triaged] - Assigned to Jay Pipes (jaypipes)
17:03:57 cdent diga: twitter suggests that jay is rather ill today
17:04:13 diga cdent: ohh
17:05:50 maciejjozefczyk mriedem: Ok, I'll paste it tomorrow, need to go
17:05:55 mdbooth cfriesen: The behaviour of that formatting is lost in the depths of time. That sounds like a bona fide bug, though.
17:06:07 diga cdent: do you have sometime, can you help me ?
17:06:07 mriedem maciejjozefczyk: o/
17:06:24 cdent diga: I might be able to yeah, what's up?
17:06:34 mdbooth cfriesen: Do other clouds format blank ephemeral disks for you? It seems like such a weird thing to do in Nova.
17:07:39 maciejjozefczyk first sight, strange, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L763 gives me https://pastebin.com/skR0X0Mm (at init)
17:07:46 mdbooth cfriesen: I don't think we can change that behaviour without at least a microversion bump, btw. Probably a cycle of ops discussion, a prominent release note, and a microversion bump.
17:07:55 maciejjozefczyk mriedem: and cpu_allocation_ratio is set as 16.0
17:08:35 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver https://review.openstack.org/526094
17:08:49 mriedem maciejjozefczyk: that's because it pulls the object from the db and runs it through ComputeNode._from_db_object which sets the 'default' allocation ratios (hardcoded in code) if not set in config
17:08:59 mdbooth cfriesen: When the ephemeral disk is too big, does the mkfs choke causing the spawn to fail?
17:09:14 maciejjozefczyk mriedem: I'll check it tomorrow why then it is changed to 0
17:09:25 maciejjozefczyk mriedem: thats the real cause, yes?
17:09:51 mriedem yeah i don't know what is changing it to 0.0 before we hit placement
17:10:11 maciejjozefczyk mriedem: I'll find it for you ;) bb
17:10:18 mriedem godspeed
17:11:42 diga cdent:
17:11:49 diga cdent: - Here I got the error after adding scenarios - http://paste.openstack.org/show/641294/
17:12:26 diga cdent: changes are made in git diff - http://paste.openstack.org/show/641293/
17:12:28 cdent diga: do you have a work in progress that you post up to gerrit so there's code to look at?
17:13:10 diga cdent: I have submit the patch yet, Let me submit the patch then
17:13:20 diga s/have/haven't
17:14:15 diga cdent: yes, I am working on this as per jaypipes suggestions
17:14:43 cdent diga: note that DbDeadLock is already handled, what's not handled is ConcurrentUpdate
17:15:28 diga cdent: okay
17:15:46 cdent _set_allocations method already has the retry handing on it
17:16:00 cdent but it is only set up for handling db api exceptions, which ConcurrentUpdate is not
17:16:28 mriedem sdague: this is the novaclient change for the file injection deprecation and userdata + rebuild stuff https://review.openstack.org/#/c/528128/ - closes out that bp and unblocks the next novaclient change in the series for the next microversion; client release freeze is creeping up so i'd like to get some reviews on this stuff
17:16:40 diga okk, I will work on it then
17:17:10 cdent diga: let me know if/how I can help
17:17:51 diga cdent: currently main challenge in nova is to reproduce the issue, db part is reproduced, but how to reproduce ConcurrentUpdate
17:18:00 diga cdent: some pointers can be helpful
17:18:03 mriedem diga: i have a devstack patch that reproduces it...
17:18:46 cdent diga: I think you can probably do something similar to what you've done in your existing test, but put the side effect on the method that increase the generation
17:18:46 mriedem https://review.openstack.org/#/c/507918/
17:19:01 cdent diga: _increment_provider_generation
17:19:41 diga cdent: okay
17:20:07 diga mriedem: I will take a look at it
17:20:47 mriedem it's probably not very helpful though for recreating a concurrent update failure in a unit test
17:21:00 mriedem it was just something i think i noticed while investigating failures in that devstack patch
17:21:14 diga mriedem: but it needs to rerun devstack with this change
17:21:40 diga mriedem: ok, got it
17:29:08 stephenfin artom: https://review.rdoproject.org/r/#/c/11283/
17:34:37 diga mriedem: Thanks for sharing the link, it will be certainly helpful
17:34:54 diga cdent: thanks for your help
17:35:24 diga cdent: mriedem : will ping you if I need any help, will update you by tomorrow
17:38:25 cfriesen mdbooth: sorry, was off in a meeting. Yes, the initial spawn fails when the ephemeral disk is too big. I wonder if we could change the default based on size, to either use something else or just not format it instead of choking.
17:39:55 mdbooth cfriesen: I think it has to be considered part of the api. I think the only change we can make to it without a microversion bump is a minimal change to make it not fail.
17:40:21 mdbooth cfriesen: So we could, for eg, not format it at all, but only in the case that we know it would cause a failure to build.
17:41:52 cfriesen mdbooth: agreed. I'll open a bug and maybe propose a fix.
18:53:51 openstackgerrit melanie witt proposed openstack/nova master: Detach volume after deleting instance with no host https://review.openstack.org/340614
18:55:18 mriedem stvnoyes: finally got that multiattach snapshot test to pass http://logs.openstack.org/86/531386/7/check/tempest-full/2f25c03/job-output.txt.gz#_2018-01-10_02_15_35_840008
18:56:01 stvnoyes excellent. I've been working on libvirt 3.10. Finally got past the dependency issues working on getting stack up now
18:56:33 mriedem for the tempest patch, it's getting big, so i might need to think about splitting it up
18:56:40 mriedem mtreinish: how do you feel about this? https://review.openstack.org/#/c/266605/25/tempest/api/compute/volumes/test_attach_volume.py
18:56:53 melwitt is anyone willing to please review the local delete patch ^ if an instance with attached volumes etc goes into error state, when it's deleted the volumes and networks aren't disconnected from the instance and have to be manually disconnected in order to be used again
18:56:56 mriedem should i do the first patch with the config option and 1 test, then add the other tests in subsequent patches?
19:01:06 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver https://review.openstack.org/526094
19:04:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406
19:04:03 openstackgerrit Matt Riedemann proposed openstack/nova master: Add index(updated_at) on migrations table. https://review.openstack.org/531132
19:04:03 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix comment in MigrationSortContext https://review.openstack.org/532368
19:05:31 ameeda Hello, when I try to deploy overcloud on baremetal using undercloud "installed at vm" I got this error "No compute node record for host undercloud: ComputeHostNotFound_Remote: Compute host undercloud could not be found." from nova-compute.log file
19:05:43 ameeda what can I do with that ?
19:10:00 openstackgerrit Mark Goddard proposed openstack/nova master: WIP: Send traits to ironic on server boot https://review.openstack.org/508116
19:10:00 openstackgerrit Mark Goddard proposed openstack/nova master: Add get_traits() method to ComputeDriver https://review.openstack.org/532287
19:10:01 openstackgerrit Mark Goddard proposed openstack/nova master: Add support to scheduler client for setting traits https://review.openstack.org/532539
19:10:01 openstackgerrit Mark Goddard proposed openstack/nova master: Call get_traits() in the resource tracker https://review.openstack.org/532290
19:10:02 openstackgerrit Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288
19:10:45 rybridges Hey guys. Got another question for you today. Is there a way to update the user-data on an instance after it is in the build state during the boot flow? I have a hunch that it is not possible because once the instance is in the build state, the user-data and config driver stuff have already been written out onto the that instance's partition on the hypervisor. Meaning in order to update the user data
19:10:47 rybridges on the instance after it is built, we would need some api capable of modifying user data on the instance. does something like that exist?

Earlier   Later