Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-30
20:18:10 mriedem mordred: do you ever have to rely on scheduler hints?
20:19:26 clarkb mriedem: I think any of that would be largely transparent to infra at least since we are just "public" cloud ocnsumers for the most part. and since infracloud is single tenant we don't need any special scheduling we just schedule to any of our nodes
20:20:40 mriedem clarkb: ok. i just figure anyone that is a user of openstack and has to rely on scheduler hints for their app to work, probably has a hell of a time making that app work across different clouds - but i have no idea if something like that exists in the wild
20:21:16 clarkb I know ovh schedules us to dedicate resources so we don't noisy neighbor their customers
20:21:25 clarkb they do this with a flavor just for us not sure if that involves scheduler hints
20:21:32 clarkb but ya we don't directly interact with scheduler hints at least
20:21:54 mriedem clarkb: yeah that wouldn't be scheduler hints, just extra specs and host aggregates
20:22:25 openstackgerrit Eric Fried proposed openstack/nova master: Parse granular resources/traits from extra_specs https://review.openstack.org/515151
20:22:25 openstackgerrit Eric Fried proposed openstack/nova master: Granularize resources_from_{flavor|request_spec} https://review.openstack.org/515223
20:22:26 openstackgerrit Eric Fried proposed openstack/nova master: ResourceRequest.to_querystring() https://review.openstack.org/515811
20:23:26 mriedem i need to know if cloud x has a scheduler hint so i can tell it which lpar i want my powervm instance on
20:23:32 mriedem because lpar 44 is the one i want
20:26:41 jaypipes mriedem: and how would a cloud user know that lpar 44 is the one they want? :)
20:27:45 sdague jaypipes: probably the same way they'd know which az they wanted, it was documented somewhere.
20:28:28 jaypipes sdague: heh
20:29:10 jaypipes sdague: my point is that the AZ is a publicly-visible grouping construct that OpenStack exposes. LPARs are not publicly-visible constructs...
20:30:05 jaypipes sdague: in the same sense that on AWS I cannot request a particular compute host (because, well, as a cloud user, I don't *know* what the compute hosts are...)
20:30:48 jaypipes sdague: but maybe you're just pulling my leg... I can't tell :)
20:31:38 mriedem i'm just using a ridiculous example
20:32:11 mriedem but yeah, just saying, one cloud/product docs could have any number of weird things tailored for their users
20:34:33 mriedem if we actually cared, we'd have some kind of scheduler_hints API to return what *this* cloud supports for scheduler hints, so it would at least be discoverable
20:35:12 jaypipes mriedem: sure, agreed. discoverable, not interoperable...
20:35:25 mriedem right, neither are flavors
20:35:32 jaypipes mriedem: ack
20:35:53 jaypipes mriedem: but standard resource classes and os-traits are getting us much closer in that respect.
20:44:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Transform keypair.import notification https://review.openstack.org/467514
21:00:33 mriedem this is fun, http://logs.openstack.org/96/516396/1/check/legacy-tempest-dsvm-py35/ecb9db4/logs/screen-n-api.txt.gz#_Oct_30_18_01_18_122380
21:00:41 mriedem can't find that request in the scheduler logs to see why NoValidHost
21:04:29 mriedem ah because we overwrite the NoValidHost message before returning to the api
21:04:34 mriedem dansmith: http://logs.openstack.org/96/516396/1/check/legacy-tempest-dsvm-py35/ecb9db4/logs/screen-n-super-cond.txt.gz?level=TRACE#_Oct_30_18_01_18_003148
21:05:44 mriedem (409 {"errors": [{"status": 409, "request_id": "req-7eec8dd2-f65c-43fa-b3df-cdf7a236aa03", "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Inventory changed while attempting to allocate: Another thread concurrently updated the data. Please retry your update "}]})
21:17:05 mriedem e-r seems to be dead so can't tell how bad this is http://status.openstack.org/elastic-recheck/
21:21:59 mriedem https://bugs.launchpad.net/nova/+bug/1728722
21:22:01 openstack Launchpad bug 1728722 in OpenStack Compute (nova) "Resize test fails in conductor during migration/instance allocation swap: "Unable to replace resource claim on source host"" [High,New]
21:22:30 mriedem ok, so what's probably happening is conductor is trying to 'claim' allocations in placement at the same time as the scheduler for a concurrent test, and capacity changed so we have to retry client-side, just like in the scheduler
21:25:13 openstackgerrit Merged openstack/nova master: rp: break functions out of _set_traits() https://review.openstack.org/509908
21:25:41 openstackgerrit Merged openstack/nova master: Fix live migration grenade ceph setup https://review.openstack.org/508271
21:25:49 openstackgerrit Merged openstack/nova master: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225
21:30:59 cfriesen mriedem: is there a bug for the fact that rebuilding a boot-from-volume instance changes the reported image without changing the volume contents? Or are we just tracking that under https://bugs.launchpad.net/nova/+bug/1482040 ?
21:31:00 openstack Launchpad bug 1482040 in OpenStack Compute (nova) "rebuild volume-backed instance with image does not replace image in root disk" [Undecided,Confirmed]
21:31:29 mriedem just tracking that under that bug
21:32:46 cfriesen is there any concensus on how to handle that? if not, should we bring it up at the Forum?
21:32:59 mriedem we should fail fast in the api if it's a volume-backed instance
21:33:27 cfriesen sounds reasonable
21:38:48 kukacz hi, is there a method how to ensure user data script execution on instance creation without making the user to use eg. the --user-data CLI argument?
21:39:39 kukacz I'm thinking of doing it based eg. on some pre-defined metadata value presence
21:43:07 mriedem kukacz: check this out https://docs.openstack.org/nova/latest/user/vendordata.html
21:43:46 mriedem configure nova for force_config_drive=True so every instance gets a config drive, and then you can setup a vendordata v2 service for injecting whatever you need into the guest
21:46:19 mriedem jaypipes: for this todo, we'd want a microversion so the client can detect if it can rely on the server doing the retries or if the client has to do them https://github.com/openstack/nova/blob/965f56d7d2ca1f668f70d24d4dcc20e418bb5b9c/nova/objects/resource_provider.py#L1887
21:46:44 mriedem jaypipes: but i'm wondering if we want to make that a hard-coded retry value in placement (or configurable), or allow the user to pass in the number of retries (probably not)
21:47:31 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/511035
21:47:35 mriedem or allow the user to request a certain number of retries but have a configurable max in placement, defaulted to 3?
21:47:58 kukacz mriedem: thanks! looking into that
21:48:17 dansmith mriedem: is that a generation mismatch thing?
21:48:21 mriedem dansmith: yup
21:48:28 cfriesen mriedem: does a rebuild in the boot-from-vol case rebuild any of the volume connection stuff in a way that wouldn't happen on a stop/start? if so then maybe we'd only want to fail if we try to rebuild to a different image
21:48:44 dansmith mriedem: so we're not retrying those in the reportclient I guess?
21:48:49 mriedem dansmith: not in put_allocations
21:48:56 mriedem which is what the migrate task uses
21:49:03 dansmith ack okay
21:49:07 mriedem the claim_resources method in the reportclient is the thing the scheduler calls and that does the double up and retry logic
21:49:15 dansmith okay
21:49:22 dansmith mriedem: you gots a bug yet?
21:49:28 mriedem so we could hacky sack a bit and have migrate call claim_resources but pass a "don't double things"
21:49:35 mriedem dansmith: yeah https://bugs.launchpad.net/nova/+bug/1728722
21:49:36 openstack Launchpad bug 1728722 in OpenStack Compute (nova) "Resize test fails in conductor during migration/instance allocation swap: "Unable to replace resource claim on source host"" [High,Triaged]
21:50:02 dansmith mriedem: nah, we can just make a wrapper for put that does it I think and refactor a bit
21:50:12 mriedem that's what i started doing,
21:50:17 dansmith mriedem: I'll work on that tomorrow
21:50:39 mriedem but then you end up copying most of the put_allocations method from placement, and have to get the response back for the status_code
21:50:45 kukacz mriedem: which release was this vendordata feature introduced in? Pike?
21:50:47 mriedem and reportclient.put doesn't have the safe_connect on it
21:50:51 mriedem but i think we said we could remove that...
21:51:00 dansmith mriedem: I think we want the wrapper in reportclient though
21:51:00 mriedem kukacz: newton i think, and enhanced in ocata
21:51:09 mriedem dansmith: ah yeah that would work
21:51:14 dansmith mriedem: I would think we could make claim_resources use the wrapped version too
21:51:16 dansmith to avoid that duplication
21:51:31 dansmith anything calling put_allocations shouldn't have to worry about generation retries
21:51:59 openstackgerrit Merged openstack/nova master: use context mgr in instance.delete https://review.openstack.org/443764
21:52:02 kukacz mriedem: great, seems this could help our usecase. thanks a lot!
21:52:22 mriedem kukacz: yw
21:54:16 melwitt mriedem: I'm not seeing where https://review.openstack.org/#/c/508163 ran the test_volume_backed_live_migration test in the results
21:55:03 mriedem melwitt: http://logs.openstack.org/63/508163/4/check/legacy-tempest-dsvm-multinode-live-migration/633c914/job-output.txt.gz#_2017-10-30_20_50_05_842520
21:55:42 melwitt mriedem: oh, thanks. I was looking at the http://logs.openstack.org/63/508163/4/check/legacy-tempest-dsvm-multinode-live-migration/633c914/logs/testr_results.html.gz and I guess it doesn't drill down that much
21:55:47 mriedem nope
21:55:52 jaypipes mriedem: I think a hard-coded server-side retry is what I would go with.
21:55:53 mriedem these are post test hook driven
21:56:01 melwitt oh :\
21:56:14 mriedem melwitt: looks like i didn't uncomment the part to run it with ceph as the ephemeral backend, but not sure that would make any difference
21:56:14 jaypipes mriedem: but right now, dinner...
21:56:22 mriedem https://review.openstack.org/#/c/508163/4/nova/tests/live_migration/hooks/run_tests.sh@67
21:56:48 melwitt oh, I see
21:57:18 melwitt I *think* that doesn't make a difference but we could do it just to be sure
22:00:14 mriedem melwitt: yeah it might given the comment for VIR_MIGRATE_NON_SHARED_INC says "Migration with non-shared storage with incremental disk copy"
22:00:23 mriedem so i guess i'll rev that test patch to see what happens
22:00:41 melwitt yeah, I think that'd be best. I'll keep an eye out for the result
22:01:11 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Run test_volume_backed_live_migration and iscsi test https://review.openstack.org/508163
22:02:46 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: Don't VIR_MIGRATE_NON_SHARED_INC without migrate_disks https://review.openstack.org/507202
22:02:46 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Run test_volume_backed_live_migration and iscsi test https://review.openstack.org/508163

Earlier   Later