| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-30 | |||
| 20:20:40 | mriedem | clarkb: ok. i just figure anyone that is a user of openstack and has to rely on scheduler hints for their app to work, probably has a hell of a time making that app work across different clouds - but i have no idea if something like that exists in the wild | |
| 20:21:16 | clarkb | I know ovh schedules us to dedicate resources so we don't noisy neighbor their customers | |
| 20:21:25 | clarkb | they do this with a flavor just for us not sure if that involves scheduler hints | |
| 20:21:32 | clarkb | but ya we don't directly interact with scheduler hints at least | |
| 20:21:54 | mriedem | clarkb: yeah that wouldn't be scheduler hints, just extra specs and host aggregates | |
| 20:22:25 | openstackgerrit | Eric Fried proposed openstack/nova master: Granularize resources_from_{flavor|request_spec} https://review.openstack.org/515223 | |
| 20:22:25 | openstackgerrit | Eric Fried proposed openstack/nova master: Parse granular resources/traits from extra_specs https://review.openstack.org/515151 | |
| 20:22:26 | openstackgerrit | Eric Fried proposed openstack/nova master: ResourceRequest.to_querystring() https://review.openstack.org/515811 | |
| 20:23:26 | mriedem | i need to know if cloud x has a scheduler hint so i can tell it which lpar i want my powervm instance on | |
| 20:23:32 | mriedem | because lpar 44 is the one i want | |
| 20:26:41 | jaypipes | mriedem: and how would a cloud user know that lpar 44 is the one they want? :) | |
| 20:27:45 | sdague | jaypipes: probably the same way they'd know which az they wanted, it was documented somewhere. | |
| 20:28:28 | jaypipes | sdague: heh | |
| 20:29:10 | jaypipes | sdague: my point is that the AZ is a publicly-visible grouping construct that OpenStack exposes. LPARs are not publicly-visible constructs... | |
| 20:30:05 | jaypipes | sdague: in the same sense that on AWS I cannot request a particular compute host (because, well, as a cloud user, I don't *know* what the compute hosts are...) | |
| 20:30:48 | jaypipes | sdague: but maybe you're just pulling my leg... I can't tell :) | |
| 20:31:38 | mriedem | i'm just using a ridiculous example | |
| 20:32:11 | mriedem | but yeah, just saying, one cloud/product docs could have any number of weird things tailored for their users | |
| 20:34:33 | mriedem | if we actually cared, we'd have some kind of scheduler_hints API to return what *this* cloud supports for scheduler hints, so it would at least be discoverable | |
| 20:35:12 | jaypipes | mriedem: sure, agreed. discoverable, not interoperable... | |
| 20:35:25 | mriedem | right, neither are flavors | |
| 20:35:32 | jaypipes | mriedem: ack | |
| 20:35:53 | jaypipes | mriedem: but standard resource classes and os-traits are getting us much closer in that respect. | |
| 20:44:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Transform keypair.import notification https://review.openstack.org/467514 | |
| 21:00:33 | mriedem | this is fun, http://logs.openstack.org/96/516396/1/check/legacy-tempest-dsvm-py35/ecb9db4/logs/screen-n-api.txt.gz#_Oct_30_18_01_18_122380 | |
| 21:00:41 | mriedem | can't find that request in the scheduler logs to see why NoValidHost | |
| 21:04:29 | mriedem | ah because we overwrite the NoValidHost message before returning to the api | |
| 21:04:34 | mriedem | dansmith: http://logs.openstack.org/96/516396/1/check/legacy-tempest-dsvm-py35/ecb9db4/logs/screen-n-super-cond.txt.gz?level=TRACE#_Oct_30_18_01_18_003148 | |
| 21:05:44 | mriedem | (409 {"errors": [{"status": 409, "request_id": "req-7eec8dd2-f65c-43fa-b3df-cdf7a236aa03", "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Inventory changed while attempting to allocate: Another thread concurrently updated the data. Please retry your update "}]}) | |
| 21:17:05 | mriedem | e-r seems to be dead so can't tell how bad this is http://status.openstack.org/elastic-recheck/ | |
| 21:21:59 | mriedem | https://bugs.launchpad.net/nova/+bug/1728722 | |
| 21:22:01 | openstack | Launchpad bug 1728722 in OpenStack Compute (nova) "Resize test fails in conductor during migration/instance allocation swap: "Unable to replace resource claim on source host"" [High,New] | |
| 21:22:30 | mriedem | ok, so what's probably happening is conductor is trying to 'claim' allocations in placement at the same time as the scheduler for a concurrent test, and capacity changed so we have to retry client-side, just like in the scheduler | |
| 21:25:13 | openstackgerrit | Merged openstack/nova master: rp: break functions out of _set_traits() https://review.openstack.org/509908 | |
| 21:25:41 | openstackgerrit | Merged openstack/nova master: Fix live migration grenade ceph setup https://review.openstack.org/508271 | |
| 21:25:49 | openstackgerrit | Merged openstack/nova master: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 21:30:59 | cfriesen | mriedem: is there a bug for the fact that rebuilding a boot-from-volume instance changes the reported image without changing the volume contents? Or are we just tracking that under https://bugs.launchpad.net/nova/+bug/1482040 ? | |
| 21:31:00 | openstack | Launchpad bug 1482040 in OpenStack Compute (nova) "rebuild volume-backed instance with image does not replace image in root disk" [Undecided,Confirmed] | |
| 21:31:29 | mriedem | just tracking that under that bug | |
| 21:32:46 | cfriesen | is there any concensus on how to handle that? if not, should we bring it up at the Forum? | |
| 21:32:59 | mriedem | we should fail fast in the api if it's a volume-backed instance | |
| 21:33:27 | cfriesen | sounds reasonable | |
| 21:38:48 | kukacz | hi, is there a method how to ensure user data script execution on instance creation without making the user to use eg. the --user-data CLI argument? | |
| 21:39:39 | kukacz | I'm thinking of doing it based eg. on some pre-defined metadata value presence | |
| 21:43:07 | mriedem | kukacz: check this out https://docs.openstack.org/nova/latest/user/vendordata.html | |
| 21:43:46 | mriedem | configure nova for force_config_drive=True so every instance gets a config drive, and then you can setup a vendordata v2 service for injecting whatever you need into the guest | |
| 21:46:19 | mriedem | jaypipes: for this todo, we'd want a microversion so the client can detect if it can rely on the server doing the retries or if the client has to do them https://github.com/openstack/nova/blob/965f56d7d2ca1f668f70d24d4dcc20e418bb5b9c/nova/objects/resource_provider.py#L1887 | |
| 21:46:44 | mriedem | jaypipes: but i'm wondering if we want to make that a hard-coded retry value in placement (or configurable), or allow the user to pass in the number of retries (probably not) | |
| 21:47:31 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/511035 | |
| 21:47:35 | mriedem | or allow the user to request a certain number of retries but have a configurable max in placement, defaulted to 3? | |
| 21:47:58 | kukacz | mriedem: thanks! looking into that | |
| 21:48:17 | dansmith | mriedem: is that a generation mismatch thing? | |
| 21:48:21 | mriedem | dansmith: yup | |
| 21:48:28 | cfriesen | mriedem: does a rebuild in the boot-from-vol case rebuild any of the volume connection stuff in a way that wouldn't happen on a stop/start? if so then maybe we'd only want to fail if we try to rebuild to a different image | |
| 21:48:44 | dansmith | mriedem: so we're not retrying those in the reportclient I guess? | |
| 21:48:49 | mriedem | dansmith: not in put_allocations | |
| 21:48:56 | mriedem | which is what the migrate task uses | |
| 21:49:03 | dansmith | ack okay | |
| 21:49:07 | mriedem | the claim_resources method in the reportclient is the thing the scheduler calls and that does the double up and retry logic | |
| 21:49:15 | dansmith | okay | |
| 21:49:22 | dansmith | mriedem: you gots a bug yet? | |
| 21:49:28 | mriedem | so we could hacky sack a bit and have migrate call claim_resources but pass a "don't double things" | |
| 21:49:35 | mriedem | dansmith: yeah https://bugs.launchpad.net/nova/+bug/1728722 | |
| 21:49:36 | openstack | Launchpad bug 1728722 in OpenStack Compute (nova) "Resize test fails in conductor during migration/instance allocation swap: "Unable to replace resource claim on source host"" [High,Triaged] | |
| 21:50:02 | dansmith | mriedem: nah, we can just make a wrapper for put that does it I think and refactor a bit | |
| 21:50:12 | mriedem | that's what i started doing, | |
| 21:50:17 | dansmith | mriedem: I'll work on that tomorrow | |
| 21:50:39 | mriedem | but then you end up copying most of the put_allocations method from placement, and have to get the response back for the status_code | |
| 21:50:45 | kukacz | mriedem: which release was this vendordata feature introduced in? Pike? | |
| 21:50:47 | mriedem | and reportclient.put doesn't have the safe_connect on it | |
| 21:50:51 | mriedem | but i think we said we could remove that... | |
| 21:51:00 | mriedem | kukacz: newton i think, and enhanced in ocata | |
| 21:51:00 | dansmith | mriedem: I think we want the wrapper in reportclient though | |
| 21:51:09 | mriedem | dansmith: ah yeah that would work | |
| 21:51:14 | dansmith | mriedem: I would think we could make claim_resources use the wrapped version too | |
| 21:51:16 | dansmith | to avoid that duplication | |
| 21:51:31 | dansmith | anything calling put_allocations shouldn't have to worry about generation retries | |
| 21:51:59 | openstackgerrit | Merged openstack/nova master: use context mgr in instance.delete https://review.openstack.org/443764 | |
| 21:52:02 | kukacz | mriedem: great, seems this could help our usecase. thanks a lot! | |
| 21:52:22 | mriedem | kukacz: yw | |
| 21:54:16 | melwitt | mriedem: I'm not seeing where https://review.openstack.org/#/c/508163 ran the test_volume_backed_live_migration test in the results | |
| 21:55:03 | mriedem | melwitt: http://logs.openstack.org/63/508163/4/check/legacy-tempest-dsvm-multinode-live-migration/633c914/job-output.txt.gz#_2017-10-30_20_50_05_842520 | |
| 21:55:42 | melwitt | mriedem: oh, thanks. I was looking at the http://logs.openstack.org/63/508163/4/check/legacy-tempest-dsvm-multinode-live-migration/633c914/logs/testr_results.html.gz and I guess it doesn't drill down that much | |
| 21:55:47 | mriedem | nope | |
| 21:55:52 | jaypipes | mriedem: I think a hard-coded server-side retry is what I would go with. | |
| 21:55:53 | mriedem | these are post test hook driven | |
| 21:56:01 | melwitt | oh :\ | |
| 21:56:14 | jaypipes | mriedem: but right now, dinner... | |
| 21:56:14 | mriedem | melwitt: looks like i didn't uncomment the part to run it with ceph as the ephemeral backend, but not sure that would make any difference | |
| 21:56:22 | mriedem | https://review.openstack.org/#/c/508163/4/nova/tests/live_migration/hooks/run_tests.sh@67 | |
| 21:56:48 | melwitt | oh, I see | |
| 21:57:18 | melwitt | I *think* that doesn't make a difference but we could do it just to be sure | |
| 22:00:14 | mriedem | melwitt: yeah it might given the comment for VIR_MIGRATE_NON_SHARED_INC says "Migration with non-shared storage with incremental disk copy" | |
| 22:00:23 | mriedem | so i guess i'll rev that test patch to see what happens | |
| 22:00:41 | melwitt | yeah, I think that'd be best. I'll keep an eye out for the result | |
| 22:01:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Run test_volume_backed_live_migration and iscsi test https://review.openstack.org/508163 | |
| 22:02:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Run test_volume_backed_live_migration and iscsi test https://review.openstack.org/508163 | |
| 22:02:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Don't VIR_MIGRATE_NON_SHARED_INC without migrate_disks https://review.openstack.org/507202 | |
| 22:03:09 | mriedem | i don't know why it says it can't merge | |
| 22:05:35 | melwitt | I never figured out why it says that | |