| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-04 | |||
| 17:57:07 | sean-k-mooney | we chose to jsut document what ws not supprotted | |
| 17:57:11 | donnyd | scp openstack:/var/lib/glance/images/8dd685fd-277d-46fb-8d1c-b05ff7c5ecaa | |
| 17:57:12 | donnyd | 8dd685fd-277d-46fb-8d1c-b05ff7c5ecaa 100% 6212MB 209.1MB/s 00:29 | |
| 17:57:16 | mriedem | most times when we know shit doesn't work we just don't fix it, don't fail fast detect it, or document it | |
| 17:57:24 | donnyd | so 2x faster than glance | |
| 17:57:38 | mriedem | over time people eventually get tired of that so they put fail fast checks in place, like the things that don't work in the api with volume-backed servers | |
| 17:57:40 | donnyd | but dansmith is correct. CPU bound on the controller side | |
| 17:58:15 | sean-k-mooney | mriedem: yes so aspiers is breakign the mold by puting the check in place with the code | |
| 17:58:16 | dansmith | donnyd: scp'ing from and to the same disks on each side presumably? | |
| 17:58:42 | mriedem | sean-k-mooney: not really, like i said, we have "this doesn't work with this type of instance" checks in the API for certain things | |
| 17:58:56 | mriedem | he's just doing it a bit differently with a decorator, it's the same idea though | |
| 17:59:14 | sean-k-mooney | ya | |
| 17:59:28 | sean-k-mooney | i understand that and that we do have check like this | |
| 17:59:41 | sean-k-mooney | aspiers: so what are you going to return to the use | |
| 17:59:47 | sean-k-mooney | a 4** | |
| 17:59:54 | mriedem | the bw provider qos stuff fails fast in the api for move ops today too | |
| 18:00:01 | sean-k-mooney | yes | |
| 18:00:10 | sean-k-mooney | i was looking ot see if i could find that | |
| 18:00:20 | donnyd | https://www.irccloud.com/pastebin/wQsy8Dyc/ | |
| 18:00:35 | donnyd | not likely to be a network issue | |
| 18:00:47 | mriedem | sean-k-mooney: for the bw provider case (supports_port_resource_request_during_move) we return a 400 | |
| 18:00:58 | sean-k-mooney | for consitnece we proablyt want to fail the migration with the same respoce code | |
| 18:01:07 | dansmith | donnyd: there are various chunk sizes for the glance backends, which may be too small by default for a fast network | |
| 18:01:13 | sean-k-mooney | ok 400 bad request makes sense i guess | |
| 18:01:19 | mriedem | i'm not sure that 400 was really correct there, 409 might have been better, but ... | |
| 18:02:17 | aspiers | sean-k-mooney: I can't even find the place where migration chooses which error codes to return. There's no mention of migration in nova.api.openstack.compute.servers | |
| 18:02:20 | sean-k-mooney | i was just looking at 409 | |
| 18:02:21 | sean-k-mooney | https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/409 | |
| 18:02:39 | donnyd | dansmith: using file backend because it performs the fastest | |
| 18:02:41 | sean-k-mooney | i think we had this conversation for bandwidth too | |
| 18:03:07 | dansmith | donnyd: yeah, there are chunk size knobs for several of them, filesystem included | |
| 18:03:07 | aspiers | oh found it | |
| 18:03:17 | aspiers | nova.api.openstack.compute.migrate_servers | |
| 18:03:25 | dansmith | donnyd: 64k by default, apparently, which is pretty dang small | |
| 18:03:49 | donnyd | Yea that is a little on the small side | |
| 18:04:07 | dansmith | for 1gE that's probably okay, but not 10gE I'd think.. | |
| 18:04:23 | sean-k-mooney | aspiers: right here https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/migrate_server.py#L61-L68 | |
| 18:04:29 | donnyd | the controller side is 40G | |
| 18:04:34 | aspiers | OK so I want the check to go in nova.api.openstack.compute.migrate_servers not nova.compute.api | |
| 18:04:36 | donnyd | and each compute is 10 | |
| 18:04:44 | dansmith | donnyd: the rbd default chunk is 8MiB, fwiw | |
| 18:05:15 | sean-k-mooney | aspiers: or in your case you want the live_migrate check below it | |
| 18:05:18 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/migrate_server.py#L127-L137 | |
| 18:05:22 | dansmith | although it's not clear if that chunk size affects streaming like the filesystem one does | |
| 18:05:58 | sean-k-mooney | aspiers: so that his proably the method you want to extend to call into hardware.py and check if sev is requested | |
| 18:06:06 | mriedem | aspiers: i don't know that it matters really; i liked the decorator b/c you don't have to copy/paste the validation logic | |
| 18:06:31 | mriedem | if you do it in nova.compute.api.API methods you'll have to handle whatever exception you raise so that the api doesn't return a 500 | |
| 18:06:43 | aspiers | mriedem: got it | |
| 18:06:58 | sean-k-mooney | mriedem: well you chould still use a decorator on thos funcitons | |
| 18:07:12 | mriedem | sean-k-mooney: i'd rather not use a decorator on those | |
| 18:07:17 | aspiers | yes, so decorate nova.compute.api.API and handle the new exception in nova.api.openstack.compute.migrate_server | |
| 18:07:19 | mriedem | the decorator for wsgi stuff is really about wsgi | |
| 18:07:27 | sean-k-mooney | ok | |
| 18:07:32 | mriedem | the compute API methods already have decorators that do sniff tests | |
| 18:07:37 | mriedem | like task_state, is locked, etc | |
| 18:07:42 | aspiers | yeah I was copying those | |
| 18:07:59 | aspiers | now we just have to spend 3 hours bikeshedding the new exception name | |
| 18:08:12 | sean-k-mooney | aspiers: or just add a method to hardware.py that raise the correct excption | |
| 18:08:24 | sean-k-mooney | then jsut call it in all the palces you need too | |
| 18:08:35 | aspiers | hardware.py doesn't know what operation is being done | |
| 18:08:37 | sean-k-mooney | most of the exception in hardware.py inherit form the http ones | |
| 18:09:04 | sean-k-mooney | yes but you can call it form the api and pass in the oppertion if its needed | |
| 18:09:23 | aspiers | that feels like mixing up layers to me | |
| 18:09:42 | aspiers | I think I have enough hints to get it done now, thanks guys | |
| 18:09:59 | sean-k-mooney | not sure i helped much | |
| 18:10:10 | mriedem | stephenfin: ok, -1 :) | |
| 18:10:14 | aspiers | sean-k-mooney: ;-) | |
| 18:13:07 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Public method to retrieve custom resources https://review.opendev.org/676029 | |
| 18:13:07 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 18:13:08 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Update provider tree with new custom resources https://review.opendev.org/676522 | |
| 18:13:49 | sean-k-mooney | dansmith: artom: so live migration form (master) src node to (artoms code) dest node with unpinned or pinned rpc | |
| 18:14:02 | sean-k-mooney | appears to work the same as if we pin to the old rpc version | |
| 18:14:05 | sean-k-mooney | e.g. no xml updates | |
| 18:15:03 | sean-k-mooney | and you can live migrate it back too | |
| 18:16:34 | mriedem | here it comes | |
| 18:17:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.opendev.org/627890 | |
| 18:17:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.opendev.org/633293 | |
| 18:17:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.opendev.org/627891 | |
| 18:17:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.opendev.org/634832 | |
| 18:17:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646 | |
| 18:17:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080 | |
| 18:17:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684 | |
| 18:17:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668 | |
| 18:17:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253 | |
| 18:17:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | |
| 18:17:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592 | |
| 18:17:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591 | |
| 18:17:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451 | |
| 18:17:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450 | |
| 18:17:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013 | |
| 18:17:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012 | |
| 18:17:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.opendev.org/637070 | |
| 18:17:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058 | |
| 18:19:22 | sean-k-mooney | while its on the dest node with the invalid pinning we get the message we expect in the update resouce providers http://paste.openstack.org/show/770652/ whic is the same behavior as master today | |
| 18:21:40 | sean-k-mooney | dansmith: artom im going to lie down and take some panadol and hopefully my headache will go away. let me know if i need to test anything else and i will do it tomorrow | |
| 18:22:00 | dansmith | sean-k-mooney: cool, thanks for these confirmations... | |
| 18:22:35 | donnyd | sean-k-mooney: using 2x the ceph defaults to start | |
| 18:22:40 | donnyd | we will see how it goes | |
| 18:22:50 | sean-k-mooney | 2x? | |
| 18:23:14 | dansmith | was that intended for me? | |