| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-07 | |||
| 19:20:21 | sean-k-mooney | We will state the limitations, if any, in Cyborg documentation. | |
| 19:20:30 | sean-k-mooney | that was the last comment on that | |
| 19:22:11 | sean-k-mooney | option 2 was "2. Gradually phase in the support for the server operations and document the limitations in the meantime but don't actively block them in the API like this change does. They either work (by chance) or they don't, but they aren't officially supported. Once they are supported, we patch them in without a new microversion as bug fixes (or just claim test support so they are no longer | |
| 19:22:13 | sean-k-mooney | considered experimental)." | |
| 19:23:18 | sean-k-mooney | gibi: dansmith ^ are ye still ok with that regarding the livemigation check | |
| 19:23:44 | sean-k-mooney | so no check for now and document. and we can add one if we want in the future | |
| 19:25:37 | dansmith | sean-k-mooney: where was that? in the spec? | |
| 19:25:46 | dansmith | it really depends on what the result is | |
| 19:25:55 | sean-k-mooney | https://review.opendev.org/#/c/674726/ | |
| 19:26:01 | sean-k-mooney | its was a nova patch | |
| 19:26:11 | dansmith | if it's data corruption, state intervention required, etc then it needs a check to be graceful | |
| 19:26:22 | dansmith | if it fails in some reasonable way then I'm not so concerned | |
| 19:26:42 | sean-k-mooney | dansmith: i think libvirt will raise an error | |
| 19:27:03 | sean-k-mooney | qemu will reject a migration if the domain has a hostdev that is not of type usb | |
| 19:27:20 | dansmith | for live migration I assume, but based on the changes we've made to the flow, I'm not sure what will happen on resize | |
| 19:27:20 | sean-k-mooney | so we will get to the migrate call and it will fail | |
| 19:27:24 | sean-k-mooney | yes | |
| 19:27:28 | dansmith | we might migrate the instance and ignore the fact that it's missing an accelerator | |
| 19:27:59 | sean-k-mooney | ya i dont know what will happen for resize evacuate or shelve | |
| 19:28:01 | sean-k-mooney | ill find out | |
| 19:28:05 | dansmith | if live fails in a predictable and recoverable way, then I'm fine without a check on that one, | |
| 19:28:22 | dansmith | which is what I said earlier that virt-specific limitations shouldn't be enforced in the api when we can help it | |
| 19:29:45 | Sundar | dansmith, sean-k-mooney: FWIW, IMHO, an explicit check for device profiles in extra specs, as in https://review.opendev.org/#/c/674726/, is probably the safest and clearest to the user. | |
| 19:30:26 | Sundar | We did say that it should be documentation only. But I don;t know if folks will read Cborg dics, or any docs, before kicking off an op. | |
| 19:31:24 | sean-k-mooney | Sundar: the main issue with that patch was it was too agressive in what it blocked | |
| 19:31:27 | Sundar | *Cyborg docs | |
| 19:31:39 | sean-k-mooney | and users never read docs until it breaks | |
| 19:31:58 | Sundar | sean-k-mooney: The details of the patch can be adjusted. | |
| 19:32:38 | Sundar | dansmith, efried, gibi, sean-k-mooney: Would you all recommend to bring back https://review.opendev.org/#/c/674726/? | |
| 19:36:02 | sean-k-mooney | i would not restore it as is. if we add code to block it we shoudl only block the operation we know do not work. | |
| 19:38:21 | sean-k-mooney | that change blocks all snapshoting and backups, and interface and volume atache/removal and other operations like rescue and lock. | |
| 19:39:53 | sean-k-mooney | blocking resize,live-migrate and evacuate might make sense. the rest i think are questionable | |
| 19:40:38 | sean-k-mooney | also blocking shelve might make sense but again we said document and fix as bugfixes so we could adress them one by one without api changes for each | |
| 19:42:44 | Sundar | "Without API changes" -- are you thinking of microversion changes for blocking now and end every unblock in the future, if we support more ops? | |
| 19:43:39 | Sundar | *and every | |
| 19:43:48 | sean-k-mooney | yes im saying we didnt want to do a microversion bump for evey one | |
| 19:47:18 | sean-k-mooney | speaking of op i just booted a vm with your latest revision so ill go test some of them | |
| 19:48:23 | Sundar | Sure. Thanks. | |
| 19:50:08 | sean-k-mooney | my isp broke my home network so its takeing longer then i hoped to test this. i might set up the multi node setup on my laptop instead of my home openstack on monday but ill do what i can on a singel node first | |
| 20:09:13 | efried | Sundar: I've always been in favor of that idea (blocking unsupported operations with a useful message rather than letting them fail "organically" and mysteriously), but I know others disagree. | |
| 20:11:25 | sean-k-mooney | for what its worth i have just done boot,stop,start,reboot,add/remove volume, rescue/unrescue and so far no errors | |
| 20:11:56 | sean-k-mooney | i am not seeing any real interaction with cyborg during those operations | |
| 20:12:42 | sean-k-mooney | which is more or less expected but im not sure if we would loss the acclerator when we regenerate the xml | |
| 20:13:26 | sean-k-mooney | i will look at this more closely next week but right now im just check to make sure the operation complette correctly | |
| 20:29:01 | Sundar | sean-k-mooney: Great. FWIW, I do most of these with FPGAs. Except rescue/unrescue with different images. The list of ops i have checked are in: https://review.opendev.org/673735 | |
| 20:29:44 | sean-k-mooney | im still makeing my way through the list. | |
| 20:30:11 | sean-k-mooney | im hoping to get access to a real server with a rush creek or vista creek next week or the week after | |
| 20:30:20 | sean-k-mooney | which one are you using again? | |
| 20:31:09 | sean-k-mooney | i have jsut done add/remove network interface, pause,unpause, suspend,resume, lock,unlock and rebuild | |
| 20:31:13 | Sundar | Rush Creek, DCP 1.2 | |
| 20:31:40 | sean-k-mooney | cool we have 1 server i think with one of each so ill ask for the rush creek system | |
| 20:31:56 | Sundar | OPAE version 1.1.2-1 | |
| 20:32:26 | Sundar | The OPAE packages come with some sample bitstreams, sp. NLB modes 0 and 3. I use both. | |
| 20:33:58 | Sundar | I can also be reached at ns1.sundar AT gmail DOT com if there is a need for more detailed responses or file transfers. | |
| 20:35:29 | Sundar | I have some utility functions to create/delete device profiles, ARQs using curl. ould be happy to share them if you prefer. | |
| 20:37:01 | sean-k-mooney | im just using the openstack client and i have a test script | |
| 20:38:26 | Sundar | The openstack client is WIP. There are patches to recast it to use openstacksdk etc. https://review.opendev.org/#/c/681391/ | |
| 20:38:48 | sean-k-mooney | yep im using that | |
| 20:39:02 | sean-k-mooney | with the openstacksdk patch too | |
| 20:39:26 | sean-k-mooney | there are a bunch of design issue with it that should be adressed but its kind of useable | |
| 20:40:11 | sean-k-mooney | like when you create a device profile you can only spify its name not the uuid but it only allows you to show a device profile by uuid not name | |
| 20:40:53 | Sundar | Yea, agreed. The APIs allow both name and uuid though. | |
| 20:41:12 | sean-k-mooney | yep so the patches are just incomplete | |
| 20:41:22 | sean-k-mooney | i have worked around that | |
| 20:42:08 | Sundar | Do you have any pointers for me to look at the rescheduling question? | |
| 20:43:37 | sean-k-mooney | i have not read the question so no. i was going to try and force it by making the vm spawn fail | |
| 20:44:29 | sean-k-mooney | ok so first bug. when i shelve the instance the arq is still bound to the host when the vm is shevle_offloaded | |
| 20:45:01 | Sundar | Yup, I have not added support for shelve. The Delete ARQ patch states what I support. | |
| 20:45:25 | sean-k-mooney | yep but we want to check them all anyway | |
| 20:46:01 | Sundar | Basically, I prioritized the basic ops. I asked some folks outside Intel what they do with FPGAs in their lab with Cyborg, and went with that. | |
| 20:47:56 | sean-k-mooney | ya which is fine. unshevel "works" but i dont see any interacation with cyborg os i suspect it would not have the accellerator attached after unshelve. | |
| 20:48:08 | sean-k-mooney | again thats fine we just need to document it | |
| 20:49:32 | Sundar | ok | |
| 20:50:25 | sean-k-mooney | ok so after unshelve the allocations do not container the fake device so ya that means the device woudl be lsot | |
| 20:54:14 | sean-k-mooney | it looks like resize to a different flavor and then back does not fix the placement allcoation | |
| 20:54:20 | sean-k-mooney | it did however complete | |
| 20:58:49 | Sundar | Yes, I have only tested with resize to the same flavor, which is of course a no op. I was just making sure that there is no basic gotcha. | |
| 20:59:30 | sean-k-mooney | this is what i tested with the fake driver today | |
| 20:59:32 | sean-k-mooney | http://paste.openstack.org/show/789306/ | |
| 20:59:56 | sean-k-mooney | Sundar: resize to same flavor should be blocked in the api | |
| 21:00:00 | sean-k-mooney | it is in the client | |
| 21:00:13 | sean-k-mooney | you can migrate but resize to same flavor is invalid | |
| 21:01:22 | Sundar | remove-vol/net: unrelated to Cyborg, right? | |
| 21:01:50 | sean-k-mooney | yes but you blocked them in your patch that check for the flavor extra spec | |
| 21:01:56 | sean-k-mooney | they seam to work fine | |
| 21:02:05 | Sundar | Suspend will not work with real FPGAs, because libvirt will error out with: Domain has assigned non-USB devices. | |
| 21:02:07 | sean-k-mooney | at least with the fake dirver | |
| 21:02:38 | sean-k-mooney | suspend shoudl do a managed save which will detach all hostdev devices | |
| 21:02:46 | Sundar | delete failed? | |
| 21:02:49 | sean-k-mooney | yes | |
| 21:03:12 | sean-k-mooney | it failed in the last versions too | |
| 21:03:34 | sean-k-mooney | the conductor explodes with an unexpected respoce form cyborg | |
| 21:03:48 | sean-k-mooney | when it tries to delete /unbind the arqs | |
| 21:04:03 | Sundar | Hmm, please send me the logs. I do deletes all the time, but your sequence of ops probably triggered something. | |
| 21:04:39 | sean-k-mooney | i found that was broken just with boot then delete two days ago | |
| 21:04:56 | sean-k-mooney | ill need to unstack and stack to be able to test it again | |
| 21:05:18 | Sundar | I'll try boot + delete. Was it hard reboot? | |
| 21:06:36 | sean-k-mooney | no just boot then delete | |
| 21:06:47 | sean-k-mooney | hardreboot seams to be fine | |
| 21:08:23 | sean-k-mooney | Sundar: https://etherpad.openstack.org/p/sean-cyborg-testing-delete-logs | |