| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-07 | |||
| 19:25:46 | dansmith | it really depends on what the result is | |
| 19:25:55 | sean-k-mooney | https://review.opendev.org/#/c/674726/ | |
| 19:26:01 | sean-k-mooney | its was a nova patch | |
| 19:26:11 | dansmith | if it's data corruption, state intervention required, etc then it needs a check to be graceful | |
| 19:26:22 | dansmith | if it fails in some reasonable way then I'm not so concerned | |
| 19:26:42 | sean-k-mooney | dansmith: i think libvirt will raise an error | |
| 19:27:03 | sean-k-mooney | qemu will reject a migration if the domain has a hostdev that is not of type usb | |
| 19:27:20 | dansmith | for live migration I assume, but based on the changes we've made to the flow, I'm not sure what will happen on resize | |
| 19:27:20 | sean-k-mooney | so we will get to the migrate call and it will fail | |
| 19:27:24 | sean-k-mooney | yes | |
| 19:27:28 | dansmith | we might migrate the instance and ignore the fact that it's missing an accelerator | |
| 19:27:59 | sean-k-mooney | ya i dont know what will happen for resize evacuate or shelve | |
| 19:28:01 | sean-k-mooney | ill find out | |
| 19:28:05 | dansmith | if live fails in a predictable and recoverable way, then I'm fine without a check on that one, | |
| 19:28:22 | dansmith | which is what I said earlier that virt-specific limitations shouldn't be enforced in the api when we can help it | |
| 19:29:45 | Sundar | dansmith, sean-k-mooney: FWIW, IMHO, an explicit check for device profiles in extra specs, as in https://review.opendev.org/#/c/674726/, is probably the safest and clearest to the user. | |
| 19:30:26 | Sundar | We did say that it should be documentation only. But I don;t know if folks will read Cborg dics, or any docs, before kicking off an op. | |
| 19:31:24 | sean-k-mooney | Sundar: the main issue with that patch was it was too agressive in what it blocked | |
| 19:31:27 | Sundar | *Cyborg docs | |
| 19:31:39 | sean-k-mooney | and users never read docs until it breaks | |
| 19:31:58 | Sundar | sean-k-mooney: The details of the patch can be adjusted. | |
| 19:32:38 | Sundar | dansmith, efried, gibi, sean-k-mooney: Would you all recommend to bring back https://review.opendev.org/#/c/674726/? | |
| 19:36:02 | sean-k-mooney | i would not restore it as is. if we add code to block it we shoudl only block the operation we know do not work. | |
| 19:38:21 | sean-k-mooney | that change blocks all snapshoting and backups, and interface and volume atache/removal and other operations like rescue and lock. | |
| 19:39:53 | sean-k-mooney | blocking resize,live-migrate and evacuate might make sense. the rest i think are questionable | |
| 19:40:38 | sean-k-mooney | also blocking shelve might make sense but again we said document and fix as bugfixes so we could adress them one by one without api changes for each | |
| 19:42:44 | Sundar | "Without API changes" -- are you thinking of microversion changes for blocking now and end every unblock in the future, if we support more ops? | |
| 19:43:39 | Sundar | *and every | |
| 19:43:48 | sean-k-mooney | yes im saying we didnt want to do a microversion bump for evey one | |
| 19:47:18 | sean-k-mooney | speaking of op i just booted a vm with your latest revision so ill go test some of them | |
| 19:48:23 | Sundar | Sure. Thanks. | |
| 19:50:08 | sean-k-mooney | my isp broke my home network so its takeing longer then i hoped to test this. i might set up the multi node setup on my laptop instead of my home openstack on monday but ill do what i can on a singel node first | |
| 20:09:13 | efried | Sundar: I've always been in favor of that idea (blocking unsupported operations with a useful message rather than letting them fail "organically" and mysteriously), but I know others disagree. | |
| 20:11:25 | sean-k-mooney | for what its worth i have just done boot,stop,start,reboot,add/remove volume, rescue/unrescue and so far no errors | |
| 20:11:56 | sean-k-mooney | i am not seeing any real interaction with cyborg during those operations | |
| 20:12:42 | sean-k-mooney | which is more or less expected but im not sure if we would loss the acclerator when we regenerate the xml | |
| 20:13:26 | sean-k-mooney | i will look at this more closely next week but right now im just check to make sure the operation complette correctly | |
| 20:29:01 | Sundar | sean-k-mooney: Great. FWIW, I do most of these with FPGAs. Except rescue/unrescue with different images. The list of ops i have checked are in: https://review.opendev.org/673735 | |
| 20:29:44 | sean-k-mooney | im still makeing my way through the list. | |
| 20:30:11 | sean-k-mooney | im hoping to get access to a real server with a rush creek or vista creek next week or the week after | |
| 20:30:20 | sean-k-mooney | which one are you using again? | |
| 20:31:09 | sean-k-mooney | i have jsut done add/remove network interface, pause,unpause, suspend,resume, lock,unlock and rebuild | |
| 20:31:13 | Sundar | Rush Creek, DCP 1.2 | |
| 20:31:40 | sean-k-mooney | cool we have 1 server i think with one of each so ill ask for the rush creek system | |
| 20:31:56 | Sundar | OPAE version 1.1.2-1 | |
| 20:32:26 | Sundar | The OPAE packages come with some sample bitstreams, sp. NLB modes 0 and 3. I use both. | |
| 20:33:58 | Sundar | I can also be reached at ns1.sundar AT gmail DOT com if there is a need for more detailed responses or file transfers. | |
| 20:35:29 | Sundar | I have some utility functions to create/delete device profiles, ARQs using curl. ould be happy to share them if you prefer. | |
| 20:37:01 | sean-k-mooney | im just using the openstack client and i have a test script | |
| 20:38:26 | Sundar | The openstack client is WIP. There are patches to recast it to use openstacksdk etc. https://review.opendev.org/#/c/681391/ | |
| 20:38:48 | sean-k-mooney | yep im using that | |
| 20:39:02 | sean-k-mooney | with the openstacksdk patch too | |
| 20:39:26 | sean-k-mooney | there are a bunch of design issue with it that should be adressed but its kind of useable | |
| 20:40:11 | sean-k-mooney | like when you create a device profile you can only spify its name not the uuid but it only allows you to show a device profile by uuid not name | |
| 20:40:53 | Sundar | Yea, agreed. The APIs allow both name and uuid though. | |
| 20:41:12 | sean-k-mooney | yep so the patches are just incomplete | |
| 20:41:22 | sean-k-mooney | i have worked around that | |
| 20:42:08 | Sundar | Do you have any pointers for me to look at the rescheduling question? | |
| 20:43:37 | sean-k-mooney | i have not read the question so no. i was going to try and force it by making the vm spawn fail | |
| 20:44:29 | sean-k-mooney | ok so first bug. when i shelve the instance the arq is still bound to the host when the vm is shevle_offloaded | |
| 20:45:01 | Sundar | Yup, I have not added support for shelve. The Delete ARQ patch states what I support. | |
| 20:45:25 | sean-k-mooney | yep but we want to check them all anyway | |
| 20:46:01 | Sundar | Basically, I prioritized the basic ops. I asked some folks outside Intel what they do with FPGAs in their lab with Cyborg, and went with that. | |
| 20:47:56 | sean-k-mooney | ya which is fine. unshevel "works" but i dont see any interacation with cyborg os i suspect it would not have the accellerator attached after unshelve. | |
| 20:48:08 | sean-k-mooney | again thats fine we just need to document it | |
| 20:49:32 | Sundar | ok | |
| 20:50:25 | sean-k-mooney | ok so after unshelve the allocations do not container the fake device so ya that means the device woudl be lsot | |
| 20:54:14 | sean-k-mooney | it looks like resize to a different flavor and then back does not fix the placement allcoation | |
| 20:54:20 | sean-k-mooney | it did however complete | |
| 20:58:49 | Sundar | Yes, I have only tested with resize to the same flavor, which is of course a no op. I was just making sure that there is no basic gotcha. | |
| 20:59:30 | sean-k-mooney | this is what i tested with the fake driver today | |
| 20:59:32 | sean-k-mooney | http://paste.openstack.org/show/789306/ | |
| 20:59:56 | sean-k-mooney | Sundar: resize to same flavor should be blocked in the api | |
| 21:00:00 | sean-k-mooney | it is in the client | |
| 21:00:13 | sean-k-mooney | you can migrate but resize to same flavor is invalid | |
| 21:01:22 | Sundar | remove-vol/net: unrelated to Cyborg, right? | |
| 21:01:50 | sean-k-mooney | yes but you blocked them in your patch that check for the flavor extra spec | |
| 21:01:56 | sean-k-mooney | they seam to work fine | |
| 21:02:05 | Sundar | Suspend will not work with real FPGAs, because libvirt will error out with: Domain has assigned non-USB devices. | |
| 21:02:07 | sean-k-mooney | at least with the fake dirver | |
| 21:02:38 | sean-k-mooney | suspend shoudl do a managed save which will detach all hostdev devices | |
| 21:02:46 | Sundar | delete failed? | |
| 21:02:49 | sean-k-mooney | yes | |
| 21:03:12 | sean-k-mooney | it failed in the last versions too | |
| 21:03:34 | sean-k-mooney | the conductor explodes with an unexpected respoce form cyborg | |
| 21:03:48 | sean-k-mooney | when it tries to delete /unbind the arqs | |
| 21:04:03 | Sundar | Hmm, please send me the logs. I do deletes all the time, but your sequence of ops probably triggered something. | |
| 21:04:39 | sean-k-mooney | i found that was broken just with boot then delete two days ago | |
| 21:04:56 | sean-k-mooney | ill need to unstack and stack to be able to test it again | |
| 21:05:18 | Sundar | I'll try boot + delete. Was it hard reboot? | |
| 21:06:36 | sean-k-mooney | no just boot then delete | |
| 21:06:47 | sean-k-mooney | hardreboot seams to be fine | |
| 21:08:23 | sean-k-mooney | Sundar: https://etherpad.openstack.org/p/sean-cyborg-testing-delete-logs | |
| 21:08:36 | sean-k-mooney | paste.openstack.org isnt loadign for me | |
| 21:08:44 | sean-k-mooney | but that is all the nova and cyborg logs | |
| 21:10:19 | sean-k-mooney | it look like the cyborg api returned a 401 | |
| 21:12:19 | Sundar | devstack@cyborg-api.service[26903]: 2020-02-07 20:56:05.719 .... Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed. | |
| 21:12:53 | sean-k-mooney | yep | |
| 21:13:35 | sean-k-mooney | so the call to cyborg either need an admin token which i dont think is correct or the token expired and you need to handel that | |
| 21:14:29 | sean-k-mooney | deleting the vm again does not fix it by they way | |