| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-07 | |||
| 19:31:39 | sean-k-mooney | and users never read docs until it breaks | |
| 19:31:58 | Sundar | sean-k-mooney: The details of the patch can be adjusted. | |
| 19:32:38 | Sundar | dansmith, efried, gibi, sean-k-mooney: Would you all recommend to bring back https://review.opendev.org/#/c/674726/? | |
| 19:36:02 | sean-k-mooney | i would not restore it as is. if we add code to block it we shoudl only block the operation we know do not work. | |
| 19:38:21 | sean-k-mooney | that change blocks all snapshoting and backups, and interface and volume atache/removal and other operations like rescue and lock. | |
| 19:39:53 | sean-k-mooney | blocking resize,live-migrate and evacuate might make sense. the rest i think are questionable | |
| 19:40:38 | sean-k-mooney | also blocking shelve might make sense but again we said document and fix as bugfixes so we could adress them one by one without api changes for each | |
| 19:42:44 | Sundar | "Without API changes" -- are you thinking of microversion changes for blocking now and end every unblock in the future, if we support more ops? | |
| 19:43:39 | Sundar | *and every | |
| 19:43:48 | sean-k-mooney | yes im saying we didnt want to do a microversion bump for evey one | |
| 19:47:18 | sean-k-mooney | speaking of op i just booted a vm with your latest revision so ill go test some of them | |
| 19:48:23 | Sundar | Sure. Thanks. | |
| 19:50:08 | sean-k-mooney | my isp broke my home network so its takeing longer then i hoped to test this. i might set up the multi node setup on my laptop instead of my home openstack on monday but ill do what i can on a singel node first | |
| 20:09:13 | efried | Sundar: I've always been in favor of that idea (blocking unsupported operations with a useful message rather than letting them fail "organically" and mysteriously), but I know others disagree. | |
| 20:11:25 | sean-k-mooney | for what its worth i have just done boot,stop,start,reboot,add/remove volume, rescue/unrescue and so far no errors | |
| 20:11:56 | sean-k-mooney | i am not seeing any real interaction with cyborg during those operations | |
| 20:12:42 | sean-k-mooney | which is more or less expected but im not sure if we would loss the acclerator when we regenerate the xml | |
| 20:13:26 | sean-k-mooney | i will look at this more closely next week but right now im just check to make sure the operation complette correctly | |
| 20:29:01 | Sundar | sean-k-mooney: Great. FWIW, I do most of these with FPGAs. Except rescue/unrescue with different images. The list of ops i have checked are in: https://review.opendev.org/673735 | |
| 20:29:44 | sean-k-mooney | im still makeing my way through the list. | |
| 20:30:11 | sean-k-mooney | im hoping to get access to a real server with a rush creek or vista creek next week or the week after | |
| 20:30:20 | sean-k-mooney | which one are you using again? | |
| 20:31:09 | sean-k-mooney | i have jsut done add/remove network interface, pause,unpause, suspend,resume, lock,unlock and rebuild | |
| 20:31:13 | Sundar | Rush Creek, DCP 1.2 | |
| 20:31:40 | sean-k-mooney | cool we have 1 server i think with one of each so ill ask for the rush creek system | |
| 20:31:56 | Sundar | OPAE version 1.1.2-1 | |
| 20:32:26 | Sundar | The OPAE packages come with some sample bitstreams, sp. NLB modes 0 and 3. I use both. | |
| 20:33:58 | Sundar | I can also be reached at ns1.sundar AT gmail DOT com if there is a need for more detailed responses or file transfers. | |
| 20:35:29 | Sundar | I have some utility functions to create/delete device profiles, ARQs using curl. ould be happy to share them if you prefer. | |
| 20:37:01 | sean-k-mooney | im just using the openstack client and i have a test script | |
| 20:38:26 | Sundar | The openstack client is WIP. There are patches to recast it to use openstacksdk etc. https://review.opendev.org/#/c/681391/ | |
| 20:38:48 | sean-k-mooney | yep im using that | |
| 20:39:02 | sean-k-mooney | with the openstacksdk patch too | |
| 20:39:26 | sean-k-mooney | there are a bunch of design issue with it that should be adressed but its kind of useable | |
| 20:40:11 | sean-k-mooney | like when you create a device profile you can only spify its name not the uuid but it only allows you to show a device profile by uuid not name | |
| 20:40:53 | Sundar | Yea, agreed. The APIs allow both name and uuid though. | |
| 20:41:12 | sean-k-mooney | yep so the patches are just incomplete | |
| 20:41:22 | sean-k-mooney | i have worked around that | |
| 20:42:08 | Sundar | Do you have any pointers for me to look at the rescheduling question? | |
| 20:43:37 | sean-k-mooney | i have not read the question so no. i was going to try and force it by making the vm spawn fail | |
| 20:44:29 | sean-k-mooney | ok so first bug. when i shelve the instance the arq is still bound to the host when the vm is shevle_offloaded | |
| 20:45:01 | Sundar | Yup, I have not added support for shelve. The Delete ARQ patch states what I support. | |
| 20:45:25 | sean-k-mooney | yep but we want to check them all anyway | |
| 20:46:01 | Sundar | Basically, I prioritized the basic ops. I asked some folks outside Intel what they do with FPGAs in their lab with Cyborg, and went with that. | |
| 20:47:56 | sean-k-mooney | ya which is fine. unshevel "works" but i dont see any interacation with cyborg os i suspect it would not have the accellerator attached after unshelve. | |
| 20:48:08 | sean-k-mooney | again thats fine we just need to document it | |
| 20:49:32 | Sundar | ok | |
| 20:50:25 | sean-k-mooney | ok so after unshelve the allocations do not container the fake device so ya that means the device woudl be lsot | |
| 20:54:14 | sean-k-mooney | it looks like resize to a different flavor and then back does not fix the placement allcoation | |
| 20:54:20 | sean-k-mooney | it did however complete | |
| 20:58:49 | Sundar | Yes, I have only tested with resize to the same flavor, which is of course a no op. I was just making sure that there is no basic gotcha. | |
| 20:59:30 | sean-k-mooney | this is what i tested with the fake driver today | |
| 20:59:32 | sean-k-mooney | http://paste.openstack.org/show/789306/ | |
| 20:59:56 | sean-k-mooney | Sundar: resize to same flavor should be blocked in the api | |
| 21:00:00 | sean-k-mooney | it is in the client | |
| 21:00:13 | sean-k-mooney | you can migrate but resize to same flavor is invalid | |
| 21:01:22 | Sundar | remove-vol/net: unrelated to Cyborg, right? | |
| 21:01:50 | sean-k-mooney | yes but you blocked them in your patch that check for the flavor extra spec | |
| 21:01:56 | sean-k-mooney | they seam to work fine | |
| 21:02:05 | Sundar | Suspend will not work with real FPGAs, because libvirt will error out with: Domain has assigned non-USB devices. | |
| 21:02:07 | sean-k-mooney | at least with the fake dirver | |
| 21:02:38 | sean-k-mooney | suspend shoudl do a managed save which will detach all hostdev devices | |
| 21:02:46 | Sundar | delete failed? | |
| 21:02:49 | sean-k-mooney | yes | |
| 21:03:12 | sean-k-mooney | it failed in the last versions too | |
| 21:03:34 | sean-k-mooney | the conductor explodes with an unexpected respoce form cyborg | |
| 21:03:48 | sean-k-mooney | when it tries to delete /unbind the arqs | |
| 21:04:03 | Sundar | Hmm, please send me the logs. I do deletes all the time, but your sequence of ops probably triggered something. | |
| 21:04:39 | sean-k-mooney | i found that was broken just with boot then delete two days ago | |
| 21:04:56 | sean-k-mooney | ill need to unstack and stack to be able to test it again | |
| 21:05:18 | Sundar | I'll try boot + delete. Was it hard reboot? | |
| 21:06:36 | sean-k-mooney | no just boot then delete | |
| 21:06:47 | sean-k-mooney | hardreboot seams to be fine | |
| 21:08:23 | sean-k-mooney | Sundar: https://etherpad.openstack.org/p/sean-cyborg-testing-delete-logs | |
| 21:08:36 | sean-k-mooney | paste.openstack.org isnt loadign for me | |
| 21:08:44 | sean-k-mooney | but that is all the nova and cyborg logs | |
| 21:10:19 | sean-k-mooney | it look like the cyborg api returned a 401 | |
| 21:12:19 | Sundar | devstack@cyborg-api.service[26903]: 2020-02-07 20:56:05.719 .... Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed. | |
| 21:12:53 | sean-k-mooney | yep | |
| 21:13:35 | sean-k-mooney | so the call to cyborg either need an admin token which i dont think is correct or the token expired and you need to handel that | |
| 21:14:29 | sean-k-mooney | deleting the vm again does not fix it by they way | |
| 21:14:40 | sean-k-mooney | but i can create and delete non cyborg vms | |
| 21:15:05 | sean-k-mooney | i get the same 401 reponce on the second attempt | |
| 21:15:13 | sean-k-mooney | but i can delete hte arq myself | |
| 21:15:58 | Sundar | What does it take to handle an expired token? BTW: https://opendev.org/openstack/cyborg/src/branch/master/cyborg/common/policy.py#L86 applies defuault rule for ARQ deletes | |
| 21:16:39 | Sundar | *default rule: rule:admin_or_owner | |
| 21:17:02 | sean-k-mooney | so there is basically two ways to try and resovle it or you can raise an error | |
| 21:17:38 | sean-k-mooney | if service tokens are configured youc an fall back to using those alternitvly you can have an admin token for the servcie in the nova conf and escalate | |
| 21:17:42 | sean-k-mooney | or you can fail | |
| 21:17:51 | sean-k-mooney | but in this can my token was not expired | |
| 21:18:14 | sean-k-mooney | when i did it again i was issue a new token and it failed again | |
| 21:18:58 | sean-k-mooney | my devstack also issue tokesn that last for an hour so it shoudl be fine | |
| 21:20:41 | sean-k-mooney | anyway im done for the day o/ | |
| 21:21:21 | Sundar | Ok. Not sure what is going wrong here. I'll try it out as non-admin. | |
| 21:21:34 | Sundar | Thanks a lot for all your efforts. ;) Have a good weekend! | |
| 21:43:38 | Sundar | efried: Re. "blocking unsupported operations", the objection seems to be that each change requires microversion changes. | |
| 21:45:49 | Sundar | FWIW, I think any further changes after this series will be few and far between. We could put in a microversion change now. Any further changes should be clumped together as much as possible, based on operator requirements. | |
| #openstack-nova - 2020-02-08 | |||
| 03:02:28 | openstackgerrit | HYSong proposed openstack/nova master: fix bug 1861964 https://review.opendev.org/706647 | |
| 03:02:28 | openstack | bug 1861964 in OpenStack Compute (nova) "availability_zone in request_specs not updated when live migrate vm from one availability zone to another" [Undecided,New] https://launchpad.net/bugs/1861964 | |
| 05:15:09 | openstackgerrit | HYSong proposed openstack/nova master: fix format https://review.opendev.org/706655 | |