Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-19
15:55:29 dansmith so,
15:55:44 dansmith it looks like the fake driver just exports devices and something in cyborg itself is what creates the providers
15:56:20 dansmith which really shouldn't happen without a parent, AFAICT, so I guess cyborg itself has a bug, both in how it is creating the provider, and that it is agreeing to create one as a root
16:10:35 sean-k-mooney dansmith: you mean in how the cyborg agent on the compute node does an rpc call to the conductor to create and update then placment RP form a perodic task
16:10:53 sean-k-mooney because yse i think that is also an architural bug
16:11:06 dansmith not sure I parsed that
16:11:24 dansmith sean-k-mooney: there should be an RP created by cyborg, which is a child of the compute node provider right?
16:11:27 sean-k-mooney the cyborg agents dont call placement to create the RPs or update them
16:11:33 sean-k-mooney yes
16:11:56 dansmith sean-k-mooney: in my stack, cyborg has created a root provider called "FakeDevice", not child under the compute node
16:12:10 sean-k-mooney in my case its a child
16:12:35 dansmith okay, not sure how that got screwed up for me, but this is basically a virgin stack
16:13:13 sean-k-mooney am it could be down to the oder in which they started
16:13:23 sean-k-mooney or if the config name is not correct in the cyborg config
16:13:33 dansmith that's a bug then,
16:13:34 dansmith because it shouldn't create it as a root if it doesn't find the parent
16:13:42 sean-k-mooney ya
16:14:03 sean-k-mooney it should likely fail to start the agent and complain loudly in the log
16:14:09 dansmith yes
16:14:22 sean-k-mooney but it think that is a sideffect fo the fact the agentd dont create or manage the RPs
16:14:22 dansmith I don't have a hostname configured in my cyborg.conf, if that's what you mean
16:14:42 dansmith it shouldn't matter
16:14:52 dansmith if you need a uuid for the parent and don't find one, you should not create a root one
16:15:09 sean-k-mooney the [DEFAULT]/host option
16:15:18 sean-k-mooney yes
16:15:21 sean-k-mooney well
16:15:23 sean-k-mooney actully no
16:15:38 sean-k-mooney so in the case where cyborg is managning a device on a node that is not a compute node you should
16:15:45 sean-k-mooney there cinder like use case
16:15:46 dansmith sure, but not in this case
16:15:51 sean-k-mooney ya
16:16:02 sean-k-mooney im not sure they have a way to tell however
16:16:09 dansmith well, then *that* is broken :)
16:16:31 sean-k-mooney ya they may have or need a is_compute_node config option
16:16:35 dansmith so are you thinking that I could delete the provider and restart the conductor and have it re-create it?
16:16:49 sean-k-mooney yes
16:17:19 sean-k-mooney i dont know if it will work but i think it should
16:18:17 dansmith 9-700a-4401-915f-3a9036bf4d4a - - - - -] Error, provider guaranine can not be found: IndexError: list index out of range
16:18:40 dansmith but my compute node provider is the fqdn
16:19:08 sean-k-mooney it could be a fqdn vs host name issue
16:19:18 dansmith the log is full of those errors, yet it's returning None from the _get_root_provider() which is letting it create it without a parent
16:19:34 dansmith so if it's really an error (as evidenced by the log level) it should not be just silently returning None and creating the provider anyway
16:19:46 dansmith is cyborg on storyboard?
16:22:00 sean-k-mooney :( yes i think so
16:22:26 dansmith guh, it's nearly empty
16:22:33 dansmith how do I file a bug? create a new story?
16:26:15 sean-k-mooney dansmith: sorry on downstream call but yes i think storybord unfrotunetly
16:27:28 dansmith lol, got a 400 and a 500 from storyboard while trying to save my bug
16:36:06 dansmith sean-k-mooney: ah, I missed it but I did get host=$short in my cyborg.conf file
16:38:25 dansmith deleting the provider, updating the conf, restarting the agent fixed me
16:42:55 sean-k-mooney cool
16:43:06 sean-k-mooney so i know what bug is
16:43:09 sean-k-mooney i can go fix it
16:43:11 dansmith devstack doesn't install a cyborg client apparently
16:43:21 dansmith sean-k-mooney: https://storyboard.openstack.org/#!/story/2007310
16:43:24 sean-k-mooney cyborg default to useing the fqdn we defualt to the hostname
16:43:29 sean-k-mooney for the conf.host value
16:43:40 stephenfin efried: when you've time, can I annoy you with some trivial but nice-to-have patches https://review.opendev.org/#/c/705652/
16:43:46 dansmith sean-k-mooney: the opposite, but yeah
16:45:01 sean-k-mooney https://review.opendev.org/#/c/696014/2/cyborg/agent/manager.py
16:45:33 dansmith sean-k-mooney: right, but conf.host is set to the short name
16:45:34 dansmith so it's a devstack config thing
16:45:45 efried stephenfin: ack, though my tenure is officially on a timer as of this morning, so I may want to dedicate myself to "important" things where possible?
16:45:56 efried not that I'm doing a great job of that anyway.
16:46:04 stephenfin :(
16:46:10 stephenfin yes, definitely in that case
16:46:52 dansmith cripes, cyborgclient can't actually do anything
16:47:41 sean-k-mooney you need a diffenrt patch that is not merged yet
16:50:07 sean-k-mooney dansmith: basicaly you need to isntall a patched version of the openstacksdk and then you need a different eversion fo cyborg client
16:50:10 sean-k-mooney ill get the links
16:51:04 dansmith fwiw, I deleted the device profile underneath an instance and now I can't delete the instance
16:51:22 sean-k-mooney ya
16:51:27 sean-k-mooney so i dont think that is the issue
16:51:41 sean-k-mooney i have not been able to delete vms with cyborg devices in general
16:51:43 dansmith no, just noting that I hit that
16:51:49 dansmith oh hah
16:52:10 sean-k-mooney i fails with auth erros for me
16:52:35 sean-k-mooney basically cyborg reject the arq delete with a 401 form keysonte midelware
16:52:42 sean-k-mooney but i can delete by hand
16:53:06 sean-k-mooney i also notice a similar issue which spawaynign vms after it had been running for an hour
16:53:11 dansmith okay
16:53:30 sean-k-mooney i think we are caching or not updating a token someher in the nova side
16:53:34 dansmith I recreated my device profile with the same name and now I'm getting a 401 from cyborg when nova tries to fetch it
16:53:42 sean-k-mooney ya
16:53:50 sean-k-mooney try restarting the nova conductor
16:55:03 dansmith I don't think it's getting to conductor
16:56:25 sean-k-mooney i guss it could be in the schduler. i restarted all the nova serivces
16:56:59 sean-k-mooney well i restart all the nova and cyborg services to be more accurate
16:58:18 dansmith if it's not getting to conductor it's not getting to scheduler
16:58:39 sean-k-mooney dansmith: you need https://review.opendev.org/#/c/681391/ and https://review.opendev.org/#/c/679914/ to get the client to work by they way
16:59:07 sean-k-mooney dansmith: where do we get the device profile form the api?
16:59:13 sean-k-mooney are we doing that in the api?
16:59:33 dansmith idk, i just don't see any traffic in any conductor logs making it look like it's even getting called
16:59:33 sean-k-mooney i assumed it would be later but i guess we might be doing an early check to ensure it exits
17:01:32 dansmith sean-k-mooney: https://review.opendev.org/#/c/631243/57/nova/compute/api.py
17:04:10 dansmith okay yeah, stale token i guess
17:04:42 dansmith had to restart n-api
17:05:03 dansmith now I get Binding failed for ARQ UUIDs
17:05:24 dansmith which is this from cyborg-api: Feb 19 09:04:37 guaranine devstack@cyborg-api.service[767]: : sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one()
17:06:27 dansmith from the stack trace, I'd guess there are two deployables for the FakeDriver RP maybe

Earlier   Later