Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-19
16:16:09 dansmith well, then *that* is broken :)
16:16:31 sean-k-mooney ya they may have or need a is_compute_node config option
16:16:35 dansmith so are you thinking that I could delete the provider and restart the conductor and have it re-create it?
16:16:49 sean-k-mooney yes
16:17:19 sean-k-mooney i dont know if it will work but i think it should
16:18:17 dansmith 9-700a-4401-915f-3a9036bf4d4a - - - - -] Error, provider guaranine can not be found: IndexError: list index out of range
16:18:40 dansmith but my compute node provider is the fqdn
16:19:08 sean-k-mooney it could be a fqdn vs host name issue
16:19:18 dansmith the log is full of those errors, yet it's returning None from the _get_root_provider() which is letting it create it without a parent
16:19:34 dansmith so if it's really an error (as evidenced by the log level) it should not be just silently returning None and creating the provider anyway
16:19:46 dansmith is cyborg on storyboard?
16:22:00 sean-k-mooney :( yes i think so
16:22:26 dansmith guh, it's nearly empty
16:22:33 dansmith how do I file a bug? create a new story?
16:26:15 sean-k-mooney dansmith: sorry on downstream call but yes i think storybord unfrotunetly
16:27:28 dansmith lol, got a 400 and a 500 from storyboard while trying to save my bug
16:36:06 dansmith sean-k-mooney: ah, I missed it but I did get host=$short in my cyborg.conf file
16:38:25 dansmith deleting the provider, updating the conf, restarting the agent fixed me
16:42:55 sean-k-mooney cool
16:43:06 sean-k-mooney so i know what bug is
16:43:09 sean-k-mooney i can go fix it
16:43:11 dansmith devstack doesn't install a cyborg client apparently
16:43:21 dansmith sean-k-mooney: https://storyboard.openstack.org/#!/story/2007310
16:43:24 sean-k-mooney cyborg default to useing the fqdn we defualt to the hostname
16:43:29 sean-k-mooney for the conf.host value
16:43:40 stephenfin efried: when you've time, can I annoy you with some trivial but nice-to-have patches https://review.opendev.org/#/c/705652/
16:43:46 dansmith sean-k-mooney: the opposite, but yeah
16:45:01 sean-k-mooney https://review.opendev.org/#/c/696014/2/cyborg/agent/manager.py
16:45:33 dansmith sean-k-mooney: right, but conf.host is set to the short name
16:45:34 dansmith so it's a devstack config thing
16:45:45 efried stephenfin: ack, though my tenure is officially on a timer as of this morning, so I may want to dedicate myself to "important" things where possible?
16:45:56 efried not that I'm doing a great job of that anyway.
16:46:04 stephenfin :(
16:46:10 stephenfin yes, definitely in that case
16:46:52 dansmith cripes, cyborgclient can't actually do anything
16:47:41 sean-k-mooney you need a diffenrt patch that is not merged yet
16:50:07 sean-k-mooney dansmith: basicaly you need to isntall a patched version of the openstacksdk and then you need a different eversion fo cyborg client
16:50:10 sean-k-mooney ill get the links
16:51:04 dansmith fwiw, I deleted the device profile underneath an instance and now I can't delete the instance
16:51:22 sean-k-mooney ya
16:51:27 sean-k-mooney so i dont think that is the issue
16:51:41 sean-k-mooney i have not been able to delete vms with cyborg devices in general
16:51:43 dansmith no, just noting that I hit that
16:51:49 dansmith oh hah
16:52:10 sean-k-mooney i fails with auth erros for me
16:52:35 sean-k-mooney basically cyborg reject the arq delete with a 401 form keysonte midelware
16:52:42 sean-k-mooney but i can delete by hand
16:53:06 sean-k-mooney i also notice a similar issue which spawaynign vms after it had been running for an hour
16:53:11 dansmith okay
16:53:30 sean-k-mooney i think we are caching or not updating a token someher in the nova side
16:53:34 dansmith I recreated my device profile with the same name and now I'm getting a 401 from cyborg when nova tries to fetch it
16:53:42 sean-k-mooney ya
16:53:50 sean-k-mooney try restarting the nova conductor
16:55:03 dansmith I don't think it's getting to conductor
16:56:25 sean-k-mooney i guss it could be in the schduler. i restarted all the nova serivces
16:56:59 sean-k-mooney well i restart all the nova and cyborg services to be more accurate
16:58:18 dansmith if it's not getting to conductor it's not getting to scheduler
16:58:39 sean-k-mooney dansmith: you need https://review.opendev.org/#/c/681391/ and https://review.opendev.org/#/c/679914/ to get the client to work by they way
16:59:07 sean-k-mooney dansmith: where do we get the device profile form the api?
16:59:13 sean-k-mooney are we doing that in the api?
16:59:33 sean-k-mooney i assumed it would be later but i guess we might be doing an early check to ensure it exits
16:59:33 dansmith idk, i just don't see any traffic in any conductor logs making it look like it's even getting called
17:01:32 dansmith sean-k-mooney: https://review.opendev.org/#/c/631243/57/nova/compute/api.py
17:04:10 dansmith okay yeah, stale token i guess
17:04:42 dansmith had to restart n-api
17:05:03 dansmith now I get Binding failed for ARQ UUIDs
17:05:24 dansmith which is this from cyborg-api: Feb 19 09:04:37 guaranine devstack@cyborg-api.service[767]: : sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one()
17:06:27 dansmith from the stack trace, I'd guess there are two deployables for the FakeDriver RP maybe
17:06:35 dansmith since they're using a dns-scoped uuid
17:06:55 efried dansmith: Don't look at this now, but please open tab or however you queue for "later": https://review.opendev.org/#/c/707064/11/upper-constraints.txt@721
17:08:36 prometheanfire efried: it'd be nice to have it looked at, but I can remove it for now and let it show up next run
17:09:20 efried prometheanfire: ack.
17:09:34 dansmith sean-k-mooney: got a booted instance after cleaning up their database
17:09:46 sean-k-mooney cool
17:10:27 dansmith so definitely some fragility here
17:12:21 openstackgerrit Eric Fried proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766
17:14:12 dansmith sean-k-mooney: documented that here: https://storyboard.openstack.org/#!/story/2007310
17:15:06 sean-k-mooney ya i guess i have had slightly better look since in my case the fqdn and host name are returned the same as just the hostname
17:16:25 sean-k-mooney but its definetly one of the "it worked on my devstack" moments
17:17:27 openstackgerrit Merged openstack/nova master: Remove universal wheel configuration https://review.opendev.org/706466
17:17:34 openstackgerrit Merged openstack/nova master: Avoid PlacementFixture silently swallowing kwargs https://review.opendev.org/701754
17:17:34 dansmith I'm less concerned about the actual hostname thing, since it's actually a devstack thing,
17:17:42 dansmith and more concerned about how fragile this is trying to get it fixed
17:17:55 dansmith efried: I don't know the answer nor why we need to name our enums
17:18:04 dansmith efried: if those names go into the DDL then I guess we need a migration for it
17:18:16 dansmith efried: if it's just client-side in the model, then I guess we can just add them/
17:18:25 dansmith s#/#?#
17:20:02 efried I figured it was going to sound something like that. If we put up a patch that adds the names, will the test results make it clear whether or not we need a migration? (I fear not...)
17:20:13 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP nova-next: Start testing the q35 machine type https://review.opendev.org/708701
17:21:35 dansmith efried: add them to the models and see if functional tests pass locally
17:21:54 dansmith and then if yeah, it all just works through grenade tests and functional then it should be okay I think
17:22:12 dansmith if the model sync test complains about schema vs. models, then... sadface
17:22:13 efried dansmith: I need to do something locally to make pg tests actually run, but I'm not sure what that is.
17:22:32 dansmith oh is this on pg?
17:22:54 efried if the test result is to be believed.
17:23:21 efried sorry, is that not clear from the context? I'll add it in...
17:23:31 efried yeah Failure in nova.tests.unit.db.test_migrations.TestNovaMigrationsPostgreSQL.test_models_sync
17:23:36 efried That means pg, right?
17:24:20 dansmith yeah sorry I just skimmed to rsplit('.')[-1]
17:24:35 dansmith so if it fails on pg, then we probably need a schema migration, which is pretty god awful

Earlier   Later