| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-19 | |||
| 16:43:24 | sean-k-mooney | cyborg default to useing the fqdn we defualt to the hostname | |
| 16:43:29 | sean-k-mooney | for the conf.host value | |
| 16:43:40 | stephenfin | efried: when you've time, can I annoy you with some trivial but nice-to-have patches https://review.opendev.org/#/c/705652/ | |
| 16:43:46 | dansmith | sean-k-mooney: the opposite, but yeah | |
| 16:45:01 | sean-k-mooney | https://review.opendev.org/#/c/696014/2/cyborg/agent/manager.py | |
| 16:45:33 | dansmith | sean-k-mooney: right, but conf.host is set to the short name | |
| 16:45:34 | dansmith | so it's a devstack config thing | |
| 16:45:45 | efried | stephenfin: ack, though my tenure is officially on a timer as of this morning, so I may want to dedicate myself to "important" things where possible? | |
| 16:45:56 | efried | not that I'm doing a great job of that anyway. | |
| 16:46:04 | stephenfin | :( | |
| 16:46:10 | stephenfin | yes, definitely in that case | |
| 16:46:52 | dansmith | cripes, cyborgclient can't actually do anything | |
| 16:47:41 | sean-k-mooney | you need a diffenrt patch that is not merged yet | |
| 16:50:07 | sean-k-mooney | dansmith: basicaly you need to isntall a patched version of the openstacksdk and then you need a different eversion fo cyborg client | |
| 16:50:10 | sean-k-mooney | ill get the links | |
| 16:51:04 | dansmith | fwiw, I deleted the device profile underneath an instance and now I can't delete the instance | |
| 16:51:22 | sean-k-mooney | ya | |
| 16:51:27 | sean-k-mooney | so i dont think that is the issue | |
| 16:51:41 | sean-k-mooney | i have not been able to delete vms with cyborg devices in general | |
| 16:51:43 | dansmith | no, just noting that I hit that | |
| 16:51:49 | dansmith | oh hah | |
| 16:52:10 | sean-k-mooney | i fails with auth erros for me | |
| 16:52:35 | sean-k-mooney | basically cyborg reject the arq delete with a 401 form keysonte midelware | |
| 16:52:42 | sean-k-mooney | but i can delete by hand | |
| 16:53:06 | sean-k-mooney | i also notice a similar issue which spawaynign vms after it had been running for an hour | |
| 16:53:11 | dansmith | okay | |
| 16:53:30 | sean-k-mooney | i think we are caching or not updating a token someher in the nova side | |
| 16:53:34 | dansmith | I recreated my device profile with the same name and now I'm getting a 401 from cyborg when nova tries to fetch it | |
| 16:53:42 | sean-k-mooney | ya | |
| 16:53:50 | sean-k-mooney | try restarting the nova conductor | |
| 16:55:03 | dansmith | I don't think it's getting to conductor | |
| 16:56:25 | sean-k-mooney | i guss it could be in the schduler. i restarted all the nova serivces | |
| 16:56:59 | sean-k-mooney | well i restart all the nova and cyborg services to be more accurate | |
| 16:58:18 | dansmith | if it's not getting to conductor it's not getting to scheduler | |
| 16:58:39 | sean-k-mooney | dansmith: you need https://review.opendev.org/#/c/681391/ and https://review.opendev.org/#/c/679914/ to get the client to work by they way | |
| 16:59:07 | sean-k-mooney | dansmith: where do we get the device profile form the api? | |
| 16:59:13 | sean-k-mooney | are we doing that in the api? | |
| 16:59:33 | dansmith | idk, i just don't see any traffic in any conductor logs making it look like it's even getting called | |
| 16:59:33 | sean-k-mooney | i assumed it would be later but i guess we might be doing an early check to ensure it exits | |
| 17:01:32 | dansmith | sean-k-mooney: https://review.opendev.org/#/c/631243/57/nova/compute/api.py | |
| 17:04:10 | dansmith | okay yeah, stale token i guess | |
| 17:04:42 | dansmith | had to restart n-api | |
| 17:05:03 | dansmith | now I get Binding failed for ARQ UUIDs | |
| 17:05:24 | dansmith | which is this from cyborg-api: Feb 19 09:04:37 guaranine devstack@cyborg-api.service[767]: : sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one() | |
| 17:06:27 | dansmith | from the stack trace, I'd guess there are two deployables for the FakeDriver RP maybe | |
| 17:06:35 | dansmith | since they're using a dns-scoped uuid | |
| 17:06:55 | efried | dansmith: Don't look at this now, but please open tab or however you queue for "later": https://review.opendev.org/#/c/707064/11/upper-constraints.txt@721 | |
| 17:08:36 | prometheanfire | efried: it'd be nice to have it looked at, but I can remove it for now and let it show up next run | |
| 17:09:20 | efried | prometheanfire: ack. | |
| 17:09:34 | dansmith | sean-k-mooney: got a booted instance after cleaning up their database | |
| 17:09:46 | sean-k-mooney | cool | |
| 17:10:27 | dansmith | so definitely some fragility here | |
| 17:12:21 | openstackgerrit | Eric Fried proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766 | |
| 17:14:12 | dansmith | sean-k-mooney: documented that here: https://storyboard.openstack.org/#!/story/2007310 | |
| 17:15:06 | sean-k-mooney | ya i guess i have had slightly better look since in my case the fqdn and host name are returned the same as just the hostname | |
| 17:16:25 | sean-k-mooney | but its definetly one of the "it worked on my devstack" moments | |
| 17:17:27 | openstackgerrit | Merged openstack/nova master: Remove universal wheel configuration https://review.opendev.org/706466 | |
| 17:17:34 | dansmith | I'm less concerned about the actual hostname thing, since it's actually a devstack thing, | |
| 17:17:34 | openstackgerrit | Merged openstack/nova master: Avoid PlacementFixture silently swallowing kwargs https://review.opendev.org/701754 | |
| 17:17:42 | dansmith | and more concerned about how fragile this is trying to get it fixed | |
| 17:17:55 | dansmith | efried: I don't know the answer nor why we need to name our enums | |
| 17:18:04 | dansmith | efried: if those names go into the DDL then I guess we need a migration for it | |
| 17:18:16 | dansmith | efried: if it's just client-side in the model, then I guess we can just add them/ | |
| 17:18:25 | dansmith | s#/#?# | |
| 17:20:02 | efried | I figured it was going to sound something like that. If we put up a patch that adds the names, will the test results make it clear whether or not we need a migration? (I fear not...) | |
| 17:20:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-next: Start testing the q35 machine type https://review.opendev.org/708701 | |
| 17:21:35 | dansmith | efried: add them to the models and see if functional tests pass locally | |
| 17:21:54 | dansmith | and then if yeah, it all just works through grenade tests and functional then it should be okay I think | |
| 17:22:12 | dansmith | if the model sync test complains about schema vs. models, then... sadface | |
| 17:22:13 | efried | dansmith: I need to do something locally to make pg tests actually run, but I'm not sure what that is. | |
| 17:22:32 | dansmith | oh is this on pg? | |
| 17:22:54 | efried | if the test result is to be believed. | |
| 17:23:21 | efried | sorry, is that not clear from the context? I'll add it in... | |
| 17:23:31 | efried | yeah Failure in nova.tests.unit.db.test_migrations.TestNovaMigrationsPostgreSQL.test_models_sync | |
| 17:23:36 | efried | That means pg, right? | |
| 17:24:20 | dansmith | yeah sorry I just skimmed to rsplit('.')[-1] | |
| 17:24:35 | dansmith | so if it fails on pg, then we probably need a schema migration, which is pretty god awful | |
| 17:24:47 | efried | prometheanfire: remind me how I would test a nova fix here: can I make the nova patch depend-on the req patch or does it have to be vice versa? | |
| 17:25:52 | dansmith | we don't use alembic, so it's super unfortunate to have to migrate people's databases to take this... :/ | |
| 17:27:09 | prometheanfire | efried: first, a reqs review would be made that included only the bump to alembic, then you'd dep on that | |
| 17:28:27 | jroll | efried: it doesn't look like I'm going to be able to finish the vTPM things in U, so you can go ahead and bump that out of Direction:Approve. sorry :( | |
| 17:28:32 | dansmith | efried: context being PG is definitely clear, I'm ignoring a call to mess with cyborg, so I just wasn't looking closely enough originally when I just hopped over to look | |
| 17:29:09 | efried | no worries, swhy I said for later, didn't want to distract you from cyborg. | |
| 17:29:26 | efried | jroll: ack, thanks for the heads up. | |
| 17:29:41 | jroll | np | |
| 17:29:45 | prometheanfire | efried: https://review.opendev.org/708706 | |
| 17:33:26 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Name Enums https://review.opendev.org/708707 | |
| 17:33:30 | efried | prometheanfire, dansmith: ^ | |
| 17:54:28 | dansmith | sean-k-mooney: so I was able to delete my cyborg instance.. were you saying you just couldn't delete once the token expired? | |
| 17:54:43 | dansmith | and I wonder if that token expiry problem is related to efried's demanding a singleton? | |
| 17:55:18 | sean-k-mooney | ya so i only tried to delete it after an hour or two of other manual tests | |
| 17:55:26 | dansmith | okay | |
| 17:55:51 | sean-k-mooney | im currently trying to fix there multi node devstack support so i dont have an env runing to test with | |
| 17:55:59 | sean-k-mooney | but it was just failing on the keystone token | |
| 17:56:03 | dansmith | gotcha | |
| 17:56:20 | dansmith | sean-k-mooney: so you're fixing the fakedriver too or just the devstack part? | |
| 17:56:40 | sean-k-mooney | i fixed the fake libvirt driver already | |
| 17:56:48 | sean-k-mooney | im not going to fix the config generation | |
| 17:56:52 | dansmith | I mean the fake cyborg driver | |
| 17:56:55 | sean-k-mooney | and i might fix the host name issue too | |