| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-19 | |||
| 16:52:42 | sean-k-mooney | but i can delete by hand | |
| 16:53:06 | sean-k-mooney | i also notice a similar issue which spawaynign vms after it had been running for an hour | |
| 16:53:11 | dansmith | okay | |
| 16:53:30 | sean-k-mooney | i think we are caching or not updating a token someher in the nova side | |
| 16:53:34 | dansmith | I recreated my device profile with the same name and now I'm getting a 401 from cyborg when nova tries to fetch it | |
| 16:53:42 | sean-k-mooney | ya | |
| 16:53:50 | sean-k-mooney | try restarting the nova conductor | |
| 16:55:03 | dansmith | I don't think it's getting to conductor | |
| 16:56:25 | sean-k-mooney | i guss it could be in the schduler. i restarted all the nova serivces | |
| 16:56:59 | sean-k-mooney | well i restart all the nova and cyborg services to be more accurate | |
| 16:58:18 | dansmith | if it's not getting to conductor it's not getting to scheduler | |
| 16:58:39 | sean-k-mooney | dansmith: you need https://review.opendev.org/#/c/681391/ and https://review.opendev.org/#/c/679914/ to get the client to work by they way | |
| 16:59:07 | sean-k-mooney | dansmith: where do we get the device profile form the api? | |
| 16:59:13 | sean-k-mooney | are we doing that in the api? | |
| 16:59:33 | sean-k-mooney | i assumed it would be later but i guess we might be doing an early check to ensure it exits | |
| 16:59:33 | dansmith | idk, i just don't see any traffic in any conductor logs making it look like it's even getting called | |
| 17:01:32 | dansmith | sean-k-mooney: https://review.opendev.org/#/c/631243/57/nova/compute/api.py | |
| 17:04:10 | dansmith | okay yeah, stale token i guess | |
| 17:04:42 | dansmith | had to restart n-api | |
| 17:05:03 | dansmith | now I get Binding failed for ARQ UUIDs | |
| 17:05:24 | dansmith | which is this from cyborg-api: Feb 19 09:04:37 guaranine devstack@cyborg-api.service[767]: : sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one() | |
| 17:06:27 | dansmith | from the stack trace, I'd guess there are two deployables for the FakeDriver RP maybe | |
| 17:06:35 | dansmith | since they're using a dns-scoped uuid | |
| 17:06:55 | efried | dansmith: Don't look at this now, but please open tab or however you queue for "later": https://review.opendev.org/#/c/707064/11/upper-constraints.txt@721 | |
| 17:08:36 | prometheanfire | efried: it'd be nice to have it looked at, but I can remove it for now and let it show up next run | |
| 17:09:20 | efried | prometheanfire: ack. | |
| 17:09:34 | dansmith | sean-k-mooney: got a booted instance after cleaning up their database | |
| 17:09:46 | sean-k-mooney | cool | |
| 17:10:27 | dansmith | so definitely some fragility here | |
| 17:12:21 | openstackgerrit | Eric Fried proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766 | |
| 17:14:12 | dansmith | sean-k-mooney: documented that here: https://storyboard.openstack.org/#!/story/2007310 | |
| 17:15:06 | sean-k-mooney | ya i guess i have had slightly better look since in my case the fqdn and host name are returned the same as just the hostname | |
| 17:16:25 | sean-k-mooney | but its definetly one of the "it worked on my devstack" moments | |
| 17:17:27 | openstackgerrit | Merged openstack/nova master: Remove universal wheel configuration https://review.opendev.org/706466 | |
| 17:17:34 | openstackgerrit | Merged openstack/nova master: Avoid PlacementFixture silently swallowing kwargs https://review.opendev.org/701754 | |
| 17:17:34 | dansmith | I'm less concerned about the actual hostname thing, since it's actually a devstack thing, | |
| 17:17:42 | dansmith | and more concerned about how fragile this is trying to get it fixed | |
| 17:17:55 | dansmith | efried: I don't know the answer nor why we need to name our enums | |
| 17:18:04 | dansmith | efried: if those names go into the DDL then I guess we need a migration for it | |
| 17:18:16 | dansmith | efried: if it's just client-side in the model, then I guess we can just add them/ | |
| 17:18:25 | dansmith | s#/#?# | |
| 17:20:02 | efried | I figured it was going to sound something like that. If we put up a patch that adds the names, will the test results make it clear whether or not we need a migration? (I fear not...) | |
| 17:20:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-next: Start testing the q35 machine type https://review.opendev.org/708701 | |
| 17:21:35 | dansmith | efried: add them to the models and see if functional tests pass locally | |
| 17:21:54 | dansmith | and then if yeah, it all just works through grenade tests and functional then it should be okay I think | |
| 17:22:12 | dansmith | if the model sync test complains about schema vs. models, then... sadface | |
| 17:22:13 | efried | dansmith: I need to do something locally to make pg tests actually run, but I'm not sure what that is. | |
| 17:22:32 | dansmith | oh is this on pg? | |
| 17:22:54 | efried | if the test result is to be believed. | |
| 17:23:21 | efried | sorry, is that not clear from the context? I'll add it in... | |
| 17:23:31 | efried | yeah Failure in nova.tests.unit.db.test_migrations.TestNovaMigrationsPostgreSQL.test_models_sync | |
| 17:23:36 | efried | That means pg, right? | |
| 17:24:20 | dansmith | yeah sorry I just skimmed to rsplit('.')[-1] | |
| 17:24:35 | dansmith | so if it fails on pg, then we probably need a schema migration, which is pretty god awful | |
| 17:24:47 | efried | prometheanfire: remind me how I would test a nova fix here: can I make the nova patch depend-on the req patch or does it have to be vice versa? | |
| 17:25:52 | dansmith | we don't use alembic, so it's super unfortunate to have to migrate people's databases to take this... :/ | |
| 17:27:09 | prometheanfire | efried: first, a reqs review would be made that included only the bump to alembic, then you'd dep on that | |
| 17:28:27 | jroll | efried: it doesn't look like I'm going to be able to finish the vTPM things in U, so you can go ahead and bump that out of Direction:Approve. sorry :( | |
| 17:28:32 | dansmith | efried: context being PG is definitely clear, I'm ignoring a call to mess with cyborg, so I just wasn't looking closely enough originally when I just hopped over to look | |
| 17:29:09 | efried | no worries, swhy I said for later, didn't want to distract you from cyborg. | |
| 17:29:26 | efried | jroll: ack, thanks for the heads up. | |
| 17:29:41 | jroll | np | |
| 17:29:45 | prometheanfire | efried: https://review.opendev.org/708706 | |
| 17:33:26 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Name Enums https://review.opendev.org/708707 | |
| 17:33:30 | efried | prometheanfire, dansmith: ^ | |
| 17:54:28 | dansmith | sean-k-mooney: so I was able to delete my cyborg instance.. were you saying you just couldn't delete once the token expired? | |
| 17:54:43 | dansmith | and I wonder if that token expiry problem is related to efried's demanding a singleton? | |
| 17:55:18 | sean-k-mooney | ya so i only tried to delete it after an hour or two of other manual tests | |
| 17:55:26 | dansmith | okay | |
| 17:55:51 | sean-k-mooney | im currently trying to fix there multi node devstack support so i dont have an env runing to test with | |
| 17:55:59 | sean-k-mooney | but it was just failing on the keystone token | |
| 17:56:03 | dansmith | gotcha | |
| 17:56:20 | dansmith | sean-k-mooney: so you're fixing the fakedriver too or just the devstack part? | |
| 17:56:40 | sean-k-mooney | i fixed the fake libvirt driver already | |
| 17:56:48 | sean-k-mooney | im not going to fix the config generation | |
| 17:56:52 | dansmith | I mean the fake cyborg driver | |
| 17:56:55 | sean-k-mooney | and i might fix the host name issue too | |
| 17:56:59 | sean-k-mooney | sorry | |
| 17:57:05 | sean-k-mooney | i ment fake cyborg dirver too | |
| 17:57:10 | dansmith | okay | |
| 17:57:18 | sean-k-mooney | im just used to typing libvirt :) | |
| 17:58:08 | sean-k-mooney | so im hopign to fix it so it stacks probly on two nodes with two agents then ill submit a DNM change to nova that adds a multi node jobs with a cyborg flavor and standard tempest | |
| 17:58:23 | dansmith | cool that sounds good | |
| 17:59:39 | sean-k-mooney | im about to go get food but ill be back in an hour or so. | |
| 18:00:09 | sean-k-mooney | i might get the job done later tonight if not ill do it in the morning i dont think it will take too much to get right | |
| 18:00:29 | dansmith | cool | |
| 18:00:44 | dansmith | we're missing a lot of replies to comments from sundar at this point | |
| 18:00:56 | dansmith | like most of the patches have a -1 on them looking for change or clarification | |
| 18:03:23 | sean-k-mooney | ya i need to set some time aside to also go through the patch chain and actully review them but i have been relying on you, gibi and erric to to that mostly and just commeinting if i know the anser and i happend to see the gerrit email notification | |
| 18:05:05 | dansmith | yeah, I'm not sure more comments will help right now, | |
| 18:05:33 | dansmith | so it might be good to hold off and focus on things we can fix, like what you're doing | |
| 18:05:55 | dansmith | I don't want to push Sundar so much that the queue of feedback seems insurmountable before he has to move on | |
| 18:06:10 | sean-k-mooney | ya makes sense | |
| 18:06:11 | dansmith | the more layers of feedback the harder it is to take action sometimes | |
| 18:07:06 | dansmith | anybody know how the reference a task in a story in a commit message? | |
| 18:07:20 | TheJulia | dansmith: task: | |
| 18:07:22 | TheJulia | and number | |
| 18:07:24 | dansmith | thanks | |
| 18:36:11 | bauzas | folks, I'm becoming crazy | |
| 18:36:20 | bauzas | (py37) [sbauza@sbauza nova]$ stestr run | |