| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-04 | |||
| 14:18:05 | sean-k-mooney | we are not fragmenting the packets properloy | |
| 14:18:15 | efried | My point: It shouldn't be up to every consuming project to "not output so much" | |
| 14:18:39 | sean-k-mooney | right well we could just use pytest to run things but some people object to that too | |
| 14:19:54 | efried | is this a mtreinish thing? | |
| 14:20:19 | mriedem | well in this case something is obviously busted while this test is running | |
| 14:20:40 | mriedem | one of the services is gone from the db for some reason, or the context it was using somehow - it's a multi-cell test so maybe it's looking in the wrong db | |
| 14:20:55 | mriedem | so every 10 seconds when it reports it's state the service is not found | |
| 14:21:06 | mriedem | and prints the traceback | |
| 14:21:17 | mtreinish | efried: patches are more than welcome if you have a fix | |
| 14:21:51 | sean-k-mooney | mtreinish: i think the issue is that its non triaival to figure out how stestr works to try and even start on the fix | |
| 14:22:27 | mtreinish | sean-k-mooney: it's non trivial to figure out how subunit is working under the covers which is why I haven't had a fix for it yet | |
| 14:23:04 | mtreinish | the stream handling code is really hairy and mostly leftover from testr | |
| 14:23:24 | sean-k-mooney | mtreinish: ya i treied to look at the stestr code to figure out how the subunit stream was being created to figure out where to fragment the packet but i got lost | |
| 14:24:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Implement cleanup_instance_network_on_host for neutron API https://review.opendev.org/697162 | |
| 14:24:11 | sean-k-mooney | that was partly because i was lookig at 2 codebase i dont undersantd and as you said the code is kind of hairy | |
| 14:24:20 | efried | okay, I wasn't criticizing you mtreinish, sorry if it came out that way. If you've already looked and it's "really hairy"... | |
| 14:25:14 | sean-k-mooney | efried: i know stephenfin melwitt and i all tried to figuire this out before and gave up | |
| 14:25:44 | efried | I just remember several patches (from mriedem I think) that tried to chip away at the problem by reducing the amount of output from our tests. | |
| 14:26:00 | mtreinish | yeah, also masayukig smcginnis and I also took a look into a couple years ago f2f at a summit and also gave up | |
| 14:26:09 | efried | gah | |
| 14:26:27 | mriedem | regardless of subunit/stestr i don't think our tests should be spewing out a buttload of random tracebacks | |
| 14:26:39 | mriedem | masking that with fixing subunit/stestr somehow just papers over those issues | |
| 14:26:48 | sean-k-mooney | so the simple solution is to just use pytest or a non subunit based test runner but there are perfromace impacts to doing that | |
| 14:27:29 | mtreinish | I have a PR which should workaround it I think: https://github.com/mtreinish/stestr/pull/271 but it introduces other behavior changes and isn't ready yet (and would be an opt-in mode) | |
| 14:28:36 | sean-k-mooney | mtreinish: playing with the schduleiing did help in the past. i notced that grouping by class helped to some extent | |
| 14:29:10 | mriedem | oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor weee | |
| 14:29:52 | mriedem | we seem to have some fun eventlet issue in this failure as well | |
| 14:31:36 | mriedem | https://gist.githubusercontent.com/mriedem/e01252253600abbef95045d73cfe2206/raw/67b6360c5766eb681cb3ad9e1901ff94fa152e98/gistfile1.txt | |
| 14:31:39 | sean-k-mooney | and some db errors " WARNING [nova.compute.manager] Failed to revert task state for instance. Error: (sqlite3.OperationalError) no such table: instance_extra'" | |
| 14:32:37 | mtreinish | which review # and job is the failure happening on? I can take another stab at fixing the issue | |
| 14:32:48 | sean-k-mooney | https://review.opendev.org/697162 | |
| 14:33:35 | mriedem | mtreinish: it's intermittent, we get all sorts of random stuff like this http://status.openstack.org/elastic-recheck/#1813147 and then we squash them as they crop up | |
| 14:34:00 | mriedem | the above gist is weird because the test is creating a server on compute1, the logs say it's ok, but the traceback looks like something is rescheduling to another host during build | |
| 14:34:01 | sean-k-mooney | mtreinish: mriedem is not really changeing anything in that that would resonably cause those errors | |
| 14:34:39 | sean-k-mooney | mriedem: could we be leaking some state betwween the different functional tests | |
| 14:35:14 | sean-k-mooney | such that this fails only with some test orderings and passes with others | |
| 14:35:54 | mriedem | yeah maybe but i don't see any other tests on the same worker that look like they would be doing reschedules | |
| 14:36:00 | mriedem | anywho, recheck! | |
| 14:36:32 | sean-k-mooney | that was why grouping the test execution by class work to "fix" the issue in the past. i ment we did not hit the issue due to how the test were run | |
| 14:36:40 | mriedem | we could probably make that ServiceNotFound in the servicegroup api not a traceback so that if this fails again like this it doesn't blow up the subunit output parser | |
| 14:37:27 | sean-k-mooney | we didnt merge that since we actully found the problem and fixed it in that case | |
| 14:53:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle ServiceNotFound in DbDriver._report_state https://review.opendev.org/697301 | |
| 15:40:20 | dansmith | efried: despite the fact that I dropped some comments on the request group patch about the testing mess, I'm going to mostly depend on you to ack the placementy request-group-y bit of that before I look too closely | |
| 15:42:26 | efried | dansmith: ack. It might depend on my series here https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:use-placement-1.34 | |
| 15:43:30 | Sundar | dansmith: Thanks for the super-quick reviews. | |
| 15:43:45 | efried | but, having been looking at that stuff pretty closely for the past week, I ought to be able to provide reasonable guidance in that regard :) | |
| 15:44:18 | dansmith | Sundar: mmm, six hours later.. not super-quick :) | |
| 15:45:59 | Sundar | dansmith: I'll set higher standards for you going forward :P. | |
| 15:46:11 | dansmith | Sundar: fair warning, I have this gut feeling that you're missing a bunch of test coverage, but I'm not in that phase of review yet, so I'm not nitpicking all the conditionals | |
| 15:47:49 | Sundar | dansmith: ack, will look more into that. | |
| 15:49:22 | stephenfin | mriedem: comment on https://review.opendev.org/#/c/697122/ | |
| 15:50:42 | stephenfin | Could I ask someone to go through the bottom few patches of the remove-nova-network series starting at [1]? There's a merge conflict coming and I'd rather not rebase/recheck the _entire_ queue [1] https://review.opendev.org/#/c/696503/ | |
| 15:50:43 | efried | My understanding stephenfin: we need to backport it as it stands. Which means if your change merges first, the backport of this one will need to be different from the master version. | |
| 15:51:49 | stephenfin | Yup, I get that. I'm tempted to suggest we just apply it against stable/train since it will soon be useless on master, but maybe that's a lot of work just to avoid some merge conflicts | |
| 15:52:00 | stephenfin | (as in StableOnly) | |
| 15:52:05 | efried | It won't be useless on master | |
| 15:52:12 | efried | we still need to cache, which we're not doing | |
| 15:52:19 | mriedem | stephenfin: replied | |
| 15:53:50 | stephenfin | mriedem: Good thing I asked :) | |
| 15:54:29 | mriedem | i think we need to just make the neutron SecurityGroupAPI a singleton | |
| 15:54:34 | mriedem | it doesn't have any state as far as i can tell | |
| 15:54:49 | mriedem | unlike SchedulerReportClient which does have some state | |
| 15:55:11 | efried | It must not have state, since we were able to cache it before with no ill effects | |
| 15:55:22 | efried | unless the state was added in the interval when we had removed the caching. | |
| 15:55:30 | mriedem | the neutron sg api you mean | |
| 15:55:37 | stephenfin | Makes sense to me. I can rework that patch accordingly | |
| 15:56:24 | stephenfin | though I'll hold off for now since that queue is approaching x-cell resize size and will kill CI if pushed | |
| 15:56:28 | stephenfin | RIP CI | |
| 15:57:25 | mriedem | the bottom 5 or so changes can use another core | |
| 15:57:33 | stephenfin | mriedem: While I have you, is this the correct thing to do? https://review.opendev.org/#/c/697152/2/doc/api_samples/os-quota-class-sets/quota-classes-show-get-resp.json | |
| 15:58:06 | stephenfin | I assume we don't support arbitrary quota classes so we can't modify the output for older microversions | |
| 15:58:37 | stephenfin | ...meaning I have to fake those nova-net'y quotas rather than just dropping them | |
| 15:58:41 | mriedem | i'm pretty sure we only support one quota class, 'default' | |
| 15:59:02 | stephenfin | sorry, arbitrary quotas...types? | |
| 15:59:20 | mriedem | yeah you can't create random quota types | |
| 16:00:03 | salmankhan | HI guys, I am facing an issue if someone can help. I upgraded the nova queens api from xenial to bionic and I am not able to see network info in "nova show" or "openstack server show" commands. However I can see the network details on dashboard or "nova interface-list" or "openstack port show" commands. Doesn't look like a database issue, but may be. If I reboot a instance there is no interface attached as well. What could be | |
| 16:00:04 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/quota_sets.py#L29 | |
| 16:00:51 | stephenfin | so I do have to keep them. okidok | |
| 16:00:59 | mriedem | that said, i'm not sure about what you're doing there with just changing all the network quota values to -1 | |
| 16:01:41 | stephenfin | that's what the noop driver was doing (unlimited) and those things are noops now | |
| 16:01:43 | stephenfin | so... | |
| 16:01:45 | openstackgerrit | Merged openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180 | |
| 16:02:02 | mriedem | the noop quota driver is just for testing | |
| 16:02:07 | mriedem | or nfv single tenant i guess | |
| 16:02:39 | mriedem | i agree that we can't just remove them from the responses and stuff without a microversion since it changes the schema | |
| 16:03:08 | mriedem | we don't proxy enforcement of those to neutron at all do we? | |
| 16:03:20 | mriedem | i know there was talk of doing that back in like havana | |
| 16:03:25 | stephenfin | nope, they're dead | |
| 16:03:38 | stephenfin | all DB model driven | |
| 16:05:36 | mriedem | yeah, i mean the -1 is a lie because we're not enforcing those. we did deprecate them from the apis in 2.35 and 2.49 and they were a lie before if using neutron anyway - you could change security group rules quota to 500 in nova and it wouldn't apply to neutron | |
| 16:06:15 | melwitt | re: the noop quota driver, I had thought that was the way for a deployment that doesn't care about quotas to bypass all the calculating (perf impact). so I was thinking not to remove it because of that. am I missing something? | |
| 16:06:45 | mriedem | how many deployments besides nfv single tenant don't care about quota? | |
| 16:08:19 | stephenfin | If -1 is a lie, what do I return instead? Just 0? | |
| 16:08:35 | stephenfin | I have to return _something_ | |
| 16:10:45 | mriedem | it's all a lie is my point i think, so -1 is probably good enough | |
| 16:11:11 | mriedem | i'm just not sure how clearly i'm thinking about this atm, or what alternatives there are | |
| 16:11:31 | mriedem | i defer to dansmith | |
| 16:11:36 | mriedem | o-) | |
| 16:12:05 | dansmith | I don't know about quota things | |
| 16:12:09 | dansmith | I defer to melwitt | |