Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-04
14:26:39 mriedem masking that with fixing subunit/stestr somehow just papers over those issues
14:26:48 sean-k-mooney so the simple solution is to just use pytest or a non subunit based test runner but there are perfromace impacts to doing that
14:27:29 mtreinish I have a PR which should workaround it I think: https://github.com/mtreinish/stestr/pull/271 but it introduces other behavior changes and isn't ready yet (and would be an opt-in mode)
14:28:36 sean-k-mooney mtreinish: playing with the schduleiing did help in the past. i notced that grouping by class helped to some extent
14:29:10 mriedem oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor weee
14:29:52 mriedem we seem to have some fun eventlet issue in this failure as well
14:31:36 mriedem https://gist.githubusercontent.com/mriedem/e01252253600abbef95045d73cfe2206/raw/67b6360c5766eb681cb3ad9e1901ff94fa152e98/gistfile1.txt
14:31:39 sean-k-mooney and some db errors " WARNING [nova.compute.manager] Failed to revert task state for instance. Error: (sqlite3.OperationalError) no such table: instance_extra'"
14:32:37 mtreinish which review # and job is the failure happening on? I can take another stab at fixing the issue
14:32:48 sean-k-mooney https://review.opendev.org/697162
14:33:35 mriedem mtreinish: it's intermittent, we get all sorts of random stuff like this http://status.openstack.org/elastic-recheck/#1813147 and then we squash them as they crop up
14:34:00 mriedem the above gist is weird because the test is creating a server on compute1, the logs say it's ok, but the traceback looks like something is rescheduling to another host during build
14:34:01 sean-k-mooney mtreinish: mriedem is not really changeing anything in that that would resonably cause those errors
14:34:39 sean-k-mooney mriedem: could we be leaking some state betwween the different functional tests
14:35:14 sean-k-mooney such that this fails only with some test orderings and passes with others
14:35:54 mriedem yeah maybe but i don't see any other tests on the same worker that look like they would be doing reschedules
14:36:00 mriedem anywho, recheck!
14:36:32 sean-k-mooney that was why grouping the test execution by class work to "fix" the issue in the past. i ment we did not hit the issue due to how the test were run
14:36:40 mriedem we could probably make that ServiceNotFound in the servicegroup api not a traceback so that if this fails again like this it doesn't blow up the subunit output parser
14:37:27 sean-k-mooney we didnt merge that since we actully found the problem and fixed it in that case
14:53:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle ServiceNotFound in DbDriver._report_state https://review.opendev.org/697301
15:40:20 dansmith efried: despite the fact that I dropped some comments on the request group patch about the testing mess, I'm going to mostly depend on you to ack the placementy request-group-y bit of that before I look too closely
15:42:26 efried dansmith: ack. It might depend on my series here https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:use-placement-1.34
15:43:30 Sundar dansmith: Thanks for the super-quick reviews.
15:43:45 efried but, having been looking at that stuff pretty closely for the past week, I ought to be able to provide reasonable guidance in that regard :)
15:44:18 dansmith Sundar: mmm, six hours later.. not super-quick :)
15:45:59 Sundar dansmith: I'll set higher standards for you going forward :P.
15:46:11 dansmith Sundar: fair warning, I have this gut feeling that you're missing a bunch of test coverage, but I'm not in that phase of review yet, so I'm not nitpicking all the conditionals
15:47:49 Sundar dansmith: ack, will look more into that.
15:49:22 stephenfin mriedem: comment on https://review.opendev.org/#/c/697122/
15:50:42 stephenfin Could I ask someone to go through the bottom few patches of the remove-nova-network series starting at [1]? There's a merge conflict coming and I'd rather not rebase/recheck the _entire_ queue [1] https://review.opendev.org/#/c/696503/
15:50:43 efried My understanding stephenfin: we need to backport it as it stands. Which means if your change merges first, the backport of this one will need to be different from the master version.
15:51:49 stephenfin Yup, I get that. I'm tempted to suggest we just apply it against stable/train since it will soon be useless on master, but maybe that's a lot of work just to avoid some merge conflicts
15:52:00 stephenfin (as in StableOnly)
15:52:05 efried It won't be useless on master
15:52:12 efried we still need to cache, which we're not doing
15:52:19 mriedem stephenfin: replied
15:53:50 stephenfin mriedem: Good thing I asked :)
15:54:29 mriedem i think we need to just make the neutron SecurityGroupAPI a singleton
15:54:34 mriedem it doesn't have any state as far as i can tell
15:54:49 mriedem unlike SchedulerReportClient which does have some state
15:55:11 efried It must not have state, since we were able to cache it before with no ill effects
15:55:22 efried unless the state was added in the interval when we had removed the caching.
15:55:30 mriedem the neutron sg api you mean
15:55:37 stephenfin Makes sense to me. I can rework that patch accordingly
15:56:24 stephenfin though I'll hold off for now since that queue is approaching x-cell resize size and will kill CI if pushed
15:56:28 stephenfin RIP CI
15:57:25 mriedem the bottom 5 or so changes can use another core
15:57:33 stephenfin mriedem: While I have you, is this the correct thing to do? https://review.opendev.org/#/c/697152/2/doc/api_samples/os-quota-class-sets/quota-classes-show-get-resp.json
15:58:06 stephenfin I assume we don't support arbitrary quota classes so we can't modify the output for older microversions
15:58:37 stephenfin ...meaning I have to fake those nova-net'y quotas rather than just dropping them
15:58:41 mriedem i'm pretty sure we only support one quota class, 'default'
15:59:02 stephenfin sorry, arbitrary quotas...types?
15:59:20 mriedem yeah you can't create random quota types
16:00:03 salmankhan HI guys, I am facing an issue if someone can help. I upgraded the nova queens api from xenial to bionic and I am not able to see network info in "nova show" or "openstack server show" commands. However I can see the network details on dashboard or "nova interface-list" or "openstack port show" commands. Doesn't look like a database issue, but may be. If I reboot a instance there is no interface attached as well. What could be
16:00:04 mriedem https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/quota_sets.py#L29
16:00:51 stephenfin so I do have to keep them. okidok
16:00:59 mriedem that said, i'm not sure about what you're doing there with just changing all the network quota values to -1
16:01:41 stephenfin that's what the noop driver was doing (unlimited) and those things are noops now
16:01:43 stephenfin so...
16:01:45 openstackgerrit Merged openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180
16:02:02 mriedem the noop quota driver is just for testing
16:02:07 mriedem or nfv single tenant i guess
16:02:39 mriedem i agree that we can't just remove them from the responses and stuff without a microversion since it changes the schema
16:03:08 mriedem we don't proxy enforcement of those to neutron at all do we?
16:03:20 mriedem i know there was talk of doing that back in like havana
16:03:25 stephenfin nope, they're dead
16:03:38 stephenfin all DB model driven
16:05:36 mriedem yeah, i mean the -1 is a lie because we're not enforcing those. we did deprecate them from the apis in 2.35 and 2.49 and they were a lie before if using neutron anyway - you could change security group rules quota to 500 in nova and it wouldn't apply to neutron
16:06:15 melwitt re: the noop quota driver, I had thought that was the way for a deployment that doesn't care about quotas to bypass all the calculating (perf impact). so I was thinking not to remove it because of that. am I missing something?
16:06:45 mriedem how many deployments besides nfv single tenant don't care about quota?
16:08:19 stephenfin If -1 is a lie, what do I return instead? Just 0?
16:08:35 stephenfin I have to return _something_
16:10:45 mriedem it's all a lie is my point i think, so -1 is probably good enough
16:11:11 mriedem i'm just not sure how clearly i'm thinking about this atm, or what alternatives there are
16:11:31 mriedem i defer to dansmith
16:11:36 mriedem o-)
16:12:05 dansmith I don't know about quota things
16:12:09 dansmith I defer to melwitt
16:15:08 melwitt I'll take a look at it
16:15:52 salmankhan Anyone??? HI guys, I am facing an issue if someone can help. I upgraded the nova queens api from xenial to bionic and I am not able to see network info in "nova show" or "openstack server show" commands. However I can see the network details on dashboard or "nova interface-list" or "openstack port show" commands. Doesn't look like a database issue, but may be. If I reboot a instance there is no interface attached as well. Wha
16:15:52 salmankhan fine.
16:17:35 mriedem the instance network info cache is probably missing
16:17:58 salmankhan how can it be fixed
16:18:02 mriedem check the instance_info_caches.network_info column for that instance
16:18:26 mriedem on queens....i'm not sure
16:18:32 mriedem there is a periodic that would heal it starting in i think stein
16:18:49 mriedem see https://review.opendev.org/#/c/679274/
16:20:26 mriedem you could also take a look at this https://review.opendev.org/#/c/640516/
16:20:55 mriedem "If someone comes forward saying "yes this fixes a thing I'm hitting" then I'd be willing to restore it."
16:21:38 mriedem that second patch won't help you on queens since it relies on the force_refresh kwarg from the previous patch
16:21:53 sean-k-mooney mriedem: for what its worth we have backported only the periodic heal downstream
16:22:06 mriedem super
16:22:09 sean-k-mooney we did not backport the data migration
16:22:18 sean-k-mooney because that caused other issues
16:22:19 mriedem and canonical is going to as well it looks like
16:23:02 sean-k-mooney we have it all the way back to newton at this point
16:23:31 sean-k-mooney although we only did newton in the last month
16:23:34 salmankhan So you saying apply this 679274, that could fix on queens?
16:24:18 sean-k-mooney only if you have fixed teh underlying network issue

Earlier   Later