| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-04 | |||
| 13:57:35 | mriedem | https://bugs.launchpad.net/tempest/+bug/1844568 ? | |
| 13:57:44 | efried | yes | |
| 13:57:51 | mriedem | sure go ahead | |
| 13:58:51 | sean-k-mooney | stephenfin: has your changes to _wait_for_state_change already merged | |
| 13:58:53 | gibi | efried: comments in https://review.opendev.org/#/c/686804 | |
| 13:59:08 | stephenfin | sean-k-mooney: yes | |
| 13:59:30 | sean-k-mooney | ok so i can drop the new class in that case but it means i have to do the rewite in the backport now | |
| 13:59:40 | efried | stephenfin: did you see vTPM conversation from earlier? | |
| 13:59:46 | stephenfin | I did not | |
| 14:00:04 | efried | stephenfin: ~1h45m back | |
| 14:00:05 | stephenfin | sean-k-mooney: You can...or you can just use the weird variant in the backports | |
| 14:00:16 | sean-k-mooney | no that is what i mean | |
| 14:00:20 | efried | 1h15, sorry | |
| 14:00:52 | sean-k-mooney | i need to use the version that takes the expected state in master and then rewrite it to use the move from state in the backports | |
| 14:01:14 | efried | gibi: thanks | |
| 14:01:41 | stephenfin | sean-k-mooney: Yeah. It only needs to be done once though (the first backport) and should only result in 4 or 5 single line changes, I'd imagine | |
| 14:01:46 | stephenfin | Not the end of the world | |
| 14:02:25 | stephenfin | efried: Let me take a look after this meeting | |
| 14:03:03 | sean-k-mooney | it might be more involed then that but ya ill rebase everythin against master tehn ill start updatign the changes to make the difference clear | |
| 14:03:30 | openstackgerrit | sean mooney proposed openstack/nova master: Block rebuild when NUMA topology changed https://review.opendev.org/687957 | |
| 14:03:44 | openstackgerrit | sean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild https://review.opendev.org/689861 | |
| 14:04:20 | sean-k-mooney | so that is just a rebase to master and ill work on the functional test changes now | |
| 14:15:16 | mriedem | i see our subunit parser failures are back on the rise | |
| 14:16:33 | mriedem | looks like this blew up https://zuul.opendev.org/t/openstack/build/d96bc0bfbb214661bcbb4b4e59b3d8bb/log/job-output.txt#3010 | |
| 14:17:12 | efried | gdi | |
| 14:17:19 | efried | has anyone considered fixing subunit? | |
| 14:17:40 | sean-k-mooney | its not subunit its self | |
| 14:17:47 | sean-k-mooney | its how stestr is using it | |
| 14:17:55 | efried | has anyone considered fixing stestr? | |
| 14:18:05 | sean-k-mooney | we are not fragmenting the packets properloy | |
| 14:18:15 | efried | My point: It shouldn't be up to every consuming project to "not output so much" | |
| 14:18:39 | sean-k-mooney | right well we could just use pytest to run things but some people object to that too | |
| 14:19:54 | efried | is this a mtreinish thing? | |
| 14:20:19 | mriedem | well in this case something is obviously busted while this test is running | |
| 14:20:40 | mriedem | one of the services is gone from the db for some reason, or the context it was using somehow - it's a multi-cell test so maybe it's looking in the wrong db | |
| 14:20:55 | mriedem | so every 10 seconds when it reports it's state the service is not found | |
| 14:21:06 | mriedem | and prints the traceback | |
| 14:21:17 | mtreinish | efried: patches are more than welcome if you have a fix | |
| 14:21:51 | sean-k-mooney | mtreinish: i think the issue is that its non triaival to figure out how stestr works to try and even start on the fix | |
| 14:22:27 | mtreinish | sean-k-mooney: it's non trivial to figure out how subunit is working under the covers which is why I haven't had a fix for it yet | |
| 14:23:04 | mtreinish | the stream handling code is really hairy and mostly leftover from testr | |
| 14:23:24 | sean-k-mooney | mtreinish: ya i treied to look at the stestr code to figure out how the subunit stream was being created to figure out where to fragment the packet but i got lost | |
| 14:24:11 | sean-k-mooney | that was partly because i was lookig at 2 codebase i dont undersantd and as you said the code is kind of hairy | |
| 14:24:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Implement cleanup_instance_network_on_host for neutron API https://review.opendev.org/697162 | |
| 14:24:20 | efried | okay, I wasn't criticizing you mtreinish, sorry if it came out that way. If you've already looked and it's "really hairy"... | |
| 14:25:14 | sean-k-mooney | efried: i know stephenfin melwitt and i all tried to figuire this out before and gave up | |
| 14:25:44 | efried | I just remember several patches (from mriedem I think) that tried to chip away at the problem by reducing the amount of output from our tests. | |
| 14:26:00 | mtreinish | yeah, also masayukig smcginnis and I also took a look into a couple years ago f2f at a summit and also gave up | |
| 14:26:09 | efried | gah | |
| 14:26:27 | mriedem | regardless of subunit/stestr i don't think our tests should be spewing out a buttload of random tracebacks | |
| 14:26:39 | mriedem | masking that with fixing subunit/stestr somehow just papers over those issues | |
| 14:26:48 | sean-k-mooney | so the simple solution is to just use pytest or a non subunit based test runner but there are perfromace impacts to doing that | |
| 14:27:29 | mtreinish | I have a PR which should workaround it I think: https://github.com/mtreinish/stestr/pull/271 but it introduces other behavior changes and isn't ready yet (and would be an opt-in mode) | |
| 14:28:36 | sean-k-mooney | mtreinish: playing with the schduleiing did help in the past. i notced that grouping by class helped to some extent | |
| 14:29:10 | mriedem | oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor weee | |
| 14:29:52 | mriedem | we seem to have some fun eventlet issue in this failure as well | |
| 14:31:36 | mriedem | https://gist.githubusercontent.com/mriedem/e01252253600abbef95045d73cfe2206/raw/67b6360c5766eb681cb3ad9e1901ff94fa152e98/gistfile1.txt | |
| 14:31:39 | sean-k-mooney | and some db errors " WARNING [nova.compute.manager] Failed to revert task state for instance. Error: (sqlite3.OperationalError) no such table: instance_extra'" | |
| 14:32:37 | mtreinish | which review # and job is the failure happening on? I can take another stab at fixing the issue | |
| 14:32:48 | sean-k-mooney | https://review.opendev.org/697162 | |
| 14:33:35 | mriedem | mtreinish: it's intermittent, we get all sorts of random stuff like this http://status.openstack.org/elastic-recheck/#1813147 and then we squash them as they crop up | |
| 14:34:00 | mriedem | the above gist is weird because the test is creating a server on compute1, the logs say it's ok, but the traceback looks like something is rescheduling to another host during build | |
| 14:34:01 | sean-k-mooney | mtreinish: mriedem is not really changeing anything in that that would resonably cause those errors | |
| 14:34:39 | sean-k-mooney | mriedem: could we be leaking some state betwween the different functional tests | |
| 14:35:14 | sean-k-mooney | such that this fails only with some test orderings and passes with others | |
| 14:35:54 | mriedem | yeah maybe but i don't see any other tests on the same worker that look like they would be doing reschedules | |
| 14:36:00 | mriedem | anywho, recheck! | |
| 14:36:32 | sean-k-mooney | that was why grouping the test execution by class work to "fix" the issue in the past. i ment we did not hit the issue due to how the test were run | |
| 14:36:40 | mriedem | we could probably make that ServiceNotFound in the servicegroup api not a traceback so that if this fails again like this it doesn't blow up the subunit output parser | |
| 14:37:27 | sean-k-mooney | we didnt merge that since we actully found the problem and fixed it in that case | |
| 14:53:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle ServiceNotFound in DbDriver._report_state https://review.opendev.org/697301 | |
| 15:40:20 | dansmith | efried: despite the fact that I dropped some comments on the request group patch about the testing mess, I'm going to mostly depend on you to ack the placementy request-group-y bit of that before I look too closely | |
| 15:42:26 | efried | dansmith: ack. It might depend on my series here https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:use-placement-1.34 | |
| 15:43:30 | Sundar | dansmith: Thanks for the super-quick reviews. | |
| 15:43:45 | efried | but, having been looking at that stuff pretty closely for the past week, I ought to be able to provide reasonable guidance in that regard :) | |
| 15:44:18 | dansmith | Sundar: mmm, six hours later.. not super-quick :) | |
| 15:45:59 | Sundar | dansmith: I'll set higher standards for you going forward :P. | |
| 15:46:11 | dansmith | Sundar: fair warning, I have this gut feeling that you're missing a bunch of test coverage, but I'm not in that phase of review yet, so I'm not nitpicking all the conditionals | |
| 15:47:49 | Sundar | dansmith: ack, will look more into that. | |
| 15:49:22 | stephenfin | mriedem: comment on https://review.opendev.org/#/c/697122/ | |
| 15:50:42 | stephenfin | Could I ask someone to go through the bottom few patches of the remove-nova-network series starting at [1]? There's a merge conflict coming and I'd rather not rebase/recheck the _entire_ queue [1] https://review.opendev.org/#/c/696503/ | |
| 15:50:43 | efried | My understanding stephenfin: we need to backport it as it stands. Which means if your change merges first, the backport of this one will need to be different from the master version. | |
| 15:51:49 | stephenfin | Yup, I get that. I'm tempted to suggest we just apply it against stable/train since it will soon be useless on master, but maybe that's a lot of work just to avoid some merge conflicts | |
| 15:52:00 | stephenfin | (as in StableOnly) | |
| 15:52:05 | efried | It won't be useless on master | |
| 15:52:12 | efried | we still need to cache, which we're not doing | |
| 15:52:19 | mriedem | stephenfin: replied | |
| 15:53:50 | stephenfin | mriedem: Good thing I asked :) | |
| 15:54:29 | mriedem | i think we need to just make the neutron SecurityGroupAPI a singleton | |
| 15:54:34 | mriedem | it doesn't have any state as far as i can tell | |
| 15:54:49 | mriedem | unlike SchedulerReportClient which does have some state | |
| 15:55:11 | efried | It must not have state, since we were able to cache it before with no ill effects | |
| 15:55:22 | efried | unless the state was added in the interval when we had removed the caching. | |
| 15:55:30 | mriedem | the neutron sg api you mean | |
| 15:55:37 | stephenfin | Makes sense to me. I can rework that patch accordingly | |
| 15:56:24 | stephenfin | though I'll hold off for now since that queue is approaching x-cell resize size and will kill CI if pushed | |
| 15:56:28 | stephenfin | RIP CI | |
| 15:57:25 | mriedem | the bottom 5 or so changes can use another core | |
| 15:57:33 | stephenfin | mriedem: While I have you, is this the correct thing to do? https://review.opendev.org/#/c/697152/2/doc/api_samples/os-quota-class-sets/quota-classes-show-get-resp.json | |
| 15:58:06 | stephenfin | I assume we don't support arbitrary quota classes so we can't modify the output for older microversions | |