| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-04 | |||
| 13:19:18 | gibi | efried: I'm hesitant, as this is raw at the edges | |
| 13:19:36 | gibi | let me go through the spec. | |
| 13:19:41 | efried | cold migrate works | |
| 13:19:45 | gibi | ohh | |
| 13:19:47 | gibi | coolio | |
| 13:20:04 | efried | and shelve/unshelve (which uses snapshots) so cross-cell resize also works. | |
| 13:21:34 | gibi | OK. let me read the spec, but it seems workable at least. And an admin can only disable the feature by not configuring it in the nova conf | |
| 13:21:44 | gibi | s/only/always/ | |
| 13:22:58 | efried | yup | |
| 13:23:04 | efried | thanks gibi | |
| 13:23:19 | gibi | efried: have fun! | |
| 13:23:23 | efried | enjoy your doc :) | |
| 13:23:33 | gibi | ... a long queue ... | |
| 13:24:06 | gibi | it is the 4th hour I'm queueing... | |
| 13:49:23 | efried | egads | |
| 13:49:39 | efried | Does Hungary have a national health service like the UK? | |
| 13:50:26 | gibi | yes, this is the type of service everyone gets automatically in Hungary | |
| 13:50:43 | efried | I remember the UK NHS being notoriously slow like that. | |
| 13:51:00 | sean-k-mooney | slow but free | |
| 13:51:14 | sean-k-mooney | and good once you are seen | |
| 13:52:02 | gibi | I have a private issurance but it does not cover this type of follow up I have to do after my surgery I had the beginning of this year | |
| 13:52:06 | gibi | so I sit and wait | |
| 13:53:16 | sean-k-mooney | efried: even without deleting the barbican key they can still do a vm delete right | |
| 13:53:16 | openstack | Launchpad bug 1855015 in OpenStack Compute (nova) "Intermittent fails in nova-next job with "Multiple possible networks found, use a Network ID to be more specific."" [Undecided,Invalid] | |
| 13:53:16 | mriedem | efried: i've invalidated https://bugs.launchpad.net/nova/+bug/1855015 since the jobs i'm looking at aren't failures for the most part | |
| 13:53:34 | mriedem | efried: there is a tempest test which is triggering the error in the api logs but the test handles the 400 response | |
| 13:53:44 | sean-k-mooney | and ya most service allow admins to delete stuff even if they dont own it so i suspect barbican and swift can do the same | |
| 13:54:25 | gibi | efried, mriedem there is a fix for the tempest multiple network conflict https://review.opendev.org/#/c/682964 | |
| 13:54:33 | efried | good lord son, you posted two more comments and changed status while I was writing my response. | |
| 13:55:08 | mriedem | gotta be quick | |
| 13:55:10 | efried | gibi: thanks, they told me about that in -neutron | |
| 13:55:19 | efried | mriedem: okay if I reopen and mark as dup? | |
| 13:56:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'FloatingIP' object https://review.opendev.org/697153 | |
| 13:56:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'SecurityGroup' object https://review.opendev.org/697155 | |
| 13:56:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'Network' object https://review.opendev.org/697154 | |
| 13:56:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Follow-up for I12a96ea659ed402cc4d1bd52a50e2e16042b6372 https://review.opendev.org/697252 | |
| 13:56:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156 | |
| 13:56:14 | efried | mriedem: unclear why it would be invalid. It caused at least three patches to fail that I was looking directly at. The rest I didn't check explicitly, but no reason to believe it would be different. | |
| 13:57:21 | mriedem | dupe against what? | |
| 13:57:35 | openstack | Launchpad bug 1844568 in tempest "[compute] "create_test_server" if networks is undefined and more than one network is present" [Undecided,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) | |
| 13:57:35 | mriedem | https://bugs.launchpad.net/tempest/+bug/1844568 ? | |
| 13:57:44 | efried | yes | |
| 13:57:51 | mriedem | sure go ahead | |
| 13:58:51 | sean-k-mooney | stephenfin: has your changes to _wait_for_state_change already merged | |
| 13:58:53 | gibi | efried: comments in https://review.opendev.org/#/c/686804 | |
| 13:59:08 | stephenfin | sean-k-mooney: yes | |
| 13:59:30 | sean-k-mooney | ok so i can drop the new class in that case but it means i have to do the rewite in the backport now | |
| 13:59:40 | efried | stephenfin: did you see vTPM conversation from earlier? | |
| 13:59:46 | stephenfin | I did not | |
| 14:00:04 | efried | stephenfin: ~1h45m back | |
| 14:00:05 | stephenfin | sean-k-mooney: You can...or you can just use the weird variant in the backports | |
| 14:00:16 | sean-k-mooney | no that is what i mean | |
| 14:00:20 | efried | 1h15, sorry | |
| 14:00:52 | sean-k-mooney | i need to use the version that takes the expected state in master and then rewrite it to use the move from state in the backports | |
| 14:01:14 | efried | gibi: thanks | |
| 14:01:41 | stephenfin | sean-k-mooney: Yeah. It only needs to be done once though (the first backport) and should only result in 4 or 5 single line changes, I'd imagine | |
| 14:01:46 | stephenfin | Not the end of the world | |
| 14:02:25 | stephenfin | efried: Let me take a look after this meeting | |
| 14:03:03 | sean-k-mooney | it might be more involed then that but ya ill rebase everythin against master tehn ill start updatign the changes to make the difference clear | |
| 14:03:30 | openstackgerrit | sean mooney proposed openstack/nova master: Block rebuild when NUMA topology changed https://review.opendev.org/687957 | |
| 14:03:44 | openstackgerrit | sean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild https://review.opendev.org/689861 | |
| 14:04:20 | sean-k-mooney | so that is just a rebase to master and ill work on the functional test changes now | |
| 14:15:16 | mriedem | i see our subunit parser failures are back on the rise | |
| 14:16:33 | mriedem | looks like this blew up https://zuul.opendev.org/t/openstack/build/d96bc0bfbb214661bcbb4b4e59b3d8bb/log/job-output.txt#3010 | |
| 14:17:12 | efried | gdi | |
| 14:17:19 | efried | has anyone considered fixing subunit? | |
| 14:17:40 | sean-k-mooney | its not subunit its self | |
| 14:17:47 | sean-k-mooney | its how stestr is using it | |
| 14:17:55 | efried | has anyone considered fixing stestr? | |
| 14:18:05 | sean-k-mooney | we are not fragmenting the packets properloy | |
| 14:18:15 | efried | My point: It shouldn't be up to every consuming project to "not output so much" | |
| 14:18:39 | sean-k-mooney | right well we could just use pytest to run things but some people object to that too | |
| 14:19:54 | efried | is this a mtreinish thing? | |
| 14:20:19 | mriedem | well in this case something is obviously busted while this test is running | |
| 14:20:40 | mriedem | one of the services is gone from the db for some reason, or the context it was using somehow - it's a multi-cell test so maybe it's looking in the wrong db | |
| 14:20:55 | mriedem | so every 10 seconds when it reports it's state the service is not found | |
| 14:21:06 | mriedem | and prints the traceback | |
| 14:21:17 | mtreinish | efried: patches are more than welcome if you have a fix | |
| 14:21:51 | sean-k-mooney | mtreinish: i think the issue is that its non triaival to figure out how stestr works to try and even start on the fix | |
| 14:22:27 | mtreinish | sean-k-mooney: it's non trivial to figure out how subunit is working under the covers which is why I haven't had a fix for it yet | |
| 14:23:04 | mtreinish | the stream handling code is really hairy and mostly leftover from testr | |
| 14:23:24 | sean-k-mooney | mtreinish: ya i treied to look at the stestr code to figure out how the subunit stream was being created to figure out where to fragment the packet but i got lost | |
| 14:24:11 | sean-k-mooney | that was partly because i was lookig at 2 codebase i dont undersantd and as you said the code is kind of hairy | |
| 14:24:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Implement cleanup_instance_network_on_host for neutron API https://review.opendev.org/697162 | |
| 14:24:20 | efried | okay, I wasn't criticizing you mtreinish, sorry if it came out that way. If you've already looked and it's "really hairy"... | |
| 14:25:14 | sean-k-mooney | efried: i know stephenfin melwitt and i all tried to figuire this out before and gave up | |
| 14:25:44 | efried | I just remember several patches (from mriedem I think) that tried to chip away at the problem by reducing the amount of output from our tests. | |
| 14:26:00 | mtreinish | yeah, also masayukig smcginnis and I also took a look into a couple years ago f2f at a summit and also gave up | |
| 14:26:09 | efried | gah | |
| 14:26:27 | mriedem | regardless of subunit/stestr i don't think our tests should be spewing out a buttload of random tracebacks | |
| 14:26:39 | mriedem | masking that with fixing subunit/stestr somehow just papers over those issues | |
| 14:26:48 | sean-k-mooney | so the simple solution is to just use pytest or a non subunit based test runner but there are perfromace impacts to doing that | |
| 14:27:29 | mtreinish | I have a PR which should workaround it I think: https://github.com/mtreinish/stestr/pull/271 but it introduces other behavior changes and isn't ready yet (and would be an opt-in mode) | |
| 14:28:36 | sean-k-mooney | mtreinish: playing with the schduleiing did help in the past. i notced that grouping by class helped to some extent | |
| 14:29:10 | mriedem | oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor weee | |
| 14:29:52 | mriedem | we seem to have some fun eventlet issue in this failure as well | |
| 14:31:36 | mriedem | https://gist.githubusercontent.com/mriedem/e01252253600abbef95045d73cfe2206/raw/67b6360c5766eb681cb3ad9e1901ff94fa152e98/gistfile1.txt | |
| 14:31:39 | sean-k-mooney | and some db errors " WARNING [nova.compute.manager] Failed to revert task state for instance. Error: (sqlite3.OperationalError) no such table: instance_extra'" | |
| 14:32:37 | mtreinish | which review # and job is the failure happening on? I can take another stab at fixing the issue | |
| 14:32:48 | sean-k-mooney | https://review.opendev.org/697162 | |